The chat and sidebar scroll-fade overlays ended their gradient at the
`transparent` keyword, which is transparent black. Safari 27 Beta
(Liquid Glass) interpolates an opaque colour to transparent black
through a grey midtone, so the fades render as two solid grey bands
(top of the chat and above the composer).
Fade each gradient to the theme colour at zero alpha instead, so every
step keeps the same hue and no grey can appear. Fixes#6457.
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Auto-install SSM kernels (causal-conv1d, mamba-ssm) for inference loads
Mamba/SSM hybrids (Nemotron-H/Nano, Falcon-H1, Granite-4.0-H, ...) lazily import
mamba_ssm / causal_conv1d during from_pretrained, so loading them for chat failed
with 'mamba-ssm is required by the Mamba model but cannot be imported'. The training
worker already wheel-first installs these before a fine-tune; the inference worker
did not. Add utils/ssm_runtime.ensure_ssm_runtime and call it from the inference load
path so the same models load for inference. Training worker is untouched; a drift
test keeps the shared detection and pinned versions in lockstep.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ssm_runtime: invalidate import caches, skip MLX, cover LoRA base
- Invalidate importlib finder caches in _is_importable and after a successful
wheel install, so a kernel installed earlier in this same process is actually
importable when the modeling code lazy-imports it during from_pretrained.
- Skip the SSM kernel install entirely on the MLX (Apple Silicon) load path:
these are CUDA/ROCm Torch kernels with no MLX use and no macOS prebuilt wheel,
so the source build would fail before the MLX backend loads the model.
- For LoRA loads, also run detection over the resolved base model, since an
adapter id like 'me/my-lora' won't match the SSM heuristics but its SSM base
(Nemotron-H, ...) is what needs the kernels.
Adds tests for cache invalidation and the MLX-skip / LoRA-base worker wiring.
* Tighten SSM autoinstall comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ssm_runtime: verify wheel imports, HIP-aware source build, build heartbeat
Address review feedback:
- Verify a prebuilt wheel actually imports before trusting it; a CUDA/ABI-mismatched
wheel now falls back to a source build instead of returning success and failing later
with the cryptic lazy-import error.
- HIP-aware source build: require hipcc on ROCm, inject clang --gcc-install-dir, and use
the 1800s timeout, mirroring the training worker (ROCm has no prebuilt wheel).
- Emit a status heartbeat every 60s during the source build so a long (ROCm) build does
not trip the orchestrator's 300s inactivity timeout.
Tests cover the wheel-not-importable fallback and the missing-hipcc ROCm bail.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make causal-conv1d best-effort and harden the SSM source build
- causal-conv1d is a fast path: models that merely want it (Qwen3-Next, LFM2)
fall back to torch, so a failed install must not reject an otherwise loadable
chat model on Windows/CPU/macOS or an ABI without a wheel. Only a true SSM
model's mamba-ssm requirement stays fatal, matching the training worker which
treats causal-conv1d as best-effort.
- The source build is reached only when not importable, including a wheel that
installed but failed to import; add --reinstall/--force-reinstall so it
replaces the broken install instead of no-opping as already satisfied.
- Add --no-cache to the ROCm uv source build to avoid reusing stale artifacts
from a partial HIP build, mirroring the training worker.
* Address review: install SSM kernels before transformers, harden import + Windows
Codex:
- Install the SSM kernels before importing transformers. run_inference_process
imported core.inference.inference (which imports unsloth/transformers) before the
load, and a sidecar transformers can evaluate its optional-backend gates against
the import state; installing causal_conv1d/mamba_ssm afterwards left those gates
unsatisfied and a Nemotron/Falcon/Granite load still failed with "mamba-ssm is
required". The initial model's kernels are now installed in run_inference_process
before the ML import, via a shared _ensure_ssm_kernels helper; _handle_load keeps
calling it (idempotent) for a LoRA's base and for later in-process loads.
- _is_importable now treats any import failure as "not importable", not only
ImportError. An ABI-incompatible native kernel (undefined symbol after a torch/CUDA
upgrade) raises OSError/RuntimeError; letting those escape reported
ssm_runtime_install_failed instead of falling back to reinstall/source build.
- Skip causal-conv1d on Windows (no prebuilt wheel), mirroring the training worker.
A causal-conv1d-only model (Qwen3-Next/LFM2) no longer drops a chat load into a
multi-minute untimed source build; it uses the torch fallback. mamba-ssm is still
attempted for true SSM hybrids.
Tests: test_ssm_runtime.py +5 (broken-kernel exceptions read as not-importable;
causal-conv1d skipped on win32 while mamba-ssm still installs). 36 passed.
* Trim comments to be more succinct
* Run security gates before installing SSM kernels
The SSM kernel auto-install is name-based (model_is_ssm is a substring match, no
config fetch), so a model id merely containing an SSM substring triggered a
native-package install (possibly a slow source build) before the malware and
remote-code consent gates ran. Extract those gates into _run_security_gates and
call it before the kernel install in both the pre-import path of
run_inference_process and in _handle_load, so a blocked or nonexistent model is
refused before any build. The gates are metadata-only and do not import
transformers, so they are safe to run before the pre-import install.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Resolve remote LoRA bases before importing transformers
_resolve_base_model only reads a local adapter_config.json, so a remote LoRA
adapter whose own id has no SSM substring but whose base is a Nemotron/Falcon/
Granite model had its base discovered only by ModelConfig in _handle_load, after
transformers was imported and its optional-backend availability snapshotted, so
the SSM kernel install there was too late. Add _remote_lora_base, a metadata-only
adapter_config.json fetch (no huggingface_hub / transformers import), and use it
in the pre-import path so the base is gated and its kernels pre-installed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate only loaded roots, tier on the resolved base, read offline LoRA cache
Three follow-ups to the pre-import resolution:
- The security gate reused the SSM target list, which for a local full fine-tune
includes the config.json-recorded base. That base is never loaded, so scanning
it could falsely block a safe local checkpoint. Gate only the model plus a
genuine LoRA base (matching _handle_load's mc.is_lora), separate from the
broader SSM-install list.
- Tier activation ran on the raw adapter id, so a remote LoRA whose base needs a
sidecar transformers version imported the default and failed. Resolve the base
once up front and activate on it.
- _remote_lora_base bailed on offline before checking the hub cache, missing a
cached adapter's base. Read the cached adapter_config.json when offline or when
the fetch fails.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the pre-import gate transformers-free; harden remote LoRA resolution
The pre-import security gate called security_load_subdirs, which imports
model_config and thus transformers, snapshotting optional-backend availability
before the SSM kernels are installed and defeating the ordering. Add
compute_subdirs to _run_security_gates and pass False in the preflight so it scans
from the root only (transformers-free); _handle_load still runs the authoritative
gate with full subdir scoping after the import.
_remote_lora_base now skips existing local relative paths (is_local_path) so a
checkpoint like outputs/run1 is never treated as a Hub repo, and distinguishes a
definitive 404 (not a LoRA -> None) from transient/offline failures (read the
cache), so a repo that is now a full model no longer resolves a stale cached base.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Probe a real model id for SSM kernels; respect HF_ENDPOINT
model_is_ssm is a substring match, so an arbitrary name could false-match and
force a mamba-ssm install that fails the load for a non-SSM model:
- a LoRA adapter id like user/falcon-h1-lora (the SSM-relevant code is the base's);
- a local checkpoint under an SSM-named parent dir, e.g. /runs/falcon-h1/llama-ckpt.
Add ssm_probe_identifier, which resolves the base (or a bare local checkpoint's
basename) and feed that to ensure_ssm_runtime from both the pre-import path and
_handle_load, so detection runs against a real model id, never an adapter id or
parent folders.
_remote_lora_base now honors HF_ENDPOINT so enterprise/mirror deployments resolve
the adapter base instead of always hitting huggingface.co.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in the pre-import SSM gate/install path
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
* Route dense NemotronH models to the transformers 5.10 tier
Dense NemotronH models (e.g. unsloth/NVIDIA-Nemotron-3-Nano-4B) describe their
layer stack with a hybrid_override_pattern that includes '-' (MLP) layers.
transformers only learned to parse that ('-' -> 'mlp' in pattern_mapping, 'mlp'
in valid_types and MIXER_TYPES) in 5.10; on 5.3/5.5 the config raises
KeyError: '-'. The model also ships auto_map remote code, so training and
inference that approve trust_remote_code load fine, but a native (TRC=False)
load such as export hits the built-in parser and fails with
'Failed to load checkpoint: -'.
Detect dense NemotronH from config.json (a '-' in hybrid_override_pattern, or
'mlp' in an expanded layers_block_type) and route it to the 5.10 tier, where the
model loads natively without remote code. Pure-MoE NemotronH configs are
unaffected and keep their existing tier.
Covers both the local config.json and the remote HF-id paths, and adds tests for
the detector and the resulting tier selection.
* Tighten _nemotron_h_needs_mlp_support docstring
* Detect dense NemotronH in nested, cached, and resolved-away configs
Three gaps could still route a dense NemotronH (MLP '-' layers) to a tier
below 5.10 and hit KeyError: '-':
- VL wrappers (e.g. NemotronH_Nano_VL_V2) keep the dense language model under
llm_config/text_config; the detector only checked the top-level model_type.
Recurse into nested language configs.
- Offline or blocked config fetches returned None for an already-downloaded
repo. Read config.json from the HF hub cache before any network.
- A local checkpoint resolves to its base before tiering, so an offline/private
base discarded the local config that revealed the dense pattern. Prefer the
higher tier of the resolved base and the original path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden NemotronH tier detection follow-ups
Address review of the nested/cached/resolved-away detection:
- The local re-check ran the full tier detector on the original path, so a bare
LoRA adapter under e.g. /runs/gemma-4-x/llama-lora could upgrade a default base
via directory-name substrings. Gate the re-check on a real local config.json so
it reads metadata, not path names.
- The HF hub cache was read before any network, so an online tier check could
serve stale config.json after the repo changed upstream. Consult the cache only
offline or after a failed fetch.
- Reading the cache imported huggingface_hub during tier detection, which runs
before a sidecar venv is activated and could pin the default-env hub into
sys.modules. Resolve the cache path with stdlib only.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments to be more succinct
* Select newest hub-cache snapshot by mtime and retry transient config fetches
The HF cache fallback in tier detection picked the lexicographically-first
snapshot when refs/main was absent (commit-pinned downloads), which can be an
older SHA than the Hub would load. Sort snapshots by mtime instead.
A transient online fetch failure cached the hub-cache fallback under the normal
(model_name, token) key, so a long-lived worker kept serving stale metadata even
after connectivity recovered. Return the fallback without memoizing it so the
next call retries the network.
* Harden config.json tier detection against auth failures and transient blips
- _load_config_json: a 401/403/404 from the raw Hub request is a definitive access
answer, not an outage. Return None instead of falling back to the HF hub cache, so
an unauthenticated or wrong-token request can never read another caller's cached
private metadata.
- _check_config_needs_510/550: only memoize the derived tier when the underlying
config read was definitive (local file, offline cache, or a completed fetch).
A transient fetch fallback is no longer pinned, so the tier is re-evaluated once
connectivity returns instead of staying stuck on the lower tier.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in tier-detection auth/cache paths
---------
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: redesign Select model dropdown to match Hub design
Make the chat Select model picker easier to scan by reusing the Hub
on-device card's visual language.
- Rows now split owner/name, add a param chip, a DotTag format pill,
a tabular size, and a Loaded marker on the active model.
- Hub models / Fine-tuned tabs reuse the Hub's exact .hub-tab-toggle
styling (selectors extended in hub.css to the selector menu).
- Add a Downloaded / Recommended / Custom section toggle on the Hub
tab to filter the list.
- Widen the popover and nudge the scrollbar toward the edge.
* Studio: move section toggle below search, size tabs to label
Put Downloaded / Recommended / Custom under the search bar in their own
row so Hub models / Fine-tuned no longer wrap. The section toggle uses a
smaller font and sizes each tab to its label instead of equal widths.
* Studio: extract pure row-meta helpers into their own module
Move splitRepoLabel, classifyMetaToken, and parseMetaTokens out of
pickers.tsx into row-meta.ts. No behaviour change; keeps the presentation
logic free of React/DOM deps so it is easy to test in isolation.
* Studio: content-size the source tabs and add section icons
Size the Hub models / Fine-tuned tabs to their labels (with side
padding) like the section toggle, instead of stretching full width. Add
a leading download, star, and folder icon to Downloaded, Recommended,
and Custom.
* Studio: stop source tabs stretching and hide empty Fine-tuned tab
The popover is a flex column, so the fit toggle stretched full width;
add w-fit/self-start so it sizes to its content. Also hide the
Fine-tuned tab when there are no fine-tuned models, defaulting to Hub
models.
* Studio: keep only fine-tuned models in the Fine-tuned tab
Local models (LM Studio, Ollama, custom folders) carry source "local"
and already show in the Hub tab's Downloaded / Custom sections, so
exclude them from the Fine-tuned tab and from its visibility count.
Extract the tab rules into source-tabs.ts.
* Studio: show local providers under Downloaded, Recommended first
Show LM Studio and other local provider models in the Downloaded
section in all modes (was chat-only). Put Recommended first and make it
the default section. Add a little more space below the search bar.
* Studio: make Recommended a sortable live Unsloth listing
Replace the static Recommended list (and its collapse chevron) with a
sort dropdown over Unsloth's own models: Recommended, Trending, Most
likes, Downloads, Recently updated. Recommended shows recently uploaded
GGUF/MLX models that fit the device (hidden if they do not); the other
sorts list all Unsloth models, badged but never hidden. Adds a sort
option to useHfModelSearch and a pure recommended-fit helper.
* Studio: size Recommended models from the repo name when metadata is missing
GGUF and MLX repos rarely expose safetensors metadata, so a large model
with no size could pass the Recommended fit check because unknown size was
treated as fitting. Parse the parameter count from the repo id, including
the Gemma E series, and hide anything we still cannot size.
* Studio: detect model capabilities and family from HF tags
Thread tags and the pipeline tag through the model search results and add a
pure helper that infers vision, reasoning and audio plus the architecture
family, falling back to repo-name keywords when tags are absent.
* Studio: add row details and inline section sorting to Select model
Give each model row more detail and make the Hub sections easier to scan:
- Show vision, reasoning and audio badges plus the architecture family tag
on each row, alongside the params, format and size.
- Drop the redundant unsloth/ prefix on the Recommended rows.
- Rename the Recommended section tab to Unsloth and enlarge the section tabs.
- Move the sort dropdown inline to the right of the tabs at a fixed width.
- Add Recent, Size and Downloaded sorting to the Downloaded and Custom tabs.
- Remove the header icons, pad the subheadings, and grow the list height.
* Studio: tune the Select model sort dropdown and trim row badges
- Recommended now lists the most recently created Unsloth repos.
- Narrow the sort dropdown, remove its border, and truncate long labels.
- Tighten the gap between the section tab icons and their labels.
- Remove the architecture family tag from rows since it repeats the name.
* Studio: extract the PillTabs toggle into a shared module
Move the segmented pill toggle out of the model selector into its own file so
the Hub picker can reuse it for a format filter without duplicating the markup.
* Studio: fix Recommended infinite scroll and add a format filter
- Re-attach the scroll observer on each loaded page so a filtered Recommended
list keeps paging until the viewport fills instead of spinning forever with
nothing new appearing.
- Add an All / GGUF / MLX / Safetensors toggle on the Unsloth listing that
filters every sort.
* Studio: default Recommended to Trending, rename Downloaded to On Device, and fade the scroll edge
Sort: default the Recommended view to Trending and add a Name option to
the On Device / Custom sort. Recent now orders by last load time while
Downloaded orders by file date, tracked in localStorage (model-usage.ts).
Formats: show the format filter on all three tabs (Unsloth, On Device,
Custom), exclude mobile GGUF builds from Recommended, and flag GGUF rows
that exceed the device with the same OOM badge as safetensors.
Polish: download-icon badge on already-downloaded Recommended rows, the
hugeicons view stroke-rounded vision badge, Search all models placeholder,
matched popover padding, and a top-edge mask fade once the list scrolls.
* Studio: size GGUF repos from gguf metadata so large ones flag OOM
Repos with no <n>B token in the name (Kimi, MiniMax) had no param count
and so never showed an OOM badge. Request the gguf expand field from
Hugging Face and read gguf.total, so those repos get a param chip and an
OOM badge when they exceed the device budget.
Keep the row name full contrast when over budget (the OOM badge already
signals the fit), shorten the format and sort dropdowns, narrow the
popover, and rename Recently updated to Recent and All formats to All.
* Studio: address selector review feedback
Add WAI-ARIA roving tabindex and Arrow Left/Right navigation to the pill
toggle so only the active tab is in the tab order. Keep the chat-only
GGUF/MLX filter for every Recommended sort, not just Recommended, so
chat-only users do not see unrunnable checkpoints under Trending. Feed
both listings' GGUF hints into repo detection so a tag-only GGUF in
Recommended expands variants instead of loading as a checkpoint.
* Studio: scope Select model search per tab and add an MLX tag
Search is now per section. The Unsloth tab searches the Unsloth HF
listing only, On Device filters downloaded and LM Studio models by name,
and Custom filters custom-folder models, each with its own empty state.
MLX repos get an MLX pill mirroring the GGUF tag. Downloaded quants in
the Unsloth and search lists get the same delete action as On Device.
Also: revert the model name to normal weight, narrow the popover to
558px so the format and sort dropdowns sit one gap-2 from the tabs,
tighten the dropdown menus to match the Projects activity Select, and
make the empty On Device state name the active format filter.
* Studio: show local ./models on the On Device tab so they stay selectable
Models under the local models directory (source models_dir) flow in as local
models but were dropped from every list: filtered out of Fine-tuned and never
re-added by the Hub picker, which kept only LM Studio and custom-folder
sources. Capture them in the local refresh and render a Local models group on
the On Device tab, with the same format, search, and chat-only GGUF rules as
the other local groups.
* Studio: add a Hub button beside the Select model search bar
Adds a Hub button next to the search bar that opens the full Hub Discover
page to browse more models. Styled like the section tabs (rounded, no
border, soft shadow with a faint top layer) and darkens on hover. Also
nudges the format and sort dropdown chevrons a touch toward the edge.
* Studio: align Select model padding and tighten the format pills
Sizes the popover to the tab cluster so the left and right padding match,
and drops the top row below the rounded corner so the Hub button lines up
with the Trending dropdown. Gives the Hub button a fixed width, lets the
list scrollbar sit inside the box, and shrinks the format pill dot with a
tighter dot-to-label gap.
* Studio: label the Hub button Search Hub and match the dropdown width
Renames the button to Search Hub, sets its width to the format and sort
dropdown width so it lines up above them, and tightens the icon gap.
* Studio: drop the vision and reasoning row badges to declutter
Removes the vision and reasoning capability icons from the model rows so
they read cleaner. Audio is kept.
* Studio: add a safetensors pill, hide diffusion models, eye on Vision
Gives safetensors rows a format pill and size so their meta matches GGUF
and MLX, drops image and video diffusion models from the listing since they
cannot run in chat, and shows an eye icon next to the Vision tag. Also
removes the em dashes from the Projects export and import labels.
* Studio: gate recommended folders on real weights and polish the selector
Only show a Recommended chip once the well-known dir actually holds
weights, so an empty LM Studio or Ollama scaffold no longer suggests
itself. _dir_has_downloaded_model checks for a GGUF/safetensors file or
a non-empty Ollama manifests store, with a bounded walk.
Selector polish: round the popover and option menus a touch more,
lighten the OOM badge in dark mode, soften the inner dropdown shadow,
even out the padding, and lift the toggle track and field triggers so
their edges read against the popover.
Also catch CogVideoX in the diffusion name fallback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: align the dark Select model panel with the sidebar
Match the popover, fields, dropdowns, tab toggle and row states to the
sidebar surface and accent so the dropdown reads as one piece in dark
mode. The active tab pill and Search Hub button sit a touch lighter
than the track, and the inner option menus drop their drop shadow for a
flatter look. Light mode is unchanged.
* Studio: re-derive the Select model tab on open
The picker remounts each time the dropdown opens, but the source tab
state did not, so a persisted fine-tuned or connected selection that
only lands in its list after an async load would reopen on Hub. Reset
the active tab to the selection-derived default on the open edge, while
still letting the user switch tabs freely within a session.
* Studio: fold Custom into On Device and polish the picker
Merge the Custom tab into On Device so custom folders sit right below
the downloaded models, with a folder shortcut on the group header.
Rename the first Hub tab to Recommended, give the format dropdown
colored dots, even out the tab row spacing, and tighten the popover
width. Align the folder browser with the app dialogs (soft surface,
roomier padding, green confirm, grey hover).
* Studio: fix On Device controls and nudge the folder browser close
The Hub redesign merge dropped the old Search Hub button styling, so the
On Device search row rendered flat. Point the search input and Search
Hub button at the shared .field-soft surface so they match the rest of
the Hub controls, and lift the folder browser close button slightly.
* Studio: run the Select model search on the Hub search stack
Point the picker at the Hub's useHubModelSearch and useHubInfiniteScroll
instead of its own useHfModelSearch/useInfiniteScroll, scoped to unsloth
so the listing matches the old one. Both the search and the recommended
feed now share the Hub implementation, so there is one search path. The
Hub result folds GGUF params into totalParams, so the dead ggufParams
fallback is dropped.
* Studio: trim the recommended sort to Recommended, Trending, Recent
Drop Downloads and Most likes from the sort dropdown.
* Studio: give the section tabs room off the rounded edge
The fit-mode toggle wrapped the tabs with no inset, so On Device sat
tight against the rounded-full edge. Add a small horizontal inset and
widen the popover a touch to fit it.
* Studio: drop the legacy HF search hooks for the Hub ones
Migrate the training model and dataset sections, export page, onboarding
steps and recipe dataset combobox off useHfModelSearch, useHfDatasetSearch
and useInfiniteScroll onto the Hub equivalents, scoped to unsloth so the
listings match. The picker reads recommended param counts off the search
results it already has instead of a separate fetch. Removes the duplicate
search stack: use-hf-model-search, use-hf-dataset-search,
use-hf-paginated-search, use-infinite-scroll, use-recommended-model-vram
and the old lib/hf-cache.
* Fix model selector section toggle proportions
Remove the fit-mode track inset so the active pill sits flush to the
track edge, matching the Hub's segmented controls.
* Tighten model selector width and tab padding
Reduce the popover width so the right edge aligns with the row, and
widen the fit-mode tab padding so On Device clears the track edge.
* Refine Recommended formats, sort width and tab padding
Recommended now suggests GGUF anywhere and MLX only on Mac, never
safetensors. Size the sort dropdown to its label so Recommended no
longer truncates, and match the On Device trailing gap to the active
pill's leading inset.
* Flush section toggle and match dropdown font to Search Hub
Drop the trailing track pad so the active pill fits the track exactly
at either end. Size the sort and format dropdown text to text-xs like
the Search Hub button, and clip long labels without an ellipsis.
* Fix sort menu checkmark overlap and lock dropdown widths
Keep the option's right padding so the selected checkmark no longer
overlaps the label, and let the open menu expand to fit it. Set the
format and sort triggers to a fixed width matching the Search Hub
button so they always line up.
* Keep section toggle and dropdowns on one row
Drop the wrap and size the Search Hub button, format and sort dropdowns
to a shared 100px so they stay equal width and fit on one row without
widening the box.
* Studio: pre-load inference settings dialog with native context
Add a gear on downloaded GGUF quant rows that opens a settings dialog
to adjust inference parameters before loading a model:
- Context length, KV cache dtype, speculative decoding and tensor
parallelism, all written to the runtime store the load call reads.
- Settings can be remembered per model in localStorage.
- The context slider ceiling and "Model supports up to N tokens" come
from the model's native context, read from GGUF metadata and returned
by /api/models/gguf-variants once a variant is downloaded.
Also drop models Studio can't run for chat (diffusion, image, video)
from the recommended feed and Hub search, plus minor selector polish
on row hover padding, Search Hub and dropdown widths, and tab spacing.
* Studio: model selector polish and memory-aware load warning
Search and listing:
- Drop the "Recommended" and "Hugging Face" section labels while
searching so results read as one list; keep the format and sort
dropdowns visible so search results can still be sorted and filtered.
- Request gguf metadata in the Hub listing so GGUF repos report a
parameter count, restoring the OOM badge for repos without a size
token in the name (Kimi, MiniMax, GLM).
Load settings dialog:
- Warn when weights plus the KV cache at the chosen context exceed
available memory. The KV size is sized by the backend's
architecture-aware estimator via a new kv-cache-estimate endpoint;
the budget uses VRAM plus system RAM. Best-effort, no warning on
failure or on auto context.
- Context Length placeholder reads "auto"; dark background slightly
lighter.
Other:
- Clicking the Custom Folders header opens the folder browser; its
title now reads "Select folder to detect models".
- On Device sort lists Downloaded last.
- Smaller chat template editor font; rounded wrapper clips the prompt
and template editor scrollbars so the right corners stay round.
* Studio: fix load dialog memory warning budget and KV dropdown width
- The memory warning never fired without a discrete GPU. useGpuInfo
returned zero system RAM in that case, so the budget was always zero.
Surface system RAM even when no GPU is present (Mac unified memory),
and have the load dialog read memory directly instead of through props.
- Give the dialog fields shrink-0 so the KV Cache Dtype value (e.g.
q8_0) is not squeezed and clipped by the row.
* Studio: fold fine-tuned models into On Device tab
Remove the Hub models and Fine-tuned source tabs. Fine-tuned models now
show as a section in the Hub tab's On Device view, above Custom Folders,
with the Train icon and a collapse toggle. The section only appears when
the user has fine-tuned models. With no external providers the lone Hub
tab hides its own toggle.
Also: tick-circle Show hidden checkbox and drop the divider above Eject;
keep run settings load params (KV cache dtype, speculative, tensor
parallel) from being clobbered by a mid-load status poll.
* Studio: stage load settings in the sidebar with a Load on selection toggle
Replace the pre-load settings popup with a staging flow in the Run settings
sidebar. The gear on a downloaded quant row now stages the model and opens
Run settings with Load model and Cancel buttons, so options like context
length, KV cache, speculative decoding and tensor parallelism are set before
the model loads. A "Remember these settings" tick reuses them next time.
Add a global Load on selection toggle in Settings, Chat tab (default on).
On: Unsloth auto-picks the best settings for your hardware and loads on
selection. Off: picking a model stages it in Run settings to customize first.
The gear always stages, regardless of the toggle.
Other polish in this change:
- Fine-tuned models live under the On Device tab, with a train icon on the
header that jumps to the Fine-tuned section.
- Default to the On Device tab when downloads exist, otherwise the last used
section.
- Standard Unsloth tooltips on the train, folder and gear icons.
- Request the gguf param count on every Hub listing fetch so Kimi, MiniMax
and GLM show a size badge.
- Search Hub hover state, scrollbar position and minor spacing fixes.
Remove the old inference load settings dialog.
* Studio: always show the fine-tuned shortcut and smooth out the picker
- Fine-tuned section and its train shortcut now always show on On Device,
with an empty state when no fine-tuned models exist yet.
- Folder icon on the header jumps to Custom Folders instead of opening the
browse popup, matching the train shortcut.
- Folder browser keeps the list mounted and dims it while refetching, so
toggling Show hidden or changing folders no longer flashes.
- Drop the tooltip hover grace area in the picker so moving between the
train, folder and gear icons switches the tooltip at once.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add quantization display options and drop the fine-tuned empty text
- Settings, Chat: 'Expand quantizations' toggle. On expands every On Device
GGUF model's quantizations by default; off keeps them behind a click
(default).
- Settings, Chat: 'Show all quantizations' toggle. On lists every quant
including ones not downloaded (default); off shows downloaded only.
- Remove the empty-state line under the Fine-tuned header; the header still
shows on its own.
* Studio: let expanded quantizations collapse on click and split the On/Off help
- With Expand quantizations on, clicking an On Device model now collapses or
re-expands its quantizations. The collapse state is in memory only, so it
resets on reload and when the setting is toggled.
- Put the Off sentence on its own line in the quantization setting descriptions.
* Studio: reorder chat settings and rename the model section
- Rename the Models section to Select model settings and move it above the
Chat menu section.
- Trim the section and Load on selection descriptions.
* Studio: tighten the On/Off lines in the model setting descriptions
Use a line break instead of separate spans so the On and Off lines sit on
consecutive lines without the extra paragraph gap.
* Studio: top-align the Load on selection toggle
Add an alignTop option to SettingsRow and use it so the toggle sits at the top
of the row next to the label, not centered against the tall description.
* Studio: put the gear hint and example chip on one line
Move the gear example chip inline with its label so it reads as a single line
instead of wrapping onto its own row.
* Studio: move the New badge from API keys to Chat settings
Add the New badge to the Chat settings tab and drop it from API keys.
* Studio: line the Load on selection toggle up with the first description line
Offset the top-aligned control past the label row so it sits next to the On
line instead of the label.
* Studio: label the chat menu item Chat with Files (RAG)
Rename the Chat with Files entry in the chat menu settings to clarify it is RAG.
* Studio: drop the pill around the gear example so it fits on one line
Remove the background and padding from the gear example chip so it sits inline
with its label at a lower height.
* Studio: fold the gear example into the description line spacing
Render the gear example inline in the same text block so its line spacing
matches the On and Off lines instead of an extra flex gap.
* Studio: scope Show all quantizations to On Device only
Gate the downloaded-only filter on an onDevice flag so Recommended and other
browse lists always show every quant, and note On Device in the setting copy.
* Studio: tidy On Device GGUF rows
- Drop the redundant Quantizations subheading under On Device models.
- Relay GGUF vision support up to the model name as a Vision badge instead.
- Drop the repo size from On Device GGUF model rows since the quants already
show their size.
* Studio: pin the eject button and tidy General settings
- Move Eject loaded model out of the scrollable list into a centered footer so
it stays in view no matter how far the list is scrolled.
- Space out and center the gear example in the Load on selection description.
- General: drop the duplicate Unsloth version section, move llama.cpp
notifications above Helper LLM, and note new models in its description.
* Studio: add left padding before the gear example
Nudge the gear example away from its label with a small left margin.
* Studio: make the eject footer a sticky bar over the list
Pin Eject loaded model to the bottom of the scroll area with the menu
background so rows scroll under it, and drop the divider line.
* Studio: drop the eject footer background, keep it a sticky button
Make the sticky eject a centered transparent button so it coexists with the
rows scrolling behind it. The wrapper ignores pointer events so only the button
is clickable.
* Studio: give the eject button a solid background
Add the menu background, a border and a soft shadow to the sticky eject button
so it reads as a floating button over the list.
* Studio: restore the eject footer block, keep hover on the button only
Bring back the full-width menu background behind the sticky eject footer, but
keep the button compact and centered so the hover stays on the button.
* Studio: show the vision badge on On Device rows without expanding
- cached-gguf listing reports has_vision (mmproj present), so the badge shows
on the model name without opening the quantizations.
- Make the vision badge icon-only with a tooltip: "This model can process
image inputs". Falls back to the expander-reported value on older backends.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make LM Studio and Local models sections collapsible
* Fade the eject footer instead of a solid block
* Wrap the vision badge in a bordered pill
* Taller model list with the eject footer pinned to the bottom
* Use purple for the vision badge to set it apart from GGUF
* Reduce the model list height
* Make the eject button inline with no background block
* Match the vision badge color to the Hub indigo tone
* Shorten the model list and square off the format tags
* Pin the eject button so it floats at the bottom of the list
* Give the floating eject button a tinted background
* Add bottom clearance so the list ends on white space under the eject button
* Match eject button to the menu background and unify the settings gear icon
* Move eject below the list and match its shadow and dark background
* Drop the min height so short model lists leave no white space
* Remove the eject button fill so it never covers the list
* Nest dropdown hover radius inside the menu corners
* Float the eject pill again and fix sort dropdown hover radius
* Make the eject button opaque in both themes on hover and dark
* Trim the model menu bottom padding so it stops clipping the last row
* Match dark eject background to the Search Hub button and pad row indicators
* Fade the model list bottom edge while rows sit below the fold
* Lift the eject button and trim the section toggle right padding
* Nudge the model list taller and run the bottom fade to the box edge
* Nudge the model list slightly taller
* Remove the eject button shadow
* Align the eject button to the right
* Widen the Search Hub and dropdowns and right-align them
* Seat the eject button at the base and restore On Device right padding
* Reduce the Search Hub and dropdown width by 4px
* Widen the model menu so the section toggle keeps its padding
* Make the eject button an icon-only button with shadow
* Tighten section tab padding to cut the grey between tabs
* Revert section tab padding back to px-3
* Remove the section toggle trailing padding
* Add an eject button beside the model selector trigger
* Shrink the in-list eject button to a smaller proportional size
* Raise the in-list eject button
* Make the trigger eject a bare icon next to the dropdown arrow
* Revert eject back to the labeled button on the right
* Place the format and sort dropdowns next to the section toggle
* Raise the eject button and shorten its label to Eject model
* Widen the gap between the toggle and dropdowns slightly
* Align Search Hub with the last dropdown via a shared-width grid
* Narrow the model menu for symmetric padding
* Stretch the search row so Search Hub lines up with the last dropdown
* Inset the list so the right padding matches the left
* Right-align dropdowns and full-width search so Search Hub meets the last dropdown
* Pack section toggle and dropdowns with a uniform gap
* Inset search row so Search Hub aligns with the Trending dropdown
* Trim model menu right padding to match the left
* Nudge model list scrollbar inward
* Move eject button to the bottom left with a light shadow
* Shorten show all quantizations description
* Keep eject button right-aligned, nudged in from the edge
* Move Connected into the section toggle as a cloud-icon tab
* Align eject button with the format tag edge
* Right-align Connected layout so Search Hub meets Trending
* Download selected models through the Hub download manager
* Add Other models section for non-Unsloth downloads
* Add directions icon and shortcut for Other models section
* Space out subheadings and gate Other models on non-Unsloth downloads
* Use direction-right icon for Other models
* Use flag icon for Other models
* Widen Connected menu so dropdowns align with Search Hub
* Model selector: truncate long quant labels and tidy layout
- Hub GGUF card: truncate long file-path quant labels with an ellipsis
instead of overflowing the row.
- Connected layout: left-pack the dropdowns and size the box so the last
dropdown's right gap matches the pill's left gap, with Search Hub on its edge.
- On Device: show MLX/Safetensors with the size on non-GGUF rows.
- Connected list rows use the same grey hover as the tabs; the selected
section tab no longer shows a hover change.
* Model selector: drop stale custom section on restore
A persisted custom section value no longer maps to a tab, so restoring it
opened the picker to an empty view. Fall back to recommended instead.
* Model selector: align the non-connected search bar with the All dropdown
Nudge the non-connected box width so the search bar's right edge meets the
All dropdown, which lands Search Hub on the last dropdown's edge.
* Studio chat model selector: remember last tab, route non-GGUF downloads through Hub, stack overlays
- Restore the last Hub section (Recommended / On Device) on every open instead of always snapping to On Device when downloads exist.
- Route uncached non-GGUF repos (safetensors / MLX) through the Hub download manager via a snapshot download, so every model download shows in the bottom-right indicator and follows Load on selection like GGUF.
- Allow safetensors in Recommended on Mac (they run locally there now), and honor the Safetensors format filter instead of dropping it via the recommendation default.
- Stack bottom-right overlays in one column so the download panel and banners never overlap.
- Add evenly spaced divider lines between the On Device subheadings.
- Pad the bottom of the list so the floating Eject pill never covers the last row.
* Studio downloads panel: widen left padding on header and rows
Bump the left inset to pl-4 while keeping pr-3 so the collapse and cancel buttons stay put.
* Studio: update cached-gguf route tests for the has_vision field
list_cached_gguf now returns has_vision per row (vision badge on On Device);
the expected dicts were missing it. True for the mmproj vision repo, False elsewhere.
* Studio: keep MLX/safetensors selectable in chat-only Mac search
The empty Recommended view allows GGUF plus MLX/safetensors on Mac, but the
curated and HF search lists dropped non-GGUF in chat-only via a GGUF-only filter,
so typing a query hid runnable Mac models. Reuse isRecommendableFormat in both
lists so search matches the empty view (chat-only non-Mac stays GGUF-only).
* Model selector: restore global model search and fix GGUF/device-fit regressions
- Search: training, export and onboarding pickers searched only the unsloth org
on a typed query. Restore the prior behavior (global Hub search with unsloth
floated first when a query is typed, curated unsloth listing when empty).
- Recommended browse: the GGUF/MLX-only gate ran before the format filter, so
the Safetensors filter and the Trending/Recent sorts always came back empty.
Apply that gate only for the Recommended sort and chat-only mode.
- GGUF metadata: request the gguf expand field through listModels so repos with
no size token in the name (Kimi, MiniMax, GLM) report a param count for the
size and OOM badge.
- Local GGUF: custom-folder and standalone ./models/*.gguf files now load
directly with the GGUF marker instead of dead-ending in the variant expander,
and scanned GGUF folders are classified via a backend model_format hint.
- Device fit: use system RAM in the budget on unified-memory hosts, and keep MLX
rows selectable on chat-only Macs.
- kv-cache-estimate: resolve the quant from the snapshot-relative path, skip MTP
drafter files, and prefer the most complete snapshot (mirrors the variant
scanner). Bound the Ollama manifest walk.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Model selector: classify suffixless local GGUF folders consistently
Complete the model_format plumbing so a GGUF folder is detected and loaded
through the same GGUF path that the format filter already uses:
- _scan_models_dir: a config.json no longer disqualifies a folder whose only
weights are .gguf, so HF GGUF repos shipping a config still classify as GGUF.
- _scan_lmstudio_dir: emit model_format for every GGUF row (LM Studio dirs
rarely carry a -GGUF suffix), via a shared _dir_model_format helper.
- Custom Folders and LM Studio rows: use localModelIsGguf (the same helper the
filter uses) so the row label, expand-vs-direct-load, and isGguf flag agree;
a suffixless GGUF folder no longer filters as GGUF but loads as non-GGUF.
Adds tests/test_local_model_format.py covering the classification rule.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio model selector: tighten section spacing
Trim each subheading's gap to its rows (pb-1.5 to pb-1) and pull the On Device
heading block tight to the controls while Recommended keeps a little top room.
* Hub: format filter fix, sort defaults, avatar and layout polish
- Format dropdown now filters the feed's Latest list too, so the default
GGUF hides fp8/safetensors and picking a format changes the rows.
- Latest Unsloth Models sorts by newest created, not recently updated.
- Sort dropdown order: Newest, Trending, Most downloads, Recently
updated, Most likes.
- Unsloth uploads with no upstream provider logo show the Unsloth avatar
instead of a colored initial.
- Owner scope pill gets a little more room before the chevron.
- README detail column lines up with the top bar (both-edges gutter).
- Long file-path quant labels truncate instead of overflowing the row.
- Model list keyboard nav no longer clips the focus ring.
- Run settings sheet: restore the Remember settings toggle and larger
Load/Cancel buttons on the staged load flow.
* Hub: hide the RAG embedding model from browse previews
The Hub discover feed and chat model selector pull from the Hugging Face
listing on the client, which the backend _is_hidden_model filter never
touches, so the RAG embedder (unsloth/bge-small-en-v1.5-GGUF) and the
llama.cpp validation probe leaked into the lists.
Added isHiddenModelId mirroring the backend needles and filtered it out of
the discover rows, the trending feed, and the selector's recommended and
Hugging Face search lists. Per-repo file and download views are untouched,
so the model is never deleted and a reinstall still shows it as already
downloaded.
* Studio: skip hidden dirs when checking a folder for downloaded models
_dir_has_downloaded_model walked the tree with rglob("*") bounded by
max_entries. rglob yields entries in arbitrary order and counts every one, so a
model directory that also holds a large hidden subtree (.git/.cache/venv) could
exhaust the budget before reaching the real weights and falsely report no model,
hiding a valid Recommended-folder chip. Replace the generic-weights pass with a
bounded BFS that skips hidden directories so their entries can't starve the walk.
Adds a regression test (50-entry .git beside the weights, max_entries=10).
* Fix/adjust model selector handling for PR #6364
* Studio: address codex review on the staging/recommended-folder paths
- chat-page auto-load: selectModel only clears pendingSelection on success, so a
failed auto-load left the hidden stage (and its edited load knobs) behind.
Abandon the stage when it still matches the failed pick.
- model picker: count fine-tuned rows in the On Device empty check so a
fine-tuned-only tab no longer shows a false 'No models on device' message
above the Fine-tuned section.
- general settings: add the remembered per-model load settings key to PREFS_KEYS
so 'Reset all local preferences' actually clears it.
- recommended-folders: recognize PyTorch .bin weights (gated by the scanner's
weight-name prefixes) so a .bin-only model folder still earns a chip; add tests.
* Studio: name-gate .bin weight detection and complete selector preference reset
Follow-up to the codex review on the model_format/recommended-folder paths:
- _dir_model_format and _scan_models_dir treated any .bin (incl. tokenizer.bin)
as a non-GGUF weight, so a suffixless GGUF folder shipping a companion .bin was
misclassified as a plain checkpoint and routed through the wrong load path.
Factor the scanner's weight-name gating into shared _is_weight_bin /
_has_non_gguf_weights helpers and use them everywhere (also in
_dir_has_downloaded_model).
- PREFS_KEYS was missing the new 'Select model settings' keys (load on selection,
expand/show-all quantizations), so 'Reset all local preferences' left them set.
- On Device cached search dropped the active format filter while a query was
typed; keep matchesFormatFilter applied so the format dropdown stays consistent.
Adds tests for the tokenizer.bin vs weight-.bin classification.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: validate Ollama blobs, gate staged context, honor RAM budget on no-GPU hosts
- recommended-folders: only count an Ollama dir once its manifest resolves to an
on-disk model blob, so a failed/pruned pull no longer surfaces an empty chip
- GGUF variant click: only seed the staged contextLength for already-downloaded
picks, so choosing an undownloaded quant from a partially cached repo still
starts its download (the staging effect short-circuits on a known context)
- device fit: classify GGUF variants against the system-RAM budget on no-GPU /
unified-memory hosts instead of reporting everything as fits, and pass
systemRamGb to every variant expander regardless of gpu.available
* Studio: scope Hub search to Recommended, fix staged non-GGUF settings, keep local MLX on Mac
- model picker: only run the Hub search hooks on the Recommended section. On
Device / Connected render local data, so typing there no longer fires HF
requests or a spinner and the local/offline flow is preserved
- chat settings: when a pick is staged, decide the GGUF-only controls from the
staged model's type, not the currently loaded model's. A staged non-GGUF Hub
repo no longer inherits a loaded GGUF's context/KV/speculative controls
- On Device: keep local MLX builds in ./models selectable on Mac (chat-only ran
GGUF/MLX only, but the filter dropped MLX before the format toggle)
---------
Co-authored-by: shimmyshimmer <info@unsloth.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* studio: persist personalization (profile + theme) server-side
Profile name/nickname/avatar and appearance (theme) were stored only in the
browser's localStorage, so every browser or device that connected to the same
Studio started from defaults and forgot the user's personalization.
Persist them server-side (single-account, stored as one JSON blob in
app_settings) so they follow the account:
- utils/personalization_settings.py + GET/PUT /api/settings/personalization,
with validation (theme/shape enums, avatar must be an image data URL capped at
512 KB) and a 'saved' flag.
- Frontend usePersonalizationSync (mounted in the root layout when signed in)
hydrates the profile + theme stores from the server when a blob exists, and
otherwise migrates the existing local settings up once so nothing is lost;
later changes are written through, debounced. Writers keep using the local
stores unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust personalization sync for PR #6516
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Studio personalization sync edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* fix: pin anyio to <4.14.0 to fix RuntimeError on Python 3.13
Fixes#6483
anyio 4.14+ introduced cancel scope changes that cause
RuntimeError on Python 3.13. Pin to <4.14.0 until the issue is
resolved upstream.
---
If this helps, consider buying me a coffee: https://buymeacoffee.com/muhamedfazalps
* Cap anyio<4.14.0 in single-env constraints so the pin holds across all install steps (#6483)
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Windows installer: fix DiskPart UAC, drive-root cache, spurious rename warning, CPU-base messaging
amd-smi gate (DiskPart UAC mid-install): the AMD torch wheel ships hipInfo.exe
inside the venv, and the bitsandbytes fix prepends that venv Scripts dir to PATH.
shutil.which("hipinfo") then found it and flipped _amd_smi_allowed() to True, so
the post-install AMD probe fell through to `amd-smi list` (the venv hipInfo failed
to report gcnArchName, which is why the arch came from the GPU-name table) and
amd-smi elevated, popping the DiskPart UAC. Fix: a hipinfo resolved inside the
active venv (sys.prefix) is the torch-wheel binary, not a HIP SDK, and must not
open the gate. Mirrored in install_python_stack.py, install_llama_prebuilt.py, and
backend utils/hardware/amd.py (the runtime VRAM poller had the same latent prompt).
TORCHINDUCTOR_CACHE_DIR: move from C:\tc to <StudioHome>\TORCHINDUCTOR_CACHE_DIR so
the inductor/Triton cache lives under the user's Studio home, not the system drive
root. Long paths are already enabled above so deep inductor paths still fit.
unsloth.exe rename: skip the rename (and its "pip may fail with WinError 32"
warning) when SKIP_STUDIO_BASE=1. In the install.ps1 flow base packages are not
reinstalled, so unsloth.exe is never rewritten; the self-rename only failed because
setup runs via unsloth.exe (the running launcher holds its own file). The
'studio update' flow still attempts it.
CPU PyTorch messaging: clarify that the CPU base is temporary and setup replaces it
with GPU ROCm wheels, and print an explicit "GPU ROCm PyTorch installed" line after
the AMD wheels land, so the log makes clear the final install is GPU-accelerated.
Adds two regression tests covering the venv-internal vs external hipInfo gate.
Verified end-to-end on a Strix Halo box (Radeon 8060S / gfx1151): install.ps1
--local from this branch completed exit 0 with no DiskPart prompt, no rename
warning, the cache under the Studio home, and "GPU ROCm PyTorch installed
(gfx1151)"; Studio then booted and detected "ROCm (HIP 7.13.99004) -- AMD Radeon
8060S Graphics".
* Windows installer: drop the unreliable unsloth.exe rename and its WinError 32 warning
setup.ps1 used to rename the running unsloth.exe out of the way before the
base-package upgrade so pip could replace it. That rename never actually
worked: setup runs *via* unsloth.exe, so renaming our own running
uv-trampoline launcher failed with a sharing violation (WinError 32) and only
printed a scary 'could not rename unsloth.exe; pip may fail with WinError 32'
warning on every Windows install and update.
It also was not needed. pip tolerates a running/locked console-script .exe: it
moves the old one aside and writes the new one. The base upgrade routes through
pip on Windows, so the upgrade succeeds (or, in the install.ps1 flow with
SKIP_STUDIO_BASE=1, the base is not touched at all) and unsloth.exe is left
intact either way.
Removing the rename block and its failed-install restore block removes the
false warning for all Windows devices in both the install and update flows.
* Windows installer: gate venv-internal hipInfo.exe in PowerShell amd-smi probe; harden venv path checks
Follow-up to PR #6296.
- install.ps1 and setup.ps1: ignore the AMD torch wheel hipInfo.exe that lives
inside the Studio venv when probing for a HIP SDK, so amd-smi no longer reopens
the DiskPart UAC during install/update. Mirrors _path_inside_venv in the Python
installers, which already do this.
- amd.py, install_llama_prebuilt.py, install_python_stack.py: normcase the venv
containment check (Windows paths are case-insensitive) and run the
HIP_PATH/ROCM_PATH candidate through it too.
- setup.ps1: fall back to a short TORCHINDUCTOR cache dir when long paths are
unavailable, and create the dir wildcard-safely.
- tests: isolate sys.prefix in the gate helper, add HIP_PATH/ROCM_PATH cases, and
assert the PowerShell venv exclusion.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Windows installer: install ROCm PyTorch directly for a known AMD arch
When the GPU arch is known (name-inferred from the GPU-name table) but ROCm
could not be probe-verified (no HIP SDK, no amd-smi), the bootstrap installed
a CPU PyTorch base that setup.ps1 then force-reinstalled as ROCm. The
repo.amd.com wheels bundle their own runtime (no HIP SDK required), which
setup.ps1 already relies on, so the CPU base was a pure wasted download/install.
- Gate the ROCm index on a known arch, not only on probe-verified ROCm, so a
mapped arch installs ROCm torch directly. Unmapped arches and no-GPU hosts
still get CPU (unchanged).
- Fall back to a CPU base if the ROCm-index install fails, so a transient
repo.amd.com outage does not abort the install (setup.ps1 retries ROCm).
- Correct the stale comment that claimed ROCm wheels need a confirmed HIP SDK.
- Add a regression test for the arch-based gate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Windows installer: correct the unsloth.exe rename-removal comment
The comment claimed the base upgrade 'routes through pip on Windows' and that
pip 'moves the old unsloth.exe aside, then writes the new one'. That is not what
the code does. install_python_stack tries uv first; on a locked launcher uv
aborts and falls back to pip, but the pip fallback strips --upgrade-package and
base.txt lists only bare unsloth/unsloth-zoo, so pip finds them already
satisfied and no-ops. The running unsloth.exe is left intact at its current
version either way. Reword the comment to describe the real uv-first /
pip-fallback-no-op behavior. No functional change.
* Windows installer: close two gaps in the venv-internal hipinfo exclusion
Review follow-up. The amd-smi/DiskPart gate could still reopen in two cases:
- setup.ps1 ran the HIP probe long before $VenvDir is assigned, so without
VIRTUAL_ENV (the `unsloth studio update` path) $venvRoots was empty and the
venv-internal hipInfo.exe was not recognized. Seed the venv root from
UNSLOTH_SETUP_PYTHON and the default Studio home too (both installers).
- The HIP_PATH/ROCM_PATH candidate was accepted without the venv filter, so an
env var pointing into the venv (AMD wheel) still set $HipSdkInstalled. Run
Test-HipinfoIsVenvInternal on the candidate as well (both installers).
Extend the PS gate test to assert both. Both .ps1 parse clean; install tests
pass (the venv-internal / HIP probe coverage at 359 passed).
* Windows installer: correct the CPU-base message for arches with no ROCm wheels
After gating the ROCm index on a known arch, a mapped arch sets $ROCmIndexUrl
and installs ROCm directly, so it no longer reaches the "temporary CPU base"
branch. That branch is now reached only by a name-inferred arch with no ROCm
wheels (e.g. RDNA2 gfx103X), where setup.ps1 does NOT install ROCm. The old
text ("setup replaces it with GPU ROCm wheels ... the final install IS
GPU-accelerated") was therefore always wrong there. Say plainly that PyTorch
stays on CPU for this GPU.
* Windows installer: seed the venv-internal hipInfo check from a custom Studio home
Test-HipinfoIsVenvInternal seeded the venv root from VIRTUAL_ENV, VenvDir, the
setup python, and the default %USERPROFILE% path only. A standalone
`unsloth studio update` with a custom UNSLOTH_STUDIO_HOME (or STUDIO_HOME alias)
and none of those set would not recognize the venv hipInfo on PATH, reopening the
amd-smi/DiskPart gate. Seed the custom home too, in both installers, and assert
it in the gate test.
* Studio installer: resolve venv aliases and expand ~ in the hipInfo venv filter
Two review points on the amd-smi/DiskPart UAC gate:
1. _path_inside_venv compared os.path.abspath of sys.prefix and the hipInfo
path, which does not resolve symlinks, junctions, or 8.3 short names. A venv
reached through an aliased path then fails the check, so its bundled
hipInfo.exe is mistaken for an external HIP SDK and amd-smi runs (the
DiskPart prompt this fix exists to suppress). Switch to os.path.realpath in
all three copies (amd.py, install_llama_prebuilt.py, install_python_stack.py).
2. setup.ps1's early venv-internal hipInfo probe seeded the venv root from a
custom Studio home (UNSLOTH_STUDIO_HOME / STUDIO_HOME) without expanding a
leading ~, while the canonical resolver does. With a tilde form,
[IO.Path]::GetFullPath kept the literal ~ relative to cwd, so the custom-home
hipInfo escaped the filter and reopened the gate. Expand ~ in the probe the
same way as the resolver.
tests/studio/install/test_pr5940_followups.py: 30 passed (adds a symlink
realpath case and a setup.ps1 tilde-expansion guard).
* Studio installer: mirror the hipInfo venv filter and ROCm wheel pins into install.ps1
Follow-up review on the same install.ps1 paths:
1. install.ps1's venv-internal hipInfo probe (Test-HipinfoIsVenvInternal)
seeded the venv root from a custom Studio home without expanding a leading
~, unlike the canonical resolver and setup.ps1. A tilde form left
[IO.Path]::GetFullPath with the literal ~ (relative to cwd), so the
custom-home hipInfo escaped the filter and reopened the amd-smi/DiskPart
gate. Expand ~ in the probe, matching the setup.ps1 fix.
2. The AMD ROCm path installed torchvision/torchaudio bare while pinning torch
to below 2.12. AMD's per-arch index publishes the companions independently
and may ship torchvision 0.27 (for torch 2.12) before removing 0.26, so a
bare resolve can pick an ABI-incompatible set and fall back to CPU. Add
torchvision/torchaudio floor maps and pass the pinned specs, mirroring
setup.ps1 and install_python_stack.py.
3. The ROCm-to-CPU fallback torch install used Invoke-InstallCommand (no
retry), the only torch step in the file without it. Switch to
Invoke-InstallCommandRetry so the recovery path survives a transient index
failure.
tests/studio/install/test_pr5940_followups.py: 33 passed (parametrized tilde
check over both installers, a torch/companion floor-map parity test, and a
CPU-fallback retry guard).
* Studio installer: scan all PATH hipinfo so the venv copy can't shadow a real HIP SDK
The amd-smi HIP-SDK probe used shutil.which("hipinfo") / Get-Command hipinfo,
which return only the first hit on PATH. The AMD torch wheel ships hipInfo.exe
inside the venv and the bnb fix (plus the Studio backend) prepend the venv
Scripts dir to PATH, so that venv-internal copy lands first. When a real HIP SDK
hipinfo sits later on PATH with HIP_PATH/ROCM_PATH unset, the first-hit probe
stopped at the venv copy, treated it as "not a HIP SDK", and closed the amd-smi
gate -- AMD users in that PATH-only SDK setup lost amd-smi telemetry and could
fall back to CPU. Scan every PATH entry and keep the first hipinfo that is not
venv-internal; only the venv copy is ignored, so the UAC/DiskPart suppression is
unchanged.
Applied to all three Python copies (install_llama_prebuilt.py,
install_python_stack.py, backend/utils/hardware/amd.py) via a new
_external_hipinfo_on_path helper, and both PowerShell callers (install.ps1,
setup.ps1) now use Get-Command hipinfo -All filtered by Test-HipinfoIsVenvInternal.
tests/studio/install/test_pr5940_followups.py: 36 passed (real-PATH scan tests, a
shadow-regression test for the exact venv-first ordering, and a parity check that
every Python copy uses the scanning helper).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio uninstallers: fix leftovers (false "removed", shared icon, llama lock)
Auditing a dual native+WSL uninstall on a real device surfaced three leftovers:
1. uninstall.ps1 removed the data dir (which holds unsloth.ico) before the
shortcuts that reference that icon, so Explorer's icon cache briefly held it
open. Remove-Item -Recurse reported success yet left the locked file, and the
dir was never re-attempted, so it orphaned with a false "removed" log.
_RemovePath now verifies the path is actually gone (retrying transient locks)
and reports honestly, and the data dir is re-swept after the shortcuts go.
2. install.sh writes a shared unsloth.ico to %LOCALAPPDATA%\Unsloth Studio for
the WSL shortcut, but uninstall.sh never removed it, orphaning the icon (and
dir) after a WSL uninstall. uninstall.sh now drops that icon and the dir when
empty, in both the powershell.exe and drvfs-fallback paths.
3. ~/.unsloth/.llama.cpp.install.lock was never removed, so the rmdir of
~/.unsloth failed and the dir lingered. Both uninstallers now remove the lock.
Verified by running both uninstallers on a real dual install: device fully clean
(no install dirs, shortcuts, PATH/registry entries, shared icon, or lock left).
* install.sh: auto-route Strix Halo WSL to an existing Ubuntu 24.04
ROCm-on-WSL is the GPU runtime for Strix Halo and only targets Ubuntu
24.04. When the installer runs in a newer default distro (e.g. 26.04) it
cannot enable the GPU and silently falls back to CPU. If a 24.04 distro
already exists, re-run the install there and stop in the current one so the
GPU path is taken without the user having to know about the distro
requirement.
Runs before venv creation so the wrong distro is left untouched, guards
against re-route loops via UNSLOTH_WSL_REROUTED, leaves a working ROCm
distro alone (librocdxg present), and skips the GGUF-only / opt-out /
non-Strix cases. When no 24.04 distro exists we keep today's behaviour:
continue to CPU and print the `wsl --install Ubuntu-24.04` guidance, never
auto-downloading a distro.
Adds tests/sh/test_strixhalo_wsl_reroute.sh (hermetic: extracts the
function, rewrites its paths to fixtures, mocks wsl.exe) covering the full
decision matrix, wired into tests/run_all.sh.
* uninstall.ps1: keep shared unsloth.ico for a surviving WSL shortcut
A dual native+WSL install shares %LOCALAPPDATA%\Unsloth Studio\unsloth.ico:
install.sh points the WSL shortcut's icon there while the native install owns the
dir. The native uninstaller removed the whole dir unconditionally, so uninstalling
native while keeping WSL left the WSL shortcut with a blank icon. The old code only
avoided this when Explorer happened to hold the icon open, which is unreliable; on a
real dual install the dir was deleted and the WSL shortcut went blank.
_RemoveDataDirKeepingWslIcon now scans the Start Menu + Desktop for a surviving
"Unsloth Studio (WSL ...).lnk" and, if found, removes everything in the data dir
except unsloth.ico (keeping the dir) instead of deleting it; with no WSL shortcut it
removes the dir as before. uninstall.sh still drops the icon and the empty dir when
WSL itself is uninstalled, so every uninstall order ends clean.
Adds tests/studio/test_uninstall_dual_install_icon.ps1 (AST-extracts the helper and
runs it against a temp dir with controlled shortcut dirs) covering the dual,
native-only, empty, and missing-dir cases, wired into the windows-inference smoke
workflow. Verified on a real dual install: native uninstall now keeps unsloth.ico
and the WSL shortcut's icon stays intact.
* installer: condense AMD/ROCm code comments (no behavior change)
Tighten the comments added for the Strix Halo native+WSL installer work so
they are shorter and clearer without losing intent: the venv-internal hipInfo
amd-smi gate, the ROCm torch/companion floor maps, the WSL 24.04 reroute, and
the dual-install uninstall icon handling. Comment-only; code paths unchanged.
107 insertions, 166 deletions across 11 files.
* install.sh: run the Strix Halo WSL reroute before any STUDIO_HOME write
The reroute fired after mkdir -p "$STUDIO_HOME" and the legacy-venv migration,
so rerouting 26.04 -> 24.04 left an empty ~/.unsloth/studio stub in the origin
distro (and ran venv migration in the distro about to be abandoned). Move the
reroute ahead of the venv section so the origin distro is left untouched, matching
the function's own comment. Behavior is identical on every non-reroute path.
* installer: fix ROCm CPU-fallback, hipinfo gate edge cases, uninstall icon, WSL 22.04
- install.ps1: clear $ROCmIndexUrl/$ROCmTorchFloor after the CPU fallback so the
flavor-repair block does not retry the failed ROCm index and abort the install;
pin the ROCm companion specs ($visionSpec/$audioSpec) in the repair path too.
- install.ps1 + setup.ps1: skip a bare drive root in Test-HipinfoIsVenvInternal so a
non-venv UNSLOTH_SETUP_PYTHON does not match the whole drive; iterate
HIP_PATH/HIP_PATH_57/ROCM_PATH and take the first non-venv hipinfo.
- amd.py, install_llama_prebuilt.py, install_python_stack.py: strip surrounding
quotes from PATH entries before probing for hipinfo.
- install.sh: pipefail the WSL reroute curl|sh; do not reroute supported Ubuntu 22.04.
- uninstall.sh: keep the shared unsloth.ico while any Unsloth shortcut (native or
another WSL distro) still references it, in both the powershell and drvfs paths.
- tests: regression coverage for all of the above.
* installer: forward reroute options, guard ROCm bootstrap, harden hipinfo gate
- install.sh: forward the caller's --package/--python/--verbose/--tauri and a custom
UNSLOTH_STUDIO_HOME into the WSL reroute (was a bare default install); bail on
--local; run the reroute BEFORE dependency/uv install so the origin distro is left
untouched; set UNSLOTH_SKIP_ROCM_WSL_SETUP after a failed reroute so the later
ROCm-on-WSL bootstrap does not install into the unsupported origin distro.
- install.ps1 + setup.ps1: Get-Command hipinfo -CommandType Application so only real
executables match (not an alias/function named hipinfo).
- uninstall.ps1: guard $env:APPDATA when building the default shortcut search dirs.
- tests: cover option forwarding, --local bail, the bootstrap guard, and the gate change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* installer: guard origin ROCm bootstrap on every CPU-only fallback; harden ~ expansion
WSL reroute: the no-wsl.exe, no-24.04-target and --local fallbacks all tell the
user the install continues CPU-only, but only the failed-reroute branch set
UNSLOTH_SKIP_ROCM_WSL_SETUP=1. The later _maybe_bootstrap_rocm_wsl gate keys off
that flag, so the other three branches could still install ROCm into the
unsupported origin distro (e.g. 26.04). Set the skip guard on all of them.
Forward UNSLOTH_ROCM_WSL_AUTO into the reroute so a Tauri/consented GPU bootstrap
carries through to the rerouted 24.04 child instead of dropping to the prompt path.
install.ps1/setup.ps1: guard the venv-probe ~ expansion on a non-empty
$env:USERPROFILE so Join-Path does not throw on a profile-less service account.
Tests: add no-wsl.exe and UNSLOTH_ROCM_WSL_AUTO reroute cases, the USERPROFILE
guard assertion, and route shell-test fixtures through a single trap-cleaned root.
* installer: pin + soften Windows ROCm Python repair, reroute to 22.04, harden gates
install_python_stack.py: the Windows AMD ROCm repair in _ensure_rocm_torch()
installed bare torch/torchvision/torchaudio via the fatal pip_install -- the same
asymmetry already fixed on the PowerShell side. A transient repo.amd.com failure
could abort the whole install even after install.ps1/setup.ps1 fell back to CPU.
Pin companions per-arch (gfx120X/Strix -> the rocm7.2 trio, mirroring the PS floor
maps) and make the retry nonfatal: keep the existing build and let the user re-run
update to retry ROCm, so the chain install.ps1 -> setup.ps1 -> stack stays CPU-safe.
install.sh: reroute now targets an installed Ubuntu 24.04 OR 22.04 (24.04 preferred);
both are AMD-supported for ROCm-on-WSL, matching the leave-alone set, so a box with
only 22.04 reaches the GPU instead of staying CPU-only.
install.ps1/setup.ps1: a bare ~ for UNSLOTH_STUDIO_HOME left an empty Join-Path child
(PS 5.1 throws); fall back to USERPROFILE directly and only join a real remainder.
_path_inside_venv (amd.py + both installers): guard a root-dir sys.prefix so commonpath
can't classify every path on the drive as venv-internal (defensive; venv never at root).
uninstall.sh: guard an empty LOCALAPPDATA in the PS-interop icon cleanup (mirror APPDATA).
Tests: add 22.04-target reroute cases, Windows ROCm pin+nonfatal coverage (text +
behavioral), root-dir guard coverage, and bare-~/LOCALAPPDATA guard assertions.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* install.sh: match WSL reroute target by exact distro name, not substring
The 24.04/22.04 reroute target was chosen with grep -F (substring), so a custom
distro such as 'Ubuntu-24.04-test' (with no exact Ubuntu-24.04) was picked as the
target; the later 'wsl -d Ubuntu-24.04' then fails and the Strix Halo install stays
CPU-only. Match whole lines (grep -ixF) and reuse the matched name so only a real
Ubuntu-24.04/22.04 is targeted. Adds substring-rejection + exact-vs-custom tests.
* install.sh: keep the WSL reroute target to Ubuntu 24.04 (helper-supported only)
The ROCm-on-WSL bootstrap (scripts/install_rocm_wsl_strixhalo.sh) dies on any
VERSION_ID other than 24.04 and pins the noble repo, so treating 22.04 as
GPU-supported let the parent report a successful reroute while the child fell
back to CPU. Drop 22.04 from the supported set and the reroute target list;
24.04 stays the sole target (keeping the exact whole-line distro match). An
already-working ROCm on any other version is still left alone by the librocdxg
check above.
tests: reroute 22.04 cases updated to the 24.04-only behavior; make the
"no wsl.exe" case hermetic so a real host wsl.exe can't leak in on dev boxes;
stop the tauri exit-order check from mis-flagging the reroute helper's
[ "$TAURI_MODE" = true ] && ... --tauri one-liner.
* installer: tighten comment wording across the Strix Halo install/uninstall paths
Condense the verbose multi-line comment blocks (amd-smi hipinfo gate, ROCm
torch install + CPU fallback, WSL reroute, uninstall icon-keep) into fewer,
clearer lines. Comments and a few docstrings only; no code, logic, or
behavior change. Verified with bash -n, the PowerShell parser, and ast.parse,
and the installer test suite still passes.
* add AGPL-3.0 SPDX headers to the .sh/.ps1 scripts missing them
Every shell and PowerShell script under the Studio/installer surface now
carries the standard SPDX-License-Identifier: AGPL-3.0-only + copyright
header (after the shebang where present): the installer (install.sh,
install.ps1), build.sh, the .github and src-tauri scripts, the installer
test suite, and the moe kernel test. Header-only, line endings preserved;
bash -n, the PowerShell parser, and the installer tests all pass.
* installer: drop the duplicate AGPL header from install.sh and install.ps1
Both already carry an SPDX-License-Identifier: AGPL-3.0-only header below
their usage comment block; the prior header pass added a second one at the
top because it only scanned the first few lines. Remove the duplicate so each
file keeps a single original header.
* installer: force-reinstall CPU fallback torch; propagate Tauri NEED_SUDO from reroute
install.ps1/setup.ps1: when the AMD ROCm wheel install fails and we fall back to a
CPU base, force-reinstall the torch/vision/audio triplet. A failed ROCm install can
leave an unpinned ROCm torch (e.g. 2.10.0+rocm on gfx110X/gfx90a) that still
satisfies the CPU torch>=2.4,<2.11.0 range, so without --force-reinstall uv keeps the
ROCm build and only swaps the companions -- a mismatched venv the flavor-repair block
won't fix. setup.ps1 scopes the forced reinstall to the ROCm-fallback path
() so the genuine CPU-only install stays fast.
install.sh: the Strix Halo WSL reroute treated every nonzero child exit as a reroute
failure and fell back to CPU. In --tauri mode the child uses exit 2 ([TAURI:NEED_SUDO])
to ask the desktop app to elevate for the target distro; capture the child's exit code
and propagate exit 2 in Tauri mode (the child already printed the NEED_SUDO line)
instead of masking it. CLI mode still falls back to CPU on a generic failure.
Tests: reroute Tauri exit-2 propagation (and non-Tauri CPU-fallback) cases;
run_func now preserves the child exit code; force-reinstall assertions for both
PowerShell installers.
Note: codex's _rr_q apostrophe finding is a false positive -- the helper already
emits POSIX-correct 'O'\''Brien' and round-trips under both sh and bash.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* setup.ps1: fix $cpuForce array collapse in the ROCm->CPU torch fallback
An if-expression assignment ($cpuForce = if ($ROCmCpuFallback) { @("--force-reinstall") })
collapses the single-element array to a scalar string, so @cpuForce splatting enumerated
it character-by-character into broken single-letter args (- - f o r c e ...), which made
uv/pip reject the install and aborted the whole Studio setup on the AMD ROCm->CPU fallback
path. Build $cpuForce as a real array assigned outside the if-expression so the splat passes
a single --force-reinstall arg. Genuine CPU-only installs stay fast (empty array, no flag).
Test now asserts the array-build form and rejects the if-expression form.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* uninstall: remove the isolated Node.js runtime (~/.unsloth/node)
The isolated Node.js runtime (install_node_prebuilt.py, added with the managed-Node
change) installs to ~/.unsloth/node in default mode -- a sibling of studio, so deleting
<studio> leaves it behind (~200MB orphaned after uninstall). Both uninstallers already
remove the other default-mode siblings (llama.cpp/.cache/.staging); add node alongside
them. uninstall.ps1 also adds it to the handle-lock sweep so a held node.exe can't block
the delete. Env/custom mode nests node under the custom root, removed with that root.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: defer llama.cpp update probes and self-heal MLX on macOS
Two macOS startup problems shared one root area in the FastAPI lifespan:
- The llama.cpp capability + freshness probes ran inline before the server
yielded, so a cold/slow/flaky network on the GitHub freshness check blocked
'Application startup complete' (~34s on CI, longer in the field). Move both
probes to a daemon thread; app.state stays None until ready (status routes
already re-probe at request time). Opt out with UNSLOTH_DISABLE_UPDATE_CHECK=1.
- Train and Export were greyed out because mlx/mlx-lm/mlx-vlm arrive only
transitively and a resolver backtrack silently drops them, so CHAT_ONLY stayed
true. Add utils/mlx_repair.py: when Apple Silicon is detected without MLX,
reinstall mlx/mlx-lm/mlx-vlm by name on a daemon thread and re-run hardware
detection (opt out UNSLOTH_DISABLE_MLX_AUTOREPAIR=1). Surface a chat_only_reason
in /api/health plus a sidebar tooltip so a greyed Train/Export explains itself
instead of failing silently.
* Studio: guard model defaults against a None model name
load_model_defaults(None) called model_name.lower() with no guard, raising
'Error loading model defaults for None' before any model is selected. Return
an empty dict for a falsy/non-str name.
* Studio: drop obsolete upstream macOS + Windows Blackwell prebuilt pins
Both pins worked around gaps in ggml-org upstream prebuilts, but Studio now
routes every GPU host and all of macOS to the unslothai/llama.cpp fork
(published_repo_for_host), which ships the needed bundles, so both pins are
dead code on the default install path:
- macOS b9415: macOS always routes to the fork (its own macOS bundles), and
host_supports_macos_minos() is the backstop. The pin only fired under an
explicit --published-repo ggml-org override.
- Windows Blackwell b9360: Windows-NVIDIA routes to the fork, whose
windows-x64-cuda13 bundle covers Blackwell (manifest max_sm 120, toolkit
13.3), so the pin's self-disable check makes it dormant on every default
install; it could only activate under the same upstream override on a
13.0-13.2 driver.
Remove the pin constants, functions, and call sites. Keep the Blackwell
capability detection (_drop_blackwell_incapable_windows_cuda, _host_is_blackwell,
_windows_cuda_attempt_covers_blackwell) that still drops a non-sm_120 cuda-12.4
build on a Blackwell host. After this, an explicit --published-repo ggml-org
override on a Blackwell 13.0-13.2 host loses its GPU fallback and lands on CPU;
the default fork path is unaffected. Update the install selection-logic and
macOS-compat unit tests for the new no-pin behavior.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: walk back deeper on the macOS upstream prebuilt path
After removing the b9415 macOS pin, the explicit --published-repo ggml-org
upstream path still used the default 2-release fallback, so a pre-macOS-26 host
behind a run of macOS-26-only builds would exhaust two too-new plans (minos is
only checked post-download) and drop to a source build before reaching a
loadable older release. Walk back as deep as the fork macOS path
(DEFAULT_MAX_MACOS_RELEASE_FALLBACKS), turning the removed static pin into
dynamic discovery. Addresses review feedback on the macOS upstream fallback.
* Studio: pin transformers during MLX self-heal so it cannot break Studio
mlx-lm/mlx-vlm declare transformers>=5, but the single-env install pins
transformers==4.57.6. The self-heal used --upgrade with no constraint, so it
could upgrade transformers in the live venv and break the rest of Studio just to
make import mlx.core pass. Pin transformers to the installed version via a
constraint file: the resolver either finds an mlx build compatible with it or
fails (we stay chat-only), never upgrading transformers underneath Studio.
Addresses review feedback on the MLX repair install.
* Studio: harden MLX self-heal against an unsupported mlx-vlm
Pinning transformers alone made uv backtrack mlx-vlm to 0.3.9 (below unsloth-zoo's
mlx-vlm>=0.4.4), which imports but breaks VLM Train/Export -- so the self-heal
could clear chat-only onto a broken stack. Mirror the main installer: set
UV_OVERRIDE=overrides-darwin-arm64.txt so a current mlx-vlm coexists with the
transformers pin, require the same minimum versions unsloth-zoo declares, and
gate/validate on a full mlx_stack_available() check (not a bare import) so an
old or partial stack stays chat-only. Addresses PR review.
* Studio: filter Blackwell-incapable CUDA in resolve_upstream_asset_choice
resolve_upstream_asset_choice returned the first windows-cuda choice unfiltered,
so a Blackwell host could be handed an sm_120-incapable cuda-12.4 build while the
sibling planners drop it. Apply _drop_blackwell_incapable_windows_cuda here too
and fall through to the CPU bundle on a Blackwell host with no capable GPU asset.
Addresses PR review.
* Studio: re-poll health so MLX self-heal reaches an open UI
The sidebar cached the initial /api/health, so a successful background MLX
self-heal (chat_only flips false) did not re-enable Train/Export until a manual
reload. While chat-only for the recoverable mlx_unavailable reason, re-poll
/api/health and stop once Train/Export become available. Addresses PR review.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make the disabled Train/Export tooltip reachable
The greyed Train/Export items pass a tooltip explaining why (e.g. MLX missing),
but a disabled <button> fires no pointer events and SidebarMenuButton only showed
tooltips while collapsed, so the explanation never appeared. Wrap a disabled
button in a focusable span and show its tooltip while expanded too; enabled items
keep the collapsed-only behavior. Addresses PR review.
* Studio: gate Train/Export on the full MLX stack, not bare mlx.core
detect_hardware enabled MLX training whenever `import mlx.core` worked, but the
MLX self-heal (utils/mlx_repair) treats a stack without mlx-lm/mlx-vlm at the
versions unsloth-zoo requires as inadequate. That asymmetry let the UI enable
Train/Export on exactly the partial/backtracked stack the self-heal is trying to
repair (greyed-in-but-broken VLM export). Gate on the same mlx_stack_available()
criterion so a partial stack stays chat-only (reason mlx_unavailable) and the
background repair restores it. Addresses PR review.
* Fix MLX repair and health auth for PR #6494
* Fix macOS upstream prebuilt fallback for PR #6494
* Fix MLX stack validation for PR #6494
* Fix MLX self-heal validation for PR #6494
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Review fixes: isolate hardware-state test, robust transformers pin
- test_chat_only_reason.py: detect_hardware() assigns module globals directly,
which monkeypatch does not revert; the autouse fixture now saves and restores
DEVICE/CHAT_ONLY/CHAT_ONLY_REASON/IS_ROCM so a chat-only verdict here cannot
leak into other backend tests (e.g. test_utils.py) on a GPU host.
- mlx_repair.py: read the transformers version from importlib.metadata instead of
importing transformers, so the install pin is not silently dropped when
transformers has valid metadata but fails to import.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix CI: model full MLX stack in dispatch tests, keep selection test offline
dispatch (macOS) job:
- detect_hardware now gates MLX on the full stack (mlx_stack_available imports
mlx_lm/mlx_vlm and checks dist versions), so faking only mlx.core makes the
apple_silicon_mlx profile resolve to CPU. The dispatch tests assert the routing
decision when the stack IS usable, so model a complete stack:
test_hardware_dispatch_matrix patches utils.mlx_repair.mlx_stack_available and
test_is_mlx_dispatch_gate patches hardware._has_usable_mlx_stack. The stack
predicate's own internals stay covered by test_mlx_repair.py.
Repo tests (CPU) job:
- test_no_cuda_attempt_on_published_path_for_13_1 fell through to a live
github_release_assets() upstream fetch after the Blackwell filter dropped every
published attempt, which the offline security scanner blocks. Stub that fetch so
the walk-back deterministically finds no usable CUDA build and raises
PrebuiltFallback without network.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden MLX self-heal: prepare transformers constraint inside the try
attempt_mlx_repair runs on a daemon thread, but _transformers_constraint_args was
called before the try. A failure there (e.g. tempfile.mkstemp on a full disk or a
bad TMPDIR) would propagate unhandled and silently kill the self-heal thread.
Move the call inside the try and initialize constraint_path so any such failure
is caught and leaves Studio chat-only instead of crashing the thread.
---------
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Use uppercase -GGUF suffix for GGUF export default names
Match the HF/Unsloth GGUF repo convention (Model-GGUF). Default export save dir
'<model>-gguf' -> '<model>-GGUF', the local-path sibling dir token '_gguf' -> '_GGUF',
and the model-name placeholder 'my-model-gguf' -> 'my-model-GGUF'. Pre-filled defaults
only; no backend/path logic change.
* Keep local GGUF sibling dir lowercase to match backend cleanup
The uppercase change to siblingGgufDirectory diverged from the backend's
hard-coded intermediate '<checkpoint>_gguf' dir (core/export/export.py), which
the export relocates GGUFs out of and then deletes. With the user's save dir
defaulting to '_GGUF', that no-longer-equal lowercase sibling would be relocated
and removed, which can delete an existing export. Revert the sibling default to
'_gguf'; the user-facing GGUF export name (buildRelativeSaveDirectory) keeps the
uppercase -GGUF token.
* Tighten GGUF sibling-dir comment
* Trim comments to be more succinct
---------
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
* Show model provider/org in the trust_remote_code consent dialog
The consent dialog showed only the trailing model name (modelName.split('/').pop()),
dropping the owner. The HF org/owner is the 'who do I trust' signal the prompt is
asking about, so render it: 'NVIDIA-Nemotron-3-Nano-4B from "unsloth"'. A null
provider for local paths and bare names leaves those renders unchanged. Applies to
the enable/blocked/malware variants (shared description block).
* Only show consent provider tag for a confident single Hub repo
Tighten parseModelDisplay so the 'from "<provider>"' tag is shown only for a
canonical owner/repo Hub id (exactly one slash, both segments non-empty) that is
not a local path and not part of a multi-repo scan. This avoids misattributing a
relative local directory name (models/llama/7b) or a LoRA base/external repo's
finding to the wrong publisher in a trust decision. Extract a ProviderSuffix
component so both description branches render the clause identically via &&.
* Tighten consent provider-tag comments
* Source the consent provider tag from the backend
The dialog inferred the provider client-side from the model id, using
scanCreatedRepos (a cleanup-only list) to detect multi-repo scope and a
regex that missed bare relative paths like a local owner/model dir. Both
could attribute the scanned code to the wrong publisher.
Move the decision to the backend, where locality and scan scope are
known: _consent_provider returns the owner only for a single, non-local,
canonical owner/repo Hub id, and the route returns it as payload[provider].
The frontend now renders scan.provider directly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Suppress consent provider tag when external auto_map code is scanned
A single Hub repo can declare an auto_map that loads code from another repo
(owner/other--module.Class). The scanner fingerprints that external repo's
Python, but security_targets still held only the primary, so the dialog
attributed the custom code to the primary publisher. Pass the external refs
collected during the scan to _consent_provider and return no provider when any
are present, so attribution is shown only for genuinely self-contained repos.
* Trim comments to be more succinct
---------
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Scan auto_map for GGUF-only repo ids in the consent gate
The trust_remote_code consent gate treated any repo classified GGUF-only
(ships .gguf, no transformers-loadable weight) as having no remote code,
so _config_has_auto_map returned False even when a config declared an
auto_map and the repo shipped the referenced .py. The evaluator then
skipped the scan/fingerprint for that target entirely.
GGUF-inertness is a property of the loader, not the repo. A GGUF
selection loads via llama.cpp, which never reads config.json/auto_map,
and that case is already short-circuited upstream by the caller's
is_gguf check (the inference route skips the remote-code preflight for a
GGUF load). Every path that reaches this helper (export, training,
non-GGUF inference) loads through transformers/Unsloth from_pretrained,
which DOES import auto_map even for a repo that only ships .gguf weights:
the custom module runs before from_pretrained fails on the missing
transformers weights. The export path has no is_gguf guard and passes the
source straight to FastLanguageModel.from_pretrained(trust_remote_code=True),
so the in-helper GGUF skip let a repo with config.json (auto_map) +
modeling_x.py + only a .gguf run unreviewed code during export.
Drop the redundant repo-level GGUF short-circuit (and the now-unused
_is_gguf_repo helper). A direct .gguf file reference stays inert via
_is_direct_gguf_file_ref because that genuinely is a single-file llama.cpp
load; repo ids are always scanned. A GGUF repo whose auto_map ships no .py
still allows via the existing empty-code path, so legitimate GGUF loads
are unaffected (and GGUF inference never reaches this helper at all). Only
a repo that actually contains a .gguf can change behavior here; non-GGUF
repos (safetensors, MLX) are byte-identical before and after.
Update the GGUF auto_map test to expect a scan, and add two regression
tests: a GGUF-only repo shipping auto_map Python is scanned and blocked,
and a transformers-style repo (safetensors / MLX .npz) with auto_map stays
scanned and blocked.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove trust_remote_code config defaults; consent dialog is the only enabler
trust_remote_code is a per-load decision that must go through the remote-code
consent dialog, which scans the auto_map code and pins the exact version. Two
pre-set paths could still enable it without the user reviewing any code, and the
GGUF consent bypass rode one of them into the export flow:
- 4 model_defaults YAMLs shipped trust_remote_code: true (GLM-4.7-Flash,
Nemotron-3-Nano-30B-A3B, PaddleOCR-VL, ERNIE-4.5-VL).
- The frontend consent hook silently enabled trust_remote_code on a clean scan
whenever the caller flagged the model as needing it.
Remove every trust_remote_code key from the model_defaults YAMLs (the loaders
already default to False when the key is absent) and delete the frontend silent
auto-enable, so trust_remote_code is only turned on after the user approves the
scanned code in the dialog.
The three models that genuinely run custom code ship auto_map, which the consent
gate detects on its own via _config_has_auto_map, so the dialog still fires for
them in inference, training, and export (Nemotron is also re-granted by the
trusted-org auto-enable in the workers). GLM-4.7-Flash has no auto_map:
glm4_moe_lite is native in transformers 5.0+ and it loads with
trust_remote_code=False, so its YAML flag was a no-op.
Adds test_yaml_trust_remote_code_removed.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop YAML sections emptied by trust_remote_code removal
Removing trust_remote_code from a model YAML whose section had no other key
left a bare `inference:` header, which PyYAML parses as None;
load_inference_config() then does `model_config.get("inference", {}).get(...)`
and crashes on the None. Drop those now-empty section headers (24 model
defaults, all the `inference:` section) so callers fall back to family/default
inference params, which is the same result those models had before (their only
inference override was trust_remote_code).
Strengthens test_yaml_trust_remote_code_removed.py to forbid any empty/None
top-level section and to load the affected models' inference config end to end.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add sweep asserting every model YAML loads via training + inference paths
Loads all model_defaults YAMLs through load_model_defaults (training) and
load_inference_config (inference) with the exact .get() access patterns the
routes use, so a malformed/None section that crashes either loader is caught.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Assert ex-TRC auto_map models still surface the consent dialog
Removing the trust_remote_code YAML default must not suppress the dialog for the
models that genuinely run custom code. The dialog is driven by the repo's auto_map
(via preflight_remote_code_consent_for_targets -> _config_has_auto_map), not the YAML
flag, so Nemotron/PaddleOCR-VL/ERNIE-4.5-VL still require consent; GLM-4.7-Flash (no
auto_map) takes no dialog and loads natively. Mocks only the Hub config + .py reader.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in consent-gate changes
* Trim comments to be more succinct
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
* studio/setup.ps1: complete Visual Studio 2026 support for the CUDA llama.cpp build
Builds on #6038 (VS 2026 / v18 detection). Once the generator is detected as
Visual Studio 18 2026, two things still broke the CUDA llama.cpp build:
- the CUDA to VS MSBuild integration copied the CUDA .targets into a hardcoded
VC\v170 (VS 2022) BuildCustomizations folder, so a VS 2026 (v180) toolchain
saw no CUDA toolset and cmake failed with "No CUDA toolset found".
- cmake was installed with no version check, but the "Visual Studio 18 2026"
generator requires CMake 4.2+.
This adds Get-VcBuildCustomizationsDir (derives v160/v170/v180 from the detected
generator, falls back to v170), a CMake 4.2 guard for the VS 2026 generator
(upgrades via winget once, else fails with a clear message), and routes both the
copy target and the failure hint through the derived path.
No behavior change for VS 2022/2019/2017: the folder resolves to v170 and the
guard is skipped. Adds windows-latest Pester unit tests (tests/studio_setup_ps1)
plus a workflow that runs them.
* Address review: make VS 2026 self-contained + gate CMake guard to source build
- Find-VsBuildTools now detects VS 2026: vswhere catalog_productLineVersion 2026
-> "Visual Studio 18 2026", and the filesystem scan covers the "18"/"2026" dirs
(incl. non-standard editions like Preview). Adapted from #6038 by
@LeoBorcherding, so the v180 BuildCustomizations path and the CMake guard are
actually reachable on a VS 2026-only host.
- Move the CMake 4.2 guard out of Phase 1 into the committed-source-build branch.
The preferred prebuilt llama.cpp path never reaches it, so a VS 2026 host on
CMake < 4.2 is no longer blocked from using the prebuilt.
- winget upgrade -> install fallback when the on-PATH cmake is not the Kitware
winget package, and log winget failures instead of swallowing them.
- Add a windows-latest Find-VsBuildTools VS 2026 discovery regression test.
* tests(vs2026): define New-FakeVsTree in BeforeAll so It blocks can see it
The Find-VsBuildTools discovery tests are Windows-only (-Skip on non-Windows),
so they first ran on the windows-latest Pester job, where New-FakeVsTree raised
CommandNotFoundException: it was defined in the Describe body, which Pester 5
executes only during discovery, so the function did not persist into the
run-phase It scope. Move it into a BeforeAll block (which runs in the run phase
and is visible to the It blocks). No production code change.
* Address review: probe cmake generator support, fall back to older VS, fix cmake PATH after winget
The VS 2026 CMake guard previously gated only on the cmake version (>= 4.2)
and hard-failed otherwise. Review on #6473 raised three real gaps:
- A VS-bundled cmake below 4.2 can still drive the VS 2026 generator. Probe
cmake --help (Test-CmakeListsGenerator / Test-CmakeCanDriveGenerator) and
accept it when the generator is advertised, not just on the version floor.
- After winget upgrade/install, an older cmake earlier on PATH kept being
resolved. Add-DefaultCmakeToPath prepends the default install dir so the new
cmake wins before re-probing.
- When cmake cannot drive VS 2026 but an older Visual Studio (2022/2019/2017)
is installed and usable, fall back to it (Get-FallbackVsGenerator) instead of
hard-failing, preserving the pre-VS-2026 build path.
Tests mock the cmake command rather than dropping a shim on PATH: PowerShell
caches its application-path table, so a real cmake on the runner (present on
windows-latest) wins over a PATH shim. A function mock is resolved first and is
cache-proof cross-platform.
* Detect VS installed under the Preview edition dir for older versions
Find-VsBuildTools already scans every subdir for VS 2026, but the older-version
(2017/2019/2022) filesystem fallback and Get-FallbackVsGenerator only checked
BuildTools/Community/Professional/Enterprise. A Preview-channel install lives
under a 'Preview' edition folder, so it was missed when vswhere was also
unavailable. Add 'Preview' to both edition lists and guard each with a Windows
Pester test.
* Add real-VS integration matrix: detect actual VS 2022 and VS 2026 in parallel
The unit tests validate VS detection logic with mocked vswhere and fake install
trees (all five versions). This adds a parallel integration job that runs the
real Find-VsBuildTools / Get-VcBuildCustomizationsDir against the Visual Studio
actually preinstalled on GitHub-hosted runners:
- windows-2022 -> real Visual Studio 2022, expect generator v170
- windows-2025-vs2026 -> real Visual Studio 2026, expect generator v180
It asserts our detection matches the real install, the install path exists, the
derived toolset matches, and that the derived v-number is a real folder on the VS
install. VS 2017/2019/2015 are retired from hosted images, so only 2022 and 2026
can be exercised against a genuine install; the rest stay covered by the mocks.
* Detect VS 2026 via vswhere: it reports productLineVersion '18', not '2026'
Real-VS CI on the windows-2025-vs2026 runner showed vswhere reports
catalog_productLineVersion='18' (the internal major) for Visual Studio 2026, not
the marketing year '2026' that VS <= 2022 report. The vswhere map only had
'2026', so on a real VS 2026 host the vswhere branch returned null and detection
survived only via the filesystem scan (Source='filesystem'); a VS 2026 installed
outside the default Program Files location would not be found at all.
Extract a pure Resolve-VsGeneratorFromLabel that accepts both the year and the
internal-major form ('18'/'17'/'16'/'15' as well as '2026'/'2022'/'2019'/'2017')
and use it for both the vswhere and filesystem branches. Add pure unit tests
(cross-platform) for the mapping, including the '18' -> VS 2026 case.
* ci: dot-source Resolve-VsGeneratorFromLabel in the real-VS integration job
Find-VsBuildTools now calls Resolve-VsGeneratorFromLabel, so the integration
step must extract it too; without it the job failed with the helper not
recognized.
* Defer Visual Studio + CMake to the llama.cpp source build (prebuilt path needs no build tools)
The Windows installer required Visual Studio Build Tools and CMake eagerly in
Phase 1 (winget install + exit 1 if absent), before the llama.cpp prebuilt-vs-
source decision. But the preferred path downloads a prebuilt llama.cpp (no
compiler), the backend only shells out to the prebuilt llama-server.exe, and
PyTorch is pip wheels -- so VS and CMake are only needed for the from-source
build last resort. The eager requirement forced every Windows user to install
multi-GB Visual Studio + CMake they never use, or the installer failed.
Change (mirrors the already-lazy Resolve-CudaToolkit / OpenSSL):
- Phase 1c/1d now only DETECT cmake / VS and log; they never winget-install or
exit. The prebuilt install runs zero build-tool installs and is unblocked on
hosts without build tools.
- New Ensure-BuildToolsForLlamaSourceBuild installs CMake (best effort) + VS
(hard requirement, exit 1 with the existing guidance if it cannot be found),
called only when a source build is actually committed, before
Resolve-CudaToolkit. git stays eager (pip needs it for git+ deps).
Tests:
- Pester: the early probe (Find-VsBuildTools) returns null without exiting when
no VS is present; Ensure-BuildToolsForLlamaSourceBuild no-ops when VS is
already detected.
- New studio-windows-no-vs-smoke.yml: Job A renames Visual Studio + vswhere away
and hides cmake, runs the real install.ps1 --local --no-torch, and asserts the
prebuilt llama.cpp installed (no source-build fallback, no VS/CMake install),
PyTorch CPU imports, the backend is healthy, and a /v1/chat/completions
inference returns a reply -- all with no Visual Studio. Job B confirms the GPU
CUDA prebuilt is available and the resolver runs without VS.
* Fix VS 2026 CUDA source build ordering and fallback VS discovery
Same fix as on the stacked base branch (studio-vs2026-cuda-msbuild):
- Move Resolve-CudaToolkit below the CMake gate/fallback in the source build path. It copies the CUDA MSBuild .targets into the current VS generator's BuildCustomizations folder, so running it before a VS 2026 to older-VS fallback left the .targets under v180 while cmake configured v170 ("No CUDA toolset found"). It now runs after the final generator is selected.
- Get-FallbackVsGenerator now queries vswhere first, matching Find-VsBuildTools, so a VS installed outside the default Program Files roots is found instead of failing with a hard exit.
- Add Pester regression tests: the source build resolves CUDA after the fallback, and the fallback queries vswhere.
* Ensure the Visual C++ Redistributable is present for the prebuilt llama.cpp and PyTorch
The prebuilt llama-server.exe and the PyTorch wheels dynamically link the MSVC runtime (VCRUNTIME140.dll, MSVCP140.dll, VCRUNTIME140_1.dll). The Universal CRT ships with Windows 10+, but the VC++ 2015-2022 redistributable does not, so a clean box can fail to launch llama-server or import torch with a missing VCRUNTIME140.dll.
- Add Test-VCRedistInstalled (System32 vcruntime140_1.dll, with a registry fallback gated on version 14.20+) and Ensure-VCRedist (winget Microsoft.VCRedist.2015+.x64, non-fatal), called as Phase 1b.5 so it runs even on the no-build-tools prebuilt path. It is a no-op when the runtime is already present, which is the common case.
- Add Pester tests for the detection: present via the DLL, present via the registry, absent, and an old 2015-only redist that is too low.
* Add a CI job that validates the VC++ runtime detection on a real Windows runner
Runs on windows-latest and windows-2025-vs2026: asserts Test-VCRedistInstalled reports present on the stock image, removes both detection signals (the System32 DLL via a redirected SystemRoot and the HKLM runtime keys, restorably) to confirm detection fires on a genuinely clean box, then does a literal uninstall/reinstall round trip with the official installer and the Ensure-VCRedist winget path. The runtime is restored before the job ends.
* Dot-source the full logging closure in the VC++ runtime CI job
Ensure-VCRedist calls step/substep, which reach Write-StudioStdoutMirror and Get-StudioAnsi; extract those too so the job does not fail with an unrecognized command. Also note that the runtime is ref-counted by Visual Studio on the hosted image, so the literal package uninstall is a no-op there (the clean-box section already proves detection fires when the runtime is genuinely absent).
* Tighten comments in setup.ps1, the VS2026 tests and workflow
Comment-only: condense the verbose helper/test/CI comments to one or two lines, drop the obvious ones, keep the non-obvious rationale. Verified comment-only by comparing the PowerShell code-token stream before and after (no code tokens changed); Pester suite still green.
* Fold the no-VS and setup.ps1 VS2026 Windows CI into studio-windows-inference-smoke.yml
Move the no-vs-cpu/no-vs-gpu-resolve and pester/vs-integration/vcredist-clean-box jobs into the existing Windows GGUF CI workflow and delete the two standalone files, so a studio change triggers one Windows workflow instead of three. Path filter gains tests/studio_setup_ps1/**; job keys and artifact names stay unique.
* CI: assert a Windows ROCm prebuilt exists in the no-VS resolve job
The no-vs-gpu-resolve job confirmed a Windows CUDA asset but never a ROCm one,
and the resolver step resolves to CPU on hosted runners (no AMD GPU), so the
AMD no-VS guarantee rode only on shared resolver code. Grep the per-gfx
windows-x64-rocm-gfx bundles in the same asset-availability step so a release
that drops the Windows ROCm prebuilts fails loudly.
---------
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
* Studio: self-heal unsloth namespace-package shadows in all subprocess workers
A directory named `unsloth` (or `unsloth_zoo`) without an __init__.py on
PYTHONPATH/sys.path, a stray source checkout or a polluted PYTHONPATH, makes
`import unsloth` resolve to an empty namespace package, so a worker's
`from unsloth import FastLanguageModel` dies with a cryptic
"cannot import name ... (unknown location)".
The LLM training path already recovered from this via `_ensure_real_packages`
in trainer.py (PR #6269), but the inference, export, and embedding-training
subprocesses imported Unsloth directly with no guard. Extract that helper into
a shared, dependency-free core/import_guards.py and call it before the Unsloth
import in every subprocess: it drops the offending sys.path entries, imports
the real packages (unsloth before unsloth_zoo so the pre-zoo GPU fixes run),
then restores sys.path. trainer.py now imports the shared helper instead of its
local copy.
Covers both unsloth and unsloth_zoo and both namespace origin forms (None and
"namespace"). The existing PR #6269 test now exercises the shared helper.
* Studio: distinguish a failed model load from no model in the attach gates
A failed load never sets the checkpoint, so the image and audio attach gates
fell through to "Load a model before adding images/audio", which reads as if
the user simply forgot to pick a model rather than that the load errored. Add a
dedicated lastModelLoadError to the chat runtime store, set only when an actual
load attempt fails (not on refresh, list, status, or unload errors, which keep
using modelsError) and cleared when the next load starts. The image gate (all
three call sites) and the audio gate now use it to report a failed load and
point at the server logs, while still blocking in exactly the same cases.
* Tighten namespace-shadow guard and load-error comments
* Studio: keep staged model's load settings visible while it loads
* Studio: pin staged load settings at click time, match loading pick by variant
* Studio: drop any stale staged pick after a successful load
* Studio: disable staged Load button while a different model loads
When a model is staged with "Load on selection" off and a different model (or a different GGUF variant of the same repo) is already loading, selectModel's in-flight-load guard matches on id + native path token only, so the click was silently deduped to a no-op. The staged Load button stayed enabled but did nothing, and the stale stage was then cleared once the other load finished.
Disable the staged Load button (showing "Another model loading...") whenever a different model is loading, so it is no longer an enabled no-op. The stage stays put and the user can retry once the in-flight load settles.
* Studio: refuse loading a different model while one is in flight
The in-flight-load guard in selectModel matched on id + native path token only, so a different GGUF variant of the same repo fell through and was silently deduped to a no-op. The earlier commit disabled the staged Load button for this case, but other entry points (e.g. selecting a different quant from the model picker with "Load on selection" on) still hit selectModel directly and no-op'd.
Make the guard variant-aware (id + GGUF variant + native path token) and, for a genuinely different model while a load is in flight, surface a "Another model is already loading" toast instead of silently returning. The load path has no clean supersession, so a second concurrent load is not started; the user is told to wait or cancel. Centralized in selectModel so every entry point is covered.
* Studio: lock staged model's load settings while it loads
The staged Load button snapshots context length, KV cache dtype, speculative
decoding, draft tokens, and tensor parallelism at click time, but the settings
sheet stays mounted during the load, so edits made while "Loading..." shows were
silently ignored and then overwritten by the load response. Disable those
controls while the staged pick is loading so the visible state matches what the
run actually uses.
* Studio: refuse to stage a model while another load is in flight
With Load on selection off, selecting a model mid-load staged it into
pendingSelection and showed a disabled "Another model loading..." button,
implying the user could retry once the load settled. The post-load cleanup then
treated that queued pick as stale and silently cancelled and cleared it. Refuse
to stage while a load (or a cancel's background unload) is in flight: stageModel
no-ops in that state so every entry point (the chat selector and the Hub) is
covered, and stageOrLoad surfaces a toast on the common path. The immediate-load
path is unchanged; selectModel already rejects a concurrent load.
* Tighten staged-load guard comments
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* CLI: stop `unsloth connect` from leaking Studio credentials to unverified servers
`unsloth connect` (and `unsloth chat`) discovered a Studio base URL from
UNSLOTH_STUDIO_URL or the default localhost port after only an unauthenticated
/api/health probe, then sent credentials to it:
- keyless connect iterated every cached API key and sent each as a bearer token
to {base}/v1/models, so a malicious or port-preempting endpoint could harvest
all of them;
- with no cached key it self-issued a Studio JWT and POSTed it to
{base}/api/auth/api-keys;
- unsloth chat sent the same self-issued JWT to the discovered base.
The key cache was a flat, global list with no binding to a server identity, so a
key minted for one Studio could be replayed to any other.
Changes:
- Scope the agent key cache per base URL so a key is only ever replayed to the
exact server it was minted for. Pre-scoping flat caches are ignored rather
than replayed (at most one extra local mint on the next launch).
- Gate every automatic credential flow to loopback bases. A non-loopback
UNSLOTH_STUDIO_URL now requires an explicit --api-key and nothing is sent
automatically. SSH-tunnelled Studios that land on 127.0.0.1 keep working.
- Mint the API key locally against the Studio auth DB instead of POSTing a
self-issued JWT over the network, so no bearer token leaves the process on the
local path.
- Apply the same loopback gate to connect_studio_server (used by unsloth chat).
Fully closing same-host loopback-port preemption needs a signed /api/health
handshake so the client can verify the server identity before sending anything;
that is tracked as a server-side follow-up.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* CLI: verify Studio server identity before auto-sending credentials
Adds a challenge-response so `unsloth connect` and `unsloth chat` can confirm a
discovered loopback endpoint is really this install's Studio (not a process
that preempted the port) before sending it a cached or freshly minted
credential. This closes the same-host loopback-preemption gap left open by the
previous commit, which could only limit the blast radius.
Server:
- storage.get_or_create_identity_secret(): a dedicated server-wide secret in
app_secrets (kept separate from the per-user JWT secret), readable only by
the same OS user.
- storage.compute_identity_proof(nonce) = HMAC-SHA256(identity secret, nonce).
- GET /api/auth/identity?nonce=<base64url>: unauthenticated, returns the proof.
The nonce is opaque to the server and the proof reveals nothing about the
secret, so answering is safe.
Client:
- verify_studio_identity(base): sends a fresh 32-byte nonce, recomputes the
expected HMAC from the local same-user secret, and constant-time compares.
Fails closed on any error.
- connect._agent_api_key gates the loopback cached-key replay and the local
mint on it; connect_studio_server (used by unsloth chat) gates the
self-issued JWT on it.
A server that cannot read this install's secret (a different OS user, or a
remote/fake endpoint) cannot produce a matching proof, so the client refuses
and falls back to an explicit --api-key.
Tests: studio/backend/tests/test_identity.py (proof determinism, secret
persistence and caching, route response and nonce validation) and additions to
test_connect.py (the verify gate refuses when unverified, an explicit key skips
the check, and an end-to-end client plus server handshake against a stub HTTP
server).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* CLI: mint through the verified server instead of the local auth DB
CodeQL (py/clear-text-storage-sensitive-data) flagged the API key written to
the per-server cache and the agent config files once it was sourced from
storage.create_api_key(); the original HTTP-minted key did not trip the query.
Now that the identity handshake cryptographically confirms the loopback
responder really is this Studio before anything is sent, minting through the
server's /api/auth/api-keys endpoint with a self-issued JWT is safe again and
restores the original, CodeQL-clean data flow. The local-DB mint path is
removed.
The security properties are unchanged: discovery is still loopback-gated and
identity-verified, the key cache is still scoped per server, and a credential
reaches the server only after the handshake has proven its identity. The only
difference from the previous commit is that the self-issued JWT is sent to the
already-verified loopback server rather than the key being minted in-process.
Tests updated to mint through the fake server again.
* CLI: address review feedback on connect credential handling
- Reuse a saved per-server key before the loopback/identity gate. Keys are
scoped per base URL, so a key the user saved with --api-key for a remote or
SSH-tunnelled Studio (whose identity secret the local handshake can't match)
is replayed only to that exact server. The loopback + identity-handshake gate
now guards just auto-minting (self-issuing a JWT and creating a new key),
which is the path that needs a cryptographically verified local Studio. Fixes
keyless reuse being impossible for remote/tunnelled Studios the user had
saved a key for.
- connect_studio_server (unsloth chat / inference): when the user explicitly set
UNSLOTH_STUDIO_URL but the server can't be safely attached (non-loopback, or
identity unverifiable), fail with a clear message instead of silently loading
the model locally. Opportunistic discovery of the local default still falls
back to a local load.
- Harden cache parsing: tolerate a corrupt or hand-edited cache where a base
maps to a non-list (which would otherwise iterate a string into
single-character "keys"), and read the cache as UTF-8.
Tests updated and added: saved-key replay without the handshake for both local
and remote bases, keyless mint still refused when the loopback server is
unverified, and connect_studio_server erroring on an explicit remote while
falling back locally on default discovery.
* CLI: harden connect handshake against relay and gate cached minted keys
Addresses review feedback on the credential handshake:
- Refuse HTTP redirects on credential-bearing requests (the identity handshake,
/v1/models, key minting, and the chat HTTP backend). A process squatting the
discovered port could 302 /api/auth/identity to the real Studio and relay its
valid proof, or bounce a bearer-token request to another base, and urllib
follows redirects by default. A shared no-redirect opener now treats any 3xx
as an error.
- Give cached keys provenance. Keys the user supplied with --api-key are "saved"
and replay without the handshake (needed for remote or SSH-tunnelled Studios
whose secret the local handshake can't match). Keys we auto-mint are "minted"
and replay only after the identity handshake, so a port squatter can't collect
a previously minted localhost key just by answering the health check. New cache
shape: servers[base] = {"saved": [...], "minted": [...]}.
Known residual: a different-OS-user process that squats the port and can also
reach a genuine same-secret Studio elsewhere on loopback can still manually relay
the identity challenge. Fully closing that needs the proof bound to the server's
real listening port, or OS-level peer-credential checks; tracked as follow-up. A
same-user attacker is out of scope, since it can already read the 0600 key cache.
Tests: redirect rejection in the handshake, minted-cache requiring the handshake
while saved-cache bypasses it, and the existing suites updated for the new cache
shape. unsloth_cli (206) and test_identity.py (5) pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* CLI: keep urllib imports function-local in the no-redirect opener
The repo's import-hoist safety linter (scripts/verify_import_hoist.py) flags
hoisting urllib to module level because it re-points the 'urllib' name in the
pre-existing HttpChatBackend._request scope. Build the no-redirect opener lazily
with function-local urllib imports instead, matching this module's convention,
and restore the local 'import urllib.request' in _request and
verify_studio_identity.
* test(identity): skip route tests when routes.auth import chain is unavailable
The identity route tests build a TestClient from routes.auth, which pulls the
whole routes package (routes/__init__ -> inference -> llama_cpp, ...). In a
minimal test matrix without the heavy backend deps, or when another test in the
same process has already broken that import chain, importing it raised and the
two route tests hard-failed. Skip in that case instead: the proof crypto is
covered by the storage-level tests, and the full backend CI still exercises the
route. No behaviour change where the deps are present (5 passed in isolation).
* test(connect): make connect tests pass on native Windows
unsloth connect supports Windows: --no-launch prints PowerShell ($env:X =
"v" / Remove-Item Env:X) instead of POSIX (export/unset), and the launch
path bridges env into a Windows agent .exe over WSLENV. The tests hardcoded
the POSIX shell forms, so on a real windows-latest runner 12 of them failed on
the assertion string even though every command exited 0.
Add OS-aware assertion helpers (_assert_env_set / _assert_env_unset) that check
the right shell syntax for the host OS, and skip the two WSL-from-Linux shim
tests on native Windows (os.name is 'posix' inside WSL, so that path can't run
there). No change on Linux/macOS (57 passed); the connect command's behaviour
is untouched. Validated on a windows-latest staging runner.
* style(connect): tighten comments in the credential-leak fix
Condense the verbose explanatory comments and multi-line docstrings added by
this PR to one or two lines each, drop a few that just restated the code, and
keep the security rationale where it is load-bearing. Comment/whitespace only;
verified with unslothai/scripts comment_tools.py (check --strip-docstrings:
6/6 'code unchanged'). Tests unchanged: connect 57 passed, identity 5 passed.
* CLI/Studio: harden the identity handshake (review round)
Addresses the latest Codex/Gemini review of the handshake:
- Store the identity secret privately. sqlite3.connect created the auth DB
world-readable under a 022 umask, so another OS user could read app_secrets
and forge proofs, defeating the same-user assumption the handshake rests on.
The auth dir and DB are now restricted to owner-only (0700/0600); the JWT
secret and password hashes there get the same protection.
- Bind the proof to the server's listening port. The stateless HMAC(secret,
nonce) was relayable: a process squatting the discovered port could proxy the
challenge to the real Studio on another port and pass it back. The proof now
covers the port the server actually listens on (from the socket, never the
Host header) and the client checks it against the port it connected to, so a
relayed proof from a different port no longer matches. Closes the manual-relay
residual left after the redirect fix.
- Cap the identity response read (the server is still unverified at that point)
and serve the identity route from a sync def so its first-call SQLite read
runs in the threadpool instead of the event loop.
Tests: port-bound proof + relayed-proof rejection added; identity (5) and
unsloth_cli (58) suites pass. Verified end to end against a real backend
(auth DB owner-only, handshake + mint still succeed).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* CLI/Studio: bind the identity proof to the connection address, not just port
Follow-up to the port binding from the last review. Binding only to the port
left a cross-address relay: a squatter on a different loopback address but the
same port (for example localhost resolving to a squatter on ::1 while the real
Studio is on 127.0.0.1) could proxy the nonce to the real Studio and pass back
a proof that still matched, since both share the port.
The proof now covers the address and the port the connection landed on:
- Server: takes the address+port from request.scope, which uvicorn populates
from getsockname, so it is the real local address the client reached even
when Studio is bound to 0.0.0.0 (verified empirically), never the
client-controlled Host header.
- Client: resolves the base host to one concrete IP, talks to exactly that IP,
and binds the proof to (IP, port). A proof relayed from a Studio on a
different address or port was computed for that other endpoint and no longer
matches the one the client dialed.
Both sides normalise the address through ipaddress so equivalent forms compare
equal. Together with the private-secret and redirect fixes, this closes the
cross-user loopback relay an attacker can mount without reading the secret.
Tests: proof now bound to host+port; relayed-proof rejection retained; identity
(5) and unsloth_cli (58) suites pass. Verified end to end against a real backend
(handshake + mint still succeed).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* CLI: pick the loopback address at discovery so localhost does not regress
find_studio_server now resolves a bare localhost base to its concrete loopback
addresses and returns the first that answers /api/health, IPv4 127.0.0.1 first
(where unsloth studio binds by default). The whole flow (health probe, identity
check, credential send) then targets that one address instead of racing
IPv4/IPv6 resolution, where localhost could resolve ::1-first and hide a Studio
bound to 127.0.0.1. A literal IP or remote name is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: use an isolated Node.js for the frontend build instead of replacing the system Node/npm
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address Node isolation review (no-Node probe crash, PATH refresh, OXC provisioning, venv python, runtime node resolver)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust Node isolation for PR #6533
* Studio Node: don't cache a negative node resolution; accept Node metadata in setup.sh ownership guard
- node_runtime: memoize only a version-adequate executable so a Node installed
by a separate-process 'studio update' is picked up without a backend restart.
- setup.sh: _studio_owned_adoptable also accepts UNSLOTH_NODE_PREBUILT_INFO.json,
matching the setup.ps1 Node ownership guard (custom-home parity).
* Studio setup.ps1: skip OXC npm install gracefully when npm is absent
Mirror setup.sh's `command -v npm` guard so a pip-installed Studio with no
system Node skips the OXC runtime install (validator degrades at runtime) instead
of exit 1 aborting the whole setup. Tighten test_node_probe_guard.ps1's probe
regex so it only matches the two system-version probes, not this new npm guard.
* Wire test_node_probe_guard.ps1 into Windows CI for PR #6533
* Harden isolated Node install and probes for PR #6533
- install_node_prebuilt.py: keep an existing, still-usable isolated Node
when nodejs.org's dist index is unreachable instead of aborting the
update on a transient outage (existing_install_usable + tolerant fetch).
- install_node_prebuilt.py: pin NPM_CONFIG_PREFIX/npm_config_prefix and
drop NODE_PATH in _run_node so any npm -g stays inside the isolated
prefix; Windows npm otherwise writes to %APPDATA%\npm.
- install_node_prebuilt.py: resolve tar hard-link targets against the
archive root (symlink targets stay link-parent relative).
- setup.ps1: wrap the system node/npm probes in try/catch so a present
but broken shim degrades to the bundled Node instead of aborting setup.
- setup.ps1: run the isolated Node install with the handed-off/venv Python
(ReusedSetupPython); the main resolver runs later and bare python may be
a Store stub this early.
- setup.sh: log when the OXC validator runtime is skipped for missing npm,
matching setup.ps1.
- node_runtime.py: move the version-floor comment onto _version_meets_floor.
- Tests for the offline-reuse and broken-shim paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim verbose comments across the Studio Node installer for PR #6533
Comments-only pass: collapse the multi-line section banners to single lines,
drop comments that restate obvious code, and tighten the remaining docstrings
and "why" notes without losing intent. No code changes (verified with an AST
comment-only check on the Python files and a non-comment-diff scan on setup.sh
and setup.ps1). Net 109 fewer lines; the install, decision, and probe-guard
suites stay green.
* Harden Node install from review: validated Python, version floor, legacy home, lock race
For PR #6533, addressing the latest review pass:
- setup.ps1: run the isolated Node install with the validated reused/venv Python.
An incompatible reused interpreter (old venv, conda, stale UNSLOTH_SETUP_PYTHON)
is no longer used; fall back to the resolved python instead.
- setup.ps1: a STUDIO_HOME/UNSLOTH_STUDIO_HOME override equal to the legacy default
now uses the legacy sibling node dir (~/.unsloth/node), matching the runtime
resolver and setup.sh, so OXC can find the Node it installed.
- install_node_prebuilt.py: reject an explicit --node-version below the floor
(^20.19 || >=22.12 || >=23) instead of installing a Node the build cannot use.
- install_node_prebuilt.py: atomically rename a stale install lock before unlinking
so two concurrent runs without filelock cannot both acquire it.
Tests added for the version floor (parametrized + explicit-below-floor rejection).
Full install suite: 937 passed, 1 skipped; setup.ps1 parses; decision tests green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address latest review: armv7l + later-fetch offline reuse for PR #6533
- install_node_prebuilt.py: reject 32-bit ARM (armv7l) up front. Node 24 LTS
ships no linux-armv7l build, so the old path failed late with a confusing
"no sha256"; it now fails fast with a clear unsupported-architecture error.
- install_node_prebuilt.py: extend the offline-reuse fallback to the SHASUMS and
archive fetches. If index.json resolves a newer Node but a later download fails
and a usable isolated Node is already on disk, keep it instead of aborting a
non-force update.
Tests added: armv7l/armhf are unsupported; a SHASUMS failure keeps an existing
usable Node and re-raises when none is present. Full install suite: 941 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add UNSLOTH_STUDIO_HOME node-dir tests (install side + resolver) for PR #6533
* Add regression tests pinning the reuse path read-only and isolating installer writes
Lock in the two invariants behind the isolated-Node design: reusing a good
system Node never mutates the user's Node/npm, and the installer's own npm
calls only ever write inside its install_dir.
- tests/studio/install/test_install_node_prebuilt_logic.py: assert _run_node
redirects NPM_CONFIG_PREFIX/npm_config_prefix into install_dir and drops an
inherited NODE_PATH; assert _ensure_npm_floor scopes the npm self-upgrade to
install_dir (never -g against the system) and is a no-op once npm meets the floor.
- tests/sh/test_system_node_readonly.sh (new, wired into studio-backend-ci.yml):
the setup.sh NODE_SOURCE=system arm runs no global install and sets no
NPM_CONFIG_PREFIX, with a positive control that the bundled arm does.
- tests/studio/test_node_decision.ps1: symmetric structural guard that the prefix
pin and the only global install (bun) live in the bundled branch, not the system arm.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: fix llama.cpp update toast tag and reload hint
The post-update toast used the job's to_tag, which is the bare bNNNN build
number (same as installed_tag), so it showed e.g. "b9726" instead of the full
release tag. Use status.latest_tag (e.g. b9726-mix-<sha>) to match the tag the
banner already shows, falling back to to_tag and then a generic label.
Also drop "Reload your model to use it." when there is nothing to reload: only
append it when a local model is loaded, since external-provider models do not
use llama.cpp.
* Fix/adjust llama update toast for PR #6493
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Recognise the Gemma 4 separate-drafter MTP family, auto-download the drafter with retry, fall back to n-gram with a clear reason when it cannot be resolved, and retry the download on reload. Gemma 3n (ships no drafter) and embedded-MTP models (Qwen) are unaffected.
Fixes#6406
* Studio: label Apple Silicon as Metal/unified memory instead of CPU-only in installers
* Studio: drop redundant aarch64 check from the macOS GPU label detection
* studio: run /generate/stream's sync generator off the event loop to avoid blocking it
* fix: close generator in finally on client disconnect in generate_stream
* Fix/adjust generate stream test for PR #6466
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust generate stream cancellation for PR #6466
* Fix/adjust generate stream cleanup for PR #6466
* fix: cancel incomplete generate stream cleanup
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
_estimate_mtp_overhead_bytes is also reached for the separate-drafter spec modes
(draft-simple / draft-eagle3) through _user_draft_via_extras. Those modes load a
small distinct drafter with its own KV -- already counted in the draft KV +
weights -- and keep no duplicated full target context; only MTP runs a second
context over the target model's own KV geometry (llama.cpp ctx_tgt). Charging the
~main-KV-sized f16 copy there over-reserved by tens of GiB on an MLA model and
needlessly shrank the advertised context, the same under-advertising #6312 set
out to fix.
Thread mtp_keeps_target_ctx through _estimate_mtp_overhead_bytes (True for MTP,
False for separate-drafter modes) and derive _engaged_is_mtp at the fit call site
so the target copy is added only when the engaged mode is actually MTP. MLA + MTP
(GLM-5.2 / DeepSeek / Kimi) is unchanged, so the GLM-5.2 OOM fix is preserved;
non-MLA and the draft-simple / draft-eagle3 paths no longer pay the copy.
test_mtp_mla_target_ctx.py adds a case asserting the separate-drafter reserve
collapses to the draft KV (no target copy) while the default MTP path keeps it.
* Studio: Auto disables MTP for MLA models (GLM-5.2 et al.); UNSLOTH_MLA_MTP_ENABLED to re-enable
Studio's Auto speculative mode promotes any embedded-MTP model >=3B to
--spec-type draft-mtp. For MLA models (GLM-5.2/DeepSeek/Kimi) that is a
regression: llama.cpp's MLA/DSA MTP path keeps a duplicated full target-KV
context and recomputes the sparse-attention indexer every draft step, so it
runs ~2x slower than no speculation (GLM-5.2 UD-IQ1_S bench: 27 vs 45 tok/s,
flat across draft depth 1..6 and 96-100% acceptance, on both prose and code).
vLLM/SGLang get a speedup from the same model, so this is a llama.cpp
implementation gap, not a model property.
Auto now drops embedded MTP for MLA models and falls back to ngram-mod (or
spec-off when the binary lacks ngram-mod), mirroring the existing sub-3B
fallback. The metadata separator is kv_lora_rank: it is present on MLA models
and absent on non-MLA embedded-MTP models (Qwen3.x-MTP), whose MTP module is
structurally identical but fast, so a "full layer" heuristic cannot tell them
apart. Qwen MTP, separate drafters (Gemma, --model-draft), and non-MTP models
are unchanged.
Explicit overrides still engage the slower MTP route: choosing MTP / MTP+Ngram
in Settings, or passing --spec-type in extra args. UNSLOTH_MLA_MTP_ENABLED=1
re-enables Auto promotion for MLA once the upstream path is optimized.
A new spec_fallback_reason value "mla_mtp_disabled" surfaces this as an
Auto-mode policy downgrade (not a binary/update problem), with a settings
banner that points users at the MTP override. It is deliberately kept out of
the "Update llama.cpp" affordance since updating does not help.
Tests: resolver-matrix rows for MLA->ngram-mod / MLA-no-ngram->off /
non-MLA-Qwen->draft-mtp / MLA-separate-drafter->draft-mtp /
non-MTP-MLA->default / forced mtp|mtp+ngram on MLA->draft-mtp / env flag;
kv_lora_rank metadata fixtures; and reload-skip coverage (Auto ngram-mod is
idempotent, forced mtp bounces a reload).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add an Open button to reveal the models folder in the file manager
* Studio: report models folder creation failures
---------
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
test_safetensors_capability_advertise: detect_reasoning_flags now returns a
reasoning_effort_levels key, so the none-template expectation must include it.
test_tensor_parallel::test_runtime_recovery_reloads_without_mtp: the assertion
raced the recovery thread, which sets _spec_fallback_reason just before its
finally clears _mtp_runtime_fallback_in_progress. Wait for the flag to clear
before asserting.
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Fix clipped card shadows in Hub trending carousel
The carousel scroller only had vertical padding, so with overflow-x set
the first and last cards had their drop shadow clipped on the horizontal
edges. Add px-2 with a matching -mx-2 so the shadow has room while the
cards stay aligned with the section heading, and scroll-px-2 so snap-start
does not scroll the padding away on load.
* Align carousel edge fades with the scroll clip edge
The shadow fix gave the scroller an -mx-2 bleed, but the left/right fade
overlays stayed pinned to the wrapper edges, 8px inside the clip edge. That
left a thin strip where a card showed beside the fade, so the fade read as a
separate block instead of blending into the background. Offset both fades by
the same 8px so their opaque edge sits on the clip edge.
* Add click-and-drag panning to the Hub card carousel
The rows only scrolled by wheel or trackpad, and grabbing a card started
a native drag of its avatar image, so the cards could not be dragged to
move the row. Add mouse drag-to-scroll (touch and pen keep native
scrolling), swallow the click a drag would otherwise fire on a card, keep
plain clicks working, and block the avatar's native drag.
* Trim carousel edge fade width from 56px to 44px
* Smooth out carousel drag panning
Scroll snap was correcting the position on every drag frame, which made
the pan feel sticky. Disable snap while a drag is active and restore it on
release so the row follows the pointer and then settles on a card.
* Drop stale carousel drag when the button is released off-element
If a press ended outside the scroller before the drag threshold was
crossed, no pointerup reached us and the drag stayed armed, so a later
buttonless mousemove would scroll the row. Bail out and clear the drag
whenever the primary button is no longer held.
* Studio: reserve the duplicated MTP target KV context for MLA models
GLM-5.2 UD-IQ1_S advertised its native 1,048,576-token context, loaded, then
crashed cublasCreate on the first generation with "CUDA error: the resource
allocation failed" on a 2x B200 box. The model loaded fine; the decode OOMed.
Cause: when MTP speculative decoding is engaged, llama.cpp keeps a second full
copy of the target model's KV context for draft verification (ctx_tgt=yes in the
spec log), at f16. On an MLA model that copy is ~the main KV again -- for GLM-5.2
at 1M ctx llama.cpp sized it at ~97.5 GiB -- but the auto-fit reserve only
counted the tiny embedded draft head (~2 GiB), 46x too low. So weights (~202 GiB)
+ main KV (~83 GiB) + a 2 GiB reserve looked like it fit in 2x182 GiB, when the
real footprint with the ~97 GiB MTP copy is ~382 GiB and overruns the cards.
Disabling speculative decoding removed the copy and the same context ran fine.
_estimate_mtp_overhead_bytes now adds the duplicated target context (the main KV
re-estimated at f16) for MLA models, so auto-fit backs the context off (or selects
more GPUs) instead of advertising one that OOMs. It is gated strictly on MLA
(kv_lora_rank present), which is exactly the family that keeps the extra copy
(GLM-5.x, DeepSeek, Kimi-K2); non-MLA MTP (Qwen, Gemma) is byte-for-byte
unchanged. The reserve stays deterministic from GGUF dims, matching #6312.
test_mtp_mla_target_ctx.py covers it: the MLA reserve includes the f16 target
copy and dominates the draft head, the copy is f16 regardless of the main cache
type and scales with context, non-MLA embedded heads keep overhead == draft KV,
and _fit_context_to_vram on the GLM-5.2 / 2x B200 budget now returns a context
below the requested 1M where the old draft-only reserve kept the full 1M.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stub structlog/loggers in MTP-MLA test so it is import-order-independent
The new test imports core.inference.llama_cpp, which pulls in orchestrator ->
structlog. In the lightweight test env structlog is absent, so when this file is
collected before test_mtp_vram_budget.py (it sorts first) or run directly,
collection aborted with ModuleNotFoundError. Install the same loggers/structlog
(+ conditional httpx) stubs the sibling MTP tests use before the import, matching
the established per-file convention.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: fix Bypass Permissions menu freeze and show decimal GB for model sizes
Bypass Permissions freeze: the warning dialog lived inside the composer
"+"/More dropdown and kept the menu mounted via onSelect preventDefault,
so confirming or cancelling the dialog left both popovers frozen open.
Lift the dialog out of the menu into a store-driven
BypassPermissionsConfirmDialog mounted at a stable spot in the composer.
The menu item now closes normally on select and just toggles a new
bypassConfirmOpen store flag, so the popovers dismiss as expected.
Model search sizes: formatBytes divided bytes by 1024 but labelled the
result "GB", so unsloth/GLM-5.2-GGUF:UD-IQ1_S showed 201.8 GB where
Hugging Face reports 217 GB. Switch the search display to decimal
(base-1000) units to match what Hugging Face reports. The GPU-fit math
stays base-1024 since VRAM capacity is binary.
* Studio: address review feedback and add GLM-5.2 high/max/disabled thinking
Review feedback on the Bypass Permissions and size-format changes:
- Mount the Bypass Permissions warning dialog once at the chat-page root
instead of inside each Composer. It is driven by global store state, so
the per-composer mount meant Compare mode (multiple composers) rendered
duplicate dialogs and the shared-composer menu had none. A single root
mount fixes both.
- Defer opening the dialog past Radix's menu-close focus restoration with
setTimeout(0), so the dropdown does not steal focus back and break the
dialog's focus trap.
- Clamp the unit index in formatBytes so units[i] cannot go out of bounds
past TB (and to absorb log() float error at exact powers of 1000).
GLM-5.2 reasoning levels:
GLM-5.2's template gates thinking with enable_thinking and also reads a
reasoning_effort level ('high' or 'max'), so it needs high / max /
disabled rather than the binary toggle it got before (its style was
detected as enable_thinking, which made 'high' unreachable). Add a new
reasoning style 'enable_thinking_effort' that reuses the effort dropdown
but, unlike gpt-oss, can be fully disabled:
- detect_reasoning_flags classifies a template that has both
enable_thinking and reasoning_effort, extracting the discrete levels
from the quoted effort literals it branches on. Templates with only one
of the two (gpt-oss, Qwen3, DeepSeek, GLM-4.6) are unchanged.
- _request_reasoning_kwargs maps the new style to enable_thinking plus an
in-range reasoning_effort; disabling sends enable_thinking=false. The
gpt-oss reasoning_effort path is left untouched.
- The backend reports reasoning_effort_levels on the load/status response;
the frontend carries them through to the effort dropdown and sends
enable_thinking + reasoning_effort for this style.
Verified: backend reasoning kwargs render the real GLM-5.2 template to
"Reasoning Effort: High/Max" (thinking) and an empty <think></think>
(disabled); tsc, eslint, i18n parity and the production build all pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review feedback on reasoning effort and formatBytes
- chat-adapter localReasoningEffort: accept 'minimal' so a template that
branches on it (extracted into reasoning_effort_levels) is sent through
instead of being coerced to 'low' and then dropped by the backend.
- formatBytes: return '0 B' for non-finite / non-positive sizes (missing
metadata -> NaN, Infinity, negatives) and clamp the unit index lower
bound to 0, so sub-1-byte values can't produce a negative index.
* Studio: hybrid reasoning none gate and decimal GB in load progress
- _request_reasoning_kwargs: for enable_thinking_effort models, treat a
raw reasoning_effort='none' (OpenAI 'no reasoning' sentinel) as the
enable_thinking=false off gate, so a direct API caller can disable
thinking even without passing enable_thinking. The frontend already
sends enable_thinking=false; this only affects raw API callers.
- use-chat-model-runtime: the download / 'X of Y GB in memory' load
progress divided bytes by 1024**3 but labelled GB, so it disagreed with
the model picker and Hugging Face. Use decimal GB (1e9) to match.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry hybrid reasoning levels on all load paths and harden formatBytes
Review follow-ups on the enable_thinking_effort work:
- Every model-load path now copies reasoning_effort_levels and derives
supportsReasoningOff, via a shared reasoningCapsFromLoad() helper. The
shared/Compare composer load and the three chat-adapter auto-load paths
previously set only reasoningStyle, so a GLM-style hybrid model loaded
through Compare or first-chat auto-load fell back to the default
low|medium|high and lost its Max / Off controls.
- The local send path clamps the effort to the loaded model's advertised
levels (clampReasoningEffortToLevels) instead of a hard-coded list. A
stale "max" carried over from an external provider no longer reaches a
pure reasoning_effort (gpt-oss) model that only accepts none|low|medium|
high, where the backend would have dropped it.
- formatBytes divides iteratively instead of via Math.log, which has float
error at exact powers of 1000 (log(1e12)/log(1000) = 3.9999... would
label 1 TB as "1000 GB"). Keeps the non-finite/non-positive guard.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: restore sidebar roundness to its pre-#6349 state
* Studio: keep train recents as a rounded rectangle, not a pill
---------
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
* Studio: fall back to text-only when a vision projector hard-crashes llama-server
The text-only mmproj fallback (#6075) only fired when llama-server printed a
recognizable projector-format error ("Unknown projector type", exit -6). An
installed llama.cpp that predates a model's projector can instead SIGSEGV
(exit -11) with no parseable output, e.g. unsloth/Qwen3.5-4B-MTP-GGUF +
mmproj-F16 on an older gfx1151 prebuilt: llama-server crashes on load, the
--fit off retry crashes the same way, and load_model gives up with a hard 500
instead of dropping vision.
Generalize the decision: a vision (--mmproj) launch killed by a signal (POSIX
returncode < 0, e.g. -11 SIGSEGV / -6 SIGABRT; Windows 0xC0000000+ access
violation) is treated like a projector incompatibility, so the load retries
once text-only. The retry is skipped if a cancel/unload is pending, mirroring
the MTP guard. Clean non-zero exits (bad GGUF, port bind) and hung processes
keep their own handling; non-vision launches are unaffected.
Reproduced and verified on gfx1151 (Radeon 8060S, ROCm 7.2.1): a current
prebuilt (llama.cpp b9596) loads the exact model + args fine, confirming the
crash is a stale prebuilt. With a wrapper that SIGSEGVs on --mmproj, Studio now
recovers: the load returns 200 (is_vision=false) and serves at ~31 tok/s
text-only instead of failing. New _is_signal_crash helper plus tests pin the
decision.
Also normalize a few em-dashes to ASCII punctuation in existing comments.
* Studio: refine mmproj hard-crash fallback (signal scope + last argv)
- Limit _is_signal_crash to genuine program faults (SIGSEGV, SIGABRT,
SIGILL, SIGFPE, SIGBUS) and Windows 0xC0000000+ statuses. SIGKILL,
SIGTERM and SIGINT no longer count, so an OOM-killer, unload or
supervisor kill is not masked as a projector incompatibility.
- Strip --mmproj from the last attempted argv so the text-only retry
keeps --fit off / --spec-default instead of resurrecting the original
spec flags (matters for MTP vision models on an older llama.cpp).
- Drop stray temp files committed by mistake and gitignore the "~" dir
so they cannot be re-added.
* Studio: tighten comments in mmproj hard-crash fallback
* Studio: retry --flash-attn off before dropping vision on a startup crash
When llama-server hard-crashes at startup, the recovery chain now tries the
least-destructive mitigation first. Flash-attention kernels SIGSEGV at load on
some ROCm/GPU builds (often inside the vision tower's attention); disabling
flash attention keeps BOTH vision and MTP, so a hard program fault with
--flash-attn on now retries once with --flash-attn off before the MTP-drop or
the text-only (mmproj-strip) fallbacks. _is_signal_crash already gates this to
genuine faults (SIGSEGV/SIGABRT/SIGILL/SIGFPE/SIGBUS), so an OOM-kill or unload
(SIGKILL/SIGTERM/SIGINT) does not trigger a retry.
Field context: a gfx1151 user crashes loading a vision GGUF even on the latest
prebuilt, so an update cannot help, and the same model and args load fine on
another gfx1151 box, pointing at a runtime/flash-attn fault. New
_with_flash_attn_off helper plus tests. Verified on hardware with a wrapper
that SIGSEGVs on --flash-attn on: Studio recovers with is_vision=true (vision
and MTP intact) instead of failing or losing vision.
* Studio: name the OOM kill on a too-large model load
When the OS kills llama-server with no diagnostic output (SIGKILL/SIGTERM,
almost always the OOM killer, e.g. a BF16 model too large for the WSL VM's
RAM cap), the recovery ladder correctly does not retry an external kill, so
this is the message the user sees. It fell through to the generic "is the
GGUF valid / out of memory" text. Make it actionable: name the signal and
point at a smaller or more quantized GGUF, a lower context length, or raising
the WSL memory limit. Output-based diagnoses still win and a hard fault keeps
the generic fallback.
* Studio: refuse a model too large for system RAM on a unified-memory APU
On gfx1150/gfx1151 APUs the weights load into shared system RAM (GGML
unified memory). _get_gpu_free_memory reports the full ROCm/APU budget as
free (often ~100 GB), but under WSL the VM's RAM cap is the real ceiling.
Studio trusted the budget, spawned a load larger than RAM, and the OS killed
it mid-flight, taking the Studio process with it (a silent "Terminated" with
no error, the model resident in RAM not VRAM).
Add a pre-flight guard on the APU path: if the weights exceed available
system RAM (psutil, then /proc/meminfo), refuse before spawning with a clear
message (smaller/more-quantized GGUF, lower context, or raise the WSL memory
limit). Weights only so KV/context auto-reduction is not double-counted;
unknown RAM never refuses; non-APU and discrete-GPU paths are untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: refine recovery ladder (keep diagnosed errors, flip all flash-attn)
Two review points on the hard-crash recovery ladder:
1. The signal-only text-only fallback stripped --mmproj on any hard fault,
even when llama-server had already printed a non-projector cause (an OOM
such as "cudaMalloc failed: out of memory", an unsupported architecture, or
a tensor-parallel limit). That masked the real error and told the user to
update llama.cpp for vision. New _output_has_nonprojector_diagnostic gates
the signal path: it fires only when no such marker is present, so a bare
SIGSEGV with no output still retries text-only, but a diagnosed OOM surfaces
the real error instead of silently dropping vision.
2. _with_flash_attn_off only flipped the first --flash-attn. llama.cpp is
last-wins, so a leftover enable from extra_args (--flash-attn on, -fa on, or
the = form) could keep flash attention on and re-crash the retry. It now
flips every occurrence and returns None only when nothing is flippable.
test_llama_cpp_mmproj_fallback.py and the classification/APU suites: 103 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: pass the text-only retry's exit code to the failure classifier
When the text-only fallback retry itself fails, read its exit code before
_kill_process() clears it and forward it to _classify_llama_start_failure, so an
OS-killed retry surfaces the actionable out-of-memory message instead of the
generic one (matching the primary failure path).
* Studio: scope APU RAM guard to selected GPUs, count MTP drafter, neutral SIGTERM
Three refinements to the startup recovery work in this PR:
- The unified-memory APU RAM guard fired whenever any visible GPU was a
gfx1150/gfx1151 APU, so on a mixed APU+dGPU host it could refuse a valid
load placed on the discrete GPU. Scope _amd_apu_wants_unified_memory to the
selected gpu_indices (physical ids, mapped via CUDA_VISIBLE_DEVICES like
_is_datacenter_gpu); None still means every visible GPU. Applied to both the
RAM guard and the GGML_CUDA_ENABLE_UNIFIED_MEMORY env set.
- The RAM guard counted only the main GGUF plus mmproj, so a separate MTP
drafter (also resident in unified system RAM, even when offloaded to CPU)
could push the load past the RAM cap and still get OS-killed mid-load. Add
the drafter weights to the APU RAM total.
- The startup classifier reported SIGTERM (-15) as 'most likely out of memory',
but SIGTERM is also how an unload/cancel or a supervisor stops the server.
Keep the OOM wording for SIGKILL (-9, the OOM killer) and report -15
neutrally.
Tests updated/added accordingly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review on the APU guard and decode-probe ladder
- Map APU physical ids via the active ROCm mask (HIP, then ROCR, then CUDA),
mirroring _get_gpu_memory, so a HIP_VISIBLE_DEVICES-selected APU is matched.
- Only add the MTP drafter to the APU RAM total when MTP will actually engage,
so a stale LLAMA_ARG_SPEC_DRAFT_MODEL cannot refuse a non-MTP load.
- After an MTP first-decode hard fault, retry --flash-attn off (keeps MTP)
before dropping speculative decoding, matching the startup rung.
- Fold the --flash-attn= / -fa= rewrite into one branch.
Tests: tensor-parallel decode-probe assertion updated for the FA-off rung.
* Studio: tighten two comments in the APU guard and RAM preflight
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: refine flash-attn retry and APU RAM guard per review
- _with_flash_attn_off now decides on the effective last-wins value: it returns
None when FA is already off (no wasted retry), and neutralizes a bare
--flash-attn / -fa (which llama.cpp reads as on) so the retry cannot re-enable
it. Length is preserved so downstream index slices stay valid.
- _amd_apu_wants_unified_memory uses 'gpu_indices is not None' so an empty
selection is respected (not treated as all-visible).
- The APU RAM refusal now checks the base model only (main + mmproj); an
optional MTP drafter is dropped by the existing MTP-drop fallback rather than
causing a hard pre-spawn refusal of an otherwise loadable model.
Tests: bare-flag / effective-off / empty-selection / HIP-mask cases added.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: free chat model VRAM at training start only when the GPU is tight
The training start route unconditionally tore down the transformers/MLX
inference subprocess before training, and never stopped the llama.cpp GGUF
server at all, so a loaded GGUF chat model kept holding VRAM for the whole
run. Conversely the HF model was always unloaded even when there was plenty
of room to keep it.
Make the unload VRAM aware and cover every inference backend:
- Add routes/training_vram.py with summarize_resident_chat(),
can_keep_chat_during_training() and free_chat_models_for_training(). The
keep/unload decision reuses the same estimator and live per device free
VRAM reader the training GPU selection already uses (auto_select_gpu_ids,
estimate_required_model_memory_gb, get_visible_gpu_utilization), so the
probe agrees with the placement computed later in start_training.
- When a chat model is resident and training fits alongside it with a
conservative margin (required_gb * 1.15 + 4 GB), keep it loaded so the
user can train and chat at the same time; on a multi GPU box training
lands on a different GPU and both coexist. Otherwise unload the HF/MLX
orchestrator and the llama.cpp GGUF server before training starts.
- The export subprocess shutdown stays unconditional and now runs first so
its freed VRAM is reflected in the decision.
Default deny: non CUDA backends, unestimable models, or any probe error
fall back to the previous always unload behavior.
Adds tests/test_training_vram_coexistence.py and updates two existing route
tests in test_gpu_selection.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: per-GPU floor for explicit GPU lists + don't unload chat on invalid gpu_ids
Address review feedback on the chat coexistence probe:
- Explicit gpu_ids mode now enforces a per-GPU floor in addition to the
aggregate free-VRAM check, mirroring auto_select_gpu_ids' min_per_gpu_N.
Without it, an uneven split such as free [45, 10] for a 40 GB job passed
the aggregate threshold and kept chat loaded even though the 10 GB GPU
could not hold its training shard, risking an OOM.
- Invalid explicit gpu_ids (ids outside the visible set, or a UUID/MIG
mask) make resolve_requested_gpu_ids raise. That request is rejected with
a 400 before training starts, so leave the resident chat model untouched
instead of unloading it.
- Tighten the target_modules / gpu_ids type hints to List[str] / List[int].
Adds tests for the per-GPU floor (uneven split unloads, even split keeps)
and for invalid gpu_ids keeping the chat model loaded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: only free chat VRAM once training will start; handle in-flight and CPU-only chat
Address the second review pass on the chat-coexistence path:
- Run the chat/export VRAM teardown as a before_spawn hook inside
TrainingBackend.start_training, fired only after the start guards pass.
Previously the route freed chat VRAM before calling start_training, so a
refused start (e.g. a lingering pump thread) would tear down the resident
chat model even though no training job began.
- Treat an in-flight HF chat load (loading_models set, no active model yet)
as not safely sizeable: free it rather than risk both OOMing as the load
keeps allocating after training starts.
- Do not count or tear down a GGUF llama-server confirmed to run entirely on
CPU (_gpu_offload_active is False): it holds no VRAM, so killing it cannot
help training fit.
Adds tests for the before_spawn hook (runs on start, skipped when a
subprocess is alive or a pump thread will not die, survives a hook error),
the in-flight load flag, and the CPU-only GGUF exclusion in both the resident
summary and the unload path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: treat any in-flight chat load (HF swap / mid-start GGUF) as unsafe to keep
Tighten the in-flight detection in summarize_resident_chat so the keep check
never sizes a load that is still allocating:
- Flag loading on ANY non-empty loading_models, not only when active_model_name
is empty. load_model adds the new model to loading_models before clearing the
old active_model_name, so a replacement load during a swap was previously
sized as a normal resident and could OOM as the new model finishes loading.
- Flag a GGUF server that is active but not yet healthy (is_loaded False) as
in-flight: it is still mmaping/offloading layers, so its final VRAM footprint
is unknown.
Consolidates the signal into a single resident["loading"] flag; the route frees
the chat model whenever it is set. Adds tests for the replacement HF load and
the mid-start GGUF cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten comments in chat/training VRAM coexistence (comments only)
* Studio: run before_spawn VRAM hook only after GPU-selection validation
Reviewers found the before_spawn hook fired before prepare_gpu_selection
validated gpu_ids (and before config build), so a refused start (invalid
gpu_ids -> 400, or a bad grad-clip value) could still tear down chat/export
VRAM. Move the hook to immediately before proc.start(), once all synchronous
validation and process construction have passed. This also fixes the route's
in-flight-chat loading branch, since that teardown runs inside the same hook.
Add test_hook_skipped_when_gpu_selection_rejects.
* Studio: recompute GPU auto-selection after the before_spawn VRAM hook
Codex P2: with before_spawn moved after prepare_gpu_selection, placement was
frozen against the pre-teardown VRAM state while the hook freed export/chat
afterward. Auto-selection could pin training onto a GPU the hook then cleared
(or onto a kept chat model). Split validation from placement: explicit gpu_ids
are still validated before the hook (raise -> 400, no teardown; explicit
placement is VRAM-independent), but VRAM-dependent auto-selection now runs
after the hook so it sees the freed memory.
Add test_auto_placement_runs_after_hook and test_explicit_placement_validated_before_hook.
* Studio: allow chatting during training (lift sidebar gate + VRAM-aware load guard) (#6335)
* Studio: allow chatting during training (lift sidebar gate + VRAM-aware load guard)
The sidebar disabled New Chat, project, and home navigation while a training
run was active, so users could not chat during training even though the backend
serves inference fine alongside a run. This removes that gate and adds a backend
guard so the one genuinely risky operation, loading a new local chat model
mid-training, is refused with a clear 409 when it would not fit beside the run.
Frontend (app-sidebar.tsx): drop the chatDisabled = isTrainingRunning gate and
its consumers. Navigation triggers no model load on its own, so chat stays
usable during training.
Backend (routes/training_vram.py, routes/inference.py): add
can_load_chat_during_training plus a load/validate guard that sizes the same
effective load the loader performs (LoRA 4-bit to 16-bit resolved first, HF auto
placement via auto_select_gpu_ids, explicit multi-GPU per-GPU floor, GGUF sized
from on-disk shards and companions or the selected remote variant). It is a
no-op when training is inactive, never blocks external providers or
already-resident models, and default-denies only on a CUDA sizing failure so a
load can never OOM the run. Validate refuses early with the real settings so the
frontend does not unload the resident chat model for a load that would be
rejected.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review feedback for chat-during-training load guard
- Run the load/validate VRAM guard via asyncio.to_thread so the sync
nvidia-smi + HF metadata work never blocks the event loop.
- Size the GGUF KV cache at the requested context (_estimate_gguf_kv_gb)
and add it to the local GGUF estimate so large-context picks are not
under-counted.
- Keep the requested quantization when adapter_config.json is malformed
(not a JSON object) instead of raising in _effective_load_in_4bit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: size the training load guard at the launcher's effective GGUF context
The GGUF KV-cache estimate used max_seq_length only, but the llama.cpp
launcher honors a user --ctx-size/-c in llama_extra_args. A load such as
max_seq_length=4096 with --ctx-size 131072 was sized against a 4k cache
while the server allocates 131k, so the guard could approve a long-context
GGUF load that then OOMs training. Size the guard's KV at the larger of
max_seq_length and the parsed --ctx-size (reusing the launcher's own
parse_ctx_override), keeping the conservative f16 cache so the estimate is
never smaller than what the server allocates.
The chat model picker also validated with the raw max_seq_length while
/load sizes with resolveLoadMaxSeqLength, so validate could pass, unload
the current model, then have /load reject the native-context load. Validate
now uses the same effective context; the load path is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: size the GGUF training guard at the server parallel-slot count
The KV-cache estimate assumed a single slot, but llama-server allocates the
cache across --parallel slots (app.state.llama_parallel_slots). On a Studio
launched with --parallel N>1 the guard under-sized the cache N-fold and could
approve a GGUF chat load that then OOMs training. Thread the same slot count
the loader uses into the guard's KV estimate; default 1 leaves single-slot
setups unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments for chat-during-training guard
* Studio: keep chat generation alive across navigation; Train spinner + Return to Chat
Hoist the base chat runtime above the routed outlet so navigating to Train (or any tab) no longer aborts an in-flight generation; only an explicit Stop cancels. Add a Train sidebar spinner and swap New Chat to Return to Chat while a run is active, with a lightweight completion watch so the spinner clears from any tab. Also respawn a chat llama-server killed mid-session and guard unreadable HF cache dirs that 500'd the hub model list.
* Studio: show Return to Chat on the Train tab whenever a chat is live
Previously the top sidebar item only swapped to Return to Chat while training was running; on the Train tab with an idle/just-finished run it stayed New Chat, which started a fresh thread and cancelled an in-flight generation. Show Return to Chat (and navigate back, preserving the run) whenever a generation is running or its thread is still active, or training is in progress.
* Studio: keep a running chat alive when starting a New Chat
Starting a New Chat (or switching threads) while a generation was in flight
remounted the single-chat runtime provider, which detached the in-flight run
and cut the previous chat off (it showed up frozen / empty when reopened).
Key the single-chat view by project instead of by thread or new-chat nonce so
the provider stays mounted and assistant-ui switches to a fresh thread in place.
The previous generation keeps streaming in the background and autosaves on
completion, and returning to that thread reattaches the live run instead of
reloading a half-saved one.
Also:
- "Return to Chat" now lands on the thread that is still generating rather than
the empty new chat that became active after New Chat.
- Skip the explicit /inference/cancel POST when an abort comes from a runtime
detach (navigation / background switch) rather than an explicit Stop, so a
backgrounded generation is never cancelled behind the scenes.
* Studio: make model export non-blocking and inline
The Export tab opened a full-screen modal that trapped focus, could not be
closed or cancelled while running, and showed no progress. It also stopped
training and unloaded the chat model before loading, so export could not run
alongside them.
Export now mirrors the training runtime pattern:
- Inline panel embedded where the Export Model button was, with no modal or
backdrop, so the rest of the UI stays usable during an export.
- Global export runtime store plus an app-root lifecycle hook, so a run keeps
going and streaming across navigation and is reflected on the Export nav item
from any tab.
- The worker log stream now stays connected across the load to export phase
boundary instead of stranding on "Waiting for worker output".
- Progress bar driven by phase and quant index (quant N of M for GGUF), with
elapsed time and a working Cancel.
- load-checkpoint no longer stops training or unloads inference; export loads in
its own subprocess in parallel and surfaces out-of-memory as a clear error.
- Add POST /api/export/cancel and is_export_active on /api/export/status.
* Studio: show Return to Chat on the Export tab too
Extend the New Chat to Return to Chat swap to the Export route so leaving a
running chat for Export offers a way back to the live generation, matching the
Train tab.
* Studio: smooth out Export animations and polish the panel
- Drop the height-based reveal animations (source switch, run panel, quant
picker, hub fields) that caused flashing and reflow; use instant swaps and
quick opacity fades instead.
- Method and quant cards now transition colors only, with no transition-all or
hover lift, so selecting a method or quant is crisp instead of jumpy.
- Auto-scroll the export panel into view when it opens and add a scroll-to-bottom
button when its output is below the fold, like Chat.
- Show Return to Chat on the Export tab while an export is running, matching how
training drives it on the Train tab.
- Surface the current phase or stage in the live output before the first worker
line arrives so the panel never looks stuck while progress is advancing.
* Studio: show Return to Chat on every non-chat tab
Generalize the Return to Chat swap from just Train/Export to any non-chat route
(Recipes, Projects, Hub, ...) so a running or active chat is always one click
away, instead of showing New Chat there.
* Studio: stream export logs over the Cloudflare tunnel; drop janky export animations
Exporting over a --secure Cloudflare quick tunnel showed "connecting..." with no
logs while the progress bar advanced. Cloudflare buffers text/event-stream and
only flushes when the stream closes, so the SSE log stream never reached the
browser during the run (direct localhost is unaffected, which is why this only
showed up over the tunnel).
Add a tunnel-safe JSON poll fallback (GET /api/export/logs?since=) that the
runtime lifecycle hook polls while a run is active. Short JSON responses are not
buffered by the proxy, so logs show up in near real time over the tunnel. It
shares the orchestrator's monotonic seq cursor with the SSE stream and the store
de-dupes by seq, so the two transports run together (SSE on localhost, poll over
the tunnel) without double-printing. A successful poll marks the panel
"streaming" instead of leaving it stuck on "connecting...".
Also remove the framer-motion AnimatePresence reveals from the export config and
run panel (quant picker, hub fields, the inline run panel, and the live log
section). The expand/slide animations flashed and felt clunky; the sections now
render in place.
* Studio: recover export over the Cloudflare tunnel when the blocking POST times out (524)
A model export over a --secure Cloudflare quick tunnel showed "Request failed
(524)" even though the export succeeded on the backend (the GGUF was written).
Cloudflare returns 524 when a single request takes longer than ~100s to respond,
and a GGUF conversion routinely runs for minutes, so the blocking per-method
export POST is cut off while the backend keeps going.
Confirm completion via short status polls instead of relying on the long POST
response (the same approach that fixed log streaming):
- The orchestrator records each finished op's outcome (status / output_path /
error) with a monotonic seq, exposed on GET /api/export/status.
- parseJson now preserves the HTTP status; a 524/520/522/523/502/503 or a
status-less network drop is classified as a recoverable transport error.
- runExport wraps each phase (load, every export method, each GGUF quant): on a
recoverable failure it keeps the run alive (logs keep streaming, the panel
shows "reconnecting...") and polls status until the still-running op finishes,
then settles from the recorded result, recovering the output path for the
success banner. A real 4xx still fails immediately; localhost still uses the
fast POST response. applyBackendStatus also settles a reloaded run from the
last-op record.
Verified over the tunnel: a 3m14s gemma-4-E4B-it GGUF export now ends on the
success banner with the output path instead of 524.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep the export method + logs visible after navigating away mid-export
While an export was running, navigating to another tab and back to Export
remounted the page and reset the local form state (exportMethod, quant levels),
so the method card showed unselected and the run panel's log area was hidden
until the card was re-clicked. The run itself lives in the global store and was
unaffected.
Seed exportMethod / quantLevels from the active run's summary via lazy useState
initializers on (re)mount, and gate the panel's log area on the live run
(isExporting / logLines / the run's method) rather than only the local form
selection. The card stays selected and the logs/progress stay visible across
navigation; nothing changes when no run is active.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: address export/training review findings
- Export: guard Start against an empty GGUF quant selection so an inline-panel
run with no quant can't settle as success with no file produced.
- Export: thread the source HF token into the background load so gated/private
HF source exports (and gated bases) authenticate, matching the consent path.
- Export: only settle a recovered (non-owned) run as a finished export when the
last backend op was an export, not a standalone load_checkpoint.
- Training: free the export subprocess whenever an export is active, not only
once a checkpoint is loaded, so an in-flight export load can't race training
for VRAM (current_checkpoint is unset during the load phase).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: polish Bypass permissions toggle and add it to chat menu settings
- Recolor the pill, menu item and settings caption from alarm red to a
bright yellow accent via a new --bypass token (light and dark).
- Use the shield icon on the composer pill (X on hover, like other pills).
- Expand the composer to two rows the moment Bypass permissions is on.
- Keep Bypass leftmost among tool pills; Compare still sits ahead of it.
- Add a Bypass permissions row to Settings > Chat menu so it can be pinned.
- Widen the "More" submenu so the label fits on one line.
* Studio: use shield-ban icon and lighten the Bypass permissions pill
- Swap the lucide shield-off icon for the Hugeicons shield-ban across the
pill, menu item and settings row.
- Lighten the pill background and deepen the yellow text a touch so the
label stays readable on a near-white tint.
* Studio: drop the Bypass permissions resting fill, keep it on hover
Show only the yellow icon and label at rest; the rounded hover pill picks
up the yellow accent like the other toggles.
* Studio: lighten the Bypass permissions hover fill
Drop the hover tint so the yellow label stays readable on hover.
---------
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Switch the sidebar profile button from rounded-full to rounded-[5px] so its hover background is a rounded rectangle instead of a pill. The collapsed icon-only state stays circular. Styling only.
* Studio: fail fast on an invalid first training batch
Training a base vision-language model (e.g. Qwen/Qwen2-VL-7B or
unsloth/Qwen2-VL-7B) on a conversational image dataset crashed on the first
step with 'Expected ... Long, Int; but got torch.cuda.FloatTensor (embedding)'.
Root cause: the base model's chat template is a flat, media-only template that
renders to an empty string for role-based messages, so UnslothVisionDataCollator
hands the processor empty text, the processor returns empty input_ids, torch
defaults the empty tensor to float32, and the embedding lookup rejects it.
Add a preflight that runs one real batch through the trainer's own tokenization
and collation right before train(), and stops the run with an actionable message
when input_ids is empty or non-integer (pointing to the instruction-tuned variant
for the base-model case). Faithful across text, vision and audio-VLM paths, and
never blocks a run whose first batch is valid.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments in the training preflight
* Stub unsloth/trl in preflight test so backend CI collection passes
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: show an actionable message when the GGUF runtime is missing
Selecting a GGUF model with no llama-server installed surfaced a generic
"Invalid model" in the UI, because validate_model's catch-all discarded the
real cause. Add LlamaServerNotFoundError (a RuntimeError subclass) raised by the
GGUF preflight in ModelConfig.from_identifier, and catch it in the validate
route so users get an actionable message: run `unsloth studio setup` to
download the prebuilt llama.cpp runtime. Other validation failures keep the safe
generic message. Adds a regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: also map missing GGUF runtime to a 400 in load_model
validate_model already surfaces the actionable 'install the runtime'
message for LlamaServerNotFoundError; load_model fell through to the
generic 500 'Failed to load model'. Catch it there too so a GGUF load
without llama-server gives the same install hint instead of a 500.
* Trim comments for PR #6327
* Studio: fix stale validate test after #6398 and surface missing GGUF runtime on /load
- test_other_runtime_errors_do_not_get_gguf_message: after merging #6398,
validate_model surfaces a RuntimeError's own message, so a plain RuntimeError
no longer returns "Invalid model". Assert it does not receive the GGUF
install message instead (the prior assertion was stale after the main merge).
- Raise LlamaServerNotFoundError (not a plain RuntimeError) at the backend
load-time missing-binary branch, after diffusion routing, so /load returns the
actionable 400 like remote validation, instead of a generic 500.
- Share LLAMA_SERVER_NOT_FOUND_DETAIL between the from_identifier preflight and
the load-time raise so the message stays in sync.
- Add a propagation regression test for the non-tensor load path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: reach the published source asset when a mix build's commit 404s
A llama.cpp "mix" prebuilt records a merge commit that is never pushed to
the fork, so the codeload/archive URLs for that commit 404. The merged
source tree is instead published as a release asset alongside the prebuilt
(llama.cpp-source-commit-<sha>.tar.gz). The installer resolves that asset
URL from the approved-checksums manifest, but when the manifest omits the
top-level repo/release_tag the URL resolves empty, hydration falls through
to the 404-ing commit archive, and the prebuilt install drops to a slow
source build (or fails outright).
Extract exact_source_asset_url() and resolve the asset's host and tag
defensively: the artifact's own repo, then the manifest repo, then the
source repo; and the manifest release tag, then the tag we actually
installed the prebuilt from (the source asset is its sibling on the same
release). Normal installs build the identical URL as before, so this only
adds a working fallback for the degenerate manifest.
Add unit coverage for the resolver, including the empty repo/release_tag
regressions.
* Studio: cover exact_source_asset_url through the real parser chain
Add TestExactSourceAssetUrl.test_resolves_through_real_parser_chain, which runs
parse_approved_release_checksums -> preferred_source_archive -> exact_source_asset_url
so a regression in the parser or source-selection wiring cannot pass while only the
hand-built helper unit tests stay green.
* Keep server-side tools enabled under --secure and on every bind
--secure binds loopback and exposes Studio only through an authenticated
Cloudflare HTTPS tunnel, but it was grouped with a raw 0.0.0.0 bind and
force-disabled all server-side tools (web search, Python, terminal). The
process tool policy overrode the client's enable_tools request, so the
model was never told the tools existed and answered in plain text. The
plain 'unsloth studio' command had no way to re-enable and printed nothing.
Tools now default on for every bind. The bind host and --secure no longer
change the tool policy; only an explicit --enable-tools/--disable-tools
forces it on or off. Both 'unsloth studio' and 'unsloth studio run' accept
the flags and the startup banner states the resolved policy.
- run.py: replace _apply_default_tool_policy(host, secure) with
_apply_cli_tool_policy(enable_tools); add an enable_tools kwarg to
run_server and --enable-tools/--disable-tools to the argparse.
- _tool_policy.py: resolve_tool_policy defaults to on for every host and
no longer prompts on a network bind.
- studio.py: drop the secure-as-public tool gating, add the flags to the
plain command, and reword the startup banner.
- Update and extend the secure-flag and tool-policy tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add tool-policy notice to plain server banner and refresh run --help
Follow-up to PR review:
- run.py: the plain 'unsloth studio' / --secure / direct run.py path went
through _emit_startup_output without any tool-policy line, so a
network-reachable launch was silent about code execution now that tools
default on. Thread enable_tools through _emit_startup_output /
_emit_secure_startup_output and print a one-line policy notice, followed by
a single stop hint.
- studio.py: the 'unsloth studio run' --enable-tools/--disable-tools and --yes
help still described the removed loopback-on/network-off default and the
confirmation prompt; reword to match the new policy.
- Add tests for the banner notice and the refreshed help text.
* Update CI tool-policy resolver tests for default-on behavior
tests/python/test_unsloth_run_tool_policy_resolver.py still asserted the
removed network-bind policy (0.0.0.0 and LAN IP default off, explicit enable
prompts and aborts on a declined prompt), so it failed the Python CI jobs.
Rewrite the truth table: every bind defaults on, explicit on/off always wins,
and the resolver never prompts (yes/silent/prompt kept for compatibility).
* Trim comments for the tool-policy change
Shorten the verbose docstrings and block comments added for --secure tool
handling; keep the security-relevant intent. Verified comment-only via an AST
diff (code unchanged).
* Add deterministic test that server-side tools execute under --secure
Drive the GGUF agentic tool loop with a fake llama-server stream and let the
real execute_tool run: python counts 1..100, terminal returns a UTC datetime,
and web_search runs through real _web_search with only the ddgs network
boundary mocked. A policy assertion pins that the post-fix --secure path
(policy None + per-request enable_tools) is what keeps these executions
reachable. No model, GPU, or live network; runs in the existing backend CI.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align _emit_startup_output banner test with the moved stop hint
The tool-policy notice now prints between the access banner and the stop
hint, so the stop hint is emitted once at the end instead of inline in the
banner (include_stop_hint is False and print_studio_stop_hint runs once).
Update the plain-localhost case to match; the mismatch and wildcard cases
already asserted this wiring.
---------
Co-authored-by: Michael Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Reap Studio child processes when the parent dies abnormally
Standalone `unsloth studio` launches orphaned cloudflared and llama-server when
the parent exited without running the cooperative shutdown path (terminal-window
close, Task Manager End Task, SIGKILL): the children reparented to init and kept
running, leaving an authenticated Cloudflare tunnel up for days.
Add utils/process_lifetime.py: a parent-owned Windows Job Object
(JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, children auto-inherit) plus Linux
PR_SET_PDEATHSIG, behind a best-effort helper that mirrors the desktop app's
windows_job.rs. initialize_parent_lifetime() runs at the top of run_server;
long-lived spawns (cloudflared, llama-server, RAG embedder, llama.cpp updater)
get the PDEATHSIG preexec, multiprocessing workers are adopted into the job, and
_graceful_shutdown plus atexit gain a terminate_all() backstop sweep. The
cooperative shutdown path is otherwise unchanged.
Verified on Linux: killing the parent now reaps cloudflared and llama-server
within ~2s instead of orphaning them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: add real Windows kill-on-job-close integration test
Spawn a parent that installs the job and a child that inherits it, terminate
the parent, and assert the child is reaped. Skipped off Windows. Also make the
liveness probe Windows-safe (os.kill(pid, 0) terminates on Windows).
* Fix Win64 handle truncation in the Job Object calls
Set explicit argtypes so the 64-bit job/process handles are not marshaled as
c_int (which truncated them on Win64, failing AssignProcessToJobObject). Assert
install success in the Windows integration test.
* Bind multiprocessing workers to parent death; harden the sweep
Review follow-ups:
- Multiprocessing workers (inference/export/training/data-recipe/Xet) cannot be
given a preexec_fn by the parent, so adopt_pid alone left them orphanable on a
Linux SIGKILL. They now bind themselves with PR_SET_PDEATHSIG at startup via
bind_current_process_to_parent_lifetime(), wired into the shared
run_without_native_path_secret entrypoint and the Xet child entry.
- Wire the previously-missed data-recipe worker through adopt_pid.
- terminate_all now honors its timeout: SIGTERM, wait, then SIGKILL the
survivors, so cooperative children can exit cleanly.
- Track adopted pids with a /proc starttime identity and add forget_pid, so the
shutdown sweep never signals a recycled pid.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: cross-session backstop to reap a leftover llama-server on startup
Builds on the parent-lifetime reaper: the Windows Job Object / PR_SET_PDEATHSIG
path kills children when the parent dies, and terminate_all() sweeps the living
parent's children. Neither covers an orphan left by an already-dead Studio:
terminate_all()'s registry is in-memory, PR_SET_PDEATHSIG has no macOS
equivalent, and both are best-effort.
This records the spawned llama-server PID to a pidfile under the active studio
root (removed on _kill_process). The startup reaper kills that exact PID first,
verifying it is still a llama-server to guard against PID reuse, then clears the
pidfile. It is path-independent, so it also catches an orphan the install-root
match would miss; the pidfile only ever names a Studio-spawned server, so
unrelated user processes (vllm, games) are never candidates. The existing
root-gated enumeration stays as a further fallback.
Adds tests: kills a recorded live server (real subprocess, verifies the actual
SIGKILL), skips a reused non-llama PID, cleans a stale/missing pidfile, and
clears the pidfile on kill.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: only reap a recorded llama-server when it is a true orphan
Harden the pidfile cross-session reaper so it cannot kill a live server
and does not crash on Windows.
- Reap the recorded PID only when its parent is gone (a genuine orphan),
so constructing a second LlamaCppBackend in-process (the helper and
advisor paths each build one) can never kill the active chat server.
The check is topology independent: it holds whether the sweep runs in
the main process or a worker.
- Record pid:starttime and verify the start-time identity before killing,
so a PID recycled to another process is never reaped.
- Fall back to SIGTERM when signal.SIGKILL is undefined (Windows), where
os.kill maps it to TerminateProcess, instead of raising and leaving the
orphan alive while clearing the record.
Update and extend the pidfile tests: a live server with a running parent
is spared and its record kept, an identity mismatch is skipped, the
record-to-reap round trip kills a matching orphan, and the Windows
SIGKILL fallback uses SIGTERM.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Michael Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Reap Studio child processes when the parent dies abnormally
Standalone `unsloth studio` launches orphaned cloudflared and llama-server when
the parent exited without running the cooperative shutdown path (terminal-window
close, Task Manager End Task, SIGKILL): the children reparented to init and kept
running, leaving an authenticated Cloudflare tunnel up for days.
Add utils/process_lifetime.py: a parent-owned Windows Job Object
(JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, children auto-inherit) plus Linux
PR_SET_PDEATHSIG, behind a best-effort helper that mirrors the desktop app's
windows_job.rs. initialize_parent_lifetime() runs at the top of run_server;
long-lived spawns (cloudflared, llama-server, RAG embedder, llama.cpp updater)
get the PDEATHSIG preexec, multiprocessing workers are adopted into the job, and
_graceful_shutdown plus atexit gain a terminate_all() backstop sweep. The
cooperative shutdown path is otherwise unchanged.
Verified on Linux: killing the parent now reaps cloudflared and llama-server
within ~2s instead of orphaning them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: add real Windows kill-on-job-close integration test
Spawn a parent that installs the job and a child that inherits it, terminate
the parent, and assert the child is reaped. Skipped off Windows. Also make the
liveness probe Windows-safe (os.kill(pid, 0) terminates on Windows).
* Fix Win64 handle truncation in the Job Object calls
Set explicit argtypes so the 64-bit job/process handles are not marshaled as
c_int (which truncated them on Win64, failing AssignProcessToJobObject). Assert
install success in the Windows integration test.
* Bind multiprocessing workers to parent death; harden the sweep
Review follow-ups:
- Multiprocessing workers (inference/export/training/data-recipe/Xet) cannot be
given a preexec_fn by the parent, so adopt_pid alone left them orphanable on a
Linux SIGKILL. They now bind themselves with PR_SET_PDEATHSIG at startup via
bind_current_process_to_parent_lifetime(), wired into the shared
run_without_native_path_secret entrypoint and the Xet child entry.
- Wire the previously-missed data-recipe worker through adopt_pid.
- terminate_all now honors its timeout: SIGTERM, wait, then SIGKILL the
survivors, so cooperative children can exit cleanly.
- Track adopted pids with a /proc starttime identity and add forget_pid, so the
shutdown sweep never signals a recycled pid.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Michael Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: scale export GGUF size estimates from the real model size
The Export page showed hardcoded, model-independent GGUF quant size
labels (Q8_0 ~8.2 GB, BF16 ~14.2 GB, ...) calibrated for an ~8B model.
For a 35B MoE model like Qwen3.6-35B-A3B (67 GiB bf16, Q8 ~34 GiB) the
picker wrongly reported Q8 ~8.2 GB. Only the displayed estimate was
wrong; the actual export via save_pretrained_gguf was always correct.
Add GET /api/models/export-size, which returns a model's MoE-aware
fp16/bf16-equivalent size and total params using the existing
estimate_fp16_model_size_bytes (safetensors -> config -> local -> vllm).
The result is memoized and degrades to nulls so a size hint can never
break the Export page.
The Export picker now scales each quant from that size
(bytes ~= fp16_bytes * bits_per_weight / 16, GiB units to match the
model selector), and renders no size when it is unknown rather than a
misleading fixed number. The Est. size summary in the page and dialog
is restored now that the value comes from the backend.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio export-size: address review feedback
- Run the size estimate off the event loop with asyncio.to_thread so a slow
Hugging Face request cannot stall other API or SSE endpoints.
- Cache only successful estimates; a transient failure (offline, gated before
credentials) is no longer pinned as unavailable until restart.
- Forward the HF token so private and gated models can be sized, and refetch
when the token changes.
- Clamp the size formatter index so sub-1-byte values cannot pick an
out-of-range unit.
* Studio export-size: address second review pass
- Send the HF token in an X-HF-Token header instead of the query string, so
it never lands in URLs, logs, or browser history.
- Key the estimate cache by model id only (the fp16 size is token independent),
so HF tokens are never retained in the cache.
- Restrict local-path sizing to known Studio roots (outputs/exports/cache/home)
so an authenticated caller cannot trigger a scan of an arbitrary directory.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio export-size: fix CI (import-hoist + isolated-load test stubs)
- Import ExportSizeResponse from models.models in routes/models.py instead of
re-exporting it through models/__init__.py, so the import-hoist lint does not
flag a newly added but un-loaded re-export (models/__init__.py is unchanged).
- Add Header and ExportSizeResponse to the stubbed fastapi / models.models in
test_export_absolute_paths.py, which loads routes/models.py in isolation.
* Studio: validate export-size local path before filesystem access
CodeQL flagged the export-size local-path guard as path injection: the
user-provided model path was resolved and stat-ed before it was checked
for containment under a Studio data root. Decide containment by lexical
normalization (normpath/abspath/expanduser, no filesystem access) and
only touch the filesystem once the path is proven to sit under a trusted
root, so an unvalidated value never reaches a filesystem call. Add a
direct containment unit test (under-root, root itself, missing, /etc,
and '..' traversal).
* Studio: trim export-size comments to be more concise
Shorten docstrings and comments on the export-size endpoint, helpers, tests,
and frontend size utilities; drop comments that just restate the code. Verified
code-identical (comments only) via AST/TS-compiler check. No behavior change.
* Studio: harden export-size local-path handling
Address review feedback on the export-size endpoint's local sizing:
- Resolve symlinks and re-verify containment in _is_sizable_local_path so a
symlink inside a Studio root can't point the sizer outside it.
- Re-validate the resolved LoRA base before sizing, so a crafted adapter
whose base_model points outside the roots can't redirect the scan.
- Skip nested checkpoint-*/global_step* snapshots when summing local weight
sizes so a run dir's intermediate checkpoints don't inflate the estimate.
- Size the checkpoint directory for full fine-tune checkpoint exports (whose
base may be a local/custom path), keeping base-model sizing for adapters.
Adds tests for the adapter-base escape, symlink escape, and nested-checkpoint
exclusion.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Michael Han