They downloaded inline inside the load, so they had none of the manager's disk
preflight, manifest verification, resume or panel progress. Picks now stage as
scoped jobs carrying the loader's own file list, then load from a warm cache.
Lets a consumer that reads a deliberate subset of a repo stage it through the
normal download manager. Keyed as "@scope" so it never collides with a quant or
with the repo's full snapshot, and the file list rides the registry so an
XET to HTTP retry respawns the same scoped job.
Reports the repos and exact files a pick needs so the download manager can stage
them with the loader's own file scope. A plain snapshot would add the packaged
root single, transformer shards and fp16 twins the loader never opens.
Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache
folder does not update: progress counted the old root while the download wrote to
the new one, and from_pretrained could split one model across both.
- The trainer keyed its persistent conditioning cache on family and
resolution only, while the keys themselves carry just the caption or
image content and crop variant. One cache directory reused for two
checkpoints, or for the same repo at a new revision, let a warm run
skip loading its encoders and train on the other model's embeddings
and latent statistics. Namespace on the base checkpoint and its
resolved revision as well. The revision helper now lives beside the
cache in diffusion_train_extras and the inference wrapper delegates to
it, so the two cannot disagree about what counts as the same source.
- The diffusion learning rate only checked positivity, but 1e309 floats
to inf and satisfies gt, so the route evicted the resident models and
started AdamW with an infinite rate: the first step destroys the
adapter while progress looks normal and the result is saved. Bound it
below 1.0, matching the LLM schema, which rejects inf for the same
reason.
Three review findings:
- The picker filtered every catalog member out of Recommended and Hub
search on the way to canonical group rows, but nothing renders those
rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and
unused). A task-scoped picker's models list is catalogToModelOptions(),
i.e. group members exclusively, so both lists came back empty and no
curated model could be discovered or downloaded. Keep the artifacts
listed until the grouped UI exists.
- The video delete guard compared only repo_id, so deleting the
companion base of a loaded GGUF video model was allowed even though it
supplies the VAE and text encoders. Compare base_repo too, matching
what the images guard already does for its companions.
- diffusers was declared unversioned while the diffusion stack requires
0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and
Z-Image pipelines), so an upgrade could keep an older release and
selecting an advertised model failed until the user upgraded by hand.
Two correctness fixes:
- The cache keyed the checkpoint and its companion base by name only, so
a Hub repo advancing to a new commit, or a local directory updated in
place, kept returning embeddings from the previous text encoder. Pair
both with a revision marker: the locally resolved commit sha for a Hub
repo, config plus text-encoder file stats for a directory. Neither
loads the encoders, so a warm run still keeps them off the GPU.
- flow_shift only checked positivity, but JSON accepts 1e309, which
floats to inf, and inf <= 0 is False while NaN fails every comparison.
The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which
poisons every sampled sigma and saves a corrupted adapter while
progress looks normal. Require a finite value.
Six review findings across the Images page and model scanning:
- The quantized (int8/fp8) load path can only attach LoRA adapters
before quantization, but the frontend load request had no loras field,
so every generation after such a load was rejected and each reload
repeated it. Send the selection with the load.
- build_prequant_checkpoint passed no family to the scheme exclusions
while recording the family in metadata, so a Qwen int8 artifact baked
the short-M text-stream linears and was then rejected wholesale by the
loader's family-keyed check.
- Registering a bare single-file checkpoint directory produced no On
Device row even though the images loader can load it; only its parent
worked. Admit that shape when nothing else matched.
- Unload left the Reapply target set, so the repair path was skipped and
Reapply reloaded the ejected model. Clear it, as the video page does.
- Both FLUX.2 bases were trusted for training but not inference, so
Deploy to Create rejected every FLUX.2 adapter.
- Outpaint allocated the grown canvas before downscaling, exceeding the
browser canvas area cap on a large photo; an over-cap canvas is
unusable, so Extend silently posted a fully transparent image and
mask. Scale the source first.
Four bugs in the batched inference path, all found by review:
- A mixed-prompt batch sent a scalar negative prompt against a prompt
list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX
true-CFG encode a batch-1 negative against batch-N latents and fail in
the transformer's text/image concat. Broadcast it to match the batch.
- The FBCache step-cache reset sat above the chunk loop. diffusers only
resets that state at the end of a successful call, so a forward that
raised (the OOM the backoff is meant to recover) left its own residual
behind and the halved retry died on a shape mismatch. Reset before
every forward instead.
- The conditioning cache keyed on the checkpoint alone, but a GGUF or
single-file load takes its text encoders from the companion base, so
the same checkpoint against a different base reused the previous
base's embeddings. Key the base too.
- Gallery records stored the base seed and the requested batch size even
when a prompts/seeds list drove the run, so restoring the second image
of seeds=[5, 99] replayed seed 5. List-driven outputs now record as
single-image recipes on their own seed.
Also bound strength above 0: every img2img pipeline derives its step
count from it, so 0 leaves zero denoising steps and either raises or, on
SDXL, crashes on empty latents.
Mount-time recovery handled only phase=completed, so reloading the page
after a multi-minute generation failed left an idle view with no
diagnosis: the backend keeps the terminal failed record only until the
next job, and nothing else survives the reload. Surface it the same way
the poll does, filtering the cancelled sentinel.
Four correctness fixes on the training side:
- The labeling grid read caption sidecars under except OSError, but a
non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad
file 500d /diffusion/dataset/{name}/images and the grid could not be
opened to repair it. Read it as no caption, matching the info summary.
- An image past Pillow's own hard limit raises DecompressionBombError,
which derives straight from Exception and so escaped the upload guard's
(OSError, UnidentifiedImageError, ValueError) and returned 500 instead
of the intended 400.
- lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0):
lora_A and lora_B receive no gradient and the run saves an untrained
adapter while reporting normal progress. Bound it below 1.0, matching
the LLM request schema.
- The train panel re-seeded the base repo on every dataset refresh
because the family object identity changes on each info fetch, so an
upload or caption save silently replaced the user's chosen base and the
run started on a different model. Track the pick and only re-seed on a
real family change.
read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption
sidecar or video sidecar 500d the info, upload and gallery routes. A flat image
now yields the all-black edge map instead of its own luminance, and the four DiT
loss knobs the trainer implements are declared so model_dump keeps them.
str(None) stored the literal "None" as the caption, so a null row counted as
captioned and would have trained on that text. Also drops an unused import.
- start_training forwards resume_source_run_id to _start_training_impl, which
reads it. Without it every start raised NameError.
- Restore main's anchor in the load-marker order test: the file now has an
earlier `if config.is_gguf:`, so indexing the first one compared the wrong
branch.
- The two diffusion tests that reach diffusers now skip when it is absent,
matching the CPU repo-test env.
- The UI smoke finds nav rows that live in the sidebar's More flyout.
Family notes, example descriptions, precision labels and every helper line are
trimmed so they stop wrapping to three lines and colliding with the next
column. The nf4 label now fits its select without truncating.
- With exactly one tab unpinned, both More and that tab are dropped, so nothing
is drawn for it (previously it rendered inline). The page stays reachable by URL.
- The customizer's New chat preview uses PencilEdit02Icon, the icon the real row
renders; Edit03Icon was a different glyph.
- "Sidebar menu" is now "Profile menu", described as the shortcuts behind your
name at the bottom of the sidebar, so it no longer reads as a second name for
the navigation section above it.
Adds a "Sidebar navigation" section to Settings -> Appearance, above the
existing profile-menu customizer, with the same drag-to-reorder + switch UI.
- New sidebarNav preference: one { id, pinned } entry per navigable row
(projects, hub, images, train, video, recipes, export), array order = render
order. Defaults match the shipped layout, so an untouched install is unchanged.
- Unpinning moves a row into the More flyout rather than hiding it, so no page
becomes unreachable. New chat and Search stay fixed as actions.
- app-sidebar now renders from one navRows descriptor map, so a pinned row and
its flyout counterpart cannot drift; the More row appears only when something
is unpinned and highlights off whatever it actually holds.
- Mirrored in the backend PersonalizationCustomization: without it the model's
extra="ignore" would drop the field, and because sync replaces local state
with the server's copy once customization is saved, the user's pin order would
reset on the next sync. The validator dedupes and back-fills like sidebarMenu
but preserves the client's order, since here order is meaningful.
Frontend typecheck, i18n parity and catalog checks pass; 32 personalization
tests pass, including a round-trip asserting a reordered list survives a save.
Resolve the drift between PR #6763 and current main:
- deletion: main moved cached-model deletion into hub/services/models/deletion.py,
so the PR's Images/Video in-use guards move there too as _diffusion_blocks_delete
and _video_blocks_delete, keeping main's fail-closed 503 contract.
- llama_keepwarm: take main's rewrite, re-apply the PR's image/video inference
suffixes so a generation in flight blocks an idle unload.
- routes/training: keep main's sidecar-swap 409 and resume_source_run_id, run
start_training in the worker thread the PR's unload hook needs.
- model picker: main renamed components/assistant-ui/model-selector ->
features/model-picker/... and rewrote pickers.tsx, so the PR's picker work is
ported onto main's version (task/catalog props, task gating of hub + cached +
local rows, single-device expanderGpuGb, fine-tuned section hidden when scoped)
rather than reverting main's pinned-models and per-model-config work.
- images/video pages: imports repointed at the new model-selector path.
- tests: delete-guard tests retargeted at the deletion service.
Typecheck, i18n parity and model-catalog checks pass.
The gguf order fix that landed on main dropped the only assertion
covering the prerequisite that llama_extra_args inheritance runs before
the GGUF branch: the inherited value (a carried --no-mmproj) shapes the
hub guard's require_mmproj, so a future reorder could reject a load
over an mmproj download the inherited arguments would disable. The
comment also misattributed the inheritance site to
_guard_chat_load_against_training.
The assertion is restored anchored on the call form
"= _resolve_inherited_extra_args(", which pins the endpoint's call site
(the bare name would match the function definition, which always
precedes the endpoint, making the check vacuous), and the comment now
names the real inheritance site. 32 tests pass.
* Studio desktop: fix loading-toast overlap and typing lag on model load
- Toaster: on desktop, offset toasts below the ~34px custom window titlebar
(top 46 when isTauri) so they no longer cover the min/max/close controls.
Web is unchanged (top 12).
- Model load: the 2s load poll wrote loadProgress state every tick, which
re-renders the whole chat page during "Starting model" (cheap in Chrome,
janky in the desktop WebView2 -> laggy typing). That state is only read by
the dismissed-toast inline status, so gate all four poll branches to write
it only when the inline view is live; while the toast is up it updates via
Sonner alone.
* Studio desktop: fix HTML canvas preview, download, and panel offset
- CSP: add frame-src for localhost/127.0.0.1 so the desktop webview can
frame the backend-served artifact preview. default-src 'self' (no
frame-src) blocked it -> "127.0.0.1 refused to connect"; web is
same-origin so it already worked.
- Download: route the canvas Download button through the native save
dialog (downloadFile) instead of a blob-anchor click, which the Tauri
WebView2 silently drops.
- Nudge the artifact panel down 8px so its top edge/shadow isn't tucked
under the window top bar.
* Studio desktop: add HTML filter for native canvas save dialog
Canvas Download saves .html via save_native_file, but save_filter() had no
html/htm case, so the native dialog fell back to the JSON/CSV/etc filter and
could block saving/browsing the .html export. Add an HTML filter and include
html/htm in the catch-all. Addresses Codex review on #7391.
* Studio desktop: unblock canvas preview in dev shell + clear header fade
- Preview: the app CSP frame-src fix wasn't enough in the tauri dev shell.
The preview endpoint sets its own frame-ancestors response header, which
only allowed 'self' tauri://localhost http://tauri.localhost -- so the
Vite dev origin (http://localhost:5173) was blocked and the frame stayed
"refused to connect". Extend the allowlist with http://localhost:* and
http://127.0.0.1:* (the endpoint only renders postMessage'd HTML in a
no-same-origin sandbox, so it exposes no server resource).
- Shadow: the artifact panel toolbar sat under the full-width
chat-header-fade; lower the panel top (mt 80->90px) so the controls clear
the fade.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix reasoning-only Qwen3.6 completions in Studio
* Address reasoning-only review findings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat(studio): add DoRA support to studio
* fix: added use_dora fast encoder LoraConfig and gated use_dora on AdapterMethod
* fix(studio) serverside normalization for use_dora=true - add note documenting use_dora is silently dropped on diffusion
* fix: dora button disabled on mac, add preflight guard on GGUF lora export, mismatch now correctly falls through to existing error instead of silently no-opping
* Studio: add dora to the WizardState LoRA variant union for consistency
* Reject --use_dora on the MLX (Apple Silicon) CLI path
---------
Co-authored-by: danielhanchen <unslothai@gmail.com>
* feat(studio): save load settings in chat presets
Presets previously stored only sampling params (temperature, top_p, etc.).
Extend them with an optional loadConfig blob that captures context length,
KV cache dtype, speculative decoding, and GPU layer knobs from the current
runtime when saving.
- Apply loadConfig when switching presets or hydrating on startup
- Show a short summary under the preset controls
- Prompt to reload when a model is already loaded
Fixes#7347
* fix(studio): persist preset loadConfig and capture GGUF context
Add ChatPresetLoadConfig to the chat settings API schema so presets with
load settings no longer 400 on save. Capture effective GGUF context from
ggufContextLength when customContextLength is cleared after auto-mode load.
* fix(studio): address Codex review on preset load settings
Coalesce default maxSeqLength/speculative/gpu knobs when capturing presets,
no-op apply for legacy presets without loadConfig, preserve GPU pin on apply,
and stop replaying stale loadConfig during settings hydration.
* Remove unused getOrderedPresets import
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* fix(studio/colab): restore iframe embed via serve_kernel_port_as_iframe
Colab's output sanitizer often strips custom <iframe> tags from
IPython.display.HTML without raising, leaving a blank cell even though
display() succeeded. The kernel-port helper is the supported embedding
path and registers the proxy correctly.
- Prefer serve_kernel_port_as_iframe; keep raw HTML iframe as fallback
- Always show the clickable link card via show_link() so the proxy URL
is visible even when iframe embedding fails
- Add regression tests for embed ordering and URL truncation
Fixes#7344
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/colab): harden iframe embed fallbacks per Codex review
Guard show_link so a display failure cannot skip embedding, and only use
serve_kernel_port_as_iframe when get_colab_url returned a real Colab proxy
URL so localhost/colabtools environments still get the HTML iframe path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/colab): stop opening Colab proxy URLs in a new tab (#7349)
Colab *.prod.colab.dev proxy hosts are session-scoped and return HTTP 404
when opened as a top-level tab or from another device. Replace the
clickable Open button for those URLs with an in-notebook ready card, keep
serve_kernel_port_as_iframe for the UI, and point users at
start(cloudflare=True) for a real shareable / new-window link.
* fix(studio/colab): use kernel iframe on real Colab when eval_js fails (#7349)
Gate serve_kernel_port_as_iframe on COLAB_RELEASE_TAG + google.colab import
instead of a successful proxyPort URL. When eval_js fails and get_colab_url
falls back to localhost, real Colab notebooks still embed via the kernel helper
(port-only). colabtools without COLAB_RELEASE_TAG keeps the HTML iframe path.
Thanks @mfielding92 for the runtime diagnosis.
* Mock top-level google package in Colab embed tests
* test(studio/colab): mock top-level google package in Colab tests
Patching only sys.modules["google.colab"] fails when no google namespace
is installed: import google.colab resolves the parent first and returns
False in _is_colab_runtime(). Add a shared helper that mocks both google
and google.colab for deterministic tests across environments.
* Tighten comments in Colab embed helpers and tests
* fix(studio/colab): default Cloudflare on Colab with durable login credentials
Colab proxy iframes often load an empty document even when the kernel helper
appends the frame, leaving users unable to reach Studio to change the bootstrap
password and blocking start(cloudflare=True).
On real Colab runtime:
- Default cloudflare to True (pass cloudflare=False to opt out)
- Finalize the random admin password and print credentials in the notebook
- Persist credentials across cell re-runs after interrupt
- Show Cloudflare link before login credentials; skip blank proxy iframe when ready
- Reuse main._IS_COLAB for runtime detection (not COLAB_RELEASE_TAG alone)
- Only trust serve_kernel_port_as_iframe on real Colab; colabtools falls back to HTML
- Keep embedding when the link card display fails
Addresses Codex review feedback on #7349 and @mfielding92's catch-22 report.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/colab): skip credential finalize when cloudflare=False
Only call _finalize_colab_admin_password() when opening a Cloudflare
tunnel. start(cloudflare=False) should not clear the bootstrap-password
gate or show a login card that references a missing tunnel link.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/colab): drop stale cached Colab credentials after password change
On a Colab rerun the finalize path redisplayed the cached first-run
password whenever the bootstrap gate was already cleared. If the admin
changed the password through the app, that cached copy no longer
authenticates, so the notebook printed dead credentials. Validate the
cached password against the current stored hash before redisplaying and
drop the cache when it no longer matches.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Fixes#7244
The Studio per-model config dropdown only surfaced bf16, q8_0, q5_1,
and q4_1 even though llama.cpp already accepts q4_0, q5_0, iq4_nl, and
f32. Add the missing options to KV_CACHE_DTYPES and align API field
descriptions with the backend _valid_cache_types set.
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* fix(studio): stop false MTP/vision capability reports (#7302)
MTP probing only inspected the first physical --spec-type help line and
treated empty/crash --help output as "lacks MTP", which false-warned on
otherwise capable builds. Parse the full --spec-type help block, fail open
when the probe is inconclusive, and stop blaming bare mmproj crashes on a
projector-format mismatch when the text-only retry also fails.
Fixes#7302
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): tighten MTP probe semantics per Codex review (#7302)
Treat nonempty --help without --spec-type as definitive no-MTP, keep only
empty/crash probes inconclusive, skip binary_no_mtp UI hint on inconclusive
loads, and stop reporting supports_mtp=True in /status for unknown probes.
* Treat failed llama-server --help probes as inconclusive (#7302)
Gate definitive no-MTP results on a zero exit code so crash diagnostics with
nonempty stderr do not re-enable the false lacks-MTP warning path.
* Add returncode to probe test mock so probe_ok gating passes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fail open in /status when the MTP probe is inconclusive
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Report missing llama-server as lacking MTP in /status
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in MTP/mmproj probe changes
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: scan HF cache snapshot loads by their repo id
Inactive Hugging Face caches (legacy, default, and previously selected
download locations) are loaded by their resolved snapshot path so they
keep using the selected cache instead of re-downloading. That path is a
local filesystem path, so evaluate_file_security exempted it with
"local path; no Hub scan" and skipped Hugging Face's pickle/malware
scan. Active caches load by repo id and are still scanned, so the same
model could dodge the gate simply by being in an inactive cache.
An HF cache snapshot keeps the canonical models--org--repo/snapshots/<rev>
layout, so recover the repo id from that path and scan it instead of
exempting it. Non-cache local paths (models directory, custom folders)
still skip the scan, and a remote ref is still scanned by repo id.
Adds a regression test that a flagged pickle in an inactive-cache
snapshot path blocks the load.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: scan the exact cached commit for inactive HF caches
An HF cache snapshot path encodes the commit, not just the repo id
(models--org--repo/snapshots/<rev>). Recover the revision alongside the
repo id and pass it to model_info and the shard-index lookup so the scan
covers the exact files that will be deserialized, rather than the repo's
default branch. Without this, a pickle in an older cached commit that was
later removed from the branch would scan clean and still load.
Extends the regression test to assert the recovered revision is forwarded
to the Hub scan.
---------
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
Seed each request with the model's recommended sampling (matching the Chat UI), add per-field override flags, ignore oversized overrides, warn when sampling pins cannot apply to a reused server, and apply pins to the completions endpoint.
* Studio: fail closed on index-referenced nested pickle shards in the offline embedding gate
The offline embedding security gate (HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE)
only scanned the direct files of each SentenceTransformer load root and never
parsed local weight indexes, so a cached snapshot whose pytorch_model.bin.index.json
maps a weight to a nested shard (e.g. shards/pytorch_model-00001-of-00001.bin) was
treated as inert and allowed. The loader then follows the index into the subdir and
unpickles the shard. The online gate already blocks index-referenced subdir pickles,
so the offline path was strictly weaker.
Parse each local weight index in a load root and follow weight_map into nested dirs,
flagging any referenced pickle-extension shard. Paths resolve lexically (normpath),
never Path.resolve(), since HF cache snapshot files symlink into blobs/ and resolving
would leave the snapshot dir and false-block every sharded model offline. An absolute
path, a .. traversal that escapes the snapshot, or an unreadable/invalid index fails
closed. The existing safetensors-sibling suppression is kept.
* Studio: classify offline indexed shards by torch.load path, not pickle extension
load_state_dict picks safetensors vs torch.load per shard by the shard's own
suffix, so two offline-gate gaps remained:
- A model.safetensors.index.json whose weight_map points at a .bin shard was
suppressed by has_base_safetensors (the index file itself matches the base
safetensors regex), yet Transformers still torch.loads that shard. Only the
pytorch index is superseded by a base safetensors now; a safetensors index is
the chosen archive, so its non-safetensors targets are always flagged.
- A pytorch index can map weights to arbitrary names (shards/payload,
weights.data); the loader torch.loads any target not ending in .safetensors.
Flag indexed shards by that rule instead of a pickle-extension allowlist.
Restrict the scan to the two torch-family indexes (tf/flax load via non-pickle
loaders). Add regression tests for both cases.
* Studio: match offline weight-index filenames case-insensitively
The index-name check compared the on-disk filename exactly, while the
surrounding weight and safetensors matches use case-insensitive rules. On a
case-insensitive volume (Windows or macOS) from_pretrained opens an oddly-cased
cache file such as PYTORCH_MODEL.BIN.INDEX.JSON when it requests the canonical
lowercase name, so the exact-case check skipped it and a nested pickle shard it
referenced was allowed through. Lower-case the index name before matching, as
the rest of the gate does, and add a regression test.
* Studio: match load_state_dict format/selection exactly in the offline index scan
Two edge cases in the offline weight-index scan:
- load_state_dict decides safetensors vs torch.load with a case-sensitive
endswith(".safetensors"), so a shard named payload.SAFETENSORS still
deserializes via torch.load. Classify indexed shard suffixes case-sensitively
to match, instead of lower-casing (which treated such a shard as inert).
- A complete direct model.safetensors is selected before either sharded index,
so a stale model.safetensors.index.json referencing a .bin shard never loads.
Skip both indexes when a direct model.safetensors is present, so an otherwise
loadable model is not over-blocked.
Add regression tests for both.
* Studio: read the offline weight index as UTF-8
Path.read_text() uses the locale default, which is cp1252 on Windows, so a
UTF-8 weight index with non-ASCII bytes raised UnicodeDecodeError and the gate
blocked an otherwise loadable model. JSON is UTF-8 by spec (and how the loader
reads it), so pin the encoding.
* Studio: resolve safetensors alternatives via the loader's own filename lookup
The offline gate decided a safetensors alternative existed by case-folding the
directory listing. On a case-sensitive filesystem that let an uppercase decoy
such as MODEL.SAFETENSORS suppress the pickle scan, yet from_pretrained asks for
the canonical lowercase model.safetensors, does not find the decoy, and selects
the pickle (a direct pytorch_model.bin or the pytorch index) and deserializes it.
Probe each alternative with (root / name).is_file() instead, mirroring the
loader: is_file() honors the platform's case rules, so a decoy suppresses only
where the loader would truly open it. Suppression must never fail open; detection
stays case-insensitive (fail closed). Add regression tests for the direct and
indexed pickle decoys (skipped on case-insensitive volumes, where no bypass
exists).
* Studio: resolve indexes and shards exactly as from_pretrained does
Two more loader-fidelity gaps in the offline index scan:
- Shard lookup normalized backslashes to forward slashes. On POSIX a backslash
is a literal filename character, so an index naming dir\payload.bin matches a
real pickle of that exact name that Transformers joins and deserializes, while
the normalized dir/payload.bin missed it. Join the raw weight_map value with
os.path.join so the probe mirrors the loader on each platform.
- Index detection case-folded the directory listing, so on a case-sensitive
filesystem an uppercase PYTORCH_MODEL.BIN.INDEX.JSON artifact the loader never
opens was treated as live and its shard blocked. Probe the canonical name with
the loader's own is_file lookup instead, so an index counts only where
from_pretrained would actually load it.
Update the uppercase-index tests to assert the correct per-filesystem behavior
and add a POSIX backslash-shard regression test.
---------
Co-authored-by: danielhanchen <unslothai@gmail.com>
* fix(studio): resolve bare git on Windows sandbox PATH
Sandboxed terminal tools rebuilt PATH as venv + System32 only, so
user-installed Git under Program Files never resolved by bare name.
Append absolute host PATH dirs after the curated prefix and inherit
PATHEXT on Windows (#7317).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): restrict sandbox PATH inheritance to Windows Git dirs (#7323)
Only append Git-for-Windows install directories from the host PATH on
Windows, instead of every absolute entry. This fixes bare `git` resolution
(#7317) without letting user-writable dirs (venv, node_modules/.bin)
shadow auto-safe terminal commands.
* Pin sandbox PATHEXT to block cwd script hijacks (#7317)
Use a fixed .EXE;.COM list instead of inheriting the host PATHEXT so
cmd cannot resolve auto-approved bare names from workdir .BAT/.CMD stubs.
* Resolve sandbox git dir via shutil.which and disable cwd exe lookup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep non-exe git launchers resolvable under restricted PATHEXT
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restrict inherited sandbox git dir to system install roots
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop SystemRoot trust and canonicalize short paths for sandbox git
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Resolve Program Files via known-folder API and append canonical git dir
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trust native Program Files on 32-bit Windows and stub program roots in tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scan PATH for a trusted git and derive native Program Files root
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop ProgramFiles env from the trusted-root fallback
* Fail closed when trusted Program Files root cannot be resolved
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* fix(studio): persist connection model selections server-side
Remote Studio clients could see saved connections but not their enabled
model lists because models lived only in browser localStorage.
Store models and available_models in llm_providers and sync them through
the providers API so alternate clients inherit the same catalog state.
Fixes#7281
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Hydrate external connections on chat startup (#7281)
Extract provider sync logic into sync-external-providers.ts and call it
from chat-page on mount so persisted model selections appear in the
Connected picker without opening Settings → Connections first.
* fix(studio): backfill connection models and preserve local options (#7298)
Address Codex P2 on remote connection persistence:
- Backfill localStorage model selections to /api/providers when backend
rows still have empty models_json (legacy upgrades)
- Carry promptCacheTtl and openaiContainerTtlMinutes through startup sync
- Await hydratePersistedSettings before syncing on ChatPage mount
Contract tests: 7 passed; npm run typecheck passed.
* Tighten comments
* Tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio STT: only load safetensors weights for custom dictation models
The STT sidecar accepts arbitrary Hugging Face owner/model repos for
custom dictation models and, when safetensors were absent, downloaded
and loaded pytorch_model.bin through WhisperForConditionalGeneration
.from_pretrained. PyTorch checkpoints are pickles that execute code
during deserialization, and this path does not run the malware gate the
normal model loader applies, so an authenticated client on an exposed
Studio instance could load a crafted Whisper-looking repo and run code
in the backend.
Restrict custom STT repos to safetensors: the snapshot selector no
longer falls back to pytorch_model.bin(.index.json), the cached-snapshot
completeness check ignores pickle weights, and the load forces
use_safetensors so a stray cached pickle still cannot execute. The five
curated Whisper defaults already ship safetensors only, so this changes
nothing for the built-in models.
* STT: reject safetensors indexes that reference non-safetensors shards
A safetensors index (model.safetensors.index.json) is attacker-supplied
JSON and can name pytorch_model-*.bin shards in its weight_map.
Transformers dispatches shard loading per file by extension, so those
.bin shards still load through torch.load (pickle) even with
use_safetensors set. Require every weight_map value to end in
.safetensors in both the snapshot selector and the completeness check so
no pickle shard is downloaded or reused.
* unsloth start/run: tool-call flags, positional model, grouped help
Expose the existing tool-call controls as first-class CLI flags on both
unsloth run and unsloth start, add positional model detection with a GGUF
quant default, and group --help into rich panels.
Flags (unsloth run): --enable-tool-call-healing/--disable-tool-call-healing
(default on), --enable-tool-call-nudging/--disable-tool-call-nudging
(default on). Resolved before any re-exec and written to the existing env
controls (UNSLOTH_DISABLE_TOOL_CALL_HEALING, UNSLOTH_TOOL_CALL_NUDGE) so the
in-venv server reads them at import; an omitted flag respects a value the
parent already set.
Flags (unsloth start): --enable-tools/--disable-tools (default off, passthrough),
plus the same healing/nudging flags (default on). start conveys them to the
auto-started run via the child env and the tools flag, so it stays correct even
if run re-execs into an older Studio venv.
Positional model: a leading org/name(:variant) token routes to --model when
--model is absent, without stealing an option value or an agent passthrough arg.
A bare GGUF repo with no variant defaults to UD-Q4_K_XL for the unsloth namespace
and Q4_K_M elsewhere, applied only on the fresh auto-serve path so attaching to a
loaded model never reloads.
Help is grouped into rich panels (Model / Server / Session for start; Model /
Server and network / Tool calls / Advanced for run) so --help reads cleanly.
Adds unit coverage for the helpers, the start command-and-env forwarding, the
positional/quant defaulting, and the run env resolution.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Positional model: reuse _is_hub_model_id so local dirs and paths are not stolen
Route a bare org/name positional to --model only when it resolves as a hub id
(via the existing _is_hub_model_id, which rejects local paths and existing
dirs), so an OpenCode project dir like owner/repo is left for the agent. Apply
the same guard to the auto-serve GGUF quant default so a local -GGUF path is
not forced to a quant it may not contain.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* unsloth start: typer floor, drop redundant GGUF quant default, respect inherited tool-call env
- Require typer>=0.12.0. The rich_help_panel options added here crash at import
on typer<0.6, and the dependency was previously unbounded.
- Stop forcing a default GGUF quant for a bare org/name-GGUF on auto-serve. The
server's own quant preference already picks UD-Q4_K_XL for Unsloth uploads and
Q4_K_M otherwise, and falls back when that exact quant is missing, so forcing a
fixed variant broke external repos that only publish Q5_K_M/Q8_0.
- Make the healing/nudging start flags tri-state so an omitted flag keeps an
operator's inherited UNSLOTH_DISABLE_TOOL_CALL_HEALING / UNSLOTH_TOOL_CALL_NUDGE
instead of overwriting it with the start defaults.
* Fix start passthrough and inherited tool settings
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
* Studio: add Voice settings tab (dictation, dictionary, read aloud)
New Voice tab in Settings, placed just before About:
- Dictation: microphone picker, browser STT engine, recognition language,
and an inline mic test with a live transcript
- Dictation dictionary: entries rewrite matching speech to their exact
spelling and casing, applied in both dictation paths
- Recent dictations: last 20 final transcripts with copy and clear, so
text can be recovered if it lands in the wrong place
- Read aloud: optional button on assistant responses with two engines,
curated system voices (novelty and legacy voices filtered, quality
ranked, capped at 20) or the TTS audio model loaded in Unsloth via
/audio/generate (e.g. Orpheus), plus speed, pitch, volume and preview
Settings persist in localStorage (unsloth_voice_settings) and are read
at call time so changes apply without reloading the runtime. Adds en
keys plus the tab label for ja, zh-CN and pt-BR.
* Studio: drop the single option STT engine select, rename TTS option
The STT engine dropdown only had one entry, so it added noise without
giving a real choice. The engine row can come back once local STT
models land. Also renames the TTS engine option Unsloth TTS model to
Load TTS model to make the action clearer.
* Studio: harden Voice settings against edge cases found in simulation
Simulated the feature across Chromium, Firefox and WebKit plus node
level unit runs and backend contract checks. Fixes from the findings:
- Dictionary rewrite used a replacement string, so entries containing
dollar patterns corrupted transcripts (A$$AP became A$AP, $& injected
the match). Switched to the callback form of String.replace
- Persisted voice settings now validate types on hydration: non string
micDeviceId, dictationLanguage and ttsVoiceURI, and non boolean
ttsEnabled fall back to defaults instead of flowing into the UI
- Dictionary entries are trimmed, capped at 120 chars and re-sanitized
on hydration
- The Test dictation panel now falls back to the default microphone
when the saved device is unplugged, matching the composer adapter
Test coverage: 46 unit assertions (dictionary regex edge cases across
unicode, word boundaries and injection, voice curation for simulated
macOS, Windows and Linux voice inventories, corrupt storage merge),
13 backend contract checks against /audio/generate on an isolated
instance, and 60 browser assertions across the three engines covering
rendering, degradation without SpeechRecognition, curation in a real
DOM, dictionary persistence with unicode and dollar entries, the
no-model preview error path and corrupt localStorage recovery.
* Studio: address Voice settings review feedback
Verified each review comment before acting. Confirmed and fixed:
- Editing a dictionary entry was broken in two ways: the store trimmed
on every keystroke so spaces could not be typed, and clearing the
field deleted the entry and unmounted the input mid edit. Updates now
keep the raw value and a blur commit trims or removes the entry
- The unplugged mic fallback checked instanceof DOMException, but a
cross browser probe showed Firefox and WebKit throw
OverconstrainedError objects that are not DOMExceptions, so the
fallback never fired there. Matching on the error name now
- When the browser ended a dictation test on its own (silence timeout),
the mic stream stayed open. All recognition end paths now stop the
tracks and save the transcript through a single finalize path
- The studio TTS audio element now releases its WAV data URL as soon as
playback ends, fails or is cancelled
- Allow microphone now reports insecure contexts (no mediaDevices)
accurately instead of claiming access was blocked
- Voice tab copy moved into i18n keys per src/i18n/AGENTS.md, so locale
overlays can translate it; en is the baseline and parity passes
- unsloth_voice_settings added to the Reset all local preferences key
list so voice preferences obey the reset
- Non default microphones note that the system default is used when the
browser speech engine cannot bind a specific device, since browsers
without the start(track) overload ignore the argument silently
Re-ran the full simulation set after the changes: 46 unit assertions,
13 backend contract checks and 60 browser assertions across Chromium,
Firefox and WebKit all pass, plus a dedicated browser probe for the
dictionary editing behavior.
* Studio: use the chat mic icon in Voice settings for consistency
The Voice tab and its buttons used the hugeicons Mic02 glyph while the
chat composer uses a custom filled mic. Extract that composer icon into
a shared lib/mic-icon component, drop the duplicate inline copies in
thread.tsx and shared-composer.tsx, and use it for the Voice tab icon
and the tab's mic buttons so the microphone looks the same everywhere.
* Studio: address second round of Voice settings review feedback
Verified each new comment against the current code first. One item was
already fixed in the previous round (recording transcripts when the
browser ends a dictation test on its own). Confirmed and fixed:
- The microphone row showed a picker with generic names when browsers
enumerate unlabeled devices before permission, leaving no way to
grant access from the row. It now branches on whether labels are
visible and shows Allow microphone otherwise
- Compare chat dictation ignored the selected microphone. It now opens
the chosen device with the same fallback rules as the main adapter,
passes the track to recognition where supported and releases the
stream when recognition ends
- Closing the Voice tab cancelled the shared speechSynthesis even when
read aloud was playing a chat message. Cleanup now only cancels when
the tab owns an active preview
- Double clicking Start test could race two recognizers and leak the
first stream. A starting flag set before the getUserMedia await makes
start reentrancy safe
- Turning off the read aloud setting mid playback removed the only stop
control. The stop button now renders whenever a message is speaking
- When an engine lacks the start(track) overload, both dictation paths
now release the selected device stream before retrying with the
default microphone instead of holding it open
- Read aloud support no longer requires Web Speech synthesis: the
Unsloth TTS engine only needs audio playback, so it stays available
in WebViews without speechSynthesis, with a clear error if the system
engine is chosen there
Not addressed here: cancelling in flight backend TTS generation on
stop. The route runs generation in a worker thread without a
cancellation path, which is shared pre existing behavior with audio
chat generation and belongs in a backend change.
All suites re-run green: 46 unit, 13 backend contract and 60 browser
matrix assertions across Chromium, Firefox and WebKit, plus probes for
the unlabeled device branch and the double click race.
* Studio: drop empty and duplicate voiceURIs so the Voice tab never renders a crashing Select item
* Studio: guard dictation mic lifecycle in Voice test and Compare composer
Release a microphone opened after the component unmounts, and stop Compare
dictation on a permission or security failure instead of silently recording
from the default device, matching the main chat adapter.
* Studio: fix dictation and read-aloud lifecycle edge cases in Voice settings
- Join final dictation chunks with a space so recorded transcripts do not merge words
- Ignore a stale recognizer onend so a quick stop then restart is not torn down
- Use previewingRef so a double click on TTS preview does not orphan the first request
- Keep the read-aloud stop control visible when a new run starts while a message is spoken
- Stop the dictionary remove button from deleting an adjacent entry on a blur then click race
* Studio: trim redundant Voice settings comments
* Studio: fix Voice preview and Compare dictation edge cases
- Only cancel the shared speechSynthesis for a system-voice preview, so stopping
a Studio preview no longer stops an unrelated chat read-aloud
- Release the Studio preview audio and its WAV data URL on normal completion
- Iterate every finalized result in Compare dictation so batched phrases are kept
- Cap persisted recent dictations to the last 20 on hydration
* Studio: use clipboard fallback for recents and release failed preview audio
- Copy recent dictations via the copyToClipboard helper so the execCommand
fallback works in Safari and insecure http LAN contexts
- Release the Studio preview audio when play() rejects, not just on ended/error
* Studio: add local speech-to-text dictation engine
Add an offline dictation engine that transcribes with a local faster-whisper
model, alongside the existing browser (Web Speech) engine. The browser engine
streams audio to Apple or Google speech services and needs internet; the new
engine runs on the server, works offline, and drives any chat model without
evicting it (it loads in the backend process, separate from the model
subprocess). It also gives Firefox dictation, which has no Web Speech support.
Backend: a lazily-loaded, kept-warm faster-whisper sidecar and three routes
under /api/inference/audio (stt/status, stt/load, transcribe). faster-whisper
is torch-free, so this does not disturb the existing model stack.
Frontend: a Dictation engine setting (browser or local model), a curated model
picker with sizes, and MediaRecorder capture posted to the transcribe route.
The model warms automatically when the engine is selected, with live status.
* Studio: stream local STT transcription as you speak
Local dictation showed nothing until you stopped, because the whole clip was
transcribed once on stop. Now the growing recording is re-transcribed on a
fast pass every second and emitted as live interim text, with an accurate
final pass on stop. Partial recordings decode fine, and the model refines
earlier words as more audio arrives.
Adds an interim flag to the transcribe route (beam 1, no VAD) for the fast
preview pass; the final stop uses the accurate path.
* Studio: make local dictation stop instant and reliable
Stopping local dictation waited for a final network transcription before the
session ended, so the stop button did not flip and a second click ended the
session early and dropped the text. Now stop commits the live transcript
immediately, releases the mic at once, and ignores a second stop while
finalizing. Previews run more often so the committed text is current.
* Studio: record local dictation in short clips for reliable streaming
Re-transcribing a growing buffer every second got slower as it grew, flooded
the backend, showed stale words, and could leave the stop button stuck waiting
on a backlog. Record short independent clips instead and transcribe each once,
appending the text as you speak. Work per clip is bounded, so stopping is
prompt (with a hard timeout as a safety net) and long dictations stay smooth.
* Studio: dictate then transcribe once on stop, ChatGPT style
Local STT dictation streamed by re-transcribing the growing clip, which
was quadratic and saturated the backend (multi-second lag), and stop only
halted the recorder without releasing the mic, so it kept recording. Record
the microphone continuously, release it the instant the user stops, and
transcribe the whole clip once. Stopping is immediate and the transcript
lands in about a second. Also add the tiny model for the fastest option.
* Studio: surface dictation and read-aloud failures instead of failing silently
- Compare dictation reports microphone and speech-recognition errors via toast,
reusing the main chat adapter's describeMediaError and describeSpeechError
- Read-aloud toasts genuine model or synthesis failures while ignoring cancellations
* Studio: ChatGPT-style recording bar for dictation
Clicking the mic now drops the composer into a dedicated recording bar
with a live waveform, a discard (X) and a confirm (tick), instead of a
plain stop button. The tick stops recording and transcribes the clip;
the X throws the recording away and keeps whatever text was already in
the composer. The model adapter taps the mic with an analyser to drive
the waveform, and the router tracks the live session so the X can cancel
it without transcribing.
* Studio: transcribe dictation while speaking, ChatGPT layout
Match ChatGPT's recording layout: the bar now renders in place of the
input with the left plus button kept, the waveform in the middle, and
the discard and confirm buttons together on the right.
Cut the post-confirm delay by transcribing in the background as the user
talks. The audio is split at natural pauses (voice-activity detection off
the same analyser that drives the waveform) and each clip is transcribed
as it is cut, so confirming only has to finish the short final tail. The
model is also warmed when recording starts so the first run never pays a
cold load.
* Studio: ChatGPT waveform, hide tools while dictating, faster STT
Make the recording UI read like ChatGPT: the waveform is now a dense row
of round dots that rise into thin centered bars, and while dictating only
the plus button shows, with the mode badge and tool toggles hidden so the
bar is just the waveform and controls.
Speed up transcription: decode greedily (beam_size=1), which is several
times faster on CPU with negligible accuracy loss on short dictation
clips, and cap background segments at 6s so the final tail after confirm
stays short.
* Studio: finish ChatGPT voice bar and low-latency STT
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: full-width waveform with a timer that freezes on stop
Use the full-width waveform for the recording bar: brighter, bigger bars
that advance on a fixed cadence (keeping peaks between advances) so they
glide instead of racing by, inset from the composer edges. Keep a visible
timer and the green confirm button, matching the ChatGPT reference, and
freeze the timer and waveform the moment the user confirms.
* Studio: fix multilingual local dictation
* Studio: speed up dictation and release local STT
* Studio: harden dictation finalization and STT decoding
* Studio: restore Firefox dictation fallback
* Studio: add dictation history manager
* Studio: manage speech model downloads
* Studio: remove em dash from voice model label
* Studio: move dictation history into Voice
* Studio: source local STT from Unsloth Whisper models
Point the dictation STT sidecar and its Model Hub download entries at
Unsloth's Hugging Face Whisper repos (small, large-v3-turbo, large-v3)
and run them through Transformers, so Studio only ever downloads
Unsloth-uploaded weights. Drop faster-whisper and the Systran/mobiuslabs
repos; keep the Model Hub as the only download path via local_files_only,
and keep PyAV for audio decoding.
Device selection uses float16 on CUDA and float32 on MPS and CPU, since
Whisper's decoder is unstable in float16 on MPS and repeats tokens.
Shorten the model picker labels to name plus download size and update the
STT tests for the new backend.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: smooth dictation waveform and keep pill height
* Studio: align STT model dropdown width and tidy voice copy
* Studio: guide to local engine when browser dictation is offline
* Studio: clarify voice section and STT model copy
* Studio: keep STT warm with training-aware eviction
* Harden STT lifecycle and browser compatibility
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix model discovery test lint
* Harden cross-browser microphone errors
* Harden cross-browser microphone errors
* Surface voice test recognition errors and fall back to Studio TTS
- Voice test now toasts non-abort speech-recognition failures instead of
ending silently, matching the main and Compare dictation paths.
- Read-aloud routes to the backend model when the runtime lacks Web Speech
synthesis (audio-only WebView), so it no longer errors immediately.
* Fix reviewed STT lifecycle races
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix read-aloud fallback controls
* Guard read-aloud stop when deleting a non-speaking message
aui.message().stopSpeaking() throws unless this message is the one being
read aloud, so calling it unconditionally rejected the delete handler before
the message was removed. Only stop speech when this message is speaking.
* Cap recent dictation transcript length before persisting
Recent dictations only limited entry count, so a long transcript stored the
full text in the persisted voice settings and a few could exceed the
localStorage quota, throwing synchronously from the uncaught dictation cleanup
path. Truncate each entry on save and on hydration, matching the dictionary cap.
* Studio: keep dictation mic clickable and guide to local model
Register the dictation adapter unconditionally so the mic stays enabled
for any engine and starts working right after switching to the local
model on an already-open thread.
When the browser engine cannot run (Firefox, Brave, non-secure origins),
clicking the mic shows a toast that points to the local speech-to-text
model instead of leaving a disabled button. The toast stacks its action
below the text with a fully rounded button.
* Studio: add bottom padding below the dictation guidance toast button
* Studio: increase bottom padding under the dictation toast button
* Studio: add bottom padding inside the dictation toast button
* Studio: add five Whisper defaults and custom model search
Add private UnslothAI Tiny and Base mirrors to the curated local STT choices while keeping Small as the default. Let users search or paste a Transformers-compatible Whisper repository and validate it end to end.
Keep short dictations in one clip to avoid repeated padded encoder work, then split longer recordings near Whisper's 30-second boundary.
Update hidden model filters and tests, including the CPU-only CI runtime stub for PyAV.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: use public Unsloth Whisper repositories
Point the Tiny and Base dictation defaults to the public unsloth repositories and remove the private mirror references from model filtering and tests.
* Studio: update Whisper download sizes
Reflect the cleaned public Tiny and Base repositories in the curated model labels.
* Studio: right-align STT model size, fix dropdown wheel scroll, refresh sizes
- Show the download size on the right of each model row so long names
like Whisper Large v3 Turbo no longer hide it
- Update curated Whisper sizes to the safetensors weights actually
downloaded: Tiny 151 MB, Base 290 MB, Small 967 MB
- Drive the model list scroll from a wheel handler so the mouse wheel
scrolls it inside the Settings dialog, not just the scrollbar
- Add a search icon and shorten the placeholder to Search model
* Studio: do not search when a dictation model is picked, shrink repo label
- Treat the filled-in model text as a selection, not a query, so choosing
a model no longer kicks off a Hugging Face search
- Make the repository line under each model name smaller
* Studio: tighten dictation model and local engine descriptions
* Studio: keep model display on pick instead of the query, shrink row text
- Guard the combobox input so selecting a model shows its name and does
not echo the typed query back or start a search
- Map the item label to the friendly display so picks fill the field
- Reduce the model name and size text in each row
* Studio: show only the model name in the dictation field, shrink size label
- Drop the download size from the search field; the name alone is shown
once a model is selected, with sizes kept in the dropdown list
- Reduce the size label text in each row
* Studio: clarify the dictation model description
* Studio: drop Hugging Face from the dictation model description
* Studio: move the dictation dictionary to its own Manage subpage
- Replace the inline entry list with a Manage row, matching Dictation
history, so a long dictionary no longer crowds Voice settings
- Add a DictationDictionaryView subpage that holds the entry editor
* Studio: match STT field font, use best voice for System default
- Bump the dictation model field text to text-sm so it matches the
engine dropdown next to it
- Resolve the System default read-aloud voice to the top curated voice
instead of the browser default, which is a robotic legacy voice on macOS
* Studio: rerank read-aloud voices and drop duplicate voice entries
- Rank by vendor quality, then the user's locale, then a preferred list of
natural voices, so the best voice leads instead of the first alphabetically
- Collapse voices that macOS reports twice under one name and language
* Studio: fold dictionary and recents into the dictation section
- Drop the separate Dictation dictionary and Recent dictations headings;
their Manage rows now sit under Dictation, split by the row divider
- Shorten the custom spellings description
* Studio: add search and sort to dictation history
- Filter saved dictations by text with a search field
- Sort by newest, oldest, or A to Z; show a no-matches message
- Keep Clear all available regardless of the current filter
* Studio: settle cancelled STT loads before training and fix dictation review items
Wait for a cancelled STT load to exit and release its memory before
reporting it freed for training, so the loader cannot still be inside
from_pretrained()/.to(device) holding VRAM when the training subprocess
starts. A load that finishes before observing the cancel now gets
unloaded so the memory is actually reclaimed.
Clear the accelerator cache before the CPU fallback in load() so a failed
CUDA/MPS load does not strand reserved VRAM once the sidecar is marked
CPU-resident.
Send the saved Hugging Face token when polling STT download progress so a
gated or private repo resolves and shows the correct Load/Downloaded
state instead of reporting missing.
Mark the composer Dictate button as type="button" so clicking it does not
also submit the draft when the composer already has text or attachments.
* Studio: pin dictation settings per session and close STT startup races
Capture the STT model and language when a dictation session starts and
pass them to every queued segment and the warm-up load, so changing the
model or language mid-recording no longer transcribes the same clip with
the wrong model or a model that is not downloaded.
Check the local runtime at the top of transcribe(), before the model
cache lookup and the bounded audio decode, so a server missing PyTorch or
Transformers returns 501 up front instead of decoding a long clip first.
Treat the training startup window as active for STT device selection.
start_training frees VRAM in before_spawn but only assigns _proc later, so
a concurrent STT load could take the GPU that was just cleared. A startup
flag now reports training active from the free until the process is live,
forcing those loads to CPU; a finally clears it on every exit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stub the STT runtime check in transcribe orchestration tests
transcribe() now verifies the local runtime up front, so the unit tests
that exercise transcription orchestration must treat the runtime as
present to keep passing where PyTorch, Transformers, and PyAV are not
installed. Stub ensure_stt_available in the shared fixture and restore
the real check in the availability and load-rejection tests.
* Harden custom Whisper dictation models
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add whisper.cpp dictation engine with per-engine downloads and history rework
Engines
- New GGML STT sidecar that runs a managed whisper-server subprocess with
idle unload, plus a pinned static build script (scripts/build_whisper_cpp.sh)
- Dictation engine picker now offers Browser, Local transcription
(whisper.cpp), and Local transcription (Transformers)
- Both local engines serve the same five curated Whisper models and download
them directly with byte-level progress reported by /audio/stt/status
- Models auto load on selection and when their download finishes
- Unload and training admission account for both engines
Benchmarks (Apple Silicon, greedy, warm, same checkpoints)
- whisper.cpp transcribes 2.4x to 5x faster than Transformers and loads in
about 0.45s vs 0.86s for Whisper Small
- whisper.cpp GGUF path is unchanged by the Transformers addition
(load 0.445s -> 0.444s, short clip 0.391s -> 0.347s, long 1.197s -> 1.129s)
Voice settings UI
- Plain curated model select replaces the searchable combobox
- Single download progress bar with transfer rate for both engines
- Dictation history now stores every dictation with Show more pagination,
a top Clear history action, and links back to the chat it was spoken into
- Archived chats dialog gets the same pagination
- Delete dialog offers deleting a dictation together with its chat
Tests: 88 backend STT tests pass, including new snapshot download coverage.
Frontend typecheck, lint, i18n parity, and production build pass.
* Merge local engines into one option and source GGML models from unslothai
Engine selection
- The dictation engine dropdown is back to two choices: Browser and Local
transcription. The selected model decides the backend: curated ids run
GGML checkpoints through whisper.cpp, searched Hugging Face repositories
run safetensors through Transformers
- Model picker lists the curated models and searches Hugging Face for other
Whisper repositories, validating them before selection. The trigger is a
plain button so the selection never renders inside a text input
- /audio/stt/status accepts a model query param so downloaded state works
for custom repositories; the engine param on load, transcribe, and
download routes is derived from the model everywhere
Model source
- Curated GGML checkpoints now download from the Unsloth-hosted
unslothai/whisper-*-GGUF repositories (one repo per model) instead of
ggerganov/whisper.cpp; cache lookups, progress totals, and in-flight blob
tracking are per-model
Fixes
- Voice settings and dictation history were not persisting: the quota-safe
localStorage wrapper was declared after the store that uses it, so the
persist storage factory failed silently. Every settings write also threw
mid-click, which kept the model picker popover from closing on selection
- is_model_downloaded now verifies config, preprocessor config, and real
weight files instead of trusting an offline snapshot lookup, so a partial
download left by an aborted fetch shows the Download button instead of
failing to load
- Removed whisper.cpp mentions from user-facing text: the ready status
shows Loaded instead of the runtime name, picker rows show the source
repository, and runtime error messages say local transcription runtime
Verified with automated browser sessions and live API checks: selection
closes the picker with no page errors, persisted settings hydrate on
reload, a stale partial snapshot triggers download then loads on MPS and
transcribes, and curated models download from the unslothai repos. 88
backend STT tests, typecheck, lint, i18n parity, and build pass.
* Skip the duplicate source line for custom models in the STT picker
A custom repository's display name is its id, so search results and the
appended current selection rendered the same string twice. The source
line now only renders when it differs from the name; curated rows keep
their name, unslothai source repository, and download size.
* Verify every shard of a sharded checkpoint in the downloaded check
A snapshot holding one of N shards (or a corrupt shard index) passed the
downloaded check and then failed at load. When model.safetensors.index.json
exists, every shard in its weight map must now be present. Found by
simulation; covered by a regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Rename stale _starting references in the pump resilience tests
The startup flag on TrainingBackend was renamed to _spawn_in_progress but
two tests added alongside it still asserted on the old name, failing the
Python 3.11 to 3.13 CI jobs.
* Make the selected model row clearly highlighted in the STT picker
The current selection was a faint background tint. It now uses the accent
background with a medium weight name. Two line rows use a small corner
radius; single line custom repo rows keep the pill shape.
* Address review feedback on STT snapshot checks, VRAM release, and dictation UX
Verify snapshot completeness in the load preflight so a partial download
fails before the audio is decoded, for curated and custom repos alike.
Drop the failed accelerator traceback before the CPU retry so the cache
clear can actually release that memory. Keep unloading the GGUF sidecar
after cancelling an in-flight Transformers load; both engines can hold
memory at once. Allow Auto language with English-only .en checkpoints,
matching the backend which sends no forced language. Keep the discard
button usable while a transcription is pending so a slow or hung request
cannot trap the composer in dictation mode. Stop linking Compare and
settings test dictations to the unrelated active single chat thread.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Move the CPU retry out of the exception handler
On Python 3.10 the interpreter exception state keeps its own reference
to the traceback, so dropping it from the caught exception was not
enough to release the failed accelerator load during the retry. Leaving
the handler before clearing the cache works on every supported version.
* Address review feedback on session handoff, chat pinning, and server lifetime
Starting a dictation from a second entry point now cancels the session
it replaces, so the old recording cannot keep the microphone open or
save a transcript with no discard button pointing at it. The linked
chat is pinned when recording starts, so switching threads while a
transcription finalizes cannot relink the transcript to the newly
opened chat. whisper-server is now bound to Studio's lifetime like the
other long-lived children: PDEATHSIG on Linux, the parent job object on
Windows, and pid adoption so the shutdown sweep reaps it; before this
it survived a Ctrl+C exit as an orphan still holding the model.
* Remove the dictation mic test from Voice settings
The composer dictate button covers the same check, so the test row, its
transcript panel, the unsupported fallback row, and their strings and
search entry are gone.
* Studio STT: gate GGUF whisper-server on training and fix dictation retry and dictionary edits
GGUF (whisper.cpp) sidecar:
- Launch whisper-server with --no-gpu while training is active, mirroring the Transformers sidecar's CPU device choice, so a mid-training dictation cannot reclaim the VRAM training just freed.
- Report is_loading() during whisper-server startup so training VRAM admission accounts for the accelerator memory it is about to bind.
- Require PyAV in is_available() so /audio/stt/status reports the engine unavailable when uploads cannot be decoded, instead of loading fine and then 501ing at transcription.
- Reject a missing model before decoding audio, matching the Transformers download preflight.
Voice settings:
- The download Retry button now restarts the download; the sidecar error is sticky until a new start(), so re-polling alone never cleared it.
Dictation dictionary:
- Tabbing from an emptied entry to its remove button no longer commit-splices the row first, which shifted indices and deleted the wrong entry.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio STT: fix curated GGUF whisper filenames to match hosted repos
The unslothai/whisper-*-GGUF repos host the checkpoint as whisper-<id>.bin,
not ggml-<id>.bin, so every curated dictation download and cached-path
lookup 404'd and the whisper.cpp engine could never load a model. Point
GGML_STT_MODELS at the real filenames and guard the naming with a test.
* Studio STT: validate a custom dictation repo before downloading it
The Transformers STT engine accepts an arbitrary owner/model repo, but the
download route handed it straight to snapshot_download, pulling a possibly large
non-Whisper repository into the shared HF cache. Confirm the repo is a Whisper
checkpoint first with the existing metadata-only validate_remote_model (no
weights); curated ids short-circuit and the GGUF engine (curated-only) is
unaffected. A non-Whisper repo now 422s before any download.
* Studio STT: preempt a still-loading GGUF server for training admission
A whisper-server still in its startup window binds accelerator memory but has no
loaded_model yet, so training admission could miss it and launch into an OOM.
Make the GGUF startup cancellable (cancel_pending_load signals an abort event and
terminates the starting process without the load lock; _wait_for_server observes
it and raises SttLoadCancelledError; wait_for_load_to_settle blocks on the lock
until the killed server is reaped), and always fold the GGUF sidecar into the
resident-STT summary so a resident Transformers model cannot mask a loading GGUF
server. free_stt_model_for_training now cancels an in-flight load and waits for it
to settle before training claims the memory.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio STT: fall back to Transformers when whisper-server is absent
A curated dictation model (including the default small) hard-pinned the GGUF
engine, but standard installs do not ship whisper-server, so every recording
501'd instead of using the Transformers engine that serves the same checkpoint
-- the GGUF sidecar's own documented contract. Add _resolve_serving_stt_engine:
a GGUF request for a curated id (the only ids GGUF accepts, all Transformers-
servable) downgrades to Transformers when whisper-server is unavailable, applied
consistently to download, load and transcribe (not unload, which targets a
specific engine). The Voice tab likewise falls back to the Transformers status so
the model is not shown unavailable and download is not blocked.
* Studio STT: hide custom Whisper caches from the legacy model pickers
The legacy /cached-models (and /cached-gguf) routes called is_hidden_model with
only the owner/model id, which cannot reach the config-based Whisper check, so a
downloaded custom (non-curated) Whisper checkpoint was still offered as a chat
model. Pass the cached snapshot path so _path_is_whisper_model inspects the repo
config and hides it, matching the discovery route.
* Studio STT: hide GGUF dictation repos, lock-free status, unload fallback, split training eviction
- Hide the curated GGUF dictation repos (unslothai/whisper-*-GGUF) from the chat
model inventory and pickers, backend and frontend. Only their Transformers
safetensors companions were hidden; the GGUF repos use a different org and a
-GGUF suffix and carry a raw .bin with no whisper config.json, so they leaked
into chat pickers.
- Make the GGUF sidecar loaded_model/device accessors lock-free, mirroring the
Transformers sidecar. transcribe() holds self._lock across the whole inference
call, so /audio/stt status polls and training admission previously blocked
behind an in-flight transcription.
- stt_unload resolves through the serving resolver: a "gguf" pick on a host
without whisper-server is served by the Transformers fallback, so unload must
target that engine or the resident model is never freed. Unload also attempts
every engine even if one raises, so a failure freeing one backend no longer
skips the other.
- free_stt_model_for_training frees the Transformers and GGUF sidecars under
independent exception boundaries so a failure unloading one no longer skips
the other before training claims the memory.
Adds tests/test_stt_review_fixes.py covering all four.
* Studio STT: resolve Auto dictation language for the model engine + snapshot process liveness
- The model dictation adapter sent the raw setting (the literal "auto") to the
backend, while the browser engine resolves Auto via resolveDictationLanguage.
A batch of non-English voice notes came back mostly English on Auto. Add
resolveModelDictationLanguage: only the literal "auto" is resolved to a
concrete locale, gated so it becomes a language the model AND Whisper can
honor (mirroring the backend's known-whisper-languages set); an explicit
language, or a locale Whisper cannot honor, stays unchanged/auto-detect. Wire
it into both adapter call sites.
- GgmlSttSidecar._process_alive() read self._process twice; a concurrent
unload() nulls it under the lock while loaded_model/device read lock-free, so
a null between the two reads called None.poll(). Snapshot once. Adds a
deterministic regression test.
* studio: tighten comments and docstrings in the dictation modules
* studio: harden dictation model downloads, GGML readiness, and recording paths
Address review findings on the STT dictation feature:
- build_whisper_cpp.sh refuses to delete a whisper.cpp tree under a custom
Studio home unless it carries the Studio ownership marker, matching the
setup.sh policy, and marks trees it creates
- _snapshot_is_complete validates every shard of a sharded PyTorch
(pytorch_model.bin.index.json) checkpoint like the safetensors path, and
requires tokenizer assets (tokenizer.json or vocab.json + merges.txt)
- custom-repo downloads pin the revision resolved at validation time and
restrict snapshot_download to the model/tokenizer/config/preprocessor file
classes Studio loads
- the GGML sidecar holds its port reservation until just before spawning
whisper-server and only accepts readiness from a responder that both looks
like whisper.cpp's server and belongs to the still-running managed child,
probing twice, so mic audio cannot be posted to a foreign local process
- the recording adapter transcribes every non-empty segment; the RMS meter
only shapes segment boundaries and can no longer discard quiet speech
- Compare-pane dictation can cancel a pending transcription on second click,
with the button relabeled while finalizing
- localStorage quota recovery halves the dictation history until the save
fits, so small histories shrink too
- the System default TTS voice resolves to the platform default voice
- new dictation UI imports go through the chat and hub feature barrels
Regression tests cover the build-script gate, sharded PyTorch and tokenizer
completeness, revision pinning and allow patterns, and the whisper-server
readiness probe.
* Fix STT download and voice picker follow-ups
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add dictation button regression coverage
* Studio: prebuilt whisper.cpp via the shared llama.cpp install core, slim bundles paired to the llama prebuilt (#7294)
* Studio STT: add prebuilt whisper.cpp (whisper-server) installer
New install_whisper_prebuilt.py downloads a per-platform whisper-server
bundle published by the unslothai/whisper.cpp prebuilt CI into the managed
whisper.cpp dir (build/bin/whisper-server) so local dictation needs no
compiler. Mirrors install_node_prebuilt.py / install_llama_prebuilt.py:
host + backend detection, sha256 pins (whisper_prebuilt_pins.json) as the
trust anchor, staging + install lock + atomic swap, traversal-safe extract,
co-located shared libs (RUNPATH=$ORIGIN), an UNSLOTH_WHISPER_PREBUILT_INFO.json
marker with idempotent "already matches", and exit codes 0/1/2/3. Not wired
into setup yet; the pins ship empty so every asset fails closed until the
first fork release is published and its digests are reviewed in.
* Studio STT: install prebuilt whisper.cpp during setup and update
Add a fail-open whisper.cpp block to setup.sh after the llama.cpp section so
`unsloth studio update` (and a fresh install) fetch the prebuilt whisper-server
into the managed whisper.cpp dir the sidecar discovers. It skips a user-set
WHISPER_SERVER_PATH/UNSLOTH_WHISPER_CPP_PATH, honors UNSLOTH_SKIP_WHISPER_INSTALL,
forwards the resolved ROCm gfx, and never aborts setup: a busy install keeps the
existing runtime, and an unavailable prebuilt stays quiet (source build is opt-in
via UNSLOTH_WHISPER_FORCE_COMPILE) since Transformers STT and browser dictation
remain. Register UNSLOTH_WHISPER_PREBUILT_INFO.json as Studio-owned evidence.
* Studio STT: harden whisper-server child env + WSL ROCm detection
- Sidecar spawns whisper-server with a scrubbed child env that prepends the
binary dir (co-located GPU libs) to the loader path, and on WSL2 ROCm loads
the system HIP first (HSA_ENABLE_DXG_DETECTION=1) so a bundle's bare-metal HIP
does not segfault on /dev/dxg. Secret-bearing vars are dropped from the child.
- find_whisper_server_binary now requires an executable, not just a file.
- Installer rocm probe passes HSA_ENABLE_DXG_DETECTION and falls back to
/opt/rocm/bin/rocminfo so a WSL ROCm host is not misdetected as CPU-only;
gfx parsing skips the gfx000 CPU agent and generic ISA lines.
- Tests for the child env (secret scrub, lib dir, WSL HIP precedence), the
executable check, and the WSL rocm detection.
* Studio STT: in-app whisper.cpp prebuilt update stack + ship pins in the wheel
Mirror the llama.cpp update stack for the whisper.cpp prebuilt so Studio can
detect and install a newer whisper-server release from inside the app:
- backend/utils/whisper_cpp_freshness.py: read UNSLOTH_WHISPER_PREBUILT_INFO.json
and compare the installed release against the newest unslothai/whisper.cpp
release. Whisper tags are v<upstream>-unsloth.<N>, so is_behind compares a
(major, minor, patch, serial) key with a strict downgrade guard; 24h cache;
fail-open.
- backend/utils/whisper_cpp_update.py: run install_whisper_prebuilt.py to fetch
and atomically swap the newest bundle, unloading the warm GGUF sidecar first.
- backend/routes/whisper.py mounted at /api/whisper (update-status + update).
- pyproject: add whisper_prebuilt_pins.json to studio package-data so the
installer's trust anchor ships in the wheel (it is a data file, not a .py
module, so package discovery alone does not include it; node_prebuilt_pins.json
is listed for the same reason). Without this a pip-installed wheel had no pins
and the prebuilt install aborted to Transformers STT.
Adds test_whisper_cpp_freshness.py (version parser, is_behind matrix + downgrade
guard, marker layouts, stale decision, fail-open).
* Studio STT: verify whisper prebuilts via the release checksum index, like llama.cpp
Re-align the whisper.cpp prebuilt installer to install_llama_prebuilt.py's trust
model: instead of a committed whisper_prebuilt_pins.json, verify every download
against the release's own whisper-prebuilt-sha256.json checksum index, fetched
from the same GitHub release.
- parse_release_checksums / fetch_release_checksums / expected_sha256_for replace
the pins layer. The index is validated for schema/component and that its
release_tag matches the resolved release; an asset absent from it, a release
that does not publish it, or a manifest sha256 that disagrees with it all fail
closed to a source build.
- resolve_release_tag now resolves the newest published release at runtime (or an
explicit --published-release-tag), matching llama and the freshness check;
removed the pinned-default and the UNSLOTH_WHISPER_ALLOW_UNVERIFIED opt-in.
- Delete studio/whisper_prebuilt_pins.json and drop its pyproject package-data
entry (nothing to ship now, same as llama which has no committed pins).
- Adds test_install_whisper_prebuilt_checksums.py (index parser, fail-closed on
uncovered asset, tampered-manifest guard, newest-release resolution).
This is a same-origin checksum (integrity, not authenticity), identical to the
llama.cpp installer; pair releases with GitHub artifact attestations for provenance.
* Resolve whisper prebuilt release via the download host (no GitHub API)
Mirror install_llama_prebuilt.py's fast path: resolve the release tag from
the releases/latest redirect and fetch the manifest + checksum index from
constructed releases/download URLs, so the common install path makes zero
api.github.com calls (unauthenticated api.github.com is capped at 60 req/hour
per IP; the download host is not). Fall back to the GitHub API only on a 404,
malformed asset, or tag mismatch.
* Studio STT: coverage-aware whisper prebuilt selection via a shared core
whisper's select_artifact returned the first os/arch/backend manifest match and
ignored the SM-coverage fields the release manifest already carries, so a
Blackwell B200 (sm_100) was served cuda12-legacy (sms 50-61) -- runnable only via
forward PTX JIT. install_llama_prebuilt.py on the same host correctly picks
cuda13-newer.
Extract the coverage-aware selection into a shared, component-agnostic core under
studio/backend/utils/prebuilt/ (selection + GPU host-capability detection), lifted
from llama's linux_cuda_choice_from_release / _artifact_covers_sms / _sm_range and
generalised over a normalised artifact. whisper's HostInfo now records the GPU
compute caps + driver CUDA version (honoring CUDA_VISIBLE_DEVICES), and
select_artifact routes CUDA/ROCm through the shared selector: every visible SM
must be covered, the tightest-covering profile wins (Blackwell-aware runtime-line
ordering), ROCm matches the gfx target exactly, and an uncovered GPU falls back to
the CPU bundle. CPU/Metal/Vulkan keep first-match. The resolver JSON, exit codes,
and "already matches" contract are unchanged.
On the B200 the installer now resolves cuda13-newer, matching llama.
* Studio STT: gate whisper CUDA selection on the on-disk runtime, like llama
The prebuilt CUDA bundles are dynamically linked and intentionally do NOT ship
libcudart/libcublas -- they load the same runtime the host already has. So the
driver's advertised CUDA version is only an upper bound: a cuda13 bundle still
needs cuda13 runtime libraries present on disk. Port llama's on-disk runtime
scan (detected_linux_runtime_lines / detected_windows_runtime_lines) into the
shared core and intersect it with the driver-compatible lines in
select_cuda_attempts. A host with a cuda13 driver but only cuda12 runtime (e.g.
torch-cuda12) now correctly gets a cuda12 bundle instead of an unloadable cuda13
one; a host with no CUDA runtime at all falls back to CPU.
Fixes a glob bug in the port (any(Path(d).glob(p) for d in dirs) tests generator
truthiness, not a match) that made every major report present; add a real
filesystem test that exercises the scan.
* studio: harden shared prebuilt core to full llama parity
Apply the review findings on the shared coverage-aware prebuilt-consumer
core so whisper.cpp selection is exactly equivalent to the llama.cpp path.
hosts.py: port llama's CUDA_VISIBLE_DEVICES handling. A GPU hidden by an
index/UUID selector now reports has_usable_nvidia False instead of staying
usable, via supports_explicit_visible_device_matching plus the physical /
explicit-match branches, and _select_visible_rows now matches rows the way
llama does (index or UUID, gpu- prefix optional) and skips unmatched tokens
rather than keeping all rows. Adds the Linux /proc/driver/nvidia/gpus
fallback and has_physical_nvidia. Adds parse_macos_version.
runtime_libs.py: the Linux on-disk scan now requires the exact libcudart /
libcublas SONAME (libcudart.so.13), not a libcudart.so.13* glob, so a bare
versioned file without the SONAME symlink no longer counts as loadable.
Hardens the ldconfig parse against an empty left-hand side.
selection.py: fix the Blackwell/torch reordering so it keys on the covering
runtime lines (falls through to the torch preference when the covering lines
were filtered out), matching linux_cuda_choice_from_release. Corrects the
compatible_runtime_lines_for_driver docstring: the bundles do not ship the
CUDA runtime, so the driver version is only an upper bound and the caller
must intersect with the on-disk scan.
install_whisper_prebuilt.py: enforce a macOS artifact's min_os (new
HostInfo.macos_version) so a bundle that cannot load on the host OS version
is dropped. Keep resolver stdout to only the JSON line by leaving logs on
stderr in --resolve-prebuilt mode, and map an unexpected probe failure to
prebuilt_available False instead of a traceback.
Tests: new host-probe suite for the visible-device logic, exact-SONAME
runtime-scan cases, macOS min_os filtering, resolver stdout-only-JSON,
exit-code mapping, and the repo key.
* studio: fix whisper prebuilt selection + launch parity gaps from review
A parallel review surfaced integration defects where the whisper path could
select or launch a bundle that cannot run on a concrete host. Each is fixed to
match install_llama_prebuilt.py.
macOS min_os: the manifest labels macOS requirements as macos-<version>
(e.g. macos-14.0), which the version parser could not read, so the guard was a
no-op and a macOS-13 host would install the macos-14 Metal bundle. Strip the
platform prefix before parsing.
ROCm gfx detection: _detect_rocm_gfx returned the first gfx token and ignored
HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES / CUDA_VISIBLE_DEVICES. Since exact
ROCm matching treats that token as the active GPU, a mixed APU + dGPU host
(gfx1151 + gfx1100) with HIP_VISIBLE_DEVICES=1 installed the wrong archive. Route
through a shared pick_rocm_gfx_target (lifted from llama) that parses per-GPU
sections and honors the visibility vars (empty / -1 -> no AMD GPU).
--rocm-gfx override: recording the arch without setting has_rocm left the host on
its CUDA/CPU path so the ROCm bundle was never picked. --rocm-gfx now implies
has_rocm and clears NVIDIA state, like llama's _apply_host_overrides.
CUDA launch env: a CUDA bundle ships the ggml CUDA backend but not
libcudart/libcublas, and the sidecar launch env exposed only the bundle dir, so
on a host whose CUDA runtime lives only in the PyTorch wheels the selection would
gate cuda usable but the server could not load it. Add the CUDA-from-PyTorch
runtime dirs to the child loader path for CUDA bundles (bundle dir still first),
mirroring binary_env.
Also normalize a manifest artifact's supported_sms defensively (parity with
llama's parser) and document that blackwell_min_toolkit_for_caps is retained for
the Phase B llama Windows path.
Not changed (verified parity, not defects): Linux/Windows min_os is enforced
nowhere in llama (macOS only); the resolver is optimistic about the checksum
index and the install path verifies.
* studio: tighten prebuilt-core code comments
* studio: lift shared prebuilt installer core out of the whisper installer
* studio: reuse the llama.cpp prebuilt installer machinery for whisper
* studio: unify llama and whisper prebuilt installers on a shared descriptor core
* studio: consolidate prebuilt installer tests into the shared core suite
Grow tests/studio/install/test_prebuilt_core.py from 62 to 164 tests so every
component-agnostic behavior runs against both descriptors: the full seven
profile CUDA release matrix (multi-GPU, on-disk runtime gating, shuffle
stability, missing SM metadata, dotted SM normalization, no-driver fallback
policy), the ROCm gfx family matrix, macOS min_os gating and its helper,
backend resolution incl. cpu-fallback precedence and Intel-mac auto detect,
checksum-index non-object and plain-lookup cases, the tar symlink/hardlink
extraction guards moved from the llama suite, and the compute-cap, visible
device, runtime-line and Blackwell helper value tables moved verbatim from
the llama characterization suites.
Delete only tests whose exact behavior the master now asserts for the same
component: 40 pure-alias helper cases in test_selection_logic.py (replaced by
value-identical master tables plus an alias-identity pin), 6 extraction moves
and the master-absorbed zip-symlink case in the llama logic suite, 3 routing
twins in test_rocm_support.py already pinned byte-for-byte in
test_selection_logic.py, the 2 Blackwell helper tables in the backend resolve
suite, 28 whisper logic tests and 10 whisper checksum tests re-asserted by
the master whisper parameterization. Wrapper wiring pins, the llama release
plan dialect, fingerprints and every llama-only behavior stay untouched.
* studio: dedupe sidecar and update helpers into the backend prebuilt package
* studio: chain whisper.cpp prebuilt updates onto the llama.cpp update flow
* studio: consume paired slim whisper prebuilts via the llama ggml runtime
* studio: serve every whisper backend from slim prebuilts
* studio: drop the whisper fat per-accelerator selection chain
unslothai/whisper.cpp releases are slim-only from v1.9.1-unsloth.2: one
ggml-less bundle per os/arch, paired to the llama.cpp prebuilt that provides
every ggml backend. Delete the whisper-side fat CUDA/ROCm/metal/vulkan
selection glue; keep slim selection + pairing, link_ggml_runtime, and one
legacy shape, the published fat CPU bundle of an explicitly pinned pre-slim
release. Exit 2 now reads as prebuilt unavailable (whisper never source
builds); setup already treats it that way.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Wire libomp runtime DLL alongside ggml in slim whisper installs
llama's clang-built windows-arm64 ggml-base.dll imports
libomp140.aarch64.dll, shipped in the llama bundle but not a system DLL.
Without it next to whisper-server.exe the loader fails with
STATUS_DLL_NOT_FOUND before main. MSVC x64 links vcomp140.dll from
System32 and Linux ggml uses system libgomp.so.1, so only windows-arm64
was affected. The empty-runtime guard still requires a real ggml
library; libomp alone is not a pairing.
* studio: drop whisper-side fat-selection support structure
Slim whisper bundles are selected per os/arch only; all accelerator
capability comes from the installed llama.cpp prebuilt, whose installer
already did the coverage-aware selection. Remove the machinery that only
existed to pick among fat per-accelerator whisper bundles:
- prebuilt_core: delete the generic CUDA/ROCm coverage selection
(select_cuda_artifact, select_rocm_artifact, ArtifactView adapters,
detected_cuda_runtime_lines, the exact-SONAME linux probe) that no
shipped component routes through; llama keeps its own selection chain
and whisper shadows select_artifact with the slim-only version.
select_artifact is now a plain os/arch/backend first-match.
- install_whisper_prebuilt: drop the HostInfo CUDA fields
(compute_caps, driver_cuda_version, torch_runtime_line) and the torch
runtime probe that populated them; nothing reachable reads them, and
the resolver payload sources runtime_line from the artifact.
- whisper_cpp_update: delete the standalone start_update job worker;
whisper applies only run as the chained phase of the combined
llama+whisper update. The status payload keeps its job field (idle).
- routes/whisper: drop the progress logger that could never fire.
- tests: remove tests of the deleted paths and tests duplicating the
descriptor-parameterized core suite or the llama freshness suite.
Contracts unchanged: resolver JSON keys, exit codes, marker fields,
pairing logs, and the pinned pre-slim fat CPU escape hatch.
* Address review feedback on the whisper prebuilt update and install paths
- Pin the chained whisper phase to the release the freshness check
offered, so the download-host latest pointer cannot reinstall an
older build in a loop
- Wire the whisper prebuilt install into setup.ps1 (Windows setup
previously skipped it entirely)
- Treat a non-executable server or missing wired ggml libraries as a
broken install instead of reporting already matches
- Keep whisper sidecar reloads out of the job-level reload flag and
resync chat state after a partial chained update that unloaded llama
- Repoint home and profile vars for the whisper-server subprocess at a
managed scratch dir and drop credential-store pointers
- Clear the prebuilt marker before the opt-in source build overwrite
- Write the prebuilt marker with explicit utf-8 encoding
* Tighten comments in the whisper prebuilt consumer
* Harden the Windows whisper setup phase and the chained update edges
- setup.ps1: honor WHISPER_SERVER_PATH / UNSLOTH_WHISPER_CPP_PATH /
UNSLOTH_SKIP_WHISPER_INSTALL, run the custom-home ownership guard
before the atomic install, and forward the release-tag pin and ROCm
hints like setup.sh
- sidecar: a cpu-selected install launches whisper-server with --no-gpu
(slim wiring links every llama backend, so the flag is what keeps a
deliberate CPU choice off the GPU)
- chained update: leave whisper unpinned on macOS (the llama phase can
walk back there, and a newest-tag pin could be an impossible pairing
on every retry) and treat installer exit 2 as kept-existing-runtime
instead of failing the combined job
- job.to_tag now comes only from the llama phase, so a whisper-only
round cannot report a llama update that never ran
* Fix slim whisper runtime follow-ups
* Address remaining whisper update reviews
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address remaining prebuilt update reviews
* Fix remaining chained update reviews
* Fix remaining whisper runtime review edges
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
---------
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
Adds a configurable Hugging Face model download cache location to Unsloth Studio, selectable from Settings, with per-cache download manifests, scoped deletion, and read-only inventory of previously selected caches.
* Studio: reject whitespace-only passwords
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reject any whitespace in passwords
* Studio: surface whitespace error in setup form, isolate auth test import
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: clear composer draft on send
* Studio: verify the Cloudflare link is reachable before printing it
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: wait for tunnel DNS propagation before verifying the public URL
* Studio: bound tunnel DNS wait and health probe by one deadline
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep composer draft when overlay send validation fails
* Studio: retry transient DoH failures while waiting for tunnel DNS
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: mask AMD GPU pins via ROCR so an unsupported iGPU can't crash llama-server
On a mixed AMD host (e.g. a discrete gfx1102 GPU next to a gfx1103 iGPU)
the bundled rocm-gfx110X llama.cpp build segfaults during HSA device
enumeration on the unsupported iGPU -- before llama-server prints a line,
so every model load fails with a bare signal and empty logs.
The GPU-subset pin masked visibility with HIP_VISIBLE_DEVICES, but HIP
filtering runs only after the HSA runtime has already enumerated (and
crashed on) every agent. Mask the subset via ROCR_VISIBLE_DEVICES (the
ROCr/HSA layer) instead, so a deselected/unsupported GPU is never
enumerated. Exactly one layer is masked (HIP cleared) to avoid the
double-mask reindex that would otherwise drop the child to CPU. The
whole-set tensor-split path and the CPU-only sentinel keep their existing
HIP behavior.
Also stop misreporting the resulting startup segfault as a vision
projector incompatibility: when the text-only mmproj retry also hard-
crashes with a signal, surface a GPU/driver init crash (with the ROCR
hint) instead of blaming the projector.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten _emit_child_gpu_visibility comments for #7272
Comment and docstring only: condense the ROCR-vs-HIP masking rationale from ~22 to ~14 lines and the call-site note from 5 to 3, keeping every technical point (HSA enumeration segfault, physical ids, the -1 sentinel). Logic is unchanged, verified by an AST compare with docstrings stripped and by exercising _emit_child_gpu_visibility against a torch/HIP stub.
* Detect AMD SDK ROCm wheels (hip=None) in _emit_child_gpu_visibility (Codex P2)
The ROCm branch gated only on torch.version.hip, but AMD SDK wheels leave that unset while encoding 'rocm' in __version__ (detect_hardware handles this the same way). On such a wheel the masking was skipped entirely, leaving only CUDA_VISIBLE_DEVICES, so on a mixed AMD box the unsupported deselected iGPU still enumerated and could crash llama-server. Now the branch also treats 'rocm' in torch.__version__ as ROCm, mirroring detect_hardware. Adds tests for the hip=None SDK wheel (ROCR + default paths) and a CUDA guard so the version-string check can't false-positive.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remap CUDA_VISIBLE_DEVICES to post-ROCR ordinals on prefer_rocr for PR #7272 (Codex P1)
On the prefer_rocr path _emit_child_gpu_visibility set ROCR_VISIBLE_DEVICES to the
physical id and cleared HIP_VISIBLE_DEVICES, but left CUDA_VISIBLE_DEVICES at the
physical id. ROCR re-indexes the visible agents from 0 and, with HIP cleared, HIP
honours CUDA_VISIBLE_DEVICES -- so a non-zero pick (e.g. GPU 1) pointed out of
range, HIP saw 0 devices, and the child fell back to CPU, defeating GPU-picker
selections other than physical GPU 0. Remap CUDA to the post-ROCR ordinals
(0..N-1); GPU 0 is unchanged, the default (HIP) path and the CPU sentinel are
untouched, and non-AMD wheels never enter this branch.
* Detect AMD SDK wheels in _resolve_visible_physical_ids for PR #7272 (Codex P2)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the HIP mask on Windows ROCm in prefer_rocr for PR #7272 (Codex P2)
* Ignore ROCR_VISIBLE_DEVICES in _resolve_visible_physical_ids on Windows for PR #7272 (Codex P2)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Preserve inherited ROCR masks in the tensor-split pin for PR #7272 (Codex P2)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Leo Borcherding <borchborchmail@gmail.com>
* studio: classify embedding models from the HF cache and honor offline mode
is_embedding_model() went straight to huggingface_hub.model_info() for any repo
id, so in offline mode (no DNS, or HF_HUB_OFFLINE set) selecting an
already-downloaded model hung on network retries that could never succeed and
training/export never started (#6817).
Check the local HF cache first: a sentence-transformers repo carries
modules.json in its snapshot (the same marker used for local paths), so a cached
model is classified with no network call. When HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE
is set, anything not positively an embedding model returns False without a
network call instead of retrying a doomed request. Online, uncached lookups still
fall through to model_info(), so tag-only embedding models (feature-extraction)
are unaffected.
Adds _embedding_marker_in_hf_cache() over the existing _iter_hf_cache_snapshots.
* studio: judge the active cached revision, harden the cache probe, stop stub leaks
Three review fixes on the cache-first embedding detection:
1. Prefer the revision refs/main resolves to. The HF cache keeps snapshots of
older revisions, so an any-snapshot scan could classify a repo by a stale
revision -- e.g. a repo that used to be a sentence-transformers model would
short-circuit even the online lookup. When refs/main is recorded, only its
snapshot is consulted; the newest-first scan remains the fallback for caches
with no ref.
2. Keep the cache probe inside the detection error boundary. The snapshot
iterator stat()s entries and could raise if a cached model is deleted
concurrently, propagating a 500 out of the config/check-embedding routes.
_embedding_marker_in_hf_cache now catches everything and reads as
not-cached, so callers keep their normal Hub/offline fallback.
3. Stub loggers/structlog in the test only when the real modules are absent
(try-import, mirroring test_windows_gpu_detection_mock), so collecting this
file first can no longer shadow the real packages for later tests in the
same pytest process.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: treat a missing active-ref snapshot as a cache miss, don't cache offline misses
Two review fixes on the cache-first embedding detection:
1. When refs/main is recorded but points at a commit whose snapshot dir is
absent (partial download / cache pruning), the recorded ref is still
authoritative: return None (cache miss) instead of falling through to scan
older snapshots, which could report a stale historical revision's
modules.json as the active one -- the same stale-cache class this helper
avoids.
2. Do not cache the offline negative. When HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE
is set and the repo is not positively an ST model from modules.json,
is_embedding_model stored False under the (model_name, hf_token) key shared
with online lookups; after the env var cleared in the same process, a
tag-only (feature-extraction) embedder returned the cached False and never
reached model_info(). The offline negative is now returned without caching.
* studio: defer online embedding detection to the Hub, re-probe offline
The local modules.json marker short-circuited is_embedding_model() even
online, so a repo that dropped (or added) the marker since it was cached
was judged by its stale local revision instead of the current remote one.
Online now treats model_info() as authoritative and uses the cache marker
only as an uncached fallback when the Hub is unreachable, so a transient
failure never poisons the memo. Offline re-probes the marker on every call
without consulting or populating the memo, so a model downloaded later in
the session (or a cached online negative that predates the download) is
detected. _embedding_marker_in_hf_cache() now treats an unreadable refs/main
(a non-FileNotFoundError OSError) as a cache miss rather than scanning stale
history -- only a genuinely missing ref enables the fallback scan.
* studio: harden offline embedding detection against empty refs, offline flips, and cache casing
- _embedding_marker_in_hf_cache: an existing-but-empty/whitespace refs/main
(a partial write or in-progress truncate-and-rewrite) now reads as a cache
miss (None) instead of falling through to scan stale snapshots; only a
genuinely missing ref enables the historical scan.
- is_embedding_model: while offline, retain a positive already confirmed online
this session (model_info only ever memoizes Hub-derived results), so
_hf_offline_if_dns_dead() flipping the process to offline mid-load can't
downgrade a verified tag-only embedder to False. Cached negatives are still
bypassed and re-probed.
- resolve_cached_repo_casing + settings route: persist the embedding model in
the casing its local HF cache dir uses. Validation accepts a case-insensitive
cache hit, but an offline SentenceTransformer load resolves the cache by exact
case, so storing the requested spelling (baai/bge-m3 vs models--BAAI--bge-m3)
made the model fail to load on a case-sensitive filesystem.
* studio: reuse the exact-match-first case resolver and preserve the default
Replace the ad-hoc resolve_cached_repo_casing with the existing
resolve_cached_repo_id_case, which already prefers the exact-case cache dir
before any case variant and tie-breaks variants deterministically -- so an
exact requested id is never rewritten to a differently cased directory just
because iterdir() happened to yield it first.
Skip the normalization entirely when the submitted model equals the default:
rewriting its casing would make set_rag_embedding_model()'s exact-string
default comparison treat it as a custom override, pinning it so later changes
to the configured default stop taking effect.
* studio: don't let a stale cache marker mask a permanent Hub error
is_embedding_model's Hub-failure fallback consulted the local modules.json
marker for ANY model_info() exception, so a permanent error -- a deleted repo,
a gated repo without credentials, or a typo that matches stale cache casing --
could pass online validation on a stale marker instead of returning the
documented 409, and the persisted model could then fail when the loader
refreshes from the Hub. Classify permanent Hub errors (RepositoryNotFound,
GatedRepo, RevisionNotFound, EntryNotFound) as False, matching the nearby
GGUF/vision detectors, and reserve the cache fallback for transient/5xx failures.
* studio: honor TRANSFORMERS_OFFLINE in the embedding preflight, skip casing for local paths
- The embedding-model save reached the offline-aware is_embedding_model() only
after two preflight helpers made direct huggingface_hub calls that honor just
HF_HUB_OFFLINE: _st_module_subdirs() downloads modules.json and the security
scan fetches Hub metadata twice. In a TRANSFORMERS_OFFLINE-only session those
blocked on network timeouts before the offline return, so saving an already
cached model stalled. Both now consult a canonical hf_env_offline() helper --
the download passes local_files_only, and the metadata-only security scan
short-circuits to its documented fail-open instead of burning both timeouts.
- Skip cache-casing normalization for local paths: a relative directory such as
"org/model" is loaded from disk, so rewriting it to a case-insensitive HF
cache collision ("Org/model") would stop resolving to that directory and be
read as a Hub repo id instead.
* studio: never skip the security scan on TRANSFORMERS_OFFLINE alone
The previous commit skipped the Hub security scan whenever either offline flag
was set, but huggingface_hub honors only HF_HUB_OFFLINE: under a
TRANSFORMERS_OFFLINE-only session the later SentenceTransformer load still
reaches the network, so the scan was being skipped while the repo's pickle could
still be downloaded and deserialized -- waving through exactly what
_guard_model_security exists to block.
Split the flags: hf_hub_offline() (HF_HUB_OFFLINE, the only one that actually
prevents a fetch) gates the security short-circuit, while hf_env_offline()
(either flag, the user's intent) is used only where local-only behavior is
forced explicitly. The SentenceTransformer load now passes local_files_only from
that intent, so TRANSFORMERS_OFFLINE genuinely stops the loader fetching instead
of merely being assumed to.
* studio: short-circuit the security preflight under either offline flag
With the loader now pinned to the local cache by local_files_only =
hf_env_offline(), a TRANSFORMERS_OFFLINE-only session can no longer fetch
anything -- yet the preflight still fell through to two model_info() attempts on
10s and 20s timeouts, stalling every save and load of an already-cached embedder
for half a minute before failing open anyway.
Skip the metadata-only scan whenever either flag is set. The scan's job is to
stop a poisoned pickle being downloaded and deserialized, and nothing can be
downloaded under that predicate; the residual case -- a model cached BEFORE it
was flagged -- is the same fail-open this function has always documented for an
unavailable scan, and is exactly what HF_HUB_OFFLINE already did.
That safety argument depends on every loader behind the gate honoring the same
predicate, so it is pinned as a test invariant instead of a comment: removing
local_files_only from the SentenceTransformer construction now fails the suite.
Drops the short-lived hf_hub_offline() helper, which no longer has a caller.
* studio: scope the offline scan bypass to callers that load local-only
The previous commit put the offline short-circuit inside _fetch_security_status,
which is the malware gate shared by every loader -- so TRANSFORMERS_OFFLINE=1
disabled it for all of them, while only the RAG embedder had been changed to
pass local_files_only. MLX inference (core/inference/worker.py -> FastMLXModel
.from_pretrained), training and export call from_pretrained with no local-only
argument, and huggingface_hub ignores that flag, so those paths could still
fetch and deserialize an unscanned model with the gate switched off.
The bypass is now an explicit local_only_load argument, defaulting to False, and
only the two RAG embedding callers -- whose loader is pinned to the local cache
by the same predicate -- opt in. Tests pin both halves: the shared gate must
still scan under either offline flag by default, and no other caller may pass
local_only_load without constraining its loader.
* studio: capture offline state once, and probe the ST cache root
Two holes in the offline embedding path:
- _get() read hf_env_offline() twice: once inside _guard_model_security and
again for local_files_only. _hf_offline_if_dns_dead() mutates the process-wide
offline vars and restores them on exit, so a concurrent load could see True in
the guard -- skipping the Hub malware scan -- and False by the time the
constructor ran, fetching and deserializing the unscanned repo and breaking
the very invariant that licenses the bypass. The value is now read once in
_get() and passed to both; _guard_model_security takes it as an argument
instead of re-deriving it.
- The cache probe searched only HF_HUB_CACHE. SentenceTransformer downloads into
SENTENCE_TRANSFORMERS_HOME when that is set, using the same
models--org--name/snapshots layout under a different root, so a model fully
present there looked uncached and was rejected with a 409 offline even though
the local-only loader could load it. Snapshot lookup now covers both roots.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: probe the cache the ST loader actually uses, and require it be loadable
Adding SENTENCE_TRANSFORMERS_HOME to the shared snapshot iterator was too broad
in one direction and too narrow in another:
- _get() builds SentenceTransformer with no cache_folder, so with ST_HOME set it
searches THAT root only, never the Hub cache. Probing the union let offline
validation pass on a repo cached only in the Hub cache, after which the loader
looked in ST_HOME and failed. The Sentence-Transformers probe now resolves to
exactly one root: ST_HOME when set, the Hub cache otherwise.
- The shared iterator is also used by the GGUF detectors, whose downloads go
through hf_hub_download with no cache_dir and therefore really do use the Hub
cache. It is back to Hub-cache-only so detection cannot pick a snapshot the
GGUF load will not find.
- Casing normalization ran through resolve_cached_repo_id_case, which scans the
Hub cache, so with ST_HOME set the requested spelling was persisted unchanged
and the exact-case offline load missed the differently cased directory that
detection had just accepted. It now resolves against the same roots detection
uses, exact match first.
- A snapshot carrying only modules.json no longer counts as cached: the online
security preflight downloads that single file itself, and a partial download
leaves it behind, so validation passed for a snapshot with no weights and the
first RAG load then failed. A hit now requires the marker plus a config and at
least one weight file.
* studio: thread the captured offline state into the module probe, fix the gate shard
- _st_module_subdirs() re-read the process env for its local_files_only. With
_hf_offline_if_dns_dead() flipping those vars from another thread, a load that
captured local_only=False could still force this probe local-only, get () back
because modules.json is not cached, and leave the scan with NO module load
roots -- a Hub-flagged pickle under 0_Transformer/ would then pass as an
unreferenced nested artifact while the loader fetched and deserialized it. It
now takes the captured predicate as an argument, and the settings route reads
the state once and uses that single value for both the probe and the scan.
- Skip ST-cache casing on the llama-server backend. Nothing there loads through
SentenceTransformer: the embedder derives a GGUF companion from the saved
spelling and fetches it from the HUB cache, so normalizing to an ST_HOME
spelling would point it at a repo _hf_gguf_backend_error() never validated
(BAAI/bge-m3-GGUF instead of the checked baai/bge-m3-GGUF).
- Fix the security-gate shard, which the signature change had broken: the direct
_guard_model_security / _st_module_subdirs callers now pass the new argument
(they were raising TypeError before reaching any assertion), and the casing
tests patch utils.models.resolve_st_cached_repo_id_case, which the route
actually calls, instead of the Hub-only resolver it no longer uses -- those
patches were being silently ignored.
* studio: accept only torch-loadable weights in the offline ST probe; fix re-export lint
_snapshot_is_loadable_st_model accepted a cached snapshot whose only weights
were .onnx (or .pt), but the RAG loader builds SentenceTransformer with the
default torch backend, so such a snapshot passed offline validation and then
failed on the first load, the exact validate-then-fail this helper exists to
prevent. Restrict _ST_WEIGHT_SUFFIXES to .safetensors and .bin and add a
regression test for an ONNX-only snapshot.
Also teach scripts/verify_import_hoist.py that names listed in a module-level
__all__ are uses, so the legitimately added resolve_st_cached_repo_id_case
re-export in utils/models/__init__.py no longer trips HOISTED-IMPORT-UNUSED.
Covered by two new self-test cases.
* studio: probe the exact repo dir and revision an offline load resolves
The cache probe modelled the cache loosely rather than modelling what
SentenceTransformer actually does with local_files_only=True:
- It merged snapshots across every case-variant repo dir and then read refs/main
from whichever held the newest one. With both models--baai--bge-m3 and
models--BAAI--bge-m3 present, a complete embedding snapshot in the directory
the loader opens could be judged by a newer partial snapshot in the other,
failing validation for a usable model. It now selects the ONE directory the
loader opens, by the same exact-case-first rule resolve_st_cached_repo_id_case
uses to choose the spelling that gets persisted.
- It fell back to scanning historical snapshots when refs/main was absent. With
local_files_only the default revision is resolved THROUGH that ref, so a
snapshot directory alone is not discoverable: the settings request succeeded
and the loader then failed at first indexing. A missing, empty or unreadable
ref is now a cache miss, and the historical scan is gone.
The tests exercise the real lookup against a built cache tree instead of
patching the snapshot iterator, so they now cover the directory selection and
ref resolution the loader depends on.
* studio: record refs/main in the ONNX-only probe test
The ONNX-only regression test predates the refs/main requirement, so after that
change it returned None (a cache miss for want of a ref) before ever reaching
the weight-format check it exists to make. Recording the ref restores its
intent: the snapshot resolves, and the answer is False because an ONNX export is
not loadable by the RAG loader's default Torch backend.
* studio: recognize base-model weight files and gate the offline positive on a materialized snapshot
_snapshot_is_loadable_st_model matched any .safetensors/.bin by suffix, so a
partial cache carrying only a commonly published non-weight bin such as
training_args.bin (or an adapter-only artifact) passed offline validation and
then failed the local_files_only load at first indexing. Match recognized Torch
base-model weight filenames (model / pytorch_model, including sharded) by name.
is_embedding_model retained an online-confirmed positive offline even when no
files were cached, so a metadata-only /check-embedding result let an uncached
repo be saved and then fail at first indexing. Retain the positive only when the
active revision is materialized locally, which still covers a downloaded tag-only
embedder whose snapshot carries no modules.json.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: require a complete weight set offline and persist embedder verdicts across restarts
Two follow-ups to the offline embedding-model classifier:
- _snapshot_is_loadable_st_model now requires a COMPLETE Torch base-model
weight set in one snapshot directory, not just any single recognized weight
file. A partially downloaded sharded model (model-00001-of-00002 without its
sibling) no longer passes offline validation and then fails at first indexing
under local_files_only. Weight files are grouped by directory and a directory
counts only when it holds a single model.safetensors / pytorch_model.bin or a
full shard set whose indices cover 1..total.
- Online-confirmed embedder verdicts are now recorded under the resolved Studio
home (embedding_verdicts.json). The session memo is lost on exit, so a
downloaded tag-only feature-extraction embedder (snapshot present but no
modules.json) was misclassified as non-embedding the first offline call after
a restart. The offline branch consults this durable allowlist in addition to
the memo, still gated on the active revision being materialized on disk, so an
uncached repo is never trusted. Writes are best-effort and only positive
verdicts are stored.
* studio: require complete weights (with shard index) and resolve default casing offline
Follow-ups to the offline embedding-model classifier from the latest review:
- Trust a recorded embedder verdict (session memo or persisted allowlist) offline
only when the active snapshot carries a COMPLETE, loadable weight set, not merely
that it is materialized. A partial download (config present, weights missing or an
incomplete shard set) makes _embedding_marker_in_hf_cache read False rather than
None, so the previous marker-is-not-None gate wrongly returned True and the
local_files_only load then failed. Split out _snapshot_has_complete_weights (config
plus complete weights, modules.json aside) and _active_snapshot_dir, and gate the
known-embedder positive on the weight set.
- Require a sharded checkpoint's index map (model.safetensors.index.json /
pytorch_model.bin.index.json) in addition to every shard before accepting it:
transformers discovers and wires shards through that index, so a complete shard set
without it fails the local-only load.
- Resolve the embedding model name to its exact cache casing in the RAG loader before
constructing SentenceTransformer. The settings route persists that spelling for a
custom override but deliberately leaves the configured default verbatim, so a
default whose casing differs from the cache dir would miss it and fail offline.
Resolving at load time covers the default too; a no-op for a local path or when
nothing case-matching is cached, and idempotent for an already-normalized override.
Adds regression tests for the partial-snapshot verdict, the missing shard index, and
the loader casing resolution; updates the offline-invariant source assertion to the
resolved-name variable.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: require a tokenizer, case-fold verdict ids, and serialize verdict writes
Three follow-ups to the offline embedding-model classifier from the latest review:
- _snapshot_has_complete_weights now also requires a tokenizer asset. A
SentenceTransformer Transformer module builds an AutoTokenizer, so a snapshot with
a complete weight set but no tokenizer.json / tokenizer_config.json / vocab still
fails the local_files_only load. The check is a permissive union over the common
fast-tokenizer, config, and WordPiece/BPE/SentencePiece assets, so an unusual but
valid layout is not rejected -- only a genuinely tokenizer-less partial download.
- The persisted embedder allowlist is now keyed case-insensitively. model_info() is
queried under the requested casing while the settings route saves the cache-resolved
casing, so an exact-string lookup missed the persisted positive after a restart
(baai/model recorded, BAAI/model looked up) and a loadable tag-only embedder was
rejected. Both persist and lookup case-fold the id.
- _persist_embedder serializes its read-modify-write under a lock and writes through a
per-thread temp file, so concurrent confirmations of different embedders no longer
drop each other's entry or collide on the temp path. Cross-process writers stay
best-effort (os.replace is atomic; a dropped verdict is only an optimization miss a
later online re-confirmation heals).
Adds regression tests for the missing-tokenizer reject, alternate tokenizer assets,
cross-casing verdict match, and concurrent verdict writes; updates the snapshot test
helpers to materialize a tokenizer alongside config and weights.
* studio: tighten comments in the offline embedding-model classifier
Comment-only pass over the PR's changed files. Collapse the long block
comments and docstrings around is_embedding_model, the cache-snapshot and
weight-completeness helpers, the embedder-verdict persistence, the offline
security gate, and the offline/casing tests to short one- or two-line forms.
Preserve the rationale (issue #6817, the local_files_only invariant, the
casing and weight-gate reasons) in far fewer words. No code changes.
* studio: drop redundant comments in the offline embedding-model classifier
Second comment-reduction pass over the offline embedding-model cache work:
delete comments and trailing notes that restate the adjacent code or an
assertion, and trim the remaining docstrings and rationale comments to their
load-bearing invariants. Comments and docstrings only; no code changes.
* studio: pin embedder verdicts to a revision, canonicalize default aliases
- A persisted verdict recorded that the Hub tagged ONE revision an embedder, but
was stored per repo. Once refs/main advanced to a complete but non-embedding
Transformer snapshot, the offline path still returned True: the settings route
accepted the updated model without force and RAG could silently load it as an
embedder. Verdicts now carry the commit they were confirmed at and are trusted
only while the active revision matches. One confirmed before the repo was
cached has no revision to compare, so the first revision observed afterwards is
pinned then -- which is what lets a later advance be caught. The persisted file
gains a {id: commit} form and still reads the previous list format.
- tokenizer_config.json no longer counts as a tokenizer asset. It only DESCRIBES
a tokenizer, so a snapshot with config, weights and just that file passed
validation and then failed AutoTokenizer.from_pretrained(local_files_only=True)
at first indexing for common BERT/GPT-style models.
- A casing-only alias of the default is canonicalized to the default up front.
Repo ids are case-insensitive but every gate here compares exact strings, so
saving "Unsloth/bge-m3" against a default of "unsloth/bge-m3" ran the
verification and scan for a custom model and then persisted an override --
after which later changes to the configured default stopped applying.
- verify_import_hoist.py replays __all__ assignments in order instead of unioning
them. Only the final value exports anything, so a later plain "=" that drops a
name must leave its import counted as unused; "+=" still extends, and an
unreadable rebind keeps the earlier names rather than flagging real re-exports.
* studio: validate the real ST load root, and pin verdicts to the Hub revision
Four ways the offline probe still disagreed with what the loader does:
- Verdicts were pinned to the LOCAL refs/main, but model_info() describes the
current HUB revision. With a stale cache the two differ, so an older snapshot
nobody verified was allowlisted. The pin is now info.sha, taken from the
ModelInfo that produced the positive. A verdict carrying no revision (a legacy
entry) is no longer trusted at all -- trusting it meant pinning whatever
happened to be cached, which is the same bug; the next online check re-records
it properly.
- config, tokenizer and weights had to exist somewhere in the snapshot, not
together. modules.json can send SentenceTransformer at 0_Transformer/, which is
loaded FROM that directory, so a cache with the config at the root and only
0_Transformer/model.safetensors passed and then failed the local-only load.
Each directory is now checked as a complete load root, which covers both the
plain HF layout and the ST module layout.
- vocab.json and merges.txt counted independently, but BPE needs the pair unless
a serialized tokenizer.json is present, so half a pair validated and then
failed AutoTokenizer.from_pretrained(local_files_only=True).
- A slashless short name like all-MiniLM-L6-v2 is a supported ST alias that the
loader resolves through the sentence-transformers/ organization, so its
snapshot is cached under that full id. Probing only the bare name reported a
miss and 409'd a model that was cached and loadable; the bare id is still tried
first, matching the loader's own order.
* studio: fail closed for an offline security scan instead of failing open
A local_only (offline) load cannot fetch Hugging Face's malware scan, and the previous
behaviour skipped the scan and failed OPEN, so a cached repo with a poisoned pickle weight
could deserialize under SentenceTransformer(local_files_only=True). Evaluate it fail-CLOSED
against the cached files instead: block a base-model pickle weight the load would deserialize
(pytorch_model.bin and its shards, in a directory with no safetensors alternative) and allow a
pickle-free (safetensors / gguf are inert) cache. A cached pickle model must be reloaded online
once to be scanned, or shipped as safetensors. Nothing cached is not a security event.
_fetch_security_status no longer needs the local_only_load skip (the offline branch is handled
in evaluate_file_security). Adds a regression test covering the safetensors-allow and
pickle-block paths with no Hub call.
* studio: only suppress an offline pickle when a loadable safetensors weight exists
The offline security gate treated any .safetensors in a directory as covering a
pickle weight, so a cache with pytorch_model.bin beside a bare adapter_model.safetensors
(or an orphan shard with no index) passed the fail-closed check even though
from_pretrained still selects and deserializes the pickle. Require a genuinely loadable
safetensors weight -- an unsharded base file or a complete indexed shard set -- before
treating the pickle as covered.
Also make the import-hoist analyzer preserve uncertainty when __all__ is extended by a
value it cannot read statically (__all__ += dynamic()), matching how it already handles
an unreadable rebind, so a dynamically-supplied re-export is not flagged HOISTED-IMPORT-UNUSED.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: scope the offline pickle scan to load paths; reset __all__ opacity on rebind
Address three review follow-ups on the offline security gate and the import-hoist analyzer:
- The offline pickle scan walked the whole snapshot, so a stray pickle in a non-load
subdirectory (archive/, nemo/) that SentenceTransformer never deserializes was blocked.
Scope it to real from_pretrained load roots -- the snapshot root, or a subdir that holds
its own config.json -- matching the online scan's load-path scoping.
- _collect_dunder_all kept a sticky opaque flag: a readable replacing assignment after an
unreadable extend (__all__ += dynamic(); __all__ = []) still credited every import, so a
genuinely unused hoist went unreported. A replacing assignment now resets opacity.
- A bare __all__: list[str] annotation has no runtime value; it was treated as an unreadable
assignment and marked the export set opaque. Skip annotation-only declarations.
* studio: recase slashless ST aliases and accept a pinned embedder after a transient failure
Two offline-detection gaps on well-formed input:
- resolve_st_cached_repo_id_case bailed on every slashless name, so a differently-cased
short alias (all-minilm-l6-v2) validated case-insensitively but was loaded verbatim; the
SentenceTransformer loader rewrites it to sentence-transformers/all-minilm-l6-v2 and looks
it up case-sensitively, missing the canonical sentence-transformers/all-MiniLM-L6-v2 cache
dir. Resolve through _st_cache_repo_dir, which follows the same org alias, and hand back the
on-disk casing.
- On a transient (non-permanent) Hub failure, is_embedding_model only accepted a cached
modules.json marker, so a downloaded tag-only embedder (no modules.json) with a verdict
pinned to the active revision was rejected even though the offline branch accepts the
identical cache. Mirror the offline branch's pinned-verdict acceptance.
* studio: scan modules.json-declared module roots in the offline pickle gate
The offline pickle scan treated only the snapshot root and config.json-bearing subdirs as
load roots, so a pickle in a non-Transformer SentenceTransformer module directory that has no
config.json (e.g. a 0_WordEmbeddings/ module: wordembedding_config.json + pytorch_model.bin)
was skipped even though the loader deserializes it. Parse modules.json (and thread through
load_subdirs) to treat every declared module directory as a load root, so such a pickle is
scanned and fail-closed offline.
* studio: classify cached non-Transformer SentenceTransformer models offline
_snapshot_has_complete_weights recognized only a Transformer-shaped load root (config +
tokenizer + weights co-located), so a fully-cached model built from a non-Transformer module
(0_WordEmbeddings uses wordembedding_config.json + embedding weights and its own tokenizer, no
HF config.json; BoW keeps its vocab in config.json) was classified non-embedding offline and
the settings endpoint returned 409.
Add _snapshot_modules_all_loadable, which parses modules.json and accepts a snapshot when every
declared module's path directory carries the files that module class's own load() reads (a
Transformer/root module still needs the full HF load root; a WordEmbeddings module needs its
config plus a complete weight set; other modules need their *_config.json), and at least one
embedding-producing module is present. It is OR-ed after the Transformer check, so it only ever
accepts more and cannot regress the existing path or reject a pruned cache.
* studio: scan PEFT adapter pickle weights in the offline security gate
from_pretrained auto-detects an adapter_config.json in the load root and deserializes the
adapter weights on top of the base model, so adapter_model.bin is a separate pickle RCE vector
that a safetensors base weight does not cover. The offline scan matched only base-model pickle
names, so an offline local-only load with safetensors base weights plus a cached
adapter_model.bin was allowed despite the live adapter pickle. Scan adapter pickles too, scoped
to a load root where adapter_config.json is present and no adapter_model.safetensors exists.
* studio: require weights for Dense/CNN/LSTM SentenceTransformer modules offline
_module_dir_is_loadable accepted a Dense, CNN, or LSTM module dir with only its config, but
those modules' load() hard-load model.safetensors else pytorch_model.bin (verified against
sentence-transformers source: no fallback, raises if neither exists) -- exactly like
WordEmbeddings. A cache with such a module's config but no weights would validate and then
fail the local_files_only load. Require a complete weight set for every weighted module, not
just WordEmbeddings.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: scan root-index subdir pickle shards offline; handle __all__.append/.extend
- The offline pickle scan followed only load-root directories, so a shard mapped by a root
pytorch_model.bin.index.json into a non-root subdirectory was skipped even though
from_pretrained follows the index weight_map and deserializes it (a layout an attacker can
craft to evade the scanner). Read the local index and scan its referenced pickle shards,
covered by a loadable base safetensors at the index root -- mirroring the online scan.
- The import-hoist analyzer ignored __all__.append("X") / __all__.extend([...]) runtime
re-export mutators, so an import added solely for one tripped HOISTED-IMPORT-UNUSED. Read
their string args like +=, and treat any other __all__ method call as opaque.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: classify StaticEmbedding offline, require WordEmbeddings tokenizer, bound model_info
- A StaticEmbedding module (e.g. sentence-transformers/static-retrieval-mrl-en-v1's
0_StaticEmbedding/) holds tokenizer.json + weights and NO config, so the config-gated
non-Transformer path 409'd it offline. Recognize it by what StaticEmbedding.load() reads: a
tokenizer.json plus a complete Torch weight set.
- WordEmbeddings.load() rebuilds its tokenizer via the configured tokenizer_class.load() from the
module dir, so a WordEmbeddings module now also requires a tokenizer artifact
(whitespacetokenizer_config.json / phrasetokenizer_config.json, or a shared HF tokenizer asset),
not just its config + weights.
- With neither offline env var set, an unbounded model_info() could hang on connect/DNS retries
for networkless users (the #6817 symptom). Bound it with a 15s timeout so a dead network fails
fast and the existing transient-failure cache fallback resolves a cached model, while a
reachable Hub still wins. (Documented caveat: a stalled DNS getaddrinfo may exceed this.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Resolve indexed safetensors shards relative to their index
_safetensors_index_complete compared shard basenames against the flat
set of files in the index directory, so an index whose weight_map names
shards in a subdirectory was treated as incomplete whenever a legacy
pytorch_model.bin sat beside it. That falsely blocked a snapshot whose
pickle weights are fully covered by a complete, loadable safetensors
shard set. Resolve each shard path relative to the index directory
instead, and add a regression test for the subdir-mapped shard case.
* Restrict offline weight-completeness check to declared load roots
_snapshot_has_complete_weights scanned every directory in a snapshot and
accepted it when ANY directory was a complete Transformer load root. When
modules.json is present a SentenceTransformer load only opens the declared
module paths, so a snapshot whose declared modules are incomplete but which
happens to contain an unrelated complete directory was accepted offline and
then failed at the first local_files_only load. Restrict the candidate
directories to the roots a load actually opens: the snapshot root plus each
modules.json module path. For a well-formed snapshot the verdict is
unchanged; only a complete directory at an undeclared path no longer vouches
for an otherwise-incomplete snapshot.
* Scan SentenceTransformer Router child module weights offline
A Router (legacy Asym) snapshot declares its child sub-modules only in
router_config.json, not the top-level modules.json, and Router.load()
deserializes each child's weights from its own subdir. A config.json-less
child such as query_0_WordEmbeddings (wordembedding_config.json plus a
pickle pytorch_model.bin loaded via torch.load) was therefore neither a
modules.json-declared load root nor a config.json-bearing dir, so the
offline gate skipped its pickle even though the loader deserializes it.
Parse router_config.json at each load root and treat every declared child
subdir as a load root (bounded BFS, so nested routers are covered), so
those child pickles are scanned. Add Router regression tests: a pickle
child blocks, a safetensors child is allowed, and a Router in a declared
subfolder is followed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Do not treat an unreferenced config subdir as an offline load root
The offline pickle gate skipped a directory only when it was neither a
declared load root nor held a config.json. Because _st_load_roots already
resolves every real load root (snapshot root, modules.json / load_subdirs
dirs, Router children), the config.json fallback only ever promoted an
UNREFERENCED subdir -- a nested checkpoint-500/ or archive/ that ships its
own config.json + pytorch_model.bin -- to a load root. from_pretrained
never descends into such a subdir and the online scan ignores the same
unindexed pickle, so offline mode wrongly blocked a model the loader reads
from a clean safetensors root. Scope the pickle to directory in roots
only, and add a regression test (a stray checkpoint-500/ no longer blocks;
a modules.json-declared module dir still does).
* Classify a root Router (Asym) model as loadable offline
_module_dir_is_loadable applied Transformer root requirements (config +
tokenizer + weights) to every root module, so a Router saved at the
snapshot root -- which carries only modules.json + router_config.json and
loads its weights from child subdirs -- was classified not loadable
offline, and is_embedding_model missed a cached Router embedder. Dispatch
on the module class before the root Transformer fallback: a Router/Asym
dir is loadable when router_config.json parses and every declared child
subdir is loadable (validated recursively through _module_dir_is_loadable,
so nested routers and every child type are covered) with at least one
embedding-producing child. This also tightens a non-root Router, which
previously validated on the mere presence of router_config.json without
checking its children. Add Router regression tests (root and declared
subfolder, complete and incomplete-child).
* Require every declared module before accepting an offline cache
_snapshot_is_loadable_st_model returned has_complete_weights OR
modules_all_loadable, so a complete 0_Transformer short-circuited the or
and vouched for the whole snapshot even when a declared sibling module was
missing its serialized weights; SentenceTransformer builds every module in
modules.json, so that snapshot passed offline validation and then failed
the local-only load. When modules.json declares a non-empty list it is now
authoritative (modules_all_loadable validates every declared module);
has_complete_weights stays the fallback only for an empty/non-list
modules.json (the plain from_pretrained root). Also add the weight-bearing
modules whose load() hard-loads via load_torch_weights and previously fell
to the config-only path -- LayerNorm, WeightedLayerPooling, SparseAutoEncoder
-- to _ST_WEIGHTED_MODULE_NAMES, with source citations and the deliberate
exclusions (Pooling/Normalize/BoW/WordWeights read no weights on load).
Add parametrized regression tests over LayerNorm/WeightedLayerPooling/Dense
(a weightless sibling rejects, a complete sibling accepts).
* Reject self-referential Router children instead of recursing forever
_router_dir_is_loadable validates each router_config.json child through
_module_dir_is_loadable, which re-enters _router_dir_is_loadable for a
Router child. A malformed types entry naming the router's own directory
(a key of ".", which normalizes to the same dir) made that recursion
never descend, so it looped until RecursionError -- breaking the
documented never-raises contract and turning a crafted/corrupted cached
model into a 500 from is_embedding_model instead of a graceful
unverifiable result. A real child reference is a subdir and always
resolves deeper, so reject any child whose resolved path is the router
dir itself. Add a regression test (a router_config naming "." as a
Router child returns False without raising).
* Treat a destructuring __all__ assignment as opaque
_collect_dunder_all detected __all__ only as a direct ast.Name assignment
target, so a binding through a destructuring target (__all__, meta = [...],
v -> an ast.Tuple) was skipped entirely, leaving an empty, non-opaque
export set. A newly hoisted import re-exported only through that assignment
was then falsely flagged HOISTED-IMPORT-UNUSED. Its value cannot be mapped
statically, so mark the export set opaque when __all__ is reached only
through a destructuring / item / attr target, matching how the collector
already handles other unreadable __all__ forms. Add a self-test case.
* Canonicalize declared module paths before scoping the offline pickle gate
A repo could declare a traversing module path such as 0/../evil in
modules.json (or a router_config child), which SentenceTransformer resolves
to evil/ and deserializes evil/pytorch_model.bin. _st_load_roots recorded
the raw snap/"0/../evil", which never equals the snap/evil that rglob
yields, so the offline pickle gate skipped that directory and a malicious
repo slipped a pickle past the newly added gate. Add _canonical_load_dir
to collapse ./ and ../ components lexically and reject an upward escape,
and route the modules.json paths, load_subdirs and router children through
it so the gate scopes the same normalized directory the loader opens. Add
regression tests for a traversing modules.json path and router child.
* Close offline embedding-classification completeness gaps
Five real offline misclassifications, each a false negative (the #6817 hang
recurs) or false positive (accepted then 409s at the local_files_only load).
Dispatch _module_dir_is_loadable on the module class before the root
Transformer fallback. A module with save_in_root=True (every InputModule:
WordEmbeddings, StaticEmbedding, SparseStaticEmbedding, Transformer, Router)
is saved at the snapshot root, so a root WordEmbeddings was wrongly held to
Transformer requirements (an HF tokenizer it never writes) and classified not
loadable.
CLIPModel is Transformer-shaped: CLIPModel.load() reads AutoModel weights plus
AutoProcessor, so a config-only CLIP dir must not validate.
SparseStaticEmbedding needs a tokenizer plus either idf.json or a complete
torch weight set (conditionally weight-bearing); a config alone is not enough.
A present but empty or malformed modules.json is not loadable and does not fall
back to a root Transformer: with modules.json present the loader never takes
the plain-Transformer path (base/model.py _load_config_modules). The tag-only
no-modules.json embedder is classified separately via
_snapshot_has_complete_weights.
Validate a sharded weight index against its weight_map (every mapped shard
present, resolved relative to the index dir) instead of trusting the index
file's mere existence, mirroring the security-side check.
Add regression tests for all five.
* Close case-folding and online-traversal holes in the offline pickle gate
Two gate bypasses where the security scan credited or scoped a path
differently from what the loader actually resolves:
The safetensors credit was case-folded. _cached_pickle_weight_files lowercases
every filename, and the loadable-safetensors and adapter checks tested those
folded keys against the exact-lowercase names. On a case-sensitive filesystem
(Linux, the Studio default) a crafted repo shipping Model.SafeTensors plus a
malicious pytorch_model.bin makes transformers and sentence-transformers miss
the exact-name model.safetensors and deserialize the pickle, while the gate
credited an inert safetensors and did not block. Credit safetensors
case-sensitively against real filenames, and drop pytorch_model.safetensors
from the credit set (transformers loads only model.safetensors, never that
name). Pickle matching stays case-insensitive (over-blocking a mis-cased
pickle the loader would not load is the safe direction).
The online scan did not canonicalize traversing paths while the offline gate
did. A repo-controlled modules.json path (threaded into the online scan via
the RAG guard) or a weight_map shard entry like 0/../evil / ../evil was
compared verbatim, so a flagged evil/pytorch_model.bin never matched and
evaded the online scan though the loader resolves and deserializes it.
Canonicalize the repo-controlled load-subdir prefixes and weight_map shards
the same way the offline gate does, so offline and online agree.
Add regression tests for both bypasses.
* Treat a conditional __all__ mutation as opaque in the import-hoist linter
_collect_dunder_all replayed only top-level module statements, so an __all__
assignment or mutation inside a module-level if / try / for / while / with /
match (or a deeper scope) was ignored, leaving the export set understated. A
newly hoisted import re-exported only through such a conditional __all__ was
then falsely flagged HOISTED-IMPORT-UNUSED, blocking a valid change. A
conditional value cannot be replayed statically, so mark the export set opaque
when __all__ is bound or mutated anywhere other than a top-level statement.
Add a self-test case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope Router child sub-modules as load roots in the online embedding scan
The RAG embedding security guard unions the SentenceTransformer module dirs
from modules.json into the load roots it scopes for the Hub scan, so a flagged
pickle directly under a Transformer module blocks. A Router (legacy Asym)
module declares its child sub-modules only in router_config.json, not in
modules.json, and Router.load() deserializes each child from its own subdir.
The online scan therefore dropped a flagged child pickle (for example
query_0_WordEmbeddings/pytorch_model.bin) as an unreferenced nested shard while
the loader still deserialized it, the counterpart to the offline gate which
already expands router children via _router_child_dirs.
_st_module_subdirs now reads router_config.json for any Router-typed module and
adds each declared child (joined onto the module path, canonicalized so a
traversing entry is dropped) to the load roots. The config is read only for a
Router-typed module, so a plain embedder pays no extra fetch, and every failure
path still returns () so the guard never bricks the embedder.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Allow a recorded-clean pickle embedder to load offline
The offline embedding security gate is fail-closed: with no network to reach
Hugging Face's scan, a cached pickle weight cannot be verified, so it is blocked
and a model the user already downloaded and used online will not load offline.
This adds a persistent cache of clean Hub verdicts so that exact content can load
offline, without weakening the gate for an unknown or never-scanned pickle.
When an embedding repo is loaded online and HF's scan returns a completed clean
verdict, the load roots are hashed and recorded under the scanned commit as an
exact map of snapshot-relative pickle name to sha256, in a per-user JSON store at
studio_root()/security/embedding_scan_verdicts.json (atomic write, 0600, thread
and cross-process locked, 30-day TTL). Offline, a cached pickle model loads only
when the active cached commit and every load-root pickle's sha256 match the
recorded verdict; a missing record, moved commit, changed or added pickle,
expired record, or any error keeps blocking. Online loads always re-query the
Hub and an authoritative unsafe verdict deletes any stale record, so a
now-flagged commit cannot keep loading on an old clean record.
The store binds repo id, full commit, and a per-file sha256 map so a locally
swapped pickle at the same commit, a branch advance, or an added load-relevant
pickle is detected. A same-user attacker who can rewrite the model cache or the
store is outside the enforceable boundary and this is documented; the sha256 is
computed just before load, so a narrow verify-to-load window remains, and a Hub
scanner false negative is recorded faithfully (safetensors stays the stronger
defense).
Recording is triggered post-load in the RAG embedder because the settings route
only validates and the pre-load guard runs before the constructor downloads;
recording is skipped when the loaded commit differs from the scanned commit. The
blocked-pickle enumerator now returns snapshot-relative Paths so two module dirs
that ship the same pickle basename are hashed and reported distinctly.
* Harden the embedding verdict cache against review findings
Tighten the offline verdict cache and its enumeration so every uncertain or
malformed input fails closed and the recorded hashes always match the files the
loader reads:
- Hash every case-colliding pickle in a load root, not one representative. On a
case-sensitive filesystem pytorch_model.bin and PYTORCH_MODEL.BIN are distinct
files; keying by lowered name dropped one and could hash a decoy instead of the
loader's target. The enumerator now returns every variant Path.
- Only persist a clean verdict for a COMPLETED, entirely-benign scan. Require
scansDone to be the boolean True (not a truthy string), filesWithIssues to be a
well-formed list, and every flagged file to be a definitively-safe level; a
pending, error, unknown, or malformed entry no longer records as clean. The
online block decision is unchanged.
- Fail closed when the offline cache cannot be inspected: an rglob error now
propagates and blocks instead of reading as pickle-free, and a snapshot that
errors on resolution (vs a clean not-cached) blocks. The offline guard also
raises instead of returning when its own inspection throws, so the constructor
never deserializes an unverified cached pickle.
- Expand online Router children recursively (bounded BFS with a seen set),
mirroring the offline load-root expansion, so a flagged grandchild pickle is
scoped online and cannot be recorded clean.
- Reject absolute and drive/UNC declared paths in the load-root canonicalizers;
the loader would resolve them outside the snapshot, so collapsing them to an
in-snapshot relative dir scoped the wrong place.
- Pin verdict recording to the scanned commit's snapshot and take the offline
verify commit from the snapshot directory name, removing a second refs/main read
and the skew it allowed.
- Drop the now-unused pickle-name wrapper.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten offline embedding classification and the pickle gate
Close a set of offline edge cases where validation accepted a cache the
local_files_only load then rejects, and one gate bypass:
- Credit a sharded model.safetensors.index.json for a pickle sibling only at a
from_pretrained root. A non-Transformer SentenceTransformer module (Dense,
WordEmbeddings, StaticEmbedding) loads via Module.load_torch_weights, which
reads model.safetensors then pytorch_model.bin and never the index, so a sharded
safetensors index in such a module dir must not vouch for its pytorch_model.bin.
- Stop counting pytorch_model.safetensors as loadable in the offline classifier:
the loader probes model.safetensors (then its index) or pytorch_model.bin, never
pytorch_model.safetensors, matching the gate that already treats it as a decoy.
- Treat a present but unreadable weight index as incomplete: transformers opens
and parses any present index, so a malformed one or one without a weight_map
fails the load rather than falling back to filename-numbered shards.
- Require the CLIP image-processor config (preprocessor_config.json) for a CLIP
module: CLIPModel.load builds a CLIPProcessor that needs it, so a tokenizer
alone is not enough.
- Require a SparseStaticEmbedding config to actually select idf.json (a path
ending .json) or ship loadable weights; a bare idf.json the config does not name
falls through to load_torch_weights and raises.
- Do not use the tag-only recorded-verdict fallback when modules.json is present:
with the file present the loader takes the modules.json path, so a present but
empty or malformed manifest must not be validated as a plain root Transformer.
- Import-hoist linter: only a module-level conditional mutation or a function that
declares global __all__ makes the export set opaque; a __all__ bound as a local
in a nested function or class no longer masks a genuinely unused hoisted import.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope Router-child pickles to their deepest load root and gate the ST offline kwarg
The online scan stripped the first matching load-subdir prefix from a flagged file, so a
nested Router child pickle (0_Router/query_0_WordEmbeddings/pytorch_model.bin) matched the
parent 0_Router root, looked like an unreferenced nested shard, and slipped the gate even
though Router.load() deserializes that child directly. Match the deepest (longest) load
subdir instead, so the child becomes root-level under its own load root and blocks.
pyproject sets no lower bound on sentence-transformers and the local_files_only constructor
arg is absent on older releases, so always forwarding it broke every embedder warm on those
installs. Pass it only for an offline load; an online warm never forwards it and works as
before, while the offline capability still requires a version that supports it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reject snapshot-escaping shard paths and credit Transformer submodule safetensors
The offline pickle enumerator joined a weight-index weight_map value straight to the load
root and followed it, so a repo-controlled index mapping "../.." into a sibling snapshot
made an offline from_pretrained deserialize an out-of-snapshot pickle, and an online load
would then hash and record that external file as the scanned commit's clean content. Reject
any shard path that escapes the snapshot root and fail closed, mirroring the canonical-root
check the online shard scan already applies.
A complete model.safetensors.index.json was credited over a sibling pickle only at the
snapshot root, but a Transformer module subdirectory (0_Transformer/) is loaded via
AutoModel.from_pretrained, which honors that shard set and never reads the pickle. Credit the
sharded index for Transformer-typed modules declared in modules.json so a cached model that
ships both a sharded safetensors checkpoint and an unused PyTorch checkpoint is no longer
falsely blocked offline. Non-Transformer modules (Dense, WordEmbeddings, StaticEmbedding) read
a flat weight with no index and keep their pickle blocked.
Limit the import-hoist verifier's global __all__ scan to the declaring function's own scope so
a nested inner-scope local __all__ no longer marks the module export set opaque and mask an
unused hoisted import.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope Router children against the snapshot and mirror the ST alias rewrite
Router.load resolves each child at Path(subfolder, model_id) relative to the Router dir, so a
nested 1_Router with a "../evil" child points at evil/ inside the snapshot and the loader
deserializes evil/pytorch_model.bin. The offline enumerator canonicalized the child against
the Router dir alone and dropped anything with "..", so that pickle was never scanned and the
gate reported the cache pickle-free. Canonicalize router children against the snapshot,
retaining in-snapshot siblings as load roots and failing closed on a child that escapes the
snapshot itself, matching the online scan which already joins the prefix before normalizing.
The security gate resolved a slashless model id by probing the bare cache dir first, but the
SentenceTransformer constructor rewrites a non-basic slashless name to sentence-transformers/
<name> and loads THAT snapshot (only the basic ORIGINAL_TRANSFORMER_MODELS load bare). With
both models--<name> and models--sentence-transformers--<name> cached, the gate inspected the
bare dir while the loader read the namespaced one, so a pickle there bypassed the local-only
gate. Mirror the constructor: try the namespaced candidate first for non-basic slashless names.
Add the same not (snapshot / modules.json).is_file() guard to the transient-Hub-failure
tag-only fallback that the offline branch already carries, so a cache whose present manifest is
empty or malformed is no longer reported as a loadable embedder.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten root shard credit, module-path escapes, and weight-set probe order
Credit the sharded safetensors index at the snapshot ROOT only when the root is actually loaded
through an AutoModel/from_pretrained path. A modules.json root module of a non-Transformer type
(StaticEmbedding / WordEmbeddings / Dense) loads via load_torch_weights, which reads
pytorch_model.bin and ignores the index, so crediting a root shard index there suppressed a live
root pickle and let the offline gate report the cache pickle-free.
Recognize the Transformer subclasses CLIPModel and MLMTransformer as index-honoring load roots
(they load via from_pretrained), so a sharded-safetensors CLIP/MLM submodule with a legacy
pytorch_model.bin sibling is no longer falsely blocked offline. Mirrors the classifier dispatch.
Fail closed on an absolute or snapshot-escaping modules.json module path (or load_subdirs entry)
instead of silently dropping it: SentenceTransformer resolves such a path outside the snapshot and
would deserialize an external pytorch_model.bin the gate cannot scan.
On the classifier side, walk the weight set in the exact from_pretrained probe order
(model.safetensors, its index, pytorch_model.bin, its index) so a pickle behind a malformed
safetensors index is no longer accepted as complete, and restrict shard names to the loader-probed
stem/ext pairs so a decoy model-*.bin / pytorch_model-*.safetensors set is not treated as loadable.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restore scripts/verify_import_hoist.py to main
The offline embedding cache fix does not depend on the __all__ scope
handling that had accumulated in this linter, so revert the file to its
main version and keep the PR focused on the feature. The feature modules
still pass the existing import hoist check unchanged.
* Reuse a shared HF cache skeleton in the offline classification tests
Extract _mk_repo and _activate helpers for the repeated snapshot cache
setup that every per-type builder duplicated, and fold the two
StaticEmbedding missing-asset cases into one parametrized test. Same 125
collected items, all still passing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reclassify embedding models from the cache on every offline call
is_embedding_model consulted its process memo before the offline branch, so an
online lookup that memoized True from tags (without caching any weights) was returned
unchanged once the session went offline -- the studio flips HF_HUB_OFFLINE in-process
on a dead DNS, and the ungated check-embedding route can populate the memo. Settings
would then accept a repo the offline loader cannot open. Run the offline
cache-marker reclassification ahead of the memo and never record it, so an offline
verdict always reflects the local cache and a later cache materialization is not
masked by a stale negative. Add regression tests.
* Tighten comments on the offline embedding path
Condense the offline-embedding helper docstrings and inline comments added in
this PR to fewer, clearer lines, keeping the non-obvious security and offline
rationale. Comments and docstrings only; no code change.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
* studio: show system-wide VRAM in the multi-GPU System tab view on ROCm
The System tab's per-GPU list comes from get_visible_gpu_utilization. When
amd-smi is unavailable (always on Windows, minimal Linux installs) it fell back
to torch, whose readings are process-local: on Windows WDDM hands each process
its own budget, so a model held by the separate llama-server process read as
~0 VRAM used even with the GPU full (#7072). The primary-GPU endpoint already
compensates with system-wide sources -- Windows Performance Counters (Task
Manager's source) and Linux DRM sysfs -- but the multi-device endpoint never
got those fallbacks.
Add per-GPU variants of both sources and overlay them onto the torch fallback:
_rocm_windows_perf_counter_vram_per_adapter_gb() attributes Dedicated Usage per
physical adapter (phys_<N> in the counter instance name), and
_rocm_linux_sysfs_vram_per_card_gb() reads mem_info_vram_{used,total} per DRM
card. _overlay_system_wide_vram() applies them to the device list, ROCm-only,
best-effort: unmatched adapters and ambiguous card counts keep the torch
figures, and NVIDIA paths are untouched.
Fixes#7072
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: match VRAM overlay sources by device, honor unified memory, unblock the loop
Five review fixes on the multi-GPU system-wide VRAM overlay:
1. Linux: match DRM cards to devices by PHYSICAL index instead of a positional
zip, so a reordering visibility mask (HIP_VISIBLE_DEVICES=1,0) no longer
swaps each card's figures onto the other GPU (which would mislead
auto_select_gpu_ids and the coexistence checks). An index with no matching
card keeps its torch figures.
2. Linux: skip the overlay for a device whose sysfs total is below torch's --
on unified-memory APUs (Strix Halo) mem_info_vram_total is only the small
dedicated slice while torch sees the GTT-backed pool, and
_apply_unified_memory_correction already defines larger-total-wins.
3. Windows: group counter instances by adapter LUID, not the phys_<N> suffix --
separate adapters each read phys_0, which collapsed every GPU into key 0.
LUIDs are mapped to 0-based positions by ascending value as the closest
stand-in for device order.
4. Windows: pair the system-wide usage with the physical capacity from
get_device_properties (as the primary-GPU fallback does) -- under WDDM
mem_get_info's "total" is the process budget, which misreported capacity
and pushed utilization to 100%.
5. Run get_visible_gpu_utilization off the event loop in the /hardware/visible
route (asyncio.to_thread, the repo's convention): the ROCm fallbacks can
shell out to PowerShell with a 5s timeout, which would stall every other
request while the System view polls.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: skip the system-wide VRAM overlay for relative GPU indices
The overlay matches its per-GPU sources (Windows perf counters, Linux sysfs) by
physical device index, but under a UUID/MIG visibility mask the torch fallback
enumerates ordinals and reports index_kind == "relative", where `index` is a
visible ordinal, not a physical id. Applying the overlay there let card/adapter
0's system-wide VRAM overwrite the torch reading of a process that actually
exposes physical GPU 1, misleading auto_select_gpu_ids and the coexistence
checks. Gate the overlay on index_kind == "physical"; relative-index paths keep
the torch fallback.
* studio: drop the unreliable Windows VRAM overlay, keep the Linux one
The multi-GPU system-wide VRAM overlay is now Linux-only. The Windows
per-adapter Performance Counter path could not be made correct: the wildcard
Get-Counter query also returns non-ROCm/iGPU adapters and LUID order is not the
ROCm device order, so an adapter's usage could be overlaid onto the wrong GPU;
and it read only Dedicated Usage, missing WDDM shared memory on unified-memory
GPUs (Strix Halo), overstating free VRAM. Rather than misattribute VRAM and
skew placement decisions, Windows keeps the process-local torch fallback (no
regression vs before this PR); Linux DRM sysfs -- matched by physical index --
still fixes#7072 for the reporter's native-Linux ROCm case.
Removes _rocm_windows_perf_counter_vram_per_adapter_gb and _torch_props_total_gb.
* studio: key sysfs VRAM by DRM card number so filtering can't renumber cards
_rocm_linux_sysfs_vram_per_card_gb dropped cards with a zero total or unreadable
files and then the overlay enumerated the compacted list, so if card0 was
dropped, card1's usage was assigned to physical GPU index 0 (equal-capacity GPUs
slip past the unified-memory total guard). Return {card_number: (used, total)}
and match a device to its card number directly: a hole stays a hole -- device 0
keeps its torch figures when card0 is absent, and card1 maps to device 1.
* studio: key system-wide VRAM by ROCm ordinal, not raw DRM card number
When a non-amdgpu adapter (Intel iGPU, a display-only card) owns an earlier
DRM slot, DRM card numbers stop equalling ROCm device ordinals -- Intel card0
plus AMD card1/card2 gives ROCm devices 0/1, so keying the sysfs overlay by
card number handed ROCm device 1 card1's data (AMD device 0) and left device 0
on stale torch figures, corrupting free-VRAM placement on equal-capacity GPUs.
Only amdgpu cards expose mem_info_vram_*, so the glob already excludes foreign
adapters; order the surviving cards by their PCI address (ROCm/HIP's default
device order, read from each card's device symlink) and key by that position --
the ROCm physical ordinal, which is what the overlay matches against dev index.
An unreadable / zero-total amdgpu card still consumes its ordinal so a later
card is never renumbered onto its slot.
* studio: skip the VRAM overlay under layered HIP-over-ROCR masks
ROCR_VISIBLE_DEVICES filters physical GPUs at the HSA/ROCr layer, and a
HIP_VISIBLE_DEVICES set on top selects WITHIN that already-filtered set
(apply_gpu_ids sets HIP while leaving an inherited ROCR mask in place). When
both are active _get_parent_visible_gpu_spec() prefers the HIP value, so the
reported device index is a ROCR-relative ordinal, not a physical GPU id --
overlaying DRM-sysfs figures by that index would pull another GPU's usage
(e.g. ROCR=2,3 + HIP=1 is physical GPU 3, but the overlay would read card 1),
and equal-capacity cards bypass the total-size safeguard. Detect layered masks
and keep torch's process-local figures there rather than risk misattribution;
a single mask still leaves the index physical and is overlaid as before.
* studio: only overlay whole-card VRAM onto 1:1 ROCm devices
The overlay guard only skipped the case where sysfs total < torch total
(unified-memory APUs), so a partitioned ROCm device (MI300 in CPX mode) --
where HIP exposes several logical devices per physical card but sysfs reports
the whole card's aggregate -- passed the guard: the card total exceeds a
partition's torch total, and the overlay overwrote the partition with
whole-card usage and capacity, letting downstream selection think a partition
had the entire card free. Require the sysfs card total to match the torch
device total (within ~10%) so a mismatch in either direction -- unified memory
(sysfs smaller) or partitioning (sysfs larger) -- keeps torch's figures.
* studio: treat CUDA-over-ROCR as layered, enumerate AMD cards by driver
Two remaining mismatches between the reported device index and the DRM card the
overlay reads:
- On ROCm the HIP layer honors CUDA_VISIBLE_DEVICES as well as
HIP_VISIBLE_DEVICES, so a CUDA mask composed over ROCR layers identically:
ROCR=2,3 with CUDA=1 is physical GPU 3, yet the spec reports the ROCR value
[2,3] and the device was labeled index 2, overlaying card 2's usage onto GPU 3.
The layered check now treats ROCR combined with either HIP or CUDA as layered.
- The ROCm device set is now enumerated by bound driver (device/driver resolves
to amdgpu) instead of by the presence of mem_info_vram_*. An AMD device with
incomplete sysfs support (some APUs expose no VRAM files at all) was omitted
by the glob entirely and shifted every later card down one ordinal, letting a
similar-capacity GPU pass the total guard with another device's usage. Such a
card now consumes its ordinal and simply yields no entry.
* studio: honor GPU_DEVICE_ORDINAL and require an unambiguous card mapping
Two remaining ways the reported device index could be matched to the wrong DRM
card:
- GPU_DEVICE_ORDINAL is a supported ROCm visibility variable that
_get_parent_visible_gpu_spec() never consults, so GPU_DEVICE_ORDINAL=1
surfaces physical GPU 1 as torch ordinal 0 and it was mislabeled index 0,
overlaying card 0's usage onto GPU 1. The mask check now covers it, and is
renamed _rocm_device_index_unreliable() to say what it actually decides.
- driver == amdgpu is only a SUPERSET of the ROCm-visible set: an amdgpu-bound
adapter HIP cannot enumerate (an unsupported older AMD GPU beside a supported
one) still took an ordinal and shifted every real compute device. There is no
torch-side PCI identity to match against, so the overlay now requires the
amdgpu card count to equal the device count -- exactly the condition under
which position-in-PCI-order is a sound 1:1 mapping. Any disagreement keeps
torch's process-local figures: less informative, never misattributed.
* studio: keep the VRAM overlay working for masked GPU subsets
The card-count guard compared the amdgpu card list against the VISIBLE device
list, so any visibility mask disabled the overlay outright: HIP_VISIBLE_DEVICES=1,3
on a four-GPU host gives two devices against four cards. Those masked GPUs then
kept reporting process-local torch usage, hiding VRAM held by llama-server and
letting the training/chat placement checks overestimate free memory -- the exact
problem the overlay exists to fix.
The count check now applies only when no visibility mask is active, which is the
case where the reported devices really are the whole host and a mismatch means an
amdgpu adapter ROCm cannot enumerate is shifting the ordinals. Under a mask the
subset is expected, so each device's physical index is validated individually
instead: the per-card lookup bounds-checks it and the total-size guard rejects a
card whose capacity does not match the device's.
* studio: match GPUs to DRM cards by PCI identity, not by position
Every mapping bug on this PR came from the same root cause: there was no
authoritative link between a reported device index and a DRM card, so the
overlay kept inferring one positionally and each heuristic broke on a new host
shape -- foreign adapters on earlier DRM slots, cards with no VRAM sysfs, and
most recently amdgpu-bound adapters HIP cannot enumerate, which the count guard
could only catch on an unmasked host and therefore missed under any mask.
Use the link ROCm itself enumerates from. KFD topology
(/sys/class/kfd/kfd/topology/nodes/<N>/properties) lists exactly the GPUs HIP
exposes -- GPU nodes in node-id order are HIP's device order -- and each carries
its PCI location, so index N there IS physical device N with a stable identity.
DRM sysfs now supplies system-wide VRAM keyed by that same PCI address, and the
overlay is a join on it.
Every previous skew becomes a failed join rather than a misattribution: an
unenumerable adapter has no KFD node so it never takes an ordinal, a foreign
adapter contributes no entry, and a masked subset resolves each physical index
directly. That removes the count heuristic and its mask exception entirely. With
no KFD topology there is no identity to join on, so the overlay is skipped rather
than guessing positionally.
* studio: require verified host visibility and AMD-only KFD nodes
Three ways the identity map could still be built on a false premise:
- The NVIDIA open kernel module registers KFD topology nodes with a positive
SIMD count, so an earlier NVIDIA node shifted every AMD ordinal and ROCm
device 1 resolved to AMD GPU 0. GPU nodes now require vendor_id 4098 (0x1002),
the same filter install.sh already applies for this exact reason.
- A GPU node with an unreadable properties file or no location_id was skipped,
which silently shifted every later ordinal. Both now fail the whole map
closed, so the overlay is disabled rather than misattributing.
- A container exposing only some render devices through device cgroups sets no
visibility variable, yet torch compacts what it can see to ordinals from zero
while the host-mounted KFD and DRM trees still list every GPU. Nothing in the
reported payload distinguishes that from a full host, and torch exposes no PCI
id to check against, so the overlay now runs only when host visibility is
positively verified: no visibility mask AND device count equal to the host GPU
count. That also subsumes the previous layered-mask and GPU_DEVICE_ORDINAL
checks, so _rocm_device_index_unreliable() is gone.
This trades coverage for correctness: masked subsets and filtered containers now
keep torch's process-local figures instead of a mapping that cannot be verified.
* Fix the multi-GPU VRAM overlay docstring for PR #7216
The docstring claimed a reordering mask keeps each card on the right GPU,
but the overlay skips any active visibility mask and keeps torch's figures.
State the actual gating instead.
* Tighten comments in the multi-GPU VRAM overlay and its tests
Collapse the verbose docstrings and inline explanations added for the Linux
ROCm system-wide VRAM overlay to succinct one-liners, keeping the non-obvious
rationale (fail-closed KFD mapping, PCI-identity join, mask gating, the 10%
whole-card guard). Comments only, no behavior change.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Improve unsloth start runtime lifecycle
* Remove speculative Gemma prompt override
* Polish model download progress output
* Refine unsloth start status output
* Clarify unsloth readiness banner
* Clarify model reuse and switching output
* Queue model switches behind active inference
* Tighten unsloth start model switching
* Reduce model switch bookkeeping
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Studio re-exec compatibility
* Recheck sidecar reservation after inference drain
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pass start marker through child environment
* Fix key redaction, switch-waiter ordering, and stop/messaging gaps for PR #7313
- Redact minted sk-unsloth keys from the startup-failure log tail: the early
key marker lands in the server log before the model load finishes, so a
load-phase crash printed a live key to the terminal
- Deregister a finished switch waiter before releasing the swap gate so a
swap on another event loop cannot count it as still queued and unload the
model the finished request is about to generate against
- Warn on same-repo quant switches: an explicit variant replaces the resident
weights for every attached session, but the repo ids match so no switch
warning was printed
- Note the agent exit code when it is nonzero so the server keep-alive
message does not read as a successful session
- Use taskkill /T in unsloth studio stop so llama-server children stop too
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in start, studio, and inference changes
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: reuse MLX prompt cache across turns instead of re-prefilling
* clean up
* key prompt cache on what the KV covers
* skip windowed KV caches past their window
* verify prefix coverage before caching KV
The first generation issued while the deferred compile is still in flight
can deviate transiently (observed once on a cold fp8 build: mean abs pixel
delta 0.063/255); once the graph is settled, same-seed same-batch-shape
images are bit-identical across runs.
Same-seed images at the same batch shape are bit-identical; a solo
regeneration with the recorded seed matches its batched rendition up to
batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255,
LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed
bit-identity across batch shapes.
Batched generation: /images/generate takes a prompts list (one image per
prompt, txt2img only) or a seeds list (one prompt, one image per seed);
the legacy batch_size path derives per-image seeds base..base+n-1 like
the native engine. Every image gets its own torch.Generator so any batch
member replays alone from its gallery recipe; the whole list runs as one
forward by default with OOM backoff that halves a failed chunk, and an
explicit batch_size caps images per forward. Validated 10-22x over
serial engines on 32-image suites with LPIPS deltas within 0.002.
Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR
(the inference sibling of the trainers' cond_cache_dir, same persistent
store) wraps encode_prompt so repeated prompts skip the text-encoder
forward entirely; verified bit-identical outputs. Bypassed while LoRA
adapters are attached; tensor-argument calls pass through uncached.
Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a
different compiled graph than the dense family) and batched calls
register every distinct (w, h, batch) chunk shape they ran, so the heavy
GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B
CFG-batched) are paid once ever.
GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in
the single-file converter; diffusers' FLUX.2 converter KeyErrors on it
and the Qwen-Image identity mapping strands the model on meta.
Unsloth