llm-compressor 0.10.x/0.12.x pins transformers<=4.57.6. Models that require a newer
transformers (e.g. Qwen3.5, which needs the transformers 5.3.0 sidecar) cannot be
FP8/FP4 quantized at all: the compressed-export subprocess imports
transformers.modeling_utils.TORCH_INIT_FUNCTIONS, which was removed in transformers 5.x,
so it dies with a cryptic ImportError, and only after the expensive 16bit merge has
already been written.
Detect this in _unsloth_save_compressed_tensors before installing llm-compressor or
merging, and raise a clear, actionable error (export to GGUF or 16-bit instead). Unsloth
Studio surfaces it as a clean 400 in the export panel rather than a confusing traceback,
and no full 16bit checkpoint is written for a model that can never be quantized this way.
- Add _LLM_COMPRESSOR_MAX_TRANSFORMERS = "4.57.6" next to _LLM_COMPRESSOR_SPEC, to bump
in lockstep with a future llm-compressor that supports newer transformers.
- Add _transformers_exceeds_llm_compressor_ceiling() helper: packaging.version compare,
fail-open on any parse error so a real quantization attempt still surfaces its own error.
- Gate _unsloth_save_compressed_tensors on it before install/merge.
* Add customizable RAG embedding model setting and reorganize settings tabs
Chat with files, project sources, and knowledge bases previously always
embedded with unsloth/bge-small-en-v1.5. This adds a Settings option to
pick any Hugging Face embedding model (or local path), with HF search
autocomplete, server-side verification that the repo is actually an
embedding model, and a save anyway escape hatch for offline or local
models. The setting persists in app_settings and applies at runtime to
both the sentence-transformers and llama-server GGUF embedder backends
without a restart.
Also reorganizes the General settings tab: Documents & RAG sits above
Uploads, Helper LLM moved above the danger zone, and Model auto-switch
(OpenAI API) moved to the bottom of the API tab.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Support local model paths on the GGUF embedder and normalize default saves
Found by simulation testing of the embedding model setting:
Local paths saved as the embedding model now work on the llama-server
GGUF backend (the default backend on macOS and CPU). A path to a .gguf
file is used directly and a directory is scanned for a variant-matching
non-mmproj .gguf, with a clear error when none exists. Previously a
local path was sent to the HF hub API and failed with a repo lookup
error.
Saving the default model explicitly no longer stores an override, so
is_custom stays false and the UI does not show a reset button for the
default value.
* Address review: stale-vector handling, GGUF derivation, save-time guards
Review follow-ups, each verified by new tests:
Re-uploading a document after an embedding model change now re-indexes
instead of deduping by content hash. Documents record the embedder that
produced their vectors (lazy embedding_model column, NULL legacy rows
keep deduping) and a mismatch replaces the old document.
A vector width change no longer bricks the dense index. ensure_vec
drops and recreates chunks_vec when the dim changes (old vectors are in
a foreign space and only block inserts) and search_dense returns empty
on a width mismatch instead of surfacing a vec0 error, so lexical
search keeps working until documents are re-uploaded.
Saving a local sentence-transformers folder with no .gguf now returns
409 with a clear message when the install embeds via llama-server,
instead of failing at first index. force still saves.
A custom RAG_EMBEDDING_MODEL env without RAG_EMBED_GGUF_REPO now
derives the -GGUF companion repo instead of silently keeping the bge
GGUF on CPU and macOS installs.
The resolved GGUF path is tagged with the repo captured at entry, so a
setting change during a download cannot mark the old model as current.
GGUF repo detection matches gguf as a whole name segment rather than a
substring, hf_token is trimmed before verification, and the settings
combobox drops a redundant state mirror of its controlled value.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shrink embedding model font to 11px in the input and dropdown
The combobox wrapper applies className to the outer input group, so the
size utility must target the inner input element; the previous text-xs
never reached it and the field rendered at the browser default.
* Show curated unsloth embedding models when the search field is empty
The empty-query listing was the global top-downloads page, which holds
no unsloth mirrors for the unsloth-first float to reorder, so the
dropdown opened on third-party models. Match the model picker: curated
unsloth listing when empty, whole-Hub search once a query is typed.
* Address review: settings resilience and index consistency
Keep the last known embedding model on settings store errors, remove the
re-entrant dim lock in the llama-server backend, accept local GGUF saves
and verify GGUF availability for HF repos on that backend, match local
path embedders exactly in model list filters, drop same-width stale
vectors from dense search, pin the embedder per ingestion job, and only
replace completed documents after the re-index succeeds.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Consolidate the GGUF repo derivation tests
* Trim to a single core embedding-model test
* Address review: GGUF repo saves and cache race
Accept a GGUF-named HF repo on the llama-server backend by verifying GGUF
availability instead of the sentence-transformers metadata gate, and guard
the settings cache with a generation counter so a read overlapping a save
cannot repopulate it with the pre-save value.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio RAG: fix RTL/Indic PDF corruption and dropped DOCX tables
The RAG parser prefers pymupdf4llm.to_markdown for PDFs, but that rebuilds text from
positioned glyphs and mangles complex-shaping scripts (RTL Arabic/Hebrew come back as
shaped Presentation Forms, Indic matras drop to U+FFFD) and can silently drop most of a
heavy-RTL page. _pdf now compares the Markdown against PyMuPDF's logical-order
get_text() per page and falls back to it when the Markdown looks corrupted (shaped
Presentation Forms or U+FFFD above a small floor/ratio) or holds far fewer letters than
the raw layer. Latin PDFs are unaffected and keep their Markdown tables/headings.
_docx walked document.paragraphs, which excludes table cells, so DOCX tables were
dropped entirely. It now walks body content in document order via iter_inner_content,
emitting each table row as pipe-joined cells (deduped across merged cells); the preview
locator already anchors on pipes.
Adds parser tests for the corruption and incompleteness fallbacks and for DOCX table
extraction. These mirror the chat document-extractor guard raised in the unslothai/
unsloth#5351 review; the RAG parser is a separate module and needed its own fix.
* RAG DOCX: keep empty table cells and collapse in-cell newlines
Skipping empty cells shifted later cells left and broke column alignment across rows;
a cell with internal paragraphs (newlines) also broke the pipe-joined row. Keep every
cell (dropping the row only when all are empty) and normalize each cell with
" ".join(split()) so multi-paragraph cells stay on one row. Adds a test for both.
* RAG DOCX: dedup merged table cells on the <w:tc> element directly
Store the shared <w:tc> lxml element in the seen set instead of its id(); it is
hashable and compares by the underlying node, so it dedups spanned/merged cells the
same way without relying on id(). Adds a merged-cell test.
* RAG DOCX: align merged cells, pad skipped grid columns, flatten nested tables
* RAG DOCX: walk cells in document order so nested tables keep in-cell position
* RAG DOCX: dedup vertically merged cells so a spanning label is indexed once
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Add a shared fits-on-device filter to the model selects
The chat model selector gains an Only show models that fit on this
device tick under its filter row, and the Hub page gains a matching
Fits device pill next to the sort menu. Both read one persisted
preference (unsloth_models_fit_on_device_only), so toggling either
applies to both.
The filter reuses the Recommended sort's existing fit math, extracted
into hfModelFitsDevice: size from safetensors metadata, GGUF param
count, or the repo name, against the 0.7 GPU + 0.7 RAM budget, with
unsizable models hidden. In the chat selector it extends the fit
filtering to the Trending and Recent sorts and to search results;
downloaded models stay visible regardless. An unknown device budget
keeps everything. The preference is cleared by Reset all local
preferences like the other picker toggles.
* Move the device-fit toggle into the sort dropdowns
* Tighten sort menu footer spacing and shorten the label
* Align the footer checkbox with the option text
* Make the footer checkbox circular with a smaller tick
* Clear menu highlight when the pointer leaves the options
* Address review: fit filter coverage and sizing
Exempt on-disk models from the Hub fit filter, apply it to the feed
trending rows and curated search results, size safetensors and MLX rows
by the quantized load estimate instead of checkpoint bytes, and replace
the native title hint with the app Tooltip.
* Make the whole device-fit row toggle the filter
* Fix gpt-oss offload_embedding and generate() logits_to_keep on fused models
offload_embedding=True moved embed_tokens to CPU but left the input/output device-shuffling forward hooks commented out ('[TODO] Doesn't seem to work!'), so an eager forward/generate with CUDA input_ids hit the CPU embedding and raised a device-mismatch RuntimeError. Re-implement them in a testable helper _install_offload_embedding_hooks that saves the origin device on the module (the pre-hook returns a new tensor, so a device stashed on the original input is lost) and runs the lookup on the embedding weight's CURRENT device. Reading the weight device at call time (not a hard-coded cpu) also handles a non-quantized (bf16) embedding that a later model.to(...) pulls back onto the GPU, which the hard-coded version broke in the opposite direction.
unsloth_base_fast_generate injected logits_to_keep/num_logits_to_keep whenever an inner submodule forward accepted it, but transformers validates generate kwargs against the top-level prepare_inputs_for_generation (plus forward when it takes kwargs). On fused/PEFT-wrapped gpt-oss this raised 'model_kwargs are not used by the model: [logits_to_keep]'. Only inject when the top level would accept it, mirroring transformers _validate_model_kwargs. Behavior is unchanged for every model that works today.
Adds tests/test_offload_embedding_hooks.py and tests/test_generate_kwarg_gate.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload hooks: store origin device on the tensor, not the shared module
The pre-hook stashed the input device on embed_tokens itself, which races when
concurrent forwards share the module (serving). Ride it on the moved tensor and
read it from the post-hook args instead: stateless and thread-safe.
* Also strip mm_token_type_ids that generate() rejects (Qwen3-VL vision GRPO)
The vision processor (Transformers 5.x path) emits mm_token_type_ids, which
Qwen3-VL's generate() then rejects in _validate_model_kwargs on transformers
4.x, so vision GRPO fails at the first rollout:
ValueError: The following `model_kwargs` are not used by the model:
['mm_token_type_ids']
Unlike logits_to_keep this is an incoming kwarg rather than one we inject, so
drop it in unsloth_base_fast_generate when the top level generate does not
accept it, reusing the same _unsloth_generate_accepts_kwarg gate. Extends the
GPU-free gate test with the accept/reject mm_token_type_ids cases (7/7 pass).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim mm_token_type_ids comment
* Trim comments in gpt-oss offload/logits fix (comment-only)
* gpt-oss offload: return embedding output to the decoder device, not the input's
When offload_embedding moves the embedding to CPU, model.device can become CPU and
inputs then arrive on CPU, so returning the output to the input device left it on CPU
and the CUDA decoder hit a device mismatch. Capture the decoder device before offload
and always return there. This also drops the per-request tensor state (stateless, so
concurrent forwards stay correct).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: refuse offload_embedding for tied word embeddings
Tied models share embed_tokens.weight with lm_head, so offloading the weight
to CPU strands the output projection there (device mismatch at generate) and
saves no VRAM since lm_head still needs it on GPU. Detect the shared weight via
get_output_embeddings and raise NotImplementedError instead of loading into a
crash. Untied models (gpt-oss, Llama-3.1-8B) offload unchanged.
Adds tests/test_offload_tied_guard.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: skip embedding offload on fast_inference (vLLM)
vLLM manages its own weights, so offload_embedding cannot apply on the
fast_inference path (previously it was silently ignored). Disable it with a
notice, mirroring the WSL and Windows skips.
* Trim offload embedding comments (comment-only)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: track decoder device live so it survives model.to()
The post-hook returned the embedding output to a device captured at load time.
If a model is loaded on CPU then moved with model.to(cuda), that device is
stale and the output lands on the wrong device. Read the decoder device live
from the (untied) output embeddings, keeping the captured device as a fallback.
Adds a stale-fallback regression test.
* Make generate-kwarg-gate cases pytest-collectable
Cases lived in run(), which pytest does not collect, so CI never exercised the
gate. Expose them as test_generate_kwarg_gate; still runnable via __main__.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: skip a meta (disk-offloaded) lm_head as the return device
A device_map that disk-offloads an untied lm_head leaves its weight on the meta
device until that module's own hook runs, so reading it as the decoder device
would move real hidden states to meta. Skip meta (and a missing weight) and fall
back to the captured device. Adds a regression test.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: opt-in OpenAI /v1 model auto-switch and idle keep-warm
The OpenAI-compatible endpoints serve whichever GGUF is loaded and ignore the
request model field, so an OpenAI client that changes model never reloads. Add
an opt-in setting that, when a /v1 request names a downloaded local GGUF
different from the loaded one, loads it before serving by reusing the existing
/load path (its dedup, tensor fallback, and threading apply). Unknown names
still serve the loaded model, so drop-in compatibility is preserved and no
remote download is triggered.
Also add an optional idle auto-unload (TTL keep-warm): a pure-ASGI middleware
tracks in-flight inference requests so a stream is never unloaded mid-response,
and a lifespan loop unloads the model after the configured idle seconds. Both
settings default off and live in the app_settings store, exposed via
GET/PUT /api/settings/openai-auto-switch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: variant-aware auto-switch, /v1/responses coverage, keep-warm load stamp
Follow-ups from review of the opt-in OpenAI auto-switch path:
1. Variant-aware dedup. _maybe_auto_switch_model compared only the repo id, so
requesting another quant of the loaded repo (e.g. Q4_K_M loaded, Q8_0 asked)
was served by the old quant. Compare hf_variant too, matching /load dedup.
2. Streaming /v1/responses now calls the auto-switch hook. It went straight into
_responses_stream and only checked is_loaded, so stream=True could serve the
old model or 400. Non-streaming already routed through chat completions; the
hook is idempotent once loaded.
3. resolve_local_gguf tries an exact id match before splitting a trailing
:VARIANT, so local ids that contain a colon (e.g. a Windows path) resolve
instead of being cut at the drive letter.
4. Idle keep-warm stamps activity on a load/swap transition. _last_active was
only refreshed by inference requests, so a model loaded after the server sat
idle past the TTL could be unloaded before its first request.
Tests cover each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make the /v1/responses auto-switch test order-independent
The new streaming-responses test passed in isolation but failed under the CI's
randomized collection order with "object has no attribute 'state'": it passed a
bare object() as the request and stubbed only one dispatcher, so an ordering
where the real dispatcher ran hit request.state. Give the request a state and
stub both dispatchers; the test still asserts the hook fires before dispatch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: assert /v1/responses auto-switch wiring on source, not at runtime
The behavioral version executed openai_responses and relied on stubbing its
callees, which a randomized collection order in CI could defeat (the real
dispatcher ran and hit request attributes). Assert on the function source that
the hook precedes both dispatchers instead; the hook's runtime behavior is
already covered by the direct _maybe_auto_switch_model tests.
* Studio: auto-switch on /v1/embeddings, GGUF-only targets, idle-unload race gate
Second-pass review follow-ups on the opt-in auto-switch path:
1. /v1/embeddings now calls the auto-switch hook before the loaded-state check,
matching the other model-bearing OpenAI endpoints (the keep-warm middleware
already treats embeddings as inference).
2. The resolver index is now GGUF-only. The local-model scanners also surface
Transformers/safetensors repos; without a filter, auto-switch could unload
the GGUF and route a request into the non-GGUF loader. _has_local_gguf checks
a direct .gguf, a models-dir folder, and the HF-cache snapshots layout.
3. Idle keep-warm now holds an asyncio gate across the idle check and the
unload, and a request bumps inflight under the same gate, so the loop can no
longer unload in the window between "looks idle" and the kill.
Tests cover each. Broader local-model source parity (LM Studio, Ollama, legacy
caches, custom scan folders) is a follow-up; missing one of those today just
falls through to the loaded model.
* Studio: variant-aware local resolver, count_tokens + audio auto-switch coverage
Third-pass review follow-ups on the opt-in auto-switch path:
1. The resolver is now variant-aware via list_local_gguf_variants. It indexes
only the quants actually on disk, recursing snapshots and quant subdirs such
as the nested per-quant folders, so a requested repo:VARIANT resolves only
when that quant is local and a bare repo resolves to a concrete local quant.
This fixes two gaps: the previous shallow glob rejected nested-variant GGUF
repos, and a request for an uncached quant could send /load down the remote
download path, breaking the local-only contract.
2. /v1/messages/count_tokens now auto-switches like its sibling /v1/messages, so
a count uses the requested model's tokenizer.
3. /api/inference/audio/generate (direct GGUF TTS) is now tracked as in-flight
inference, so the idle loop cannot unload the model mid-generation.
Tests cover each. Two reviewer items are left as follow-ups: indexing the
remaining local sources (LM Studio, Ollama, legacy/default caches, custom scan
folders), which fails safe today by falling through to the loaded model; and
fully serializing concurrent different-model requests, an inherent limit of the
single-slot llama backend that the opt-in feature is not designed around.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make local GGUF resolver fail-safe so a bad model name cannot 500
The auto-switch hook calls resolve_local_gguf without its own guard, and
/v1/completions and /v1/embeddings pass body.get("model") through unchanged.
A non-string model (e.g. {"model": 123}) or any internal scan failure would
then raise out of the resolver and turn a request that would otherwise be
served by the loaded model into a 500, breaking the drop-in compatibility the
feature is built on.
Guard the resolver at its boundary: reject non-string input up front and wrap
the lookup so any failure returns None (fall through to the loaded model).
Add regression tests for both paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: per-model launch flags for auto-switched GGUF models
* Studio: list switch-eligible GGUFs in /v1/models when auto-switch is on
* Studio: settings UI for OpenAI model auto-switch and idle auto-unload
* Studio: show save error over the disabled-idle hint in auto-switch settings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address gemini review (case-insensitive /v1/models retrieve, idle-input empty guard)
* Studio: address codex review (deterministic override args, exclude probe/embedding models from discovery)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep-warm count_tokens, gate idle on auto-switch, drop hidden models
Three hardening fixes to the opt-in auto-switch path surfaced while reviewing
the work that builds on it:
1. count_tokens keep-warm. /v1/messages/count_tokens counts via the loaded
tokenizer and already auto-switches, but the keep-warm middleware did not
track it, so idle auto-unload could free the model mid-count. It is now a
tracked in-flight path.
2. "Off means unchanged" for idle unload. get_auto_unload_idle_seconds now
reports 0 while auto-switch is disabled. Idle unload only makes sense with
auto-switch on (an unloaded model returns only via the next request's swap),
so a stray TTL can no longer trigger a destructive unload while the feature
is off, keeping the disabled state identical to pre-feature behavior.
3. Hidden models are not switch targets. The resolver index now skips what
Studio hides from its own pickers (the llama.cpp validation probe, RAG
embedding weights) via _is_hidden_model, so they can never be auto-switched
to by name.
Tests added for each.
* Studio: bare-id reuse, responses validation order, in-flight tracking
Review follow-ups after folding in the per-model overrides and discovery work:
1. A bare model id (no :VARIANT) is now satisfied by any loaded quant of that
repo. Previously a bare name resolved to the largest local quant, so it could
force a slow reload when a different quant of the same repo was already
serving. An explicit repo:VARIANT request still honors the quant.
2. /v1/responses now runs the auto-switch hook after the empty-input validation
so a request that 400s can no longer trigger a multi-minute model load before
being rejected. The hook still precedes both dispatchers, so streaming
requests switch.
3. The keep-warm middleware now tracks in-flight requests whenever auto-switch
is enabled rather than only when the idle TTL is already positive, so a stream
that starts with the TTL at 0 is still protected if idle-unload is enabled
mid-stream. Off still passes straight through.
Tests added for each.
* Studio: tighten auto-switch code comments
Comment/docstring-only pass over the OpenAI auto-switch feature: collapse
multi-line blocks, drop a comment that restated the gate it sits next to, and
trim verbose docstrings on internal helpers while keeping the load-bearing
rationale (concurrency, API behavior, drop-in compat, gotchas). No logic
change: verified comment-only with the AST/printer signature check.
* Studio: bind auto-switch locks per running loop
Review follow-up. The auto-switch swap lock and the keep-warm unload gate were
module-level asyncio.Lock objects. That is safe under the single uvicorn loop
and on Python 3.10+ (the Lock resolves the running loop lazily on acquire), but
a module-level Lock binds to one loop on pre-3.10, which can raise a loop
mismatch in multi-loop runners. Resolve each lock through a per-loop accessor
backed by a WeakKeyDictionary so every running loop gets its own Lock and stale
loops are collected. No behavior change under the server's single loop.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch re-review fixes (body codes, coverage, swap, alias, tracking)
Follow-ups from a second review pass over the opt-in OpenAI auto-switch feature:
1. OFF-state status codes: /v1/completions and /v1/embeddings moved the body
read ahead of the loaded-state check, so a malformed/empty body with no model
loaded returned 500 instead of the prior 503. A shared helper reads the body
defensively (an unparseable/non-dict body yields no model), and the handler
re-reads after the 503 gate to surface the original parse error exactly as
before. OFF behavior is unchanged.
2. Local-model coverage: the resolver index only scanned ./models and the active
HF cache, while the model picker also lists the legacy/default HF caches, LM
Studio dirs, and user scan folders. A request for one of those named models
silently served the loaded model instead. _build_index now scans the same
roots (Ollama's symlink-creating scanner is skipped on the request path), and
resolution is offloaded with asyncio.to_thread so the wider scan never blocks
the event loop.
3. Swap vs in-flight stream: a cross-model swap killed the llama-server while
another client was still streaming from it. The hook now tracks how many
requests are streaming on the loaded model (in-flight minus those still inside
the hook) and returns 409 instead of swapping while one is active. Concurrent
same-model requests never reach this path, so they are unaffected.
4. Idle-unload + alias: after idle-unload freed the model, an unknown/alias name
resolved to nothing and 503'd, though it served the active model before the
TTL. Idle-unload now remembers the freed id and an alias request reloads it
(only an already-local model, so no remote download), cleared once a model is
loaded again.
5. In-flight tracking: the keep-warm middleware tracked in-flight only while the
feature was on, so a stream started while off could be unloaded if idle-unload
was enabled mid-stream. It now tracks on every inference path; counting is
cheap and invisible to clients.
Tests added for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove stray async_task_outputs files committed by mistake
* Studio: auto-switch review round 3 (revert swap guard, hardening)
Addressing a third review pass:
- Revert the cross-model swap guard. It counted keep-warm in-flight (which
includes external-provider calls that never touch the local model) and so
could 409 a local swap spuriously, and it still left a same-model request able
to start streaming on the model a concurrent swap was unloading. A correct fix
needs a request-lifetime reader/writer barrier; a partial guard was worse than
the honest single-slot behavior, so concurrent different-model use is back to
being serialized (documented), like llama-swap's single slot.
- Non-string request model (e.g. {"model": 123} on a raw-body endpoint) is now
treated as absent, so it falls through instead of raising in the membership
checks once an idle-unload stash exists.
- Idle-unload now stashes and replays the freed quant: an alias reload restores
the exact (id, variant) that was freed rather than the largest local quant.
- Anthropic /v1/messages validates max_tokens before the auto-switch hook, so a
request that 400s never triggers a model load.
- Keep-warm tracks a pending count for requests waiting on the unload gate, so
the idle loop cannot unload the model out from under a request that is blocked
on the gate but not yet counted as in-flight.
- The idle-unload task is awaited after cancel on shutdown to avoid pending-task
warnings.
- The resolver's HF cache scan is None-safe and logs at debug instead of letting
a bad root abort the whole index build.
- upsert_app_setting_map_entry rolls back explicitly on error.
Tests updated/added for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep saved idle-unload seconds when auto-switch is toggled off
* Studio: auto-switch hardening (thread-safe lock maps, body validation)
Defensive fixes from review:
- Guard the per-loop WeakKeyDictionary get-or-create for both the unload gate and
the auto-switch lock with a threading lock, since WeakKeyDictionary mutation is
not thread-safe when two event loops run on different threads.
- Build the resolver index under the cache lock so concurrent callers with an
expired cache don't all run the multi-dir scan at once.
- /v1/completions and /v1/embeddings return a clean 400 for a valid JSON body
that is not an object (e.g. a list), instead of a 500 from body.get(...).
- The keep-warm middleware only tracks POST requests (inference is always POST),
so CORS preflight (OPTIONS) is not counted, and tolerates a None path.
Tests added for the list-body 400 and the non-POST skip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch review round 4 (local-path load, swap guard, idle fixes)
From a 10-reviewer pass:
- HF-cache entries now load by a concrete local path, not the bare repo id. The
resolver records a load_path (the snapshot dir for a models--* cache repo, the
file/dir otherwise) so /load takes the local branch and can never trigger a
download to satisfy a partial cache. The advertised loader_id (repo id) is kept
as the launch-override key. resolve_local_gguf now returns
(load_path, variant, loader_id).
- Re-add a single-slot swap guard: a cross-model swap returns 409 model_switch_busy
while another inference request is active rather than killing its stream (the
caller is excluded from the count), and holds the keep-warm gate across the load
so no new inference starts mid-swap. Concurrent same-model requests never reach
this path. A residual spurious 409 is possible while a concurrent or external-
provider request is active; that is the documented single-slot tradeoff.
- Idle keep-warm tracks (model_identifier, hf_variant): reloading the same repo at
a different quant counts as a fresh model, so it is not unloaded before one TTL.
- Track Studio's own /api/inference/generate/stream so the idle loop can't unload
the model mid-stream on that route.
- A successful manual /load clears the idle-unload reload stash synchronously, not
only on the next idle poll.
Also merged origin/main (the branch had fallen behind, which would have reverted
unrelated files on merge). Tests added/updated for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch review round 5 (concurrency, identity, load gate)
From a 10-reviewer pass (9 request-changes, 1 approve):
- Concurrent same-target requests load once instead of each returning 409. The
count-based busy guard could not tell "another request wants the same model"
(safe, load once) from "another request is using the loaded model" (refuse).
Track in-flight auto-switch requests per (target, variant) and subtract
same-target waiters from the busy count; a cross-model swap still 409s while a
genuinely different request is active.
- Fix the identity confusion introduced when round 4 began loading by concrete
local path: the backend identifier became a filesystem path. Record the
advertised repo id on the backend after an auto-switch load and use it so
(a) a model loaded manually by repo id is recognized as already serving
(no spurious reswap/409), (b) /v1/models reports the repo id, never a host
path or a duplicate, and (c) the idle-unload stash keeps the override keyed by
the repo id, so an alias reload after TTL keeps the user's saved launch flags.
- Gate the manual /load route with the keep-warm lifecycle gate so idle
auto-unload can't unload a model mid-load. load_model now wraps _load_model_impl
in the gate; auto-switch calls _load_model_impl directly since it already holds
the gate.
- Restore default-off parity on Anthropic /v1/messages: an unloaded backend with
auto-switch disabled 503s before the max_tokens 400 check, as it did pre-feature.
When the feature is on, request-shape validation still runs before any load.
Tests added for each; full backend suite diff vs baseline is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch review round 6 (concurrency ordering, leaks, unload gate)
From a second 10-reviewer pass (8 request-changes, 2 approve):
- Same-target concurrency: register a waiter by the raw requested model before
the (slow) resolve, and exclude pending requests from the swap busy count. The
middleware counts a concurrent same-model request as in-flight before it
resolves and joins the resolved-target waiter map, so the prior fix could still
409 it. The guard now subtracts max(same resolved-target, same raw-request)
waiters and ignores pending (a pending request is blocked in the middleware,
not generating, so a swap can't interrupt it).
- External-provider requests no longer block a local swap. The keep-warm
middleware counts every inference-path POST, but external-provider chat returns
before the auto-switch hook and never touches the local GGUF. The chat handler
now untracks itself before proxying, so its in-flight stream can't trip
model_switch_busy on a concurrent local auto-switch. The middleware skips its
own end-decrement for an untracked request.
- Manual /unload is gated like load and idle-unload: it holds the lifecycle gate
and returns 409 rather than tearing down llama-server while an inference request
is in flight.
- Response model id no longer leaks the load path. /v1/models already advertised
the repo id; chat, completions, embeddings, Anthropic messages, and audio
response bodies now use the same _llama_public_model_id helper instead of the
concrete on-disk model_identifier.
- Chat completions validates the non-system-message requirement before the
auto-switch hook (as /responses and /messages already do), so an invalid
request can't swap the resident model before returning 400.
Tests added for each; full backend suite diff vs baseline is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch review round 7 (teardown policy, Unsloth-active swap, training)
From a third 10-reviewer pass (9 request-changes, 1 approve), all on the same
asymmetric-teardown theme. Resolved per the intended policy that only automatic
paths defer to an active stream; deliberate user actions stay interrupting:
- Revert the manual /unload in-flight guard added last round. A manual /load or
/unload is a deliberate action and tears down immediately, as before; only the
automatic idle-unload loop and auto-switch defer to an active request. This
removes the asymmetry the reviewers flagged (manual /load, the /unload Unsloth
branch, and the opposite-backend swaps inside _load_model_impl) by not
extending the guard to deliberate paths, rather than spreading it.
- Auto-switch now refuses a swap whenever another inference request is in flight,
not only when a GGUF is already loaded. _load_model_impl also unloads an active
Unsloth/transformers backend before loading a GGUF, so the busy guard must cover
that case too; otherwise an Unsloth stream could be killed by an auto-switch.
- Refuse API-initiated training while inference is active. When Studio is driven
as an inference API (sk-unsloth key auth), POST /api/training/start returns 409
if a request is in flight, since training frees VRAM by unloading the chat
model and would kill the stream. The Studio UI (session auth) still starts
training and coexists/frees VRAM as before. A mixed UI+API session is not yet
special-cased. Adds auth.authentication.authenticated_via_api_key.
Tests added/updated for each; full backend suite diff vs baseline is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add UNSLOTH_MODEL_IDLE_TTL env override for idle-unload
Borrowed from PR 6517: a startup env var that sets the idle-unload TTL without
the settings UI. Unlike the stored setting (gated on auto-switch), the env value
is a standalone default that enables idle-unload even with auto-switch off, for
headless/container deploys. An explicit UI/API value still overrides it and stays
gated. The settings GET reflects the env default when nothing is stored.
* Studio: auto-switch fixes from review (paths, embeddings input, env idle reload)
- /v1/models advertises a client-facing alias instead of a filesystem path:
the ./models and LM Studio scanners report the on-disk path as the model id,
so the index now prefers model_id/display_name as the advertised/override id
and keeps the concrete path internal as load_path, still resolvable by path.
- /v1/embeddings validates input before auto-switch: a request with a model but
no input now 400s before the hook (like chat/responses/messages), so an
invalid embeddings request cannot unload or swap the resident model.
- Standalone UNSLOTH_MODEL_IDLE_TTL reloads the freed model: the hook now runs
when auto-switch or idle-unload is active, and with auto-switch off it skips
the resolver and only restores the idle-unloaded model, so the first idle
timeout no longer leaves later /v1 requests with nothing loaded.
- Do not resurrect a stale GGUF over an active Unsloth model: the reload-stash
path bails when a non-GGUF backend is loaded, so an unknown /v1 name cannot
tear down a live Transformers/Unsloth model.
- Defensive HF cache scan: each cache root's resolve/dedup is wrapped so a
missing or malformed root skips that root rather than aborting the index.
- Single-model retrieve checks the id is a string before lowercasing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix automatic-load asymmetry, audio reload, preview, idle timer
The standalone UNSLOTH_MODEL_IDLE_TTL reload is a second automatic-load
trigger, but several validate-before-switch guards and reload hooks only
checked the auto-switch toggle. Add a shared _automatic_model_load_may_run()
(auto-switch on, or idle TTL > 0) and route every guard through it.
- /v1/completions validates prompt before any automatic load (it was the one
model-bearing route with no pre-check).
- /v1/chat/completions and /v1/embeddings pre-checks gate on the shared
predicate so a standalone idle TTL cannot reload then reject.
- /v1/messages no longer 503s before the reload hook can restore an idle-freed
model when auto-switch is off.
- Raw completions/embeddings with no model field pass a non-empty sentinel so
the idle-stash reload runs, restoring the legacy "omit model, use loaded" path.
- /api/inference/audio/generate gains the reload hook (after message validation)
so an idle-freed audio GGUF is restored.
- Public preview opts out of auto-switch via a request-scope flag, so a caller's
model field cannot swap away from the pinned checkpoint; preview chat streams
are now matched by _is_inference_path so idle-unload cannot kill them.
- Keep-warm no longer stamps activity on request start, and external-provider
untracking decrements without restamping, so periodic external traffic can no
longer keep the local GGUF warm forever.
Merges origin/main (the branch had fallen behind, which also brought in the
preview route the review flagged).
* Studio: surface model auto-switch in the API tab and demo it in examples
The OpenAI auto-switch toggle previously lived only in Settings -> General.
Add the same toggle to the API tab's usage-examples panel (it shares the
settings cache), and make the examples reflect it: when on, the Python
examples append a second call naming a different downloaded GGUF (so the
model field visibly selects which model serves), and the curl examples gain
a one-line note. Reuses the existing settings API client and i18n keys.
* Studio: harden OpenAI auto-switch reload-only path and Anthropic tool validation
- Omitted-model raw-body requests pass a reload-only sentinel so the idle-stash
reload still restores an idle-freed model, but the resolver never matches a
downloaded GGUF literally named "default".
- Reject malformed Anthropic client tools before _maybe_auto_switch_model so an
invalid request can no longer evict the loaded model.
* Studio: extend auto-switch reload-only and tool validation to schema endpoints
- Schema-backed endpoints (chat completions, responses, count_tokens, messages,
audio) defaulted an omitted model to "default" and passed it to the switch
hook, so a downloaded GGUF named "default" could be swapped to. Route the hook
through a helper that switches only on an explicitly set model, else reload-only.
- Propagate the explicit-set status when building the chat request from a
Responses request, so the non-streaming chat re-check stays reload-only too.
- Validate Responses function tools before the switch hook so a malformed tool
returns 400 without evicting the loaded model.
* Studio: serialize auto-switch swaps across event loops with a process-wide gate
The auto-switch lock is a per-event-loop asyncio.Lock, so two /v1 swaps on
different loops in one process could both pass it and race the single model slot
(the backend and _load_model_impl are process-wide). Add a process-wide
threading gate around the swap, acquired off the loop so a cross-loop wait never
blocks it, layered with the existing per-loop lock. Add a cross-loop test that
fails without the gate (two slow loads overlap) and passes with it.
* Studio: make the auto-switch swap gate wait cancellation-safe
_acquire_swap_gate awaited asyncio.to_thread(lock.acquire) when another loop held
the process-wide gate. to_thread cancellation doesn't stop the worker thread, so a
/v1 request cancelled mid-wait (client disconnect during a cross-loop swap) would
have its thread acquire the gate after the fact, while the finally that releases it
never runs -- permanently deadlocking later auto-switch swaps.
Poll a non-blocking acquire off a short asyncio.sleep instead: it still keeps the
wait off the loop and serializes across loops, but a cancel now lands during the
sleep, when the gate is not held, so nothing leaks. Add a test that deadlocks the
to_thread variant (it times out) and passes with the poll.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: validate modality and tool-confirmation before auto-switch
Two more request shapes could load a named GGUF and only then 400, evicting the
resident model:
- An image request naming a different text-only GGUF. The switch hook now takes
require_vision and rejects a swap to a non-vision target before loading it; a
GGUF's vision capability is its companion mmproj, knowable without a load, and
matches the post-load guard. Only the resolver branch is checked, never the
reload-stash restore.
- confirm_tool_calls=true with stream=false and local tools. /v1/chat/completions
now rejects that shape before the hook, mirroring the local tool path's
bypass_permissions exemption and intent signal.
The vision probe threads the ambient HF token to keep the capability-probe
invariant. Reload-only and idle-reload paths are unaffected.
* Studio: extend validate-before-switch and make the lifecycle gate process-wide
- /v1/messages/count_tokens now rejects malformed client tools before the switch
hook, like /messages (shared _validate_anthropic_client_tools helper), so a
count request can't evict the loaded model.
- /v1/chat/completions rejects a malformed tool_choice forcing object (a
{"type":"function","function":{}} with no name) before the switch hook.
- The inference lifecycle gate that blocks new inference during a swap is now
process-wide (a poll-acquired threading lock, cancellation-safe), not a
per-loop asyncio lock, so a request on another event loop can't start inference
while a swap tears the single backend down.
- Usage examples no longer hard-code a switch-demo repo most users lack; the
model is an explicit placeholder the user replaces.
* Studio: extend the auto-switch modality guard to /v1/responses and /v1/messages
The pre-load vision check that guards /v1/chat/completions now also runs on
/v1/responses and /v1/messages, so an image request naming a text-only GGUF is
rejected before the swap and never evicts the resident vision model. Run the
vision capability probe off the event loop. Make the /v1/models retrieve
loaded fast-path case-insensitive, and never advertise a host path from the
resolver. Remove the dead list_switch_eligible_ids helper, superseded by the
/v1/models catalog.
* Studio: filter /v1/models to GGUF, per-loop catalog lock, reject system-only Responses
Address review findings on the auto-switch path:
- /v1/models advertises only GGUF models the API can actually switch to; a
safetensors/LoRA entry would be selectable but never loadable via llama.cpp.
- The /v1/models catalog cache uses a per-loop lock (like the auto-switch path)
so a second event loop awaiting it can't hang in a multi-loop process.
- /v1/responses rejects system/developer-only input before the switch, mirroring
chat, so an invalid request can't evict the resident model.
- _build_index guards each scan source on its own so one bad root drops only
that source; the vision probe logs a real detection failure instead of
swallowing it.
* Studio: list cached GGUFs in /v1/models by inspecting files, not model_format
The HF-cache scanner leaves model_format unset for GGUF snapshots, so the
previous model_format == "gguf" filter dropped every downloaded HF-cache GGUF
from /v1/models and the retrieve fallback. Decide GGUF-ness from the on-disk
files via the resolver (info_has_local_gguf) instead, run off the event loop, so
the catalog advertises exactly what /v1 can serve.
* Studio: fix /v1/messages/count_tokens route binding plus auto-switch review fixes
The @router.post decorator for /messages/count_tokens had been separated from
anthropic_count_tokens by the _validate_anthropic_client_tools helper, so the
route bound to the validator and dropped its auth dependency. Move the decorator
back onto the handler. Add route-binding tests asserting each /v1 endpoint maps
to its handler with the auth dependency, so a decorator/handler split is caught
at the route level (the direct-call tests missed it).
Also from review:
- update_openai_auto_switch writes both settings keys in one transaction so a PUT
can't leave one updated and the other stale (drop the now-unused single setters).
- max_seq_length override rejects 0 at the boundary (ge=1) instead of accepting
then silently dropping it.
- Document that embeddings auto-switch is best-effort: GGUF pooling has no cheap
pre-load probe like vision's mmproj, so a guard would false-reject GGUF embedders.
- Add a positive idle-unload test (loop frees the model and stashes it for reload).
* Studio: validate Responses tool_choice + Anthropic mixed tools before switch, filter Ollama from catalog
More auto-switch review findings:
- /v1/responses rejects a forcing-function tool_choice with no name before the
switch, mirroring chat, so a malformed request can't evict the resident model.
- /v1/messages rejects mixing Anthropic server tools with custom client tools
before the switch (the check depends only on the payload, so it moves up cleanly).
- /v1/models no longer advertises Ollama-link models: info_has_local_gguf excludes
.studio_links / ollama_links entries, which the resolver skips and can't switch
to, so an advertised id never silently falls through.
* Studio: guard chat audio input before switch; surface env-backed idle unload in settings UI
A chat request carrying audio_base64 rides the same companion mmproj
projector as a vision request, so a text-only target cannot serve it
either. Flag require_vision for audio input as well so the multimodal
probe runs before the switch and a rejected request never evicts the
working model. Generalize the reject message to cover image and audio.
The settings response now reports idle_unload_active (effective TTL > 0)
so the UI can distinguish idle-unload that is active via the
UNSLOTH_MODEL_IDLE_TTL env var from the case where it needs the toggle
enabled.
* Studio: harden auto-switch eviction guards (count_tokens vision, TTS reload-only, mmproj/stash)
Four eviction/correctness fixes on the opt-in /v1 auto-switch path:
- /v1/messages/count_tokens now carries the same require_vision guard as
/messages, so an image count naming a text-only GGUF can't evict a loaded
vision model for a swap that can't serve the request.
- /audio/generate is now reload-only. A local GGUF's audio-input capability
is not a cheap pre-load probe (the companion mmproj signal can't tell an
audio projector from a vision one, and codec TTS ships no projector), so
resolving the client model could load a text/vision-only target and evict
the working audio model before the audio check fails. Only the idle-stash
restore runs here; switching TTS models is an explicit /load.
- The resolver no longer treats a standalone mmproj .gguf as a servable
model. _scan_models_dir's standalone-file pass does not filter mmproj the
way its directory scan does, so /v1/models could advertise a projector and
a switch could load it over the real weights.
- A non-GGUF (Transformers/Unsloth) load and a deliberate /unload now clear
the idle reload stash, so a manual load/unload is never superseded by a
stale idle-freed GGUF that the next /v1 request resurrects.
* Studio: report advertised repo id consistently after an auto-switch
Two model-id reporting fixes so an auto-switched cached HF GGUF is named by
its repo id everywhere, not its snapshot path:
- Streamed /v1/responses envelopes now derive the model id from
_llama_public_model_id (which prefers _openai_advertised_id) instead of the
raw model_identifier. After an auto-switch the identifier is the snapshot
path while the repo id lives in _openai_advertised_id, so the stream used to
report a snapshot basename while /v1/models, chat completions, and
non-streaming Responses all reported the repo id.
- When an advertised alias already resolves to the loaded model (a model
loaded by local path, requested by its repo or LM Studio id), the
already-serving early return now records the alias as the advertised id, so
/v1/models and responses report the alias and mark it loaded instead of the
path-derived basename. Resolver branch only; safe lock-free because an
in-flight request blocks any concurrent swap via the single-slot busy guard.
* Studio: validate request shapes before auto-switch (prompt/input/audio/mcp confirm)
Four more validate-before-switch guards so a deterministic client error never
evicts the resident model on the opt-in /v1 auto-switch path:
- /v1/completions rejects an object/number prompt (only a string or array is
valid) before the switch, instead of loading the named GGUF and letting
llama-server reject the shape afterward.
- /v1/embeddings rejects an object/number input the same way.
- Chat rejects an oversized audio_base64 upload (413) before the switch. The
size cap is a cheap, target-independent length check; the decode itself
stays post-switch to avoid decoding a valid upload twice.
- The chat confirm-without-stream pre-switch guard now mirrors the tool loop's
actual enablement: _effective_enable_tools (honoring a CLI --enable-tools
policy) and mcp_enabled (which opens the tool loop on its own but defers to a
CLI --disable-tools policy). Previously a confirm+no-stream request with only
mcp_enabled slipped past and 400'd after the swap.
* Studio: fix model-id retrieval, streaming n>1, resolver cache TTL, keep-warm auth
Four fixes from review:
- GET /v1/models/{id} legacy raw-path fallback now maps the raw identifier to
the same public id its /v1/models entry uses. After an auto-switch load the
identifier is the snapshot path while the entry is keyed by the advertised
repo id, so a client that cached the old absolute path no longer 404s on a
model that is in fact loaded.
- stream=true with n>1 is now rejected before the switch. Only the
non-streaming GGUF path returns multiple choices, so streaming n>1 is invalid
on every local serving path; both fields are known pre-switch, so it must not
load model B only to 400 and evict model A. Non-streaming n>1 stays
post-switch where the serving path decides.
- The resolver index cache is stamped after _build_index, not with the pre-scan
timestamp. On installs with enough local models for the multi-root scan to
exceed the 5s TTL, the cache was stored already expired and every request
rebuilt it.
- The keep-warm middleware no longer stamps model activity for 401/403
responses. It runs before FastAPI auth, so unauthenticated probes used to
refresh the idle timer without touching llama.cpp; they now decrement the
in-flight count without keeping the model warm.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
Extracted and narrowed from unslothai/unsloth#6543 by @TheJagStudio.
This keeps the startup/banner and text file encoding hardening separate from the already-merged Python code-exec UTF-8 fix in #6548.
Co-authored-by: Jagrat Patel <81472856+TheJagStudio@users.noreply.github.com>
* Pin llm-compressor auto-install to a vetted version range
install_llm_compressor() auto-installs llm-compressor on first use of an FP8/FP4
compressed export when it is not already present. The install command used the bare
package name, so pip resolved to whatever the configured index served; a compromised,
dependency-confused, or inflated-version ("999.0.0") release could then run under the
Unsloth process at install and import time.
Bound the automatic install to a vetted range
(_LLM_COMPRESSOR_SPEC = "llmcompressor>=0.8.0,<0.13"), which the oneshot /
QuantizationModifier API this uses supports, so pip can no longer jump to an arbitrary
future or inflated version. An already-installed newer llm-compressor is still used
as-is (the import short-circuits), so this only constrains the auto-install, never a
user's own install.
Add UNSLOTH_DISABLE_LLM_COMPRESSOR_AUTOINSTALL=1 to forbid the automatic install
entirely and require a manual, vetted install, for locked-down or air-gapped
environments. Update the manual-install hints to the pinned spec.
Add tests/saving/test_llm_compressor_install_pin.py: static (ast) guards that the spec
stays a bounded pin, that the install command never passes an unpinned llmcompressor
literal, and that the opt-out env gate is evaluated before any install runs.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Loosen llm-compressor auto-install ceiling to <1.0 so new models still export
The earlier <0.13 ceiling was too tight: brand-new architectures (for example
Qwen3_5ForConditionalGeneration / qwen3_5, gemma-4 MoE) can require a newer
llm-compressor, and _unsloth_save_compressed_tensors already fails with
"requires a newer llm-compressor" when a scheme is unavailable. Capping the
auto-install at 0.12 would block getting that newer release and break compressed
export for new models.
Widen to llmcompressor>=0.8.0,<1.0. pip still auto-installs the latest 0.x
(where new-architecture support lands), while the <1.0 ceiling continues to block
a jump to an inflated-version ("999.0.0") or 1.0+ dependency-confusion release.
An already-installed newer llm-compressor is still used as-is (the import
short-circuits), and UNSLOTH_DISABLE_LLM_COMPRESSOR_AUTOINSTALL still forbids the
automatic install entirely for locked-down environments.
* Lower llm-compressor floor to 0.6.0 so supported old torch still resolves
The >=0.8.0 floor conflicts with the torch this install pins in its constraints
file. Unsloth supports torch>=2.4, but llm-compressor 0.7.0+ require torch>=2.7
(0.10+ need >=2.9, 0.12+ need >=2.10). On a supported torch 2.4-2.6 box pip then
has no candidate in [0.8.0, 1.0) and FP8/FP4 export fails before quantization.
Lower the floor to 0.6.0 (its metadata only needs torch>=1.7), which never
conflicts with any supported torch. pip still prefers the newest compatible
release, so modern torch continues to get the latest 0.x (0.12.0). The <1.0
ceiling that blocks an inflated-version supply-chain jump is unchanged.
Add a regression test asserting the floor stays <= 0.6.0.
* Cap llm-compressor auto-install ceiling to a vetted minor (<0.13)
A bare <1.0 ceiling still admits any 0.x, so an inflated "0.999.0" served by a
compromised or misconfigured index would win pip's highest-version selection --
the same dependency-confusion this pin is meant to block. Cap the ceiling to the
current vetted minor (<0.13) so that jump is blocked; bump it deliberately, after
vetting, when a newer llm-compressor is needed (e.g. for a brand-new architecture
scheme). Current new models are unaffected: 0.12.0 is < 0.13 and supports them.
The 0.6.0 floor (torch>=1.7 compatible) is unchanged, so resolution still works
across Unsloth's whole supported torch range (2.4 -> 0.6.0 ... 2.12 -> 0.12.0).
Add a regression test asserting the ceiling admits the current vetted release but
blocks an inflated 0.x and the next major.
* Trim comments in the llm-compressor pin (comment-only, no code change)
* Cap llm-compressor auto-install to the exact vetted patch (<=0.12.0)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* scan_packages: key baseline on matched-code hash
The baseline matched on (package, package-relative file, check), which
excluded the matched code, so a future finding of the same check in the
same file was suppressed regardless of what the code did. A malicious
future version of an already-baselined package could place a payload in
the same file under the same check and pass the enforcing gate.
Key the baseline on a hash of the matched code too. The hash is over the
deduped, sorted set of matched spans with L<NN>: line markers stripped, so
version bumps, line shifts and match reordering stay stable while new or
changed flagged code reopens the finding. Version is left out of the key so
routine dependency bumps do not reopen every entry. The hash is capped and
recomputable from the stored evidence.
Regenerate scan_packages_baseline.json against the current dependency set;
the hf-stack, studio and extras scan shards pass enforcing (no active
CRITICAL or HIGH).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: refresh baseline for newer unsloth-zoo release
A newer unsloth-zoo published after the first regenerate added
tests/test_mlx_save_export_regressions.py, a benign test fixture
(temporary_location="/tmp/ignored") that trips the /tmp dropper check.
Regenerate the hf-stack shard against the current set so the entry is
allowlisted; studio and extras are unchanged.
* scan_packages: harden baseline loading against malformed JSON
Guard against a non-dict top-level baseline and non-dict entries so a
corrupt or hand-edited allowlist warns and fails closed instead of
crashing with AttributeError, and treat an explicit evidence: null as
empty.
* scan_packages: hash the full match set, keep indentation, strip only the marker
Address the evidence-hash review feedback:
- Capture every matching line, not the first three, so a payload appended
after existing matches in a baselined file and check reopens the finding
instead of riding the sample.
- Preserve leading indentation so a flagged line moved out of a guarded block
reads as changed.
- Strip only each span's prefix up to the first L<NN>: marker, so an L<NN>:
inside the matched code is kept and a change to it reopens the finding.
Evidence and its hash are stored in full and stay recomputable from the stored
field. Regenerate the baseline; hf-stack, studio and extras pass enforcing with
no active CRITICAL or HIGH.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: bind baseline evidence to full matched code
Address review feedback on the evidence-hash baseline key:
- Split evidence only on real span delimiters (" | " before an L<NN>:
marker, or a newline), so a bitwise-or or union type in matched code
is no longer split apart into separate spans.
- Record matched lines in full (drop the 160-char per-line cap) and
record every distinct multiline match, so code appended past the cap
or a second cross-line match reopens the finding instead of riding the
first one.
- Give the large-JS-bundle and .pth base64-blob findings a content
digest instead of empty or prefix-only evidence, and record all .pth
import lines, so a changed bundle, blob or import no longer inherits a
baselined empty or truncated key.
- Warn when a loaded baseline has entries without evidence_hash so a
legacy baseline is regenerated rather than silently degraded.
Regenerate scripts/scan_packages_baseline.json against the current dep
set and add regression tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: harden multiline and duplicate evidence handling
Follow-up hardening so the evidence hash tracks the full matched code:
- For DOTALL patterns that match across lines, record every line the match
spans (not just the start line), so a change on a continuation line (the
URL inside a baselined C2 loop, a swapped credential path) reopens the
finding. A pathological greedy span is bounded to its head line plus a
digest of the rest.
- Keep duplicate spans in the canonical evidence so a second identical
matched line in a new code path changes the key instead of deduping away.
- Anchor the evidence prefix to strip only a genuine leading label or
line-number marker, leaving a marker-like "L<NN>:" inside raw .pth code
intact.
- Make the legacy-baseline warning explicit that entries without an
evidence_hash reopen rather than suppress under a coarse key.
Regenerate scripts/scan_packages_baseline.json (same finding set; entries
for same-file repeated checks are now tracked separately) and add tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: bind every combo and large finding to its full content
Close the remaining asymmetric-evidence gaps so a changed payload cannot
ride a reviewed baseline entry:
- Digest a capped multiline span from the code without line markers, so a
pure line shift stays stable while a continuation-line change reopens.
- Give the "Unusually large executable .pth" finding a content digest
instead of keying on byte size and import-line count alone.
- Record both contributing signals for the JS credential+network stealer,
the shell credential+network and persistence-hook combos, and the hidden
network+exec docstring payload, so changing the network/exec side reopens.
- Allow punctuation in an evidence label prefix so a "network+exec:" label
is stripped and line shifts do not change the key.
Regenerate scripts/scan_packages_baseline.json and add tests for each case.
* scan_packages: bind remaining Python combos; key npm baseline on evidence
Python scanner: the openssl+key, anti-analysis, DNS-exfil and base64+exec+blob
combos recorded only one contributing signal, so a changed payload on the other
side could ride a reviewed baseline entry. Each now binds every co-occurring
signal (and the blob is digested, since it can sit on a separate line from the
decode call).
npm scanner: scan_npm_packages.py keyed its allowlist on (package, path,
pattern) only, the same coarse-key bypass the Python scanner just closed. Add an
evidence hash to the key (schema v3, fail-closed on older baselines) and store
full evidence. The committed baseline stays empty by design.
Regenerate scripts/scan_packages_baseline.json and add tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_npm_packages: bind full blob evidence and harden baseline loader
Follow-up on the npm evidence-hash key:
- _evidence now records every match and, when a snippet is truncated for
display, appends a digest of the full match. The obfuscated-blob key was
hashing only the truncated first-match snippet, so a changed payload tail or
an appended blob in the same package/file/pattern could ride a reviewed entry.
- _load_baseline guards that the root is an object, entries is a list, and each
entry is a dict before reading it, so a malformed baseline warns and fails
closed instead of raising AttributeError.
Add tests for a changed blob tail reopening the key and for malformed entries.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: symmetric baseline-loader guards; bind npm outbound host context
- Python _load_baseline now rejects a non-list "entries" with a warning instead
of raising TypeError, matching the npm loader.
- npm cred-surface-host (outbound) records the host with its URL path / fetch
call / host config, so a changed outbound path, headers or body reopens the
key rather than riding the bare host literal.
Add tests for both.
* scan_npm_packages: migrate v2 baselines and bind host-config outbound context
- _load_baseline now migrates schema v2 entries by recomputing the evidence
hash from stored evidence (with a legacy warning), matching the Python
loader, instead of discarding them; only pre-v2 basename schemas are rejected.
- The cred-surface-host (outbound) host-config branch now captures the whole
line (path, headers, body), so a changed outbound payload on the same
hostname line reopens the key instead of riding the bare host snippet.
Add tests for v2 migration and the host-config context binding.
* scan packages: bind PEM key bodies and npm windowed evidence to baseline keys
scan_packages: embedded-key findings now pin the full PEM block (BEGIN..END)
via a content digest, so a key body swapped under the same marker reopens the
finding instead of riding the unchanged BEGIN line. Single-line and DER keys
were already bound by their full matched line; marker-only references with no
END block (validation header lists) are unaffected, so the committed baseline
is unchanged.
scan_npm_packages: _evidence now digests the full containing line whenever the
shown snippet is only a window into it (short match on a long line, or a
truncated payload), so a changed payload tail outside the display window
reopens the key. The npm baseline is empty, so this changes no suppressions.
Adds regression tests for both cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan packages: bind multi-line evidence and every blob to baseline keys
_extract_evidence now extends each single-line match over its bracket
continuations, so a multi-line call binds its argument lines and a changed
URL or body on a continuation line reopens the key. After the per-line pass it
also records cross-line matches the scan cannot otherwise see (a DOTALL regex,
or a multi-line construct appended under a check that already had a one-line
match), so an appended multiline payload reopens instead of riding the key.
_blob_digest hashes every large base64 blob (not just the first) for the
base64+exec finding and the .pth large-blob finding, so an appended or swapped
second encoded payload reopens; single-blob files keep the same digest.
scan_npm_packages _evidence digests the full logical line (the matched line
plus its bracket-continuation lines), so a multi-line fetch's option and header
lines bind and a changed payload on a following line reopens the outbound key.
Regenerated the Python baseline: same package/file/check set, 24 entries pick
up the wider multi-line evidence. Adds regression tests for each case.
* scan packages: stop giant greedy spans from binding a whole-file digest
When a greedy DOTALL pattern (reverse shell socket...subprocess, C2 loop) has
its anchor tokens far apart, the match span covers the whole file. Digesting
that span bound thousands of unrelated lines, so the evidence hash drifted on
any edit between the anchors (a dependency bump reshuffling the file), which
made a baselined finding reopen on an upstream release. The multiline pass now
skips an oversized span when the per-line pass already bound the signal lines,
so the evidence is the stable matched lines; a genuinely appended multi-line
construct stays under the cap and is still recorded.
Regenerated the Python baseline against Python 3.12 (the version the scan CI
shards run) so the resolved dependency set matches CI. Same package/file/check
set. Adds a regression test.
* scan packages: tighten evidence binding (order, string brackets, span size)
Address review follow-ups on the evidence extraction:
- _canon_evidence keeps discovery (line) order instead of sorting. Line-shift
stability already comes from stripping the L<NN>: markers, so order stays
significant and reordering matched lines (a multi-line call's arguments)
reopens the finding.
- _logical_line_end (Python) and _logical_line_text (npm) blank string literals
before counting brackets, so a ) inside a string argument does not close the
logical line early and drop later argument lines.
- The oversized-span skip now only drops a giant whole-file bridge (over 60
lines); a genuinely appended multi-line construct is recorded so its payload
reopens, rather than riding an existing one-line match.
- npm _logical_line_text binds the enclosing bracket group, so a host-config
object whose { is on a prior line binds its path/headers/body lines.
Regenerated the Python baseline (Python 3.12, matching the scan CI shards):
same package/file/check set. Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan npm packages: normalize and bound the logical-line digest
- _evidence whitespace-normalizes the logical line before digesting (matching
_evidence_hash), so a formatter-only reindent of the bound continuation lines
does not change the sha256 suffix and reopen an unchanged finding.
- _logical_line_text follows a bracket group to its close up to a hard 200-line
cap (digest input only), so a config object longer than the backward window
still binds its whole tail instead of silently truncating.
Adds regression tests. npm baseline is empty, so no regeneration is needed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan: cap single-line evidence and widen npm opener window
Cap each rendered evidence line at 200 chars in scan_packages.py: a long
or minified one-line file is shown as a bounded prefix plus a sha256 of the
full line, so a packed payload cannot dump unbounded content into the CI
logs or baseline while a change past the cutoff still changes the digest
and reopens the finding. Mirrors how the npm scanner bounds its snippets.
Widen the npm backward opener window (_MAX_CONT_LINES 12 to 200, symmetric
with the forward cap) so a host deep inside a large options object binds
the whole object, not just its own line; a changed path, header, or body on
any property reopens.
Regenerate the Python baseline with Python 3.12: only the protobuf
nspkg.pth and unsloth-zoo compiler.py evidence change, both from the new
line cap; the package/file/check key set is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan: bind all host contexts, deep call continuations, far-back npm openers
Three fail-closed evidence gaps surfaced by review of the previous round.
scan_npm_packages.py: measure the forward bracket-group cap from the matched
line (idx + _MAX_GROUP_LINES) instead of the opener, so an opener found near
the widened backward limit no longer consumes the forward budget and drops
the path, headers, or body that follow the host.
scan_npm_packages.py: _outbound_host_evidence now records every outbound
context form for a host (URL, fetch-context, host-config), claiming each
non-overlapping match in form order, so a separate host-config request added
beside an already-baselined URL changes the evidence and reopens the key.
The common single-context case keeps its existing snippet.
scan_packages.py: follow a matched Python call over its continuations up to a
separate _MAX_CALL_LINES (40), decoupled from the 12-line display threshold,
so a multi-line requests.post( binds its whole argument list in the digest
and a changed body deep in the call reopens; bounded so a miscounted bracket
cannot swallow unrelated code. No baseline change: the current dependency set
has no matched call that closes between 13 and 40 lines, confirmed by a
Python 3.12 regenerate that produced a byte-identical baseline.
* scan: clamp npm depth, pin large bundles, follow backslash and bound .pth dump
Four fail-closed evidence gaps surfaced by review of the previous round.
scan_npm_packages.py: clamp the backward opener scan at depth 0 so a leading
unmatched closer (a preceding block whose opener is outside the backward
window) no longer drives depth negative and masks the real enclosing opener
that follows; a host-config object after such a block now binds and a changed
path reopens.
scan_packages.py: a large JS bundle now pins its whole content even when
another JS heuristic already fired. The bundle digest was only added when no
other finding existed; it is now appended to every finding's evidence on a
large bundle, so an unchanged obfuscation signature no longer lets changed
payload elsewhere ride the matched-line key.
scan_packages.py: _logical_line_end follows explicit backslash line
continuations, so a call split with a backslash before its parenthesis binds
the continuation line (URL/body) instead of returning at the zero-depth API
line.
scan_packages.py: the catch-all .pth import evidence is bounded through
_cap_line (prefix plus a digest of every line) so a large .pth of benign
imports cannot dump the whole member into the logs or baseline while an
appended or swapped import still reopens.
Baseline regenerated with Python 3.12: key set unchanged; one entry
(unsloth-zoo compiler.py) gains the backslash-continued banner lines now
bound by the continuation fix.
* scan: handle multi-line strings, lifecycle bodies, and de-quadratic evidence
Addresses a review round plus a performance audit of the evidence extractor.
Correctness (fail-closed):
- Bind the UNION of the single-line-blanked and multi-line-blanked bracket spans
in both scanners. The multi-line view blanks a triple-quoted Python string or a
backtick template literal that spans lines, so a `)` inside such a string no
longer closes the enclosing call early and drop later arguments. The single-line
view still counts a payload embedded INSIDE a string, so a dropper that hides a
call in a string keeps its argument lines bound. Taking the larger span never
shrinks the binding below either view, avoiding a fail-open regression.
- cred-env-in-lifecycle now pins the whole lifecycle script body via a digest, so
a changed non-token line (e.g. adding a curl exfil beside the token reference)
reopens, not just a change on the token line.
Performance / DoS (the scanner runs on attacker-controlled package files up to the
64 MiB / 16 MiB member caps, with no per-file time budget):
- _extract_evidence precomputes newline offsets once and maps match offsets with
bisect, removing the O(matches) whole-file content.count per match that made the
finditer fallback quadratic (a crafted minified file went from ~13 s/MiB and
hours at the cap to linear).
- npm _index_text splits and string-blanks the file once per evidence call instead
of per match (was O(matches x file) time and allocation).
- Bound evidence output: _MAX_EVIDENCE_SPANS (Python) and _MAX_EVIDENCE_MATCHES
(npm) fold the remainder into a digest so a file with thousands of matches cannot
build a multi-megabyte evidence/baseline blob while an added/removed match past
the cap still changes the key.
- _outbound_host_evidence caps matches per form and bounds the overlap claim so a
host repeated many times cannot make it quadratic.
No baseline change: a Python 3.12 regenerate is byte-identical (the union equals the
legacy single-line span for every current dependency file; the cap thresholds sit
above the largest real entry), so these are forward-looking hardening with no drift.
* scan: count all overflow matches, bind their context, blank JS regex literals
Follow-ups on the evidence output caps from the previous commit.
- _outbound_host_evidence no longer truncates each pattern's match iterator with
islice; it iterates every match and runs the overlap dedup only while the
display list is below the cap (so claimed stays bounded and the check is O(cap)
per match, not quadratic), folding every match past the cap into the overflow
digest. A host context beyond the 64th is counted again, so it reopens.
- The overflow digest (both scanners, via a shared _overflow_digest) binds each
overflow match's logical-line context, not just the regex match text, so a
changed payload on an over-cap line reopens even with the matched token
unchanged.
- The multi-line JS blanked view now blanks regex-literal bodies (tracking the
previous significant char for regex-vs-division and char classes for a literal
`/` inside `[...]`), so a `)` inside `/)/` no longer closes an outbound call
early. The bound span is the union of the single-line and multi-line views, so
an imperfect regex decision only ever grows the span, never shrinks it.
- The Python overflow digest canonicalizes spans (strips L<NN>: markers via
_canon_evidence) before hashing, restoring line-shift stability for the
over-cap region.
No baseline change: the overflow branches only trigger above the per-finding caps
(above the largest real entry), and the npm baseline is empty, so a Python 3.12
regenerate is byte-identical.
* scan: refresh baseline for ipython interactiveshell.py span drift
A newer ipython release changed the filesystem-enumeration span in
IPython/core/interactiveshell.py, so its content digest no longer matched the
baselined evidence and the studio scan shard flagged it as a non-baselined
CRITICAL. Regenerated with Python 3.12: only the ipython entry's evidence_hash
changes; the package/file/check key set is unchanged, and a studio enforcing
spot-check exits 0.
* Bound scanner evidence memory: stream overflow spans and cap lifecycle baseline size
scan_packages.py: _extract_evidence no longer materializes a rendered span
per match before slicing at the display cap. Once out holds _MAX_EVIDENCE_SPANS
spans, further spans fold straight into a running digest, so a minified or
padded file with hundreds of thousands of matching lines keeps memory bounded
to the display cap instead of the match count. The fold reproduces
_canon_evidence(" | ".join(overflow)) byte for byte, so the overflow digest and
every baseline key are unchanged.
scan_npm_packages.py: lifecycle-fetch-exec and cred-path-in-lifecycle stored the
entire install script body as evidence, so --write-baseline on a package with a
multi-MiB lifecycle script bloated the baseline JSON. Both now store a bounded
matched snippet plus a body-sha256 digest, matching cred-env-in-lifecycle. The
digest still binds the whole body, so a change to any line reopens the finding.
Adds tests for the streamed overflow bound and the bounded-but-reopens lifecycle
evidence. Baseline unchanged (byte-identical Python evidence; npm baseline empty).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make npm bracket-group scan order-aware so a same-line close-then-open binds
_scan_group counted brackets with a per-line net (opens minus closes), which
collapses intra-line order: a line that closes a prior block and then opens the
host-config object on the same line, e.g. `}); const opts = {`, nets to <= 0, so
the trailing `{` was dropped and the group started at the hostname line. A
changed path/headers on the following lines then hashed to the same evidence and
could ride an existing baseline key.
Replace the net count with an order-aware (L, R) reduction per line (L closers
needing an opener to the left, R openers needing a closer to the right) and apply
it in order in both the backward and forward scans, clamping stray closers at 0.
The trailing opener now stays visible so the whole object binds and a changed
payload reopens. Per-line cost is unchanged (one C-level bracket findall), so the
existing outbound-host evidence is byte-identical on all prior shapes; only the
previously-dropped same-line case changes. Adds a regression test for it.
* Harden scanner evidence: bound memory and bind Python call tails fail-closed
Five fixes across both scanners, none of which change the committed baseline (a
full regen of all three pip shards produced a byte-identical 185-key set).
scan_npm_packages.py: _evidence and _outbound_host_evidence collected every regex
match into a list before applying the 64-match display cap, so a text file under
the size cap that repeats a cheap signal (such as NPM_TOKEN) millions of times
could allocate a huge list of re.Match objects and stall or OOM before the
overflow digest ran. They now stream from finditer and fold overflow as matches
arrive via a shared _fold_overflow_match helper, byte-identical to the prior
digest.
scan_packages.py:
- _extract_evidence kept inserting every unique over-cap span into the seen set
even after it stopped appending to the display list, so a generated file with
millions of one-line matches still grew that set unbounded. It now tracks spans
only while filling the display list (per-line spans are unique by line number,
so dropping them past the cap cannot miss a dedup).
- _scan_line_end counted brackets with a per-line net, so a continued statement
that closes on the same line it opens a flagged call (a leading "]" before
"requests.post(") had the call's open paren cancelled and bound only the opener
line. It now applies brackets in order via _bracket_lr (leading closers clamp at
0), matching the npm bracket fix.
- a single-quoted string continued by a trailing backslash was not tracked across
lines, so a close paren inside the continued string on the next line closed the
call early; _blank_code_strings now carries the continuation.
- a call with more argument lines than the soft cap was hashed only through the
cap, so a changed data=/headers tail past it stayed suppressed; a closing call
is now followed to its real close under a 200-line hard limit (a never-closing
opener still stops at the 40-line soft cap so it cannot swallow the file).
Adds regression tests for each. npm baseline is empty; the Python baseline is
unchanged (verified byte-identical by regenerating all three shards).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Bind giant DOTALL span anchors and add context to constant IOC evidence
Two fail-closed gaps where a changed payload could keep the same evidence hash
and stay suppressed by the baseline.
scan_packages.py: a giant greedy DOTALL span (a cross-line IOC match bridging
more than 60 lines, e.g. RE_TEMP_EXEC matching a /tmp line and a much-later
subprocess line) was dropped entirely once the per-line pass had any match, so an
appended cross-line payload -- a new /tmp line plus a later subprocess line that
share no single line, so the per-line pass never binds them -- produced the same
evidence and rode the key. The span is no longer dropped: it is bound by its head
and tail anchor lines plus a digest over just those (no line numbers, so a pure
line shift is stable). An added or moved anchor reopens the finding, while churn
in the bridged interior stays stable, so this does not reintroduce whole-file
drift. Two baseline entries (multiprocess test, unsloth-zoo scanner file) carry
such a span and are refreshed; a full three-shard regen confirmed only those two
keys change.
scan_npm_packages.py: known-ioc-string and cred-surface-host (always-bad) recorded
only the bare needle/host as evidence, so a reviewed tarball that kept the IOC
string while altering the adjacent fetch/exfil body produced an identical key.
They now bind matched-line context: known-ioc-string via the matched line and its
bracket-group continuation, cred-surface-host (always-bad) via the outbound call
context (path/headers/body, falling back to the bare host when not in an outbound
call). A changed payload on the same call now reopens.
Adds regression tests for each. npm baseline is empty; the Python baseline updates
only the two giant-span entries.
* Hash giant-span interiors, bind exec/eval trigger, JS content, intra-literal whitespace
Four fail-closed gaps where a changed payload could keep the same evidence hash.
scan_packages.py:
- A giant bridged DOTALL span was bound only by its head and tail anchors, so a
cross-line payload inserted into the bridged interior between unchanged outer
anchors kept the same key. The whole span content is now digested (via _render),
so any interior change reopens; a pure line shift stays stable because the digest
is over the markerless code. Two baseline entries (multiprocess test, unsloth-zoo
scanner file) carry such a span; with full-interior binding, multiprocess
resolved at two versions across shards now yields two distinct entries where the
anchor digest had collapsed them into one.
- The exec/eval-with-hidden-payload findings omitted the visible exec/eval line
that makes the hidden string executable, so flipping a harmless eval("1+1") to
exec(__doc__) kept the same key while arming the payload. The trigger line from
the real-code view is now bound into the evidence.
- check_js_file extracted evidence with the Python-string-aware extractor, which
does not blank JS backtick template literals, so a template containing a close
paren closed a call's bracket span early and omitted later option/body lines. The
full file content digest is now pinned to every JS finding (not just large
bundles), binding the whole call.
scan_npm_packages.py: the evidence canon collapsed all whitespace via split(),
erasing whitespace inside JS string literals along with harmless indentation, so a
changed request body 'a b' -> 'a b' kept the same key. A new _canon_preserve_strings
collapses whitespace only OUTSIDE string literals (reindent-stable) while preserving
it INSIDE single/double/backtick literals (intra-payload edits reopen). Used for the
evidence hash and the logical-line digests.
Adds regression tests for each. npm baseline is empty; the Python baseline updates
the two giant-span entries and adds the second multiprocess version's entry.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio RAG: disable trust_env on loopback llama-server httpx clients
The RAG embedder health probe (embed_llama_server.py), its pooled httpx.Client, and the vision captioner (captioner.py) call the local 127.0.0.1 llama-server with httpx's default trust_env=True, so an ambient HTTP(S)_PROXY that returns 503 for loopback breaks embedder startup and captioning. Set trust_env=False on these loopback clients, matching the existing fix on the main llama_cpp and inference clients. External provider calls are untouched.
Follow-up to the loopback trust_env fix; covers the remaining local llama-server clients in the RAG path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio RAG tests: accept trust_env kwarg in captioner httpx.post mocks
The loopback captioner now passes trust_env=False; update the _vision_complete
fake_post stubs to accept it and assert it is False.
* Trim comments in Studio RAG trust_env fix (comment-only)
* RAG trust_env test: explicit UTF-8 read + scan all package .py files
Addresses review: utf-8 open avoids a Windows decode error, and scanning every
.py in core/rag catches any future file that adds an httpx call.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* MLX CI: find llama-cli where save_pretrained_gguf actually installs it
The GGUF reload step hardcoded the CWD-relative paths llama.cpp/llama-cli and
llama.cpp/build/bin/llama-cli, but save_pretrained_gguf builds and installs llama.cpp
under unsloth_zoo's LLAMA_CPP_DEFAULT_DIR ($UNSLOTH_LLAMA_CPP_PATH, else
~/.unsloth/llama.cpp), so the reload could not find the binary and failed the Mac M1
job with "llama-cli not found". _find_llama_cli now searches that install directory
(and honors the env override) before falling back to the old CWD layout, with a
recursive glob as a last resort. The search is a strict superset of the previous
paths, so it cannot regress a layout that already worked.
* MLX CI: return an absolute llama-cli path from the locator
Resolve the located binary to an absolute path. If UNSLOTH_LLAMA_CPP_PATH is a
relative directory (e.g. "."), Path(".") / "llama-cli" normalizes to the bare name
"llama-cli", and subprocess.run treats a separator-less argument as a PATH lookup
rather than a file to execute, raising FileNotFoundError. resolve() makes the returned
path absolute so it always runs the intended binary.
* MLX CI: give llama-cli EOF on stdin so GGUF reload cannot hang
With the binary now found, the GGUF reload actually invokes llama-cli and it timed
out after 300s generating 24 tokens on a 270m model, which is a stdin block rather
than slow generation: subprocess.run captured stdout/stderr but left stdin inherited,
so -no-cnv still left llama-cli waiting for interactive input. Pass
stdin=subprocess.DEVNULL so it receives an immediate EOF and runs the single prompt to
completion.
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Condense the verbose comments and docstrings added by the recent
chat template, GPT-OSS detection, PEFT tensor-parallel, and Studio
inference proxy fixes. Comments and whitespace only; no code changes.
check_js_file put the bundle's KB size inside the finding's check label, which is
part of the baseline match key (package, file, check). When tensorboard's
projector_binary.js grew from 1918 KB to 1933 KB, the reviewed baseline entry stopped
matching and the benign HIGH resurfaced, red-failing the studio and extras
scan-packages shards. Move the size into the evidence field (shown for review, not
matched) and keep the check label constant, then update the one tensorboard baseline
entry to the size-agnostic label. The finding is suppressed again and will not
re-break when the bundle grows by a few KB. Scanner self-tests pass unchanged.
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* add models for /update endpoint
* add logic for identifying out of date hf models
* add endpoint for updating hf models
* add relevant field to GgufVariantDetail
* make exception handling better
* add update_available flag for cached_models, and moved /update endpoint from inference -> models
* hook up /update endpoint on the frontend
* implement update scenarios for the model picker
* fix bug where downloaded flag for an older revision was being wrongly set to false
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix import and make hf calls async
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* remove has_vision from UpdateRequest
* fix ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* clear cancel event before updating gguf variant
* set _cancel_event back if it was set initially
* add hf_token to get_paths_info
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: harden model update endpoint and update checks
- update_hf_model: pass snapshot_download local_dir (local_path is not a
valid kwarg and 500s when updating bicodec audio models)
- get_gguf_variants: wrap the remote update check so a network, rate-limit,
gated, or offline failure degrades to "no update info" instead of failing
the whole variant listing, matching list_cached_models
- add regression tests for both paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: HF model update detection and Update action for cached models
Surface an "Update available" cue and a managed Update action for cached
on-device models. /api/hub/update-status compares each cached main GGUF
file's local blobs against the remote main revision using set membership
across all cached revisions, so a repo that was already updated (and still
holds the old snapshot alongside the new one) is not falsely flagged.
The Update action re-downloads through the download manager so it shows in
the Downloads panel with progress and cancel. The frontend wires the Update
button into the GGUF, on-device, and model-selector cards and keeps the
quant label fully visible when the action buttons crowd the row.
Adds regression tests for the multi-revision update check.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: accept force_download kwarg in hf_xet_fallback test double
The download seam now passes force_download to the attempt callable; the _FakeAttempt mock did not accept it, failing 6 tests with TypeError. Add the keyword (default False) so the scripted-results double matches the seam.
* Fix Studio model update regressions
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address Studio update review feedback
* Address Studio update edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Share GGUF update status helper
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix GGUF update detection and cache cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix cached GGUF update badges
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: shimmyshimmer <107991372+shimmyshimmer@users.noreply.github.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* fix: keep LoRA reloads working with PEFT 0.19
* test: exercise the PEFT tensor-parallel symbol extractor
* test: prove the full PEFT tensor-parallel seam
* fix: harden PEFT tensor-parallel shims
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: fall back when PEFT tensor-parallel source inspection fails
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* i18n: register Japanese language support in messages
* i18n: add Japanese locale support
* Update studio/frontend/src/i18n/locales/ja.ts
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/frontend/src/i18n/locales/ja.ts
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* test(i18n): add ja locale to parity check
* i18n: fill remaining missing keys for Japanese locale
* i18n: fix terminal string localization in ja locale
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* fix(studio/llama_cpp): disable trust_env on the loopback health probe
_wait_for_health() polls http://127.0.0.1:<port>/health with the default
httpx trust_env=True, so an ambient HTTP(S)_PROXY in the environment is
applied to the loopback request. A proxy that returns 503 for 127.0.0.1
makes every probe fail, so the loop runs until timeout and Studio load
hangs (trust_env=False returns 200 immediately).
Pass trust_env=False so the local readiness probe never goes through a
proxy. This mirrors the existing trust_env=False handling in the sibling
llama_http / external_provider HTTP clients.
* test(offline_gguf_cache): accept trust_env kwarg in fake_get mock
_wait_for_health now calls httpx.get(..., trust_env=False); update the retry test's fake_get to accept the kwarg so it doesn't raise TypeError.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/llama_cpp): bypass proxies for loopback clients
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/routes): bypass proxies for llama streams
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* studio: announce Cloudflare tunnel state and warn about public exposure on startup
The startup banner only printed a line when a tunnel URL was up, so a plain
`unsloth studio -H 0.0.0.0` launch silently created a public trycloudflare.com
URL with no indication that Studio had become reachable from the internet. The
only hint at the tunnel was the CLI help, shown when an invalid command was typed.
Make the banner always state the tunnel state for wildcard binds:
- ON: the public URL plus a warning that anyone with it can reach Studio from
outside the network, and that --no-cloudflare keeps it local-only.
- FAILED: requested but did not start (local network only).
- OFF: --no-cloudflare was passed (local network only).
Secure mode keeps its existing wording (the authenticated tunnel is intended and
--no-cloudflare is not valid there). Clarify the --cloudflare help text in both
the argparse and typer definitions. Default behavior is unchanged.
Also surface the state on the `unsloth studio run` banner, which runs the server
with silent=True and prints its own banner: it now calls _print_cloudflare_line
too, so the ON/OFF/FAILED notice and public-exposure warning are no longer
skipped on that path (previously it only echoed the URL when a tunnel was up).
For the OFF and FAILED notices, do not claim "local network only" when the
reachability probe just confirmed the raw port is reachable from the public
internet: --no-cloudflare and a failed tunnel disable only the Cloudflare link,
not the wildcard bind, so the message is reworded to flag the public raw port.
* Fix/adjust Cloudflare banner warnings for PR #6515
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust Cloudflare banner comments for PR #6515
* Fix/adjust IPv6 Cloudflare tunnel gate for PR #6515
* Fix/adjust Cloudflare review comments for PR #6515
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix silent run Cloudflare notice
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: quick eject from the model selector
Add a one-click eject shortcut to the loaded-model pill so users do not
have to open the picker to unload a model.
- The loaded-status indicator shows a green checkmark at rest and swaps to
a red eject icon on pill hover, with an "Eject model" tooltip. Clicking
it ejects without opening the picker.
- On Device tab now uses the placeholder "Search local models" instead of
"Search Unsloth models".
- The picker's "Eject model" button uses medium font weight.
* Studio: drop unused group/eject marker class on the eject control
* Studio: make the inline eject control valid HTML
The eject shortcut was a focusable span (role/tabIndex) nested inside the
trigger button. A button's content model forbids focusable descendants, so
make it a plain decorative span (aria-hidden, no role/tabIndex) that keeps
the mouse shortcut. Keyboard and screen-reader users eject via the picker's
"Eject model" button.
* Studio: disable the inline eject shortcut on touch devices
On touch (no hover) the red eject icon and title tooltip never reveal, so
tapping the loaded pill could unload the model with no visible affordance.
Add [@media(hover:none)]:pointer-events-none so taps fall through to the
trigger and open the picker; touch users eject from the picker instead.
---------
Co-authored-by: shimmyshimmer <shimmyshimmer@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* Add whole-document context mode to RAG chat attachments
Thread-attached files are injected in full when they fit a token budget,
instead of only top-K retrieved chunks, so the model reads the entire file
for summarize/reason-over-document requests. Oversized files fall back to
top-K retrieval so the context window is never blown. KB and project
corpora are unchanged (still retrieval).
- core/rag/store.py: all_chunks_for_scope returns every completed-document
chunk for a scope, ordered document-then-index, joined with filename.
- core/rag/tool.py: whole_document_context renders the chunks as the same
<chunk> blocks + citation source-map retrieval produces, returns None
when empty or over budget.
- core/inference/tools.py: build_rag_autoinject tries whole-document first
for thread scopes, falls through to search_for_autoinject otherwise.
- core/rag/config.py: THREAD_WHOLE_DOC + WHOLE_DOC_MAX_TOKENS (env-tunable).
- tests/test_rag_whole_document.py: store ordering, whole-doc render +
budget cutoff, auto-inject whole-doc vs top-K fallback, KB never whole-doc.
* Add scanned-PDF OCR fallback to RAG ingestion
A PDF page with no extractable text layer (a scanned or image-only page)
previously ingested as empty, so image PDFs were invisible to retrieval and
whole-document context. Such pages are now rendered and transcribed by the
loaded vision model during ingestion, so they become searchable and readable
like any other page. This restores OCR for the RAG document flow without a
separate extraction pipeline.
- core/rag/parsers.py: render_pdf_pages renders whole pages (1-based) to PNG.
- core/rag/captioner.py: factor the shared vision call into _vision_complete;
add _ocr_one + ocr_pages (transcribe rendered pages, OCR_MAX_PAGES bound).
- core/rag/ingestion.py: _ocr_scanned_pages runs right after parse, replacing
text on near-empty PDF pages. No-op when OCR is off, no page is scanned, or
no vision model is loaded (degrades like figure captioning).
- core/rag/config.py: OCR_SCANNED, OCR_MIN_CHARS, OCR_MAX_PAGES, OCR_DPI,
OCR_TIMEOUT_S, OCR_MAX_TOKENS (env-tunable).
- tests/test_rag_ocr_fallback.py: page render, ocr_pages gating + cap, scanned
PDF end-to-end OCR into chunks + whole-doc, born-digital skips OCR, disabled
leaves the page empty.
* Broaden OCR prompt to figures/tables and guard against repetition runaway
The OCR prompt now asks the vision model to also transcribe text inside figures,
diagrams, charts and tables, so labels and table cells on scanned pages are
indexed rather than skipped. Verified on real documents that this does not
regress plain-text transcription.
Some vision models loop on sparse images (e.g. a title-only cover) and emit the
same line hundreds of times. _collapse_runaway caps any run of identical
consecutive lines so a pathological page cannot flood the index; legitimate
short repeats (a label appearing a few times) survive. Applied in ocr_pages.
* Restrict whole-document injection to thread attachments only
whole_document_context resolved the combined project+thread scope, so a project
chat (the frontend sends both thread_id and project_id) injected the entire
project corpus in full, contradicting the design that project and KB corpora stay
retrieval-only. A large project corpus could also push the total over budget and
drop a small thread attachment back to top-K.
Resolve the thread scope alone in whole_document_context, and in
build_rag_autoinject only enter whole-doc mode when a thread attachment is present
and no KB is selected (a KB pick is exclusive: search that corpus). Project
sources and KBs keep top-K retrieval. Adds regression tests for the mixed
project+thread payload, the budget isolation, and KB precedence.
* Address review: keep project retrieval, harden budget + OCR guards
Follow-up to the 8-reviewer pass on the whole-document + OCR work.
- Preserve project grounding in project chats. The thread-scope-only fix made
whole-doc exclusive of retrieval, so a thread attachment silently dropped the
project corpus for that turn. build_rag_autoinject now whole-docs the thread
attachment AND retrieves the project sources top-K, merged under one citation
numbering via tool.render_sources. KB selection stays exclusive.
- Budget: a NULL/zero token_count no longer bypasses the cap (length-based
fallback in _row_token_count), so a malformed huge doc can't inject in full.
- OCR runaway guard: _collapse_runaway now also caps each distinct line at a
generous total across the page (not just consecutive), bounding the
interleaved/alternating loops weak models emit; blank-line floods collapse too.
- OCR: warn when a scanned PDF exceeds OCR_MAX_PAGES (pages past the cap stay
untranscribed) instead of silently dropping them.
- Document the known limits: OCR'd pages have no PDF highlight regions; vision
models need a micro-batch >= image tokens (Gemma-family) or the server aborts.
- Tests for project-retrieval composition, NULL-token budget, and interleaved
runaway; drop the now-superseded exclude-project test.
* Add OCR toggle to RAG retrieval settings
Make scanned-PDF OCR user-controllable per upload instead of only via the
RAG_OCR_SCANNED config default. The retrieval settings panel gains an OCR
scanned pages switch (persisted in localStorage, on by default); the chosen
value is read fresh at upload time and sent with each document upload.
Backend: the three upload routes accept an optional ocr form field and pass it
through start_ingestion to _ocr_scanned_pages, which now treats None as use the
config default and an explicit bool as an override. The on/off policy lives only
in _ocr_scanned_pages now, so ocr_pages no longer re-checks the config (that
double gate would have blocked a per-upload ocr=True while the default was off).
Tests cover both override directions (force on while config off, force off while
config on).
* Add "Describe figures & charts" toggle with chart-aware captions
Surface RAG figure captioning as a user control and make it actually useful for
graphs and plots. The figure detection already clustered vector drawings and
raster images into regions and rendered them, but captioning was off by default,
had no UI, and used a thin generic prompt.
Accuracy: the caption prompt now asks for chart type, axis titles and units,
legend or series, salient trends and readable values, and table columns, while
forbidding invented numbers. The token budget is configurable (CAPTION_MAX_TOKENS)
and captions pass through the same runaway guard as OCR so a looping vision model
cannot flood the index.
Control: a per-upload caption override threads from the three upload routes through
start_ingestion and _run, with the on/off policy single-sourced in _run (caption
self-gating removed from caption_images, mirroring the OCR change) so a force-on
override works when the config default is off. The frontend adds a "Describe
figures & charts" switch in the retrieval settings, persisted in localStorage and
sent with each upload. Default on; it is a no-op without a vision model and bounded
to CAPTION_MAX_IMAGES figures per document.
Tests cover the new caption_images contract, the runaway guard on captions, the
chart-aware prompt and token budget (and that OCR keeps its own prompt and budget),
and both override directions end to end through ingestion.
* Generalize figure understanding: transcribe-first prompt + high-DPI tiling
Make figure/chart description work across any visual and any model strength, not
just a strong VLM on simple figures. Two changes, validated by a recall benchmark
on authoritative documents (ResNet/Attention papers, USDA, UN UDHR).
1. Transcribe-first caption prompt. The caption now asks the model to transcribe
every visible label verbatim (titles, axis labels and units, legends, every
box/node/arrow label, table cells, equations) and then add a one-line summary,
instead of only describing the figure. Transcription is the most model-robust
visual task, so weak models that cannot reason about a chart still recover its
labels.
2. High-DPI tiling of figure pages. Figure-bearing pages are rendered as an
overlapping grid of high-DPI tiles (plus a full-page pass for context); each
tile is transcribed, then merged and de-duplicated. This keeps small diagram
labels legible and covers every sub-figure without relying on exact region
detection, which previously missed sub-figures and small labels.
Supporting changes: figure render DPI 130 -> 200 with a clip margin so edge labels
are not lost; vision calls are deterministic (temperature 0) so transcription does
not randomly drop labels; the repetition guard now applies to captions too. New
config knobs: FIGURE_DPI, FIGURE_MARGIN_FRAC, FIGURE_TILE_ROWS/COLS, FIGURE_TILE_
OVERLAP, FIGURE_FULLPAGE, CAPTION_MAX_PAGES, larger CAPTION_MAX_TOKENS, and
CAPTION_MAX_IMAGES as a per-document tile budget.
Measured figure context recall (per-label, dense academic figures):
Qwen2.5-VL: 0.50 -> 0.83 (overall 0.81 -> 0.94)
Gemma-4-E2B (weak): ~0 with loops -> 0.83 (overall 0.91)
Born-digital text and scanned-page recall are unchanged (no regression).
parsers gains _figure_boxes (shared detection), pages_with_figures, and
render_pdf_figure_tiles; captioner gains merge_page_captions and a temperature
parameter; ingestion routes figure captioning through the tiled path.
* Fix RAG review issues: whole-doc budget pre-check, figure gating, empty re-ingest, vision auth
Whole-document context now runs a cheap token-sum pre-check (store.scope_token_estimate)
before hydrating every chunk's text, so an attachment that cannot fit the budget is
rejected without loading the whole corpus into memory. The estimate mirrors
all_chunks_for_scope's filter and the per-row token-count fallback exactly.
Ingestion skips all figure work (PDF rasterization and detection, not just the caption
call) unless a vision model is loaded, so a text-only deployment pays nothing. When OCR
is enabled, scanned/image-only pages are excluded from figure tiling since OCR already
transcribes them whole, avoiding double vision work and overlapping index entries; a
scanned figure page is still tiled when OCR is off.
start_ingestion no longer dedupes forever to a prior ingest that produced zero chunks
(e.g. a scanned PDF uploaded before a vision model was loaded): the empty record is
dropped and the content is re-ingested.
Vision OCR and caption requests now send the backend Authorization header, so they
match the chat endpoint and do not 401 under direct-stream (--api-key) mode.
Adds tests for the budget estimate, scanned-page exclusion, the vision-model gate, the
empty re-ingest path, and the auth-header passthrough.
* Trim RAG vision-ingestion comments and docstrings
Tighten the verbose multi-line docstrings and comments added across the RAG vision
ingestion work (captioner, config, parsers, ingestion, store, tool, build_rag_autoinject,
the RAG tests, and the chat-store/upload-hook frontend toggles) to one or two lines while
keeping their intent. No code changed: verified comment/docstring-only against the prior
commit, and the RAG test suite still passes.
* Fix figure-tiling exclusion and client dedupe for re-ingestable docs
Figure tiling now excludes only the pages OCR actually transcribed, not every
text-less page. _ocr_scanned_pages returns the set of pages it OCR'd, and _run passes
that to pages_with_figures as exclude_pages (replacing the ocr_on-keyed min_text_chars
heuristic). A scanned page that OCR skipped (past OCR_MAX_PAGES, or whose OCR returned
empty) is no longer dropped from captioning, so a chart on such a page still gets a
caption.
The document panel's upload dedupe no longer skips re-selecting a file whose only
matching doc completed with zero chunks. Such a doc is re-ingestable (e.g. a scan
attached before a vision model loaded), and the backend re-ingests on the same content
hash, so the client must let it reach the backend; healthy or still-indexing docs are
still skipped. The SSE complete frame's chunk count is recorded on the doc so the
check is exact.
Adds a regression test for the un-OCR'd scanned figure page and updates the
pages_with_figures test to the exclude_pages interface.
* Address review findings: whole-doc budget guard, job numChunks, dead code, upload cap
whole_document_context now treats a non-positive max_tokens as "never inject" instead
of injecting the whole corpus unbounded, so RAG_WHOLE_DOC_MAX_TOKENS=0 tightens rather
than disables the budget (the real off switch stays RAG_THREAD_WHOLE_DOC=0).
The job-status endpoint and get_job_status now expose num_chunks (joined from the
document), and the upload hook threads it through the SSE-fallback completion paths
(reconcile + poll). Previously a document that finished via the connection-cap fallback
had no chunk count client-side, so the re-ingest dedupe wrongly treated it as empty and
re-uploaded it. IndexJob/JobEvent gain the field and the untyped cast is dropped.
Removes the dead render_pdf_figures function (superseded by the tiling path), its test,
and the unused FIGURE_MARGIN_FRAC config knob.
Adds an upload size cap (RAG_MAX_UPLOAD_BYTES, default 200 MB; 413 on exceed with the
partial file cleaned up) so a pathological file can't drive unbounded parse + vision
work. render_pdf_figure_tiles clamps rows/cols to >= 1 (no ZeroDivisionError on a
misconfigured grid). Captioning progress is reported after OCR so the bar is monotonic.
sqlite connections set busy_timeout=5000 so a long figure/scan ingest holding its
connection doesn't make a concurrent ingest/read fail with "database is locked".
Adds tests for the non-positive budget, the zero-grid clamp, job-status num_chunks, and
the oversize-upload rejection.
* Extract PDF text as layout-aware Markdown via pymupdf4llm
parsers._pdf now extracts each PDF page as Markdown with pymupdf4llm.to_markdown
(page_chunks=True) instead of flat page.get_text("text"), so tables, headings and lists
keep their structure in the indexed chunks and retrieve far better (a table's cells stay
associated with their row instead of flattening into a token stream). Gated by
RAG_PDF_MARKDOWN (default on); falls back to plain PyMuPDF text when the toggle is off,
pymupdf4llm is missing, extraction fails, or a page yields no Markdown. The scanned-page
OCR and figure-tiling passes operate on rendered pixels and are unaffected; docx/html/txt
keep their existing extractors.
The preview-highlight locator already strips Markdown punctuation when building anchors;
it now also splits anchor tokens on pipes so a Markdown table row still anchors to the
raw PDF word stream.
Declares pymupdf4llm as a studio/RAG dependency (was only transitively present via the
data-designer plugin). Adds parser tests (Markdown table reaches the page text, the
plain-text fallback, the missing-lib fallback) and a locator test for table-pipe anchoring.
* Pin pymupdf4llm to 0.3.4 so the package scan does not pull onnxruntime
The lockstep pymupdf4llm 1.27.x line makes pymupdf-layout a hard dependency,
which in turn pulls onnxruntime (plus numpy/networkx/protobuf). The security-audit
pip scan-packages job resolves requirements --with-deps, so adding pymupdf4llm to
no-torch-runtime.txt and studio.txt surfaced onnxruntime's un-baselined CRITICAL
finding and flipped the hf-stack shard from pass to fail.
pymupdf4llm 0.3.x keeps pymupdf-layout behind an optional [layout] extra, so a plain
install resolves to pymupdf + tabulate only and never touches onnxruntime. 0.3.4
requires pymupdf>=1.27.1, satisfied by our pinned pymupdf==1.27.2.3, and to_markdown
(page_chunks=True) produces equivalent layout-aware Markdown on real PDFs (verified on
the Attention, ResNet and USDA documents). Production already installs these files
--no-deps, so onnxruntime was never shipped at runtime; this only fixes the scanner.
The parser test now asserts Markdown markup (heading or table pipes) rather than table
pipes specifically, since 0.3.4 emits a heading but not a pipe table on the tiny
borderless synthetic fixture; both markers are absent from the plain-text fallback.
* Fix RAG whole-doc review findings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address RAG whole-doc review follow-ups
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address RAG review follow-up edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reserve image budget for whole-document RAG
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire imatrix GGUF option and FP8/NVFP4 compressed export into the export UI
GGUF export gains an importance-matrix toggle. When enabled it auto-downloads the
upstream Unsloth imatrix for the base model (or uses a custom path), which unlocks
the IQ low-bit quants iq2_xxs, iq2_m, iq3_xxs and iq4_xs. Merged export gains an
FP8 / NVFP4 compressed-tensors precision selector that runs llm-compressor for vLLM.
Backend threads imatrix_file through routes -> orchestrator -> worker -> export_gguf
(both the local save and the hub push), and maps the new compressed format_type
values onto the fp8/nvfp4 save_method, reporting the "<dir>-<suffix>" sibling output
directory. Frontend adds the imatrix Switch on the GGUF card and a merged precision
picker on the merged card, threaded through the export runtime store.
Depends on unslothai/unsloth#6706 (save.py imatrix_file and compressed-tensors
export) and unslothai/unsloth-zoo#839 (quantize_gguf imatrix flag).
* Studio export: guard imatrix/compressed against older unsloth builds and force imatrix for IQ quants
Addresses review feedback on the export wiring:
- GGUF: pass imatrix_file only when set, so a plain no-imatrix export (e.g. Q4_K_M) no
longer fails with an unexpected-keyword error against an unsloth build that predates the
imatrix_file parameter. When imatrix is requested but unsupported, return a clear
upgrade message instead of a TypeError.
- Merged: gate FP8/NVFP4 compressed-tensors export on the installed unsloth actually
supporting it, returning a clear message rather than a cryptic save_method failure.
- Frontend: IQ quants (iq2_xxs, iq2_m, iq3_xxs, iq4_xs) are imatrix-only, so force the
imatrix on when one is selected and lock the toggle, instead of submitting an IQ quant
with no imatrix that llama.cpp would reject.
Extends the backend tests for the new capability guards and the conditional kwarg wiring.
* Studio: upload compressed merged models to the Hub without recompressing
For an FP8/NVFP4 Hub export the model is already produced locally in the "<dir>-<suffix>"
output. Uploading it directly with HfApi.upload_folder (mirroring export_base_model) avoids
re-running the expensive compressed-tensors quantization a second time inside
push_to_hub_merged, which for NVFP4 also re-runs calibration and risks OOM. Falls back to
push_to_hub_merged when there is no local compressed output to reuse.
* Add FP8/FP4 compressed export to save_pretrained_merged
Adds compressed-tensors export (for vLLM) to save_pretrained_merged /
push_to_hub_merged via llm-compressor, alongside the existing lora /
merged_16bit / merged_4bit / gguf / torchao paths:
model.save_pretrained_merged("model", tokenizer, save_method="fp8")
Supported save_method values: fp8 (FP8_DYNAMIC), mxfp4, nvfp4 (W4A4) and
mxfp8. The LoRA is merged to 16bit at save_directory, then a quantized
checkpoint is written to save_directory + "-<fmt>". nvfp4 needs a small
calibration set (defaults to ultrachat, overridable via calibration_dataset).
Notes:
- llm-compressor is installed lazily on first use, pinning the current torch
and transformers via a constraints file so they are not upgraded (a plain
install pulls transformers>=5 and breaks Unsloth).
- Quantization runs in a separate process (unsloth/_compressed_quantize.py,
launched by file path) so Unsloth's transformers attention patches do not
interfere with the forward llm-compressor runs during calibration, mirroring
how GGUF export shells out to llama.cpp.
- mxfp8 needs a newer llm-compressor (transformers>=5); it is recognised and
raises a clear error until that stack is available.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: main-process guard, calibration subsampling, tokenizer + dtype handling
- Route the 16bit merge through unsloth_generic_save for both LoRA and full
finetuned models, so non-PEFT models are written in 16bit consistently
instead of saving the original (possibly quantized) weights directly.
- Honor is_main_process: only the main process quantizes and writes the
compressed output, so distributed ranks do not race on the same dirs.
- Subsample an in-memory calibration Dataset before save_to_disk so large
training sets are not fully copied to a temp dir.
- Tolerate a missing tokenizer in the converter (data-free exports); still
require one for calibration based schemes.
- Open config.json via a context manager in both files.
- Drop the redundant nvfp4 entry from the unsupported-name check (fp4 covers it).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add direct LoRA to GGUF export and harden FP8/FP4 compressed export
- Run llm-compressor install and scheme check before the 16bit merge so
unsupported schemes (e.g. mxfp8) fail fast without writing a checkpoint
- Only the main process installs, merges, quantizes and uploads; isolate
hub pushes to a temp dir and clean all temp dirs in a finally
- Forward standard save kwargs (state_dict, max_shard_size, ...) to the merge
- Fall back to the first dataset split for Hub calibration ids
- Export LoRA adapters to GGUF via convert_lora_to_gguf.py: modernize
save_pretrained_ggml/push_to_hub_ggml and add save_method="lora" to
save_pretrained_gguf/push_to_hub_gguf; resolve base from the adapter config
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix LoRA GGUF shell-injection test and compressed export trailing-slash path
- Update tests/saving/test_save_shell_injection.py for the new delegation: the
LoRA to GGUF conversion now lives in _unsloth_save_lora_gguf, so assert it
passes argv as a list with no shell=True and that the legacy ggml wrappers
delegate to it instead of calling subprocess.Popen directly
- Normalize the local save_directory before building the "<dir>-<fmt>" sibling
so a trailing slash no longer nests the compressed output inside the 16bit dir
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Polish FP8/FP4 and LoRA GGUF export after review
- Warn (not silently downgrade) when an explicit quantization_method is not a
valid LoRA GGUF outtype; default stays f16
- Correct the inference hardware note: MXFP8 is 8-bit (cc >= 8.9), only FP4
needs Blackwell for full activation quantization
- Document that a local fp8/fp4 save keeps the 16bit merge at save_directory
and writes the quantized checkpoint to save_directory + "-<fmt>"
* Use sequential calibration pipeline and validate Hub access early
- nvfp4 calibration no longer forces the memory-hungry "basic" pipeline. The
quantization runs in a clean subprocess, so llm-compressor's default
sequential pipeline (layer-by-layer onloading) works and lets large models
that do not fit at once still calibrate; fall back to "basic" only if tracing
fails
- For push_to_hub compressed exports, create/validate the repo up front so a bad
token or denied repo fails before the merge and quantization instead of after
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden compressed export: explicit sequential pipeline, base-tokenizer calibration, GPU memory
- nvfp4 calibration now passes pipeline="sequential" explicitly (layer-by-layer
onloading) instead of relying on the inferred default, with a "basic" fallback
- Calibration datasets with a messages column no longer require a chat template:
base / non-chat tokenizers fall back to concatenating message contents
- Free the in-memory model's CUDA memory before the quantize subprocess loads its
own copy from disk (best-effort, single-device non-quantized only; restored
afterward), so a single GPU need not hold two copies at once
- Create the calibration temp dir in the system temp location instead of next to
the save directory, avoiding stray dirs in the workspace
* Free the failed calibration model before the basic-pipeline retry
In the sequential -> basic NVFP4 fallback, release the partially-processed model
and clear the CUDA cache before loading a fresh copy, so the retry does not
transiently hold two model copies on the GPU.
* Harden calibration data handling and compressed-export edge cases
- Calibration messages without a chat template now handle multimodal (list)
content, None content, and null message rows instead of crashing on join
- Raise a clear error when the calibration dataset is empty after subsampling
- Reset llm-compressor's global session before freeing the model in the
sequential -> basic NVFP4 fallback, so the old model is actually released
- LoRA GGUF export accepts a single-element list quantization_method
- Attach datasets metadata to the pushed repo on compressed hub exports
- Warn (instead of silently) if the model cannot be restored to its device
- Raise a clear error if the LoRA base model id cannot be determined
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Handle DatasetDict calibration, MoE routers, and MTP models in compressed export
- Reduce an in-memory DatasetDict calibration set to a single split before row
subsampling, so save_to_disk does not copy every split to the temp dir
- For MoE models, keep the router/gate unquantized and pass
moe_calibrate_all_experts so every expert is calibrated
- Warn when a model carries MTP / speculative-decoding tensors that the
compressed export does not include
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Support many more compressed-tensors schemes and address review
- Expand save_method to cover the full set of compressed-tensors preset schemes:
FP8 (dynamic/static/block), INT8, W8A8, W8A16, W4A16(+asym), W4A8, W4AFP8,
MXFP4(+A16), NVFP4(+A16), plus the gated MXFP8; calibration is used only for
the static-activation schemes (FP8 static, NVFP4)
- Broaden the near-miss save_method error to cover int/w-prefixed names
- MoE: also keep the Qwen shared-expert gate unquantized
- Strip non-model-input columns from already-tokenized calibration data so the
collator does not choke on a leftover messages column
- Forward the Hub token to the LoRA converter and the quantize subprocess so
gated/private base models and calibration datasets work without a global login
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Collapse compressed-tensors export help line so ruff-format converges
The print line in print_quantization_methods needed two ruff-format passes to
reach a fixpoint (merge implicit string concat, then collapse the single-arg
print). pre-commit.ci applies one pass per run, so it kept reformatting. Land
the converged single-line form directly.
* Add CPU-only regression tests for the export API
Cover all export paths without a GPU, for slow CPU-only CI:
- pure-function checks of the compressed-tensors scheme registry and save_method
normalization (aliases, calibration flags, near-miss errors)
- AST checks that every merged saver dispatches compressed export, the GGUF savers
expose the lora branch, torchao routes PTQ/QAT, the public methods stay attached,
and the export subprocesses remain shell-safe (argv list, sys.executable, no shell)
- monkeypatched dispatch checks that fp8/nvfp4/merged_16bit, the LoRA-GGUF outtype
resolution, and torchao PTQ/QAT reach the right helper with the right arguments
* Run the CPU-only export tests in consolidated CI
tests/saving is --ignored by the Repo tests (CPU) job, so the new GPU-free export
tests are added by path to consolidated-tests-ci.yml (collection sanity + Bucket-A run),
alongside the existing CPU saving tests, so they actually execute on CPU CI.
* Add GPU GGUF export + llama-cli inference smoke test
tests/saving/test_gguf_export_and_inference.py: skipif no CUDA. Trains a tiny
phrase-imprinting LoRA, exports a full-model q8_0 GGUF (merge -> convert_hf_to_gguf
-> llama-quantize), asserts a valid GGUF (magic + size), and - when a llama-cli
binary is available - runs one bounded generation (byte cap + watchdog kill) and
asserts the trained phrase round-trips through HF -> GGUF -> quantize -> inference.
The llama-cli step skips gracefully since the export only builds llama-quantize.
* Fix variant mismatch in compressed (FP8/FP4) export
save_pretrained_merged(..., save_method=fp8/nvfp4, variant=...) forwarded
the variant into the intermediate 16bit merge, so Transformers wrote
variant-named shards (model.<variant>.safetensors). The converter
subprocess then reloaded that directory with the default weight filenames,
so the compressed export failed after doing the merge.
Pop the variant out of the intermediate merge (internal staging that the
subprocess reloads with default names) and forward it via --variant so it
is applied to the final compressed checkpoint instead. Add a CPU AST guard
for the contract.
* Harden export paths from review
- install_llm_compressor: fall back to uv pip when this interpreter has no
pip seeded (uv-created/relocatable venvs), instead of failing with
No module named pip.
- LoRA GGUF export: if convert_lora_to_gguf.py is missing (a prebuilt or
reused CWD llama.cpp install carries binaries but not the converter
script), force a dedicated source checkout that ships it.
- push_to_hub_gguf(save_method=lora): return on non-main ranks, matching the
local save_pretrained_gguf lora branch, so only rank 0 converts/uploads.
- compressed export VLM detection: require a vision_config or a
ForVisionText2Text architecture; a bare *ForConditionalGeneration also
matches text seq2seq models (T5/BART/Whisper) and is no longer treated as
a VLM on its own.
- GGUF GPU smoke test: drop SFTConfig(max_length=1024), which raises under
newer TRL padding-free training; length enforcement is not needed here.
* Add imatrix option to GGUF export, enabling IQ low-bit quants
save_pretrained_gguf / push_to_hub_gguf gain imatrix_file:
None -> no imatrix (unchanged)
'/path' -> pass to llama-quantize --imatrix (a *.gguf_file is renamed to *.gguf)
True -> download the upstream unsloth/<base>-GGUF imatrix (imatrix_unsloth.dat or
.gguf_file), raising a clear error if none exists
An importance matrix unlocks the IQ low-bit quants (iq2_xxs, iq4_xs, ...), which were hard
disabled before. They are gated: requesting one without an imatrix raises a clear error.
- _resolve_imatrix_file resolves path/True (PEFT base first, normalized via get_model_name,
derives unsloth/<base>-GGUF, copies out of the HF cache before renaming *.gguf_file).
- IMATRIX_QUANTS registry replaces the old commented-out IQ entries; save_to_gguf accepts a
resolved imatrix and threads it into the quantize calls.
- The --imatrix flag is emitted by unsloth_zoo's quantize_gguf (companion change). save.py
fails fast with an upgrade hint if the installed unsloth_zoo lacks the imatrix kwarg.
Tests: tests/saving/test_imatrix_export.py (CPU: resolution, repo derivation, IQ gate,
--imatrix wiring) wired into CI; tests/saving/test_gguf_export_and_inference.py extended with
GPU iq2_xxs/iq4_xs export + inference. Verified end to end on Llama-3.2-1B: imatrix
auto-downloaded, iq2_xxs/iq4_xs exported and run via llama.cpp.
Note: requires the companion unsloth_zoo quantize_gguf imatrix change.
* Address imatrix/compressed review feedback: unsloth org GGUF repo, fail-fast, calibration split
- imatrix auto-resolve (imatrix_file=True): derive the upstream repo as unsloth/<base>-GGUF
instead of <org>/<base>-GGUF, so official bases (e.g. meta-llama/Llama-3.1-8B-Instruct) find
the matching Unsloth GGUF imatrix repo rather than failing on a nonexistent meta-llama/...-GGUF.
- Resolve/validate the imatrix before the 16-bit merge in save_pretrained_gguf, so a bad path or
an unavailable upstream imatrix fails fast instead of after a long, multi-GB merge.
- Compressed calibration: when a Hub dataset has no "train" split, resolve the first split name
and slice it, instead of materializing the whole dataset just to take num_samples rows. Keeps
the original materialize-then-subselect path as a last resort.
Tests: add unsloth/<base>-GGUF mapping for an official base id, and create the imatrix file in the
quantize_gguf flag test (quantize_gguf now validates the imatrix exists).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: name the missing extractor when a Recipes upload fails
A missing optional dependency (pymupdf4llm for PDF, mammoth for DOCX) was
reported as a generic "Text extraction failed", which gives the user nothing to
act on. Catch ImportError and surface the package name instead.
* Studio: narrow missing extractor error handling
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* (feat) Add project names to studio training runs to avoid models being overwritten when doing similar training runs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/frontend/src/features/export/export-page.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/frontend/src/features/export/export-page.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/frontend/src/features/export/export-page.tsx
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* better project name sanitization, removed duplicated project name normalization
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* implement checkpoint scanning utilities and tests for base model inference
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard project_name against null and use leading important modifiers
* Fix/adjust training project names for PR #6512
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust training project names for PR #6512
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address project-name review feedback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Show project names in training recents
* Keep GGUF export directories source-specific
---------
Co-authored-by: NZ-Linix <nz-linix@outlook.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: NZ-Linix <linus.ordowski@outlook.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Speed up Studio desktop startup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address Studio startup review findings
* Keep orphaned run cleanup before readiness
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Improve code quality & performance: fix typos, compile regex & cache fields
- Fix typos across core files (repeatted → repeated, splitted → split, etc.)
- Compile regex patterns once as class attributes in TextPreprocessor
- Cache text fields/columns in RawTextDataLoader
- Improve comments (re-use → reuse)
* Use immutable raw text field constants
---------
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Local models in the Studio Hub tab (Custom folders, LM Studio, and
Local models sections) did not reveal their on-disk path on hover,
unlike the Fine-tuned rows which already do. Each of these rows maps
over a LocalModelInfo with a required path, so pass tooltipText built
from the model name and path via a small shared localPathTooltip
helper, matching the existing FT-row tooltip format.
Refs #6382
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* cascade user message deletion to include assistant reply
* Fix comment typo in delete-thread-message
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
FalconH1RMSNormGated is imported from transformers but never referenced in
unsloth/models/falcon_h1.py. The unused hoist trips the import-hoist lint gate
on the merge commit of every open PR (the gate lints PR-head merged into main),
so clearing it here unblocks those PRs.
* Studio: stop handing CI/user secrets to downloaded llama.cpp binaries
The macOS prebuilt path installs llama.cpp from the unslothai/llama.cpp
fork's latest (unpinned, mutable) release and then executes the
downloaded llama-server / llama-quantize binaries during install-time
validation. binary_env() built that child environment from a full
os.environ.copy(), so a compromised or tampered prebuilt would inherit
every secret in the process: HF_TOKEN and the workflow GitHub tokens in
CI, and HF / cloud credentials for end users running install.sh /
setup.sh.
We publish prebuilts daily, so pinning a release tag is not workable.
Instead, neutralise the impact: these binaries have no reason to read any
token, so strip secret-bearing variables (exact names plus
TOKEN/SECRET/PASSWORD/CREDENTIAL/PRIVATE_KEY/API_KEY markers) before
handing the env to a downloaded binary. The installer's own GitHub and
Hugging Face API calls read os.environ directly, so authentication and
release-API rate limiting are unaffected; PATH, LD_LIBRARY_PATH,
DYLD_LIBRARY_PATH and CUDA/ROCm vars are preserved. One change covers the
install-time validation path for all six macOS workflows and end users.
Follow-up (separate, sequenced): publish build-provenance attestations
from the fork's prebuilt workflows and verify them in CI, so a forged
release is rejected rather than merely starved of secrets.
* Strip KUBECONFIG, SSH_AUTH_SOCK, and PASSPHRASE-marked vars from binary env
Extend the deny-list per PR review: KUBECONFIG and SSH_AUTH_SOCK are
credential pointers/capabilities a downloaded binary never needs, and a
PASSPHRASE marker catches SSH_PASSPHRASE / GPG_PASSPHRASE. Tests updated.
* Studio: also scrub proxy/index env vars and URL-embedded credentials before running prebuilt binaries
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope mlx-ci secrets to the install + download commands for PR #6696
Drop the ambient step-level env block and pass GH/GITHUB/HF tokens only
on the installer and GGUF-download commands, so the directly invoked
llama-quantize / llama-server smoke runs see no secrets. The installer
still reads tokens from os.environ for the releases API and probe fetch.
* Trim verbose comments around the secret-env scrubber for PR #6696
Comment-only: condense the block comments added across this PR. Logic
unchanged (comment_tools.py check confirms code-only signature equal).
* Redirect HOME / cache pointers to an empty dir for prebuilt binaries (PR #6696)
Address Codex P2: stripping token env vars still let a tampered binary
read on-disk token stores (~/.cache/huggingface/token, ~/.aws/credentials,
~/.config/gh) through $HOME and the cache/config pointers. Point HOME plus
the HF / XDG / Windows home pointers at a single empty throwaway dir for
the downloaded-binary env. Defense in depth: a binary resolving the real
home via getpwuid is out of scope and needs OS sandboxing.
* Close residual credential-probe gaps for PR #6696
Address the latest Codex review:
- Strip token-only URL userinfo too (scheme://ghp_token@host), not just
the user:pass form.
- Redirect HOMEDRIVE/HOMEPATH alongside USERPROFILE so a Windows binary
cannot reconstruct the real profile from %HOMEDRIVE%%HOMEPATH%.
- Drop explicit credential-file pointers (NETRC, PIP_CONFIG_FILE,
DOCKER_CONFIG, GIT_CONFIG_GLOBAL) that live outside HOME.
- Probe ldd with a secret-free env: linux_runtime_dirs ran ldd on the
untrusted prebuilt with the inherited os.environ, and ldd may execute
the binary, so it could observe HF_TOKEN/GITHUB_TOKEN during the probe.
Factored the shared scrub into secret_free_environ().
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Separate token-bearing install from binary smoke; drop CI command files (PR #6696)
Address the two P1s in the latest review:
- mlx-ci: GitHub bakes secrets into the run-script text, so inline token
assignments in a step that later runs the prebuilt let a tampered binary
read them from the script. Split into a token-bearing install + download
step that never launches a binary, and a secret-free smoke step that runs
llama-quantize / llama-server.
- secret_free_environ now drops the GitHub Actions command files
(GITHUB_ENV, GITHUB_PATH, GITHUB_OUTPUT, GITHUB_STEP_SUMMARY, BASH_ENV) and
the smoke step unsets them, so a tampered prebuilt cannot inject PATH/env
into the later token-bearing MLX steps.
* Run the prebuilt smoke last, after all token-bearing steps (PR #6696)
Address the P1 workspace-poisoning vector: even with no secrets in its env,
a tampered prebuilt could edit the checkout or installed modules, and the
later HF_TOKEN MLX steps would then execute that poisoned code on push
builds. Move the prebuilt install + smoke to the end of the job so the
untrusted binary runs after every token-bearing step, leaving nothing for it
to corrupt. The MLX GGUF reload uses a source-built llama-cli, not this
prebuilt, so nothing depends on the earlier position.
* Trim comments around the secret-env scrubber and prebuilt CI steps (PR #6696)
Comment-only: condense the security-rationale block comments and merge the
duplicated prebuilt-step description in mlx-ci. Logic unchanged
(comment_tools.py check confirms the code-only signature is equal; install
suite still passes).
* Authenticate the GGUF export release-API lookup with the read-only GITHUB_TOKEN (PR #6696)
* Rename env scrubber off the secret-named identifier CodeQL flags as a clear-text sink (PR #6696)
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: restore tensor parallelism for vision/mmproj GGUFs
#6416 disabled --split-mode tensor for any GGUF that ships an mmproj projector to
dodge a GGML_ASSERT crash (#6415) seen on an older llama.cpp build with consumer
Blackwell (sm_120). The blanket skip silently dropped tensor_parallel=true for
every multimodal/MTP GGUF (e.g. Qwen3.6-35B-A3B-MTP); on hardware where the model
fits on one GPU the load then collapsed to a single GPU. mmproj + --split-mode
tensor works on current builds (verified end to end on B200/sm_100), so the skip
was disabling a working configuration.
Make the vision skip self-healing per binary:
- attempt tensor for vision models by default
- skip upfront only on a binary already seen to abort on tensor + mmproj this
session (_vision_tensor_split_aborts), recorded when such a launch crashes at
startup (_record_vision_tensor_split_abort). Process scoped, so a studio update
re-probes the new build. The route-level layer-split fallback stays the net.
- add _select_gpus(min_gpus=...) so a downgraded tensor request can keep multiple
GPUs instead of collapsing to one (default 1, no behavior change).
Add tests/test_tp_vision_regression.py: an AST allowlist guard over the
tensor_parallel drop sites (which would have flagged #6416), plus cache and
_select_gpus coverage. No GPU required.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review on vision tensor-parallel self-healing
Three fixes from the PR review:
- Record a vision-tensor abort only after every startup retry fails. The first
version cached the binary on the first spawn crash, which on every build
(including capable ones) is the benign --fit step abort that the existing
--fit off retry resolves. That poisoned the cache so the next vision load in
the same process skipped tensor. Recording now happens at the post-retry
failure block (after fit-off, flash-attn-off and MTP-drop), so a binary that
actually works is never cached.
- Gate the record on the tensor/mmproj crash signature: a hard signal fault
(_is_signal_crash) with no non-tensor cause (_output_has_nonprojector_diagnostic
excludes OOM and unknown-arch), so an OOM, bad extra args, or MTP/flash-attn
crash no longer marks an otherwise capable binary incompatible.
- Preserve the multi-GPU request on the cached downgrade. The vision gate now
raises _layer_min_gpus to the visible GPU count and threads it through the
layer-split GPU selection (_select_gpus min_gpus and the subset loops), so a
downgraded tensor request still spreads across GPUs instead of collapsing to a
single card the model happens to fit.
Verified two vision+tensor loads in one backend process both tensor-split across
4 GPUs (the benign fit abort no longer poisons the cache). Tests updated.
* Studio: harden vision tensor-parallel self-healing (review round 2)
Address the second review round on the vision/mmproj tensor-parallel fix:
- Preserve vision on the first load: a --split-mode tensor + --mmproj
GGML_ASSERT now raises so the route-level tensor->layer fallback retries
layer split with the projector intact, instead of stripping --mmproj and
silently loading text-only (which returned success and skipped the fallback,
losing vision on the first load until the next cached load).
- Symmetric multi-GPU preservation: the pooled-VRAM tensor downgrade now raises
_layer_min_gpus from the usable tensor GPUs like the vision downgrade, so it
no longer collapses a multi-GPU request to a single card.
- Base the layer fallback minimum on usable GPUs: _select_gpus caps min_gpus to
the count of cards with usable VRAM, so a downgrade never forces a nearly-full
card in (or trips --fit) just to hit the count.
- Re-probe after in-app updates: key the per-binary abort cache on (path, mtime)
like _capability_cache, so POST /api/llama/update swapping the binary in place
(no backend restart) re-probes the new build instead of inheriting the old
build's abort.
- Bump _layer_min_gpus for a known-bad vision binary independent of the tensor
drop, so the route fallback's layer retry (tensor already off) still spreads
across GPUs.
Adds deterministic non-GPU regression tests for each.
* Studio: gate cached-vision layer minimum on the current tensor request
The cached-vision _layer_min_gpus bump fired for every later vision load on a
binary recorded as tensor+mmproj-incompatible, including loads that did not
request tensor parallelism. A plain non-tensor vision load that fits on one card
would then grab every GPU just because an earlier TP attempt aborted in the same
backend process.
Re-tie the bump to the current tensor request (back inside the tensor-drop
guard), so only a downgraded tensor request preserves the multi-GPU spread; a
non-tensor vision load minimizes device count as before.
* Studio: preserve GPU count + confirm assert on vision tensor fallback
Third review round on the vision/mmproj tensor-parallel fix:
- Preserve multi-GPU on the first tensor->layer fallback. The route-level retry
runs tensor-off, so the in-function downgrades can't see the original tensor
request and a fits-on-one-card model loaded the first successful fallback on a
single GPU. The GGUF load closure now passes preserve_multi_gpu_on_layer (the
toggle asked for tensor, this attempt is layer) and load_model raises
_layer_min_gpus for it, so the downgrade still spreads across GPUs.
- Cap the auto-context layer loops to usable GPUs. They bypass _select_gpus, so a
raised _layer_min_gpus could force a nearly-full card into the subset (or trip
--fit). They now start from _auto_min_gpus, capped to the GPUs with usable VRAM.
- Confirm the tensor/mmproj assert before caching. Recording (and the layer-retry
raise) now require the ggml assert marker via _is_tensor_split_assert, not the
bare-signal predicate shared with the projector-incompat branch, so a corrupt
or too-new projector that SIGSEGVs independent of split mode is no longer cached
as tensor/mmproj-incompatible.
Adds deterministic non-GPU regression tests for each.
* Studio: extend multi-GPU fallback to extra/env tensor + overhead-aware cap
Fourth review round on the vision/mmproj tensor-parallel fix:
- Preserve multi-GPU fallback for all tensor requests, not just the UI toggle.
Tensor can also be requested via --split-mode tensor in extra args or an
inherited LLAMA_ARG_SPLIT_MODE=tensor env; the fallback retries those too, so
the preserve_multi_gpu_on_layer hint now keys off _effective_tensor_parallel
(the same check the fallback uses), comparing the overall request against the
current attempt instead of only request.tensor_parallel.
- Cap the auto-context layer fallback to GPUs that can pay the per-device layer
overhead. The cap counted any card with positive usable VRAM, so a nearly-full
GPU with a few MiB free stayed eligible and could be exposed to llama.cpp and
OOM. It now mirrors _select_gpus: a card counts only if usable VRAM exceeds the
per-device pipeline overhead.
Adds deterministic non-GPU regression tests for both.
* Studio: match the #6415 split-axis assert + replay layer-preserve hint
Fifth review round on the vision/mmproj tensor-parallel fix:
- Narrow the tensor/mmproj crash signature. _is_tensor_split_assert matched any
GGML_ASSERT/GGML_ABORT, so an unrelated invariant a corrupt GGUF or projector
trips with --mmproj present could be cached as tensor/mmproj-incompatible. It
now matches the specific #6415 warmup assertion
(GGML_ASSERT(src_ss[0].axis != GGML_BACKEND_SPLIT_AXIS_0) in ggml-backend-meta),
whose split-axis signature is inherent to tensor splitting. A reworded future
assert just re-crashes-then-falls-back (vision preserved via layer split)
instead of poisoning the cache for other models.
- Persist the layer-preserve hint for respawns. A successful tensor->layer
fallback committed _last_load_kwargs without preserve_multi_gpu_on_layer, so
_respawn_if_dead replayed only --split-mode layer + tensor_parallel=False and a
mid-session respawn of a fits-on-one-card model came back single-GPU. The hint
is now in the replay snapshot, so recovery keeps the multi-GPU placement.
Adds deterministic non-GPU regression tests for both.
* Studio: tighten comments on the vision tensor-parallel fix
Make the comments and docstrings added by this PR succinct: collapse the
multi-line block comments in llama_cpp.py / inference.py to one or two lines,
trim the verbose test docstrings (the names and assert messages already carry the
intent), and shorten the module docstring. No code changes; verified comment-only
with scripts/comment_tools.py check --strip-docstrings.
* Studio: cache vision tensor abort only on the split-axis token
_is_tensor_split_assert also accepted any GGML_ASSERT/GGML_ABORT from
ggml-backend-meta, but that file holds many asserts, so an unrelated
scheduler/projector/model invariant on an --mmproj launch could cache the binary
as tensor/mmproj-incompatible and make later compatible vision models skip tensor
parallelism. Match the GGML_BACKEND_SPLIT_AXIS_* token itself (unique to the
#6415 warmup assert), not the source file name.
* Studio: don't leak the httpx test stub into later tests
The regression module stubbed httpx via sys.modules.setdefault, which installs
the lightweight stub even when real httpx is present but not yet imported. The
stub then persists for the whole pytest process, so provider/HF tests collected
later (importing httpx or huggingface_hub.errors) got a module missing
HTTPError/Response. Mirror the neighboring llama_cpp helper tests: import real
httpx first and only fall back to a stub on ImportError.
* Studio: latch the #6415 tensor-split abort on the first spawn, key it per model
The self-heal recorded the --split-mode tensor abort only in the post-retry
failure block, after the flash-attn-off retry. But SPLIT_MODE_TENSOR requires
flash_attn, so the flash-off retry can't run tensor and its output no longer
carries the warmup split-axis assert (ggml-backend-meta :541). The record
therefore never fired on the real reproducer and the crash loop repeated on
every load (reported by oobabooga on #6659).
Latch instead on the first spawn that shows the signal crash + split-axis
marker: record it, kill the process, and raise straight to the route's layer
fallback, skipping the futile flash-attn/MTP retry ladder for this crash.
The crash is a tensor-split geometry limit (e.g. MQA n_head_kv=1 splitting to
GGML_BACKEND_SPLIT_AXIS_0), not a vision/mmproj property: it reproduces without
--mmproj and even single-GPU tensor. So drop the vision/mmproj scoping, rename
_vision_tensor_* -> _tensor_split_*, and key the session cache on
(binary, mtime, model) rather than (binary, mtime) so one model's abort no
longer skips tensor for every other model on the same build.
Regression tests updated to pin the early-spawn record, the per-model cache,
and that an unrelated ggml-backend-meta assert is not treated as the marker.
* Studio: reload on explicit tensor-off after a multi-GPU layer fallback
When a tensor load is downgraded to layer but kept multi-GPU to honor the
tensor request (preserve_multi_gpu_on_layer, the geometry-cache gate, or the
budget downgrade), the server reports tensor_parallel=False with --split-mode
layer stored. A later Apply that explicitly turns the tensor toggle off then
matched the loaded state and deduped to already_loaded, so Studio kept the
fallback's all-GPU CUDA_VISIBLE_DEVICES placement instead of re-selecting
normal placement (a single GPU for a model that fits on one card).
Latch a _layer_preserves_tensor_intent flag in load_model whenever a tensor
request is downgraded to layer with the multi-GPU floor raised
(_layer_min_gpus > 1), clear it when tensor stays on or on unload, and force a
reload in _request_matches_loaded_settings when the user explicitly turns the
tensor toggle off while that flag is set. An Apply that does not touch the
toggle still dedupes, so a working multi-GPU layer server is not churned.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address reviewer.py findings on the tensor-split self-heal
P1 (dedup): tensor intent can be dropped via extras, not only the toggle. An
explicit llama_extra_args=["--split-mode", "layer"] matches the stored fallback
extras, so _request_matches_loaded_settings deduped to the preserved all-GPU
placement instead of reloading. Now reload when layer_preserves_tensor_intent
and the user explicitly drops tensor via the toggle OR via extras
(_effective_tensor_parallel of the explicit extras is false).
P1 (downgrade symmetry): the len(tp_gpus) < 2 compute-buffer downgrade cleared
tensor_parallel without raising _layer_min_gpus, unlike the budget and geometry
downgrades. GPUs below tensor's replicated compute-buffer reserve can still take
layer split's lower overhead, so keep the multi-GPU request (len(gpus) >= 2) and
let _select_gpus cap unusable cards.
P2 (cache key): key the tensor-split abort cache on st_mtime_ns, so a binary
replaced in place within the same second after an abort is re-probed instead of
inheriting the stale entry.
P2 (test hygiene): load routes/inference.py via importlib in the regression
tests instead of importing the routes package, which runs routes/__init__.py and
pulls in every router (e.g. python-multipart). Added regression coverage for the
extras-off reload, the compute-buffer multi-GPU preservation, and the same-second
nanosecond cache invalidation.
* Studio: record the tensor-split abort on the Windows CRT abort exit too
The first-spawn split-axis latch only recorded when _is_signal_crash matched
(POSIX signal or 0xC0000000+ NTSTATUS). On MSVC builds GGML_ASSERT terminates
through the CRT abort() path with exit code 3, which is neither, so the cache
never filled on Windows and every later load of the same bad binary/model
repeated the tensor crash before falling back to layer.
The split-axis marker is definitive, so accept either a signal crash or the
Windows abort() exit (3) when the marker is present. Add _is_abort_exit and a
unit test, and assert the early latch honors it.
* Studio: fix UnboundLocalError on --fit-on fallback, reload backend fast path
Two follow-ups from review on the tensor-split self-heal:
UnboundLocalError: _layer_min_gpus was initialized inside the GPU-selection try.
If NVML probing or GGUF/mmproj sizing raised, the except path logged "using
--fit on" and fell through to the command builder, where the new
self._layer_preserves_tensor_intent = _layer_min_gpus > 1 then raised, turning a
safe --fit-on layer fallback into a hard load failure. Bind _layer_min_gpus
before the try so the except path always has it.
Backend fast path: _request_matches_loaded_settings forces a reload when a
preserved tensor->layer fallback gets an explicit tensor-off request, but
load_model's own _already_in_target_state still matched the tensor-off/layer
settings and short-circuited, so the placement re-selection never ran. Mirror
the guard there: reload when layer_preserves_tensor_intent and the request drops
tensor intent. The flag clears on that reload, so there's no loop.
Added regression coverage for both.
* Studio: testable tensor-split record decision; skip futile fit-off retry
Follow-ups from a deeper review of the tensor-split self-heal:
Extract the record decision into _should_record_tensor_split_abort(rc, output)
(marker AND (signal crash OR Windows abort)) and call it from the early latch.
The combined boolean was only covered by source-inspection substring checks, so
an or->and typo would silently stop recording on Windows (CRT abort exit 3 is
not a signal) with every test still green. Add a behavioral test over the
POSIX / Windows / NTSTATUS / clean-exit / SIGKILL / no-marker matrix.
Skip the --fit off retry inside _spawn_and_wait when the crash already shows the
split-axis marker: that abort is fit-independent, so the retry just warms up and
crashes a second time before the latch records it. Skipping it lets the caller
latch immediately and corrects the latch comment.
Also clarify the dedup-guard comments (toggle read from model_fields_set vs
extras via _effective_tensor_parallel without env; the backend fast path is
intentionally broader and only ever forces a reload).
* Studio: don't reload-loop tensor-off requests under env tensor
The preserved-fallback reload guard fired on the raw tensor toggle, ignoring
LLAMA_ARG_SPLIT_MODE=tensor. For an env-driven tensor user, an explicit
tensor_parallel=false request then forced a reload that re-engaged tensor via
the env and re-created the same preserved layer fallback, so every /load
reloaded -- bypassing the env-downgrade matching that exists to avoid exactly
this loop.
Gate the guard on the env-aware effective tensor state: reload only when an
explicit toggle/extras change leaves _effective_tensor_parallel (which consults
the env) off. If the env still forces tensor, fall through to the existing
env-downgrade match, which dedupes instead of looping. Added a regression test
with LLAMA_ARG_SPLIT_MODE=tensor set.
* Studio: tighten comments and test docstrings on the TP self-heal
Condense the verbose comments and test docstrings added across the review rounds
into fewer, succinct lines without changing their intent: the early-latch and
downgrade-site rationale, the cache/key and helper docstrings, the dedup-guard
comments, and the per-test docstrings. No code changes (AST-verified comments
and docstrings only); tests and lint unchanged.
* Studio: clear preserved tensor flag on diffusion; carry it across non-drop reloads
Two follow-ups on the preserved-fallback machinery:
Diffusion: the DiffusionGemma path early-returns from load_model before the
command builder that sets/clears _layer_preserves_tensor_intent, so the flag
from a prior tensor->layer fallback leaked onto a later diffusion load and
forced needless reloads of the diffusion server on tensor-off/extra Applies.
Clear it when starting diffusion.
Settings reload: the preserve hint was recomputed only from the new request, so
a reload for an unrelated setting (e.g. max_seq_length) with the tensor toggle
omitted dropped a preserved multi-GPU layer placement back to one GPU. Carry
llama_backend.layer_preserves_tensor_intent into the hint when the request is
not an explicit tensor-off/extras-off drop, so a fitting model stays multi-GPU.
Added regression tests for the diffusion clear, the carry-forward, and the
updated tensor-intent computation.
* Studio: gate the preserve carry-forward on the same model being loaded
The tensor-intent carry-forward read llama_backend.layer_preserves_tensor_intent
without checking it belonged to the model being loaded. On a direct model switch
(load B without an explicit /unload of A), the flag is still set from A's
downgrade (it isn't reset until B's load_model reaches the command builder, after
the route reads it), so a plain load of B got preserve_multi_gpu_on_layer=True
and was spread across all GPUs even though it fits on one and the user never
requested tensor for it. The backend dedup doesn't have this leak (it checks
model_identifier first); the leak was only in the route hint.
Extract the decision into _carry_preserved_tensor_intent(preserved, same_model,
explicit_drop) and gate it on the backend still holding the same model. Add a
behavioral truth-table test (catches a `not` inversion and a missing same-model
guard) and tighten the compute-buffer downgrade test to bound its source window.
* Studio: match the HF quant too when carrying preserved tensor intent
The same-model guard on the preserve carry-forward compared only model_identifier,
which is variant-agnostic for HF repos. A later load of the same repo with a
different gguf_variant (which already bypassed dedupe on the variant mismatch)
was treated as the same model, so a request that omits tensor settings inherited
the prior variant's preserved intent and forced multi-GPU layer placement for a
quant that never requested tensor. Also require the loaded hf_variant to match for
HF repos (local direct-file loads already differ by model_identifier path). Added
a regression test for the variant guard.
* Studio: match the loaded GGUF by path too when carrying preserved tensor intent
A local directory holding multiple GGUF variants keeps one variant-agnostic
model_identifier (the directory) while config.gguf_file selects the file, so the
same-model guard let variant B inherit variant A's preserved tensor->layer
fallback and forced B onto multi-GPU. Mirror _already_in_target_state's identity
logic: match by resolved path when both sides have a local file, else by HF
variant. #6659
* Studio: let implicit same-settings reloads dedupe after a preserved fallback
The backend _already_in_target_state mirror forced a reload on ANY effective
tensor-off request once a tensor->layer fallback was preserved. In the HF
auto-pick / local-directory flows the route-level dedup is skipped, so an
identical /load with tensor omitted reached this guard and reloaded every time
even without an explicit drop. Thread the route's preserve_multi_gpu_on_layer
decision in so only an explicit drop reloads; implicit carry-forward dedupes. #6659
* Studio: only an explicit tensor/split-mode change drops preserved intent
The explicit-drop test treated request.llama_extra_args is not None as a drop,
so a same-model reload that merely added an unrelated pass-through arg (e.g.
--top-k 20) without touching the tensor field or --split-mode disabled the
carry-forward and collapsed a fitting model back to one GPU. A drop now requires
an explicit tensor_parallel field change or a non-tensor --split-mode override,
via a shared _is_explicit_tensor_drop helper used by both the already-loaded
dedup and the load carry-forward so the two readers agree. #6659
* Studio: treat an explicit clear of extras as a tensor drop
When tensor intent was extras-driven (--split-mode tensor) and fell back to a
preserved layer split, a later request that explicitly clears extras
(llama_extra_args=[]) but omits tensor_parallel left the empty list with no
split-mode override, so the carry-forward kept the model pinned multi-GPU instead
of returning to normal layer selection. _is_explicit_tensor_drop now also counts
an explicit empty-list clear as a drop, while an unrelated extra (--top-k) or
inherit (None) still carries the preserved intent. #6659
* Studio: don't treat the UI's tensor_parallel echo as a tensor drop
The Studio frontend always sends tensor_parallel and copies the /load response's
resolved value back into its state, so after a tensor->layer fallback every
ctx/settings reload carries tensor_parallel=false even though the user never
changed it. Keying the drop on the field (or on an empty extras clear) collapsed
the preserved multi-GPU placement on the next reload. A fallback also always
stores --split-mode layer, never a tensor split mode, so a clear never wipes
tensor intent. _is_explicit_tensor_drop now drops only on an explicit non-tensor
--split-mode override; the bare field echo, an empty clear, an unrelated extra,
and inherit all keep the preserved placement, and --split-mode tensor /
tensor_parallel=true re-engage tensor. #6659
* Studio: match the resolved config.identifier when carrying tensor intent
The same-model guard for the carry-forward compared the raw request id, but
ModelConfig.from_identifier normalizes it (adds the unsloth/ prefix for a
shorthand, fixes repo-id case) before load_model stores config.identifier. So a
ctx/settings reload using the shorthand id missed the match, dropped
_carry_preserved_tensor_intent, and could collapse a preserved multi-GPU layer
placement to one GPU. Compare against config.identifier (what the backend stores),
keeping it symmetric with _already_in_target_state. #6659
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix fast_inference crash on ABI-broken vLLM: force-load compiled extensions in the broken-vLLM probe
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Broaden broken-vLLM probe: catch non-libcudart .so failures and _moe_C_stable_libtorch
* Revert stray reformat of the PDL fix log line
* Trim verbose comments in the broken-vLLM probe
* Drop non-existent vllm._moe_C_stable_libtorch from the broken-vLLM probe
* Shorten comments in broken vLLM extension detection
Condense the docstrings and inline comments for the lazy-loaded vLLM probe
and the new regression test while keeping the rationale. Comments only, no
code changes (verified with an AST signature check and the existing tests).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Two intermittent Studio CI failures, both runner-environment flakes unrelated
to test logic:
Windows 'Studio install + inference without Visual Studio': the 'Hide Visual
Studio + CMake' step renames C:\Program Files\Microsoft Visual Studio to
simulate a host with no build tools. A background handle on a Program Files
directory (Defender scan or an MSBuild node) makes Rename-Item intermittently
fail with 'Access is denied', and $ErrorActionPreference = Stop turns that into
a hard job failure. Wrap the VS and cmake renames in both Hide steps in a short
Rename-WithRetry (6 tries, 3s apart) to ride out the transient lock.
macOS 'Chat UI Tests': the re-login goto to /login can be interrupted by the
SPA auth guard redirecting to the same /login URL, which Playwright reports as
'Navigation to .../login is interrupted by another navigation to .../login'.
The goto already tolerated ERR_ABORTED; broaden it to also tolerate the same-URL
interrupt (the password-field wait right after confirms we landed on /login),
and add the same signature to the two Playwright flake-retry harnesses as a
safety net for any other navigation.
Validated: playwright_chat_ui.py parses + byte-compiles, both workflow YAMLs
parse, bash -n on the retry harnesses, PowerShell AST parse on all pwsh steps,
and a functional check of Rename-WithRetry (succeeds, and rethrows after
exhausting retries).
On torch >= 2.11 torchao tries to dlopen each prebuilt _C*.so and logs a
per-file "Failed to load .../_C*.so" WARNING via the torchao logger when one
cannot load. This happens on an ABI tag mismatch in the prebuilt wheel (for
example a cp310 .so under a cp312 runtime, as on Colab) or when the kernel
targets an arch the GPU does not have (mxfp8 needs FP8 hardware, _C_cutlass_90a
is Hopper/SM90 only). torchao falls back to its non-cpp paths and Unsloth's
bnb-4bit / Triton kernels do not use these, so the warning is cosmetic.
Add a HideLoggingMessage filter on the same torchao logger that already filters
the torch < 2.11 "Skipping import of cpp extensions" message, so only these
records are dropped rather than raising the whole logger to ERROR.
pre-commit-ci[bot]