* add models for /update endpoint
* add logic for identifying out of date hf models
* add endpoint for updating hf models
* add relevant field to GgufVariantDetail
* make exception handling better
* add update_available flag for cached_models, and moved /update endpoint from inference -> models
* hook up /update endpoint on the frontend
* implement update scenarios for the model picker
* fix bug where downloaded flag for an older revision was being wrongly set to false
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix import and make hf calls async
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* remove has_vision from UpdateRequest
* fix ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* clear cancel event before updating gguf variant
* set _cancel_event back if it was set initially
* add hf_token to get_paths_info
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: harden model update endpoint and update checks
- update_hf_model: pass snapshot_download local_dir (local_path is not a
valid kwarg and 500s when updating bicodec audio models)
- get_gguf_variants: wrap the remote update check so a network, rate-limit,
gated, or offline failure degrades to "no update info" instead of failing
the whole variant listing, matching list_cached_models
- add regression tests for both paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: HF model update detection and Update action for cached models
Surface an "Update available" cue and a managed Update action for cached
on-device models. /api/hub/update-status compares each cached main GGUF
file's local blobs against the remote main revision using set membership
across all cached revisions, so a repo that was already updated (and still
holds the old snapshot alongside the new one) is not falsely flagged.
The Update action re-downloads through the download manager so it shows in
the Downloads panel with progress and cancel. The frontend wires the Update
button into the GGUF, on-device, and model-selector cards and keeps the
quant label fully visible when the action buttons crowd the row.
Adds regression tests for the multi-revision update check.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: accept force_download kwarg in hf_xet_fallback test double
The download seam now passes force_download to the attempt callable; the _FakeAttempt mock did not accept it, failing 6 tests with TypeError. Add the keyword (default False) so the scripted-results double matches the seam.
* Fix Studio model update regressions
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address Studio update review feedback
* Address Studio update edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Share GGUF update status helper
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix GGUF update detection and cache cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix cached GGUF update badges
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: shimmyshimmer <107991372+shimmyshimmer@users.noreply.github.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: wire imatrix GGUF option and FP8/NVFP4 compressed export into the export UI
GGUF export gains an importance-matrix toggle. When enabled it auto-downloads the
upstream Unsloth imatrix for the base model (or uses a custom path), which unlocks
the IQ low-bit quants iq2_xxs, iq2_m, iq3_xxs and iq4_xs. Merged export gains an
FP8 / NVFP4 compressed-tensors precision selector that runs llm-compressor for vLLM.
Backend threads imatrix_file through routes -> orchestrator -> worker -> export_gguf
(both the local save and the hub push), and maps the new compressed format_type
values onto the fp8/nvfp4 save_method, reporting the "<dir>-<suffix>" sibling output
directory. Frontend adds the imatrix Switch on the GGUF card and a merged precision
picker on the merged card, threaded through the export runtime store.
Depends on unslothai/unsloth#6706 (save.py imatrix_file and compressed-tensors
export) and unslothai/unsloth-zoo#839 (quantize_gguf imatrix flag).
* Studio export: guard imatrix/compressed against older unsloth builds and force imatrix for IQ quants
Addresses review feedback on the export wiring:
- GGUF: pass imatrix_file only when set, so a plain no-imatrix export (e.g. Q4_K_M) no
longer fails with an unexpected-keyword error against an unsloth build that predates the
imatrix_file parameter. When imatrix is requested but unsupported, return a clear
upgrade message instead of a TypeError.
- Merged: gate FP8/NVFP4 compressed-tensors export on the installed unsloth actually
supporting it, returning a clear message rather than a cryptic save_method failure.
- Frontend: IQ quants (iq2_xxs, iq2_m, iq3_xxs, iq4_xs) are imatrix-only, so force the
imatrix on when one is selected and lock the toggle, instead of submitting an IQ quant
with no imatrix that llama.cpp would reject.
Extends the backend tests for the new capability guards and the conditional kwarg wiring.
* Studio: upload compressed merged models to the Hub without recompressing
For an FP8/NVFP4 Hub export the model is already produced locally in the "<dir>-<suffix>"
output. Uploading it directly with HfApi.upload_folder (mirroring export_base_model) avoids
re-running the expensive compressed-tensors quantization a second time inside
push_to_hub_merged, which for NVFP4 also re-runs calibration and risks OOM. Falls back to
push_to_hub_merged when there is no local compressed output to reuse.
* (feat) Add project names to studio training runs to avoid models being overwritten when doing similar training runs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/frontend/src/features/export/export-page.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/frontend/src/features/export/export-page.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/frontend/src/features/export/export-page.tsx
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* better project name sanitization, removed duplicated project name normalization
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* implement checkpoint scanning utilities and tests for base model inference
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard project_name against null and use leading important modifiers
* Fix/adjust training project names for PR #6512
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust training project names for PR #6512
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address project-name review feedback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Show project names in training recents
* Keep GGUF export directories source-specific
---------
Co-authored-by: NZ-Linix <nz-linix@outlook.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: NZ-Linix <linus.ordowski@outlook.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: require signed capability tokens for /p preview links
The public /p preview routes added in #6486 run model load and chat
generation as the admin user with no authentication. The only gate is the
preview ref, a deterministic outputs-root path (run or run/checkpoint) that
is guessable rather than secret. On a network-reachable Studio (--secure
tunnel or -H 0.0.0.0), an unauthenticated caller who guesses a ref can
consume GPU and probe a private fine-tuned checkpoint.
Make the share link an unguessable, revocable capability:
- Sign the canonical ref with a dedicated server-side secret (HMAC-SHA256,
stored in app_secrets, independent of the JWT/login secret).
- Require a valid token on every /p chat, models, and page request before
resolving a checkpoint or loading a model; missing or invalid tokens get a
generic 404 so the surface never confirms a ref exists.
- Accept the token via ?k= (browser link and preview page) or
Authorization: Bearer (OpenAI-compatible clients).
- Rotate the secret to revoke every outstanding link
(POST /api/settings/preview-links/rotate).
- Clamp preview generation (max_tokens/max_completion_tokens <= 1024, n = 1)
and set Referrer-Policy: no-referrer on the page so the token is not
leaked via Referer.
Training history hands the authenticated owner the signed token, and the
copy-link button builds /p/{ref}?k={sig}.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor a lower caller token limit in the preview clamp
Codex review: when only the legacy max_tokens was sent, the clamp left
max_completion_tokens at the 1024 default, and _effective_max_tokens prefers
max_completion_tokens, so a request like max_tokens=16 could still generate up
to 1024 tokens. Derive one effective limit (max_completion_tokens wins, else the
legacy max_tokens) and pin both fields to it so a caller's lower limit is kept.
* Studio: add preview kill switch, rate limit, and revoke-links UI
Follow-ups to the /p preview capability work:
- Public-sharing kill switch: a persisted setting (default on) gates the public
/p surface. When off, every preview request 404s even with a valid token, and
the owner UI stops offering share links. GET/PUT /api/settings/preview-sharing;
enforced in _verify_or_404.
- Per-IP rate limit on the preview chat route: a coarse in-process sliding-window
limiter (20 req/min/IP) returns 429 + Retry-After before the GPU lock is taken.
Client IP honors X-Forwarded-For only when UNSLOTH_STUDIO_TRUST_FORWARDED is
set, matching the login limiter's trust model.
- Settings UI: a "Preview sharing" section with the public-sharing toggle and a
"Revoke all preview links" button (confirm dialog) that rotates the secret.
Tests cover the kill switch (404 when off), the 429 path, the sliding window,
client-IP trust behavior, and the setting default.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix preview-fields sharing arg and refresh sigs after revoke
Codex review:
- P1: get_training_run_detail and update_training_run called _preview_fields
with only output_dir after it gained a required sharing_on parameter, raising
a 500 TypeError once get_run succeeded. Pass get_preview_sharing_enabled() at
both sites; add a detail-endpoint regression test.
- P2: after rotating the preview secret from settings, the history grid still
held stale preview_sig values, so a freshly copied link would 404. Emit
emitTrainingRunsChanged() after a successful revoke so the grid refetches
freshly signed refs.
* Studio: harden preview sharing controls (Codex review)
- Fail closed: a read failure on the preview-sharing kill switch now returns
False instead of defaulting to enabled, so an unavailable settings DB can't
reopen the public surface. A missing key still defaults to enabled.
- Per-IP rate limit behind the managed Cloudflare tunnel: client_ip now honors
CF-Connecting-IP when the socket peer is loopback, so tunneled visitors are
keyed by their real IP instead of collapsing onto the local cloudflared peer.
- GET /p no longer mints key/share_url when sharing is disabled; it returns
sharing_enabled=false so clients don't distribute links that 404.
- Settings UI: toggling public sharing emits the training-runs-changed event so
the history grid shows/hides Copy preview link without a manual refresh.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden preview rate limiter and IP keying (Opus review)
From a two-agent review of the PR:
- Rate limiter no longer evicts an active bucket when the table is full: a flood
of distinct keys could otherwise cycle out a throttled bucket and reset its
counter. Evict only aged-out buckets; if the table is full of live clients,
fail closed (deny the new key) instead.
- client_ip keys on the rightmost (proxy-appended) X-Forwarded-For hop when the
trust env is set; the leftmost is client-spoofable. Documented the
append/overwrite-proxy assumption.
- _verify_or_404 checks the capability token before the kill-switch DB read, so
unauthenticated /p spam can't be used as an unbounded settings-DB sink and the
response is identical regardless of the sharing on/off state.
Tests: nested run/checkpoint happy path + wrong-ref rejection, the eviction
fail-closed behavior, and route-level coverage for the rotate / preview-sharing
settings endpoints.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant
(fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao
NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8
stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run
before placement; status reports the mode actually engaged. This is the lean
realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the
3045-line port.
Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the
bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE
option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and
both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default);
size it per model with the Phase 5 quality harness. diffusion_bench gains
--text-encoder-quant.
129 CPU tests pass.
Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3)
storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16
compute dtype while normalisations and embeddings stay full precision. Applied
before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder
dense). status reports which encoders were cast.
Verified on Z-Image (B200, balanced/group mode where the encoder stays resident):
generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload)
at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR
vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off
by default and documented as such, with the Phase 5 harness to size the cost.
127 CPU tests pass.
Add a speed_mode knob (off by default, so the render path stays bit-identical):
default applies channels_last VAE + regional torch.compile of the denoiser's
repeated block where eligible; max also enables TF32 matmul and fused QKV. Regional
compile is gated off for the GGUF transformer (dequantises per-op) and for families
flagged not compile-friendly (a new supports_torch_compile flag, False for Z-Image),
so it activates automatically only once a non-GGUF bf16 transformer is loaded. Speed
optims run before placement/offload, per the diffusers composition order. status now
reports speed_mode + the optims actually engaged.
Verified on Z-Image (B200): default -> ['channels_last'], max -> ['channels_last',
'tf32'], compile correctly skipped for GGUF; generation works in every mode.
121 CPU tests pass.
Add a streamed 'group' offload tier (diffusers apply_group_offloading, block_level,
use_stream) that keeps the transformer flowing through the GPU a few blocks at a
time while the text encoder / VAE stay resident, and fix VAE tiling to drive the
VAE submodule (pipelines like Z-Image expose enable_tiling on pipe.vae, not the
pipeline). apply_memory_plan now returns the (policy, tiling) actually engaged so
status never overstates either, and group falls back to whole-module offload when
the transformer can't be streamed.
Measured on Z-Image (B200), all lossless (PSNR inf vs resident): balanced/group
cuts generation peak VRAM 32% (15951 -> 10840 MB) at near-resident speed (2.07 ->
2.99s); low_vram/model cuts it 48% (-> 8318 MB) but is slower (7.99s). Mode names
now match that tradeoff: balanced = stream the transformer, low_vram = offload
every component. auto picks group when the companions fit resident, else model.
112 CPU tests pass.
Add a lean, backend-agnostic memory policy that picks a CPU-offload policy and
VAE tiling/slicing from measured free device memory vs the model's estimated
resident footprint, then applies it to the built pipeline. auto stays resident
when the model fits (byte-identical to the prior resident path), and falls to
whole-module offload when tight; fast/balanced/low_vram are explicit overrides.
Sequential submodule offload is unreliable for GGUF transformers on diffusers
0.38, so it falls back to whole-module offload and status reports the policy
actually engaged.
Verified on Z-Image-Turbo Q4_K_M (B200): auto reproduces the resident image with
no VRAM/latency regression (PSNR inf); balanced/low_vram cut generation peak VRAM
47.9% (15951 -> 8318 MB) with byte-identical output, at the expected latency cost.
73 prior + 35 new CPU tests pass.
* checkpoint preview endpoint
* harden new preview endpoints
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address review
* Studio preview: pin adapter, guard streaming submit, robust copy-link
Harden the public per-checkpoint preview surface:
- Pin use_adapter=True in the preview payload sanitizer. Otherwise an
unauthenticated /p caller can POST use_adapter=false, which calls
disable_adapter_layers() on the shared in-memory model without restoring
it; since load_model skips reloads for the same checkpoint, every later
visitor (the page never sends the field) keeps getting base-model output
instead of the fine-tuned checkpoint. Forcing it on also re-enables a
previously disabled adapter and no-ops on merged checkpoints.
- Ignore preview-page submits while a response is streaming. The send
button was disabled but the Enter handler still called requestSubmit(),
so a second request could start before the first reply landed in msgs and
reorder the chat history. Both the keydown and submit handlers now honor
the disabled button.
- Keep the cloudflare-URL polling loop alive across transient startup fetch
errors instead of letting one rejection halt it.
- Build the copy-link from a backend preview_ref (output dir relative to
outputs_root, gated on previewability and the two-segment /p route limit)
so a nested output dir no longer copies a basename-only link that 404s.
Expose preview_ref on training run summaries.
Add route-level security tests (path traversal, payload sanitization,
asset containment, CSP header, HTML title escaping, streaming lock held
until drained) and preview_ref unit tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio preview: Safari-safe submit and adapter pin only for LoRA
Follow-ups from cross-browser and route simulations:
- Preview page: send the message from a shared send() helper called by both
the form submit and the Enter key, instead of form.requestSubmit(). The
latter throws on Safari < 16 and older iOS, which broke Enter-to-send there.
Verified across Chromium, Firefox and WebKit with Playwright.
- Only pin use_adapter=True when the resolved checkpoint is a LoRA adapter
(adapter_config.json present); for a merged checkpoint strip it to None.
A merged model has no adapter to toggle, so forcing it on only produced a
per-request "not a PeftModel" warning. The cross-request base-model
contamination fix still holds for LoRA previews.
Add a merged-checkpoint test asserting use_adapter is stripped to None.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio preview: trim verbose comments
Tighten comments across the preview routes, page, checkpoint helpers, and tests
to short single-line notes; drop ones that just restate the code. No behavior
change (verified comment/docstring-only with comment_tools.py check).
* Harden preview routes for PR #6486
- Return a generic 400 detail on a rejected preview path so the public /p
route never echoes the absolute install path (the real reason is logged
server-side instead).
- Strip confirm_tool_calls, session_id and rag_scope in the preview payload
sanitizer so the public surface stays inert regardless of the tool gate.
- Use Path.is_relative_to for the asset containment check, matching the rest
of the codebase.
- Add img-src 'self' and font-src 'self' to the preview page CSP.
- Preview page: on a mid-stream error keep the streamed text, flag the break,
and restore the prompt so the user can retry; drop the unused --font-sans var.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Fix Gemma 4 GGUF OpenAI API streams
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Avoid duplicate Responses stream disconnect watcher
* Keep reasoning-only Responses output hidden
* Address Gemma stream review comments
* Avoid Responses stream task-group cleanup
* Harden OpenAI chat completion streams
* Address OpenAI stream review issues
* Clean up Studio OpenAI stream helpers
* Fix Studio passthrough cold stream timeout
* Fix tool parser compatibility exports lint
* Preserve audio stream disconnect cancellation
* Avoid synthetic finish after passthrough errors
* Address stream cleanup and Gemma parser reviews
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gemma 4: parse bare-string tool args and keep safetensors tools for native <|tool_call>
- Quote bare unquoted string values in Gemma native tool-call args (e.g.
{location:Tokyo,unit:celsius}) so they parse; JSON scalars stay typed.
- Stop _detect_safetensors_features from suppressing supports_tools for
templates that emit Gemma native <|tool_call>, which the shared parser
now reads.
- Add tests for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden Gemma tool-call parsing and stream-error detection
Address three issues in the Gemma-native tool-call path:
- _quote_gemma_object_keys stopped a bare (unquoted) string value at the
first comma, so an argument like `location:New York, NY` was split
mid-value and the synthesized JSON failed to parse, dropping the whole
tool call. A bare value now ends only at `}` or a comma that begins the
next `key:` pair.
- parse_tool_calls_from_text scanned the entire response for Gemma markers
even inside a tool call already parsed from a `<tool_call>{...}` JSON
block, so a marker-like string inside an argument (data) was promoted to
a second, unintended tool call. Matches inside an already-consumed call
span are now skipped.
- _openai_passthrough_stream relied on _monitor_openai_sse_line to flag a
stream error, which returns early when monitor_id is None
(skip_api_monitor), so an upstream error chunk left saw_stream_error
unset and the synthetic-finish guard emitted a successful finish_reason
after a failed stream. Error chunks are now detected independently of API
monitoring.
Adds tests/test_gemma_tool_parse_edge_cases.py covering the comma and
marker-injection cases.
* Emit the terminal finish_reason chunk in GGUF streams
The OpenAI chat-completions GGUF tool stream and plain stream both built a
final ChatCompletionChunk carrying finish_reason but never yielded it, so
clients received the optional usage chunk and [DONE] with no chunk carrying
finish_reason. OpenAI-compatible consumers rely on that terminal choice to
distinguish stop/length/tool_calls. Yield it before the usage chunk and
[DONE], matching the other streaming paths.
* Parse tool calls in document order and skip nested markers both ways
Unify the JSON- and Gemma-format tool-call passes into a single
position-ordered scan:
- Calls are now emitted in byte order across both formats, so a mixed
output like `<|tool_call>call:create{...}<tool_call|> ... <tool_call>
{"name":"read",...}</tool_call>` executes create before read, matching
the order they appear in (tools run in returned order).
- A candidate that starts inside an already-accepted call's span is
skipped, in both directions: a JSON marker inside a Gemma argument and a
Gemma marker inside a JSON argument are treated as data, not promoted to
a second executable tool call.
Extends tests/test_gemma_tool_parse_edge_cases.py with the ordering and
JSON-in-Gemma nesting cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Quote bare Gemma array elements; order finish before trailing usage
- _quote_gemma_object_keys skipped array values, so a Gemma call with a
bare-string array argument like labels:[bug,ui] produced invalid JSON and
the whole tool call was dropped. Array values are now scanned and bare
string elements quoted, while numbers, quoted strings, and JSON literals
are preserved.
- In the OpenAI passthrough stream, a trailing usage-only chunk
(stream_options.include_usage) that arrived before any finish chunk was
relayed before the synthetic finish, producing usage -> finish -> [DONE].
Emit the synthetic finish before that usage chunk so the order matches the
other streams (finish -> usage -> [DONE]).
Extends tests/test_gemma_tool_parse_edge_cases.py with the bare-array cases.
* Harden Gemma array parsing, XML-parameter guard, and stream teardown
Address five review findings on the Gemma tool-call and OpenAI passthrough
streaming paths:
- parse_tool_calls_from_text collected JSON and Gemma markers without the
_inside_open_parameter guard, so a marker embedded in an existing
<function=...><parameter=...> value was promoted to a separate tool call.
Candidates that start inside an open XML parameter are now skipped, matching
the guard the XML-style parser already applies.
- _quote_gemma_array_elements preserved array elements starting with { or [
verbatim, so an array of objects (items:[{path:a}]) or a nested array failed
json.loads and the whole call was dropped. Object and nested-array elements
are now normalised recursively.
- _openai_passthrough_stream synthesized a finish chunk before a trailing
usage-only chunk and set saw_finish_reason, which made the EOF guard skip the
[DONE] sentinel. The EOF path now emits [DONE] whenever the upstream omitted
it, even after a finish chunk was already synthesized.
- /generate/stream drove generation through asyncio.to_thread with no
disconnect watcher, so a client disconnect during a long generation went
unnoticed until the next send. It now runs _await_disconnect_then_cancel
against the request, matching the other local streaming endpoints.
- _SameTaskStreamingResponse closed the body iterator with aclose() on a
send-side disconnect, raising GeneratorExit so the generators' cancellation
handlers (which finish the api_monitor entry) never ran. It now throws
CancelledError, falling back to aclose() when athrow is unavailable.
Extends tests/test_gemma_tool_parse_edge_cases.py with array-of-objects,
nested-array, and marker-inside-XML-parameter cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Watch disconnects on Anthropic streams; keep timestamps in Gemma values
Two follow-ups on the streaming and tool-parse paths:
- _anthropic_tool_stream and _anthropic_plain_stream drove generation through
asyncio.to_thread(next, gen, ...) and only polled is_disconnected() between
events, so a client disconnect during prefill or a long generation/tool step
held the decode slot until the next event or a failed send. Both now run the
_await_disconnect_then_cancel watcher used by the other local streams, stop it
in finally, and break promptly when cancel_event is set.
- _GEMMA_NEXT_KEY_RE treated any comma followed by word-chars-then-colon as the
next key, so a bare value such as "meet at 10:00, 11:00 tomorrow" was split
into bogus keys. The next-key token must now be identifier-shaped (start with
a letter or underscore), so a comma before a timestamp, ratio, or other
numeric-then-colon text stays part of the value.
Adds a timestamp-in-bare-value regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard nested markers, reset on disconnect, clean unstarted streams
Three follow-ups on the tool-parse and streaming paths:
- parse_tool_calls_from_text only skipped markers that fell inside a span it
had already parsed successfully, so when an unquoted Gemma argument contained
a literal marker (code:<|tool_call>call:terminal{...}<tool_call|>) the outer
object failed to normalize, its span was never recorded, and the inner marker
was promoted to a standalone terminal call. Candidates nested inside any other
candidate's brace span are now skipped regardless of whether the enclosing
candidate parsed, so a marker in malformed outer data is never executed.
- /generate/stream skipped backend.reset_generation_state() when the disconnect
watcher set cancel_event between chunks: the loop broke and the finally's reset
is guarded on cancel_event being unset. A subprocess backend kept decoding
after the client left. The cancel-break path now resets the backend.
- _SameTaskStreamingResponse threw CancelledError / called aclose() on the body
iterator on a send-side disconnect, but neither runs the try/finally of a
generator that never started (early disconnect on http.response.start), so the
passthrough's eagerly-opened upstream httpx stream and cancel-registry entry
leaked. It now tracks whether the body started and, when it did not, runs an
optional unstarted_cleanup hook; the OpenAI passthrough wires it to close the
upstream resp/client and exit the cancel tracker.
Adds a nested-unquoted-marker regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Add HF dataset streaming mode to Studio
* Added default value for datasetStreaming in training-config-store.ts
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Handle None max_steps for streaming validation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fast-fail streaming validation and guard incompatible modes
Reject dataset_streaming at the API boundary when hf_dataset is empty,
the dataset is vision/audio, or max_steps is not set. Probe eval split
with get_dataset_split_names before the streaming load so typos fail
immediately instead of mid-training. Guard column_names=None after map
on iterables. Hide the UI toggle for non-text configurations and clear
the stale flag when config becomes incompatible.
* studio: add streaming dataset tests, iterable helper, and streaming template/format support (WIP)
Work-in-progress on top of feat/studio-dataset-streaming-mode (PR #4946):
- new test_training_streaming.py and iterable.py dataset helper
- streaming support in chat_templates.py and format_conversion.py
- additional streaming guards in trainer.py / models / routes
- frontend streaming wiring in params-section and training-config-store
Committed to preserve uncommitted work before merging latest main.
* studio: fix review-team findings for streaming + main merge
BLOCKER: streaming + raw-text/CPT crashed on len(IterableDataset). Guard it in the
start route (reject format_type=="raw" or training_type=="Continued Pretraining")
and in isStreamingSupported (datasetFormat !== "raw").
Also:
- models/training.py: validate hf_dataset/subset/split (charset+length, block ..//);
cap dataset slice indices (le=1e9); note validator ordering
- chat_templates.py: guard _apply_custom_mapping .map() for streaming
- trainer.py: warn when packing+streaming
- training-config-store.ts: persist-migration bump to v11 (standalone datasetStreaming
backfill); add isVisionModel to NON_PERSISTED; toast on silent streamingCompatiblePatch
mutations in the 4 indirect setters
- tests: route rejections (max_steps, raw/cpt), slice cap, unsafe hf_dataset
* studio: enable raw-text/CPT dataset streaming + streaming UX polish
- raw_text: keep the lazy filter but skip len()-based row counting for
IterableDatasets so raw-text / CPT can stream; guard the eval-size log
- routes/trainer: drop the raw/CPT streaming block; add a defensive
not-streaming guard on the eval auto-split (train_test_split)
- dataset-section: streaming toggle is visible-but-disabled and lists the
exact unmet requirement(s) in its tooltip; block embedding models
- training-start-overlay: show "streaming (no full download)" instead of a
stuck download bar for streaming runs
- trim the streaming test suite to the high-value cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: address streaming review (MLX/embedding guards, sliced eval split, rehydrate timing)
- routes: reject dataset_streaming for embedding training and on Apple Silicon
(MLX); both loaders materialize the full dataset instead of streaming
- trainer: validate the base eval split name so streaming eval accepts HF slice
syntax such as "validation[:1000]"
- training-config-store: defer the onRehydrateStorage setState to a microtask so
it doesn't hit the store's TDZ during synchronous hydration
- test: streaming start rejects embedding models
* studio: harden HF dataset streaming (column_names, split slicing, empty/eval bounds, gating)
Address a deeper streaming review:
- raw_text: resolve_column_names() guards IterableDataset.column_names=None
(from_generator / unresolved features) so raw-text and CPT streaming no longer
raise TypeError before training
- models/routes: reject HF slice syntax in train_split/eval_split when streaming
(load_dataset(streaming=True) raises "Bad split"); reject mixed sources
(local/S3) and embedding/MLX streaming at the API, not just in the UI
- trainer: an empty post-slice/filter stream fails preflight with a clear message;
streaming eval is capped (STREAMING_EVAL_MAX_SAMPLES) so each eval terminates;
the manual-slice shortcut falls back to a regular load when train_split is sliced
- format_conversion: streaming conversions preflight the first mapped row so
format errors surface before training, not mid-iteration
- frontend: block streaming on Apple Silicon; clear datasetStreaming when a
dataset is detected as image/audio at start
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix CI for streaming PR (lint blocker + no-torch sandbox + preflight test)
- trainer.py: drop unused `IterableDataset` import (hoist safety-net blocker).
- test_training_streaming.py: only select real classes (isinstance type) when
locating the trainer class, so a MagicMock-stubbed global is never passed to
object.__new__ (fixes TypeError on the Python 3.10-3.13 jobs).
- no-torch import sandboxes (test_e2e_no_torch_sandbox.py,
test_studio_import_no_torch.py): teach the chat_templates/format_conversion
exec stubs and the full-import-chain copy list about the new `.iterable`
module so the AFTER/runtime cases import without torch again.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
* Studio: redesign Select model dropdown to match Hub design
Make the chat Select model picker easier to scan by reusing the Hub
on-device card's visual language.
- Rows now split owner/name, add a param chip, a DotTag format pill,
a tabular size, and a Loaded marker on the active model.
- Hub models / Fine-tuned tabs reuse the Hub's exact .hub-tab-toggle
styling (selectors extended in hub.css to the selector menu).
- Add a Downloaded / Recommended / Custom section toggle on the Hub
tab to filter the list.
- Widen the popover and nudge the scrollbar toward the edge.
* Studio: move section toggle below search, size tabs to label
Put Downloaded / Recommended / Custom under the search bar in their own
row so Hub models / Fine-tuned no longer wrap. The section toggle uses a
smaller font and sizes each tab to its label instead of equal widths.
* Studio: extract pure row-meta helpers into their own module
Move splitRepoLabel, classifyMetaToken, and parseMetaTokens out of
pickers.tsx into row-meta.ts. No behaviour change; keeps the presentation
logic free of React/DOM deps so it is easy to test in isolation.
* Studio: content-size the source tabs and add section icons
Size the Hub models / Fine-tuned tabs to their labels (with side
padding) like the section toggle, instead of stretching full width. Add
a leading download, star, and folder icon to Downloaded, Recommended,
and Custom.
* Studio: stop source tabs stretching and hide empty Fine-tuned tab
The popover is a flex column, so the fit toggle stretched full width;
add w-fit/self-start so it sizes to its content. Also hide the
Fine-tuned tab when there are no fine-tuned models, defaulting to Hub
models.
* Studio: keep only fine-tuned models in the Fine-tuned tab
Local models (LM Studio, Ollama, custom folders) carry source "local"
and already show in the Hub tab's Downloaded / Custom sections, so
exclude them from the Fine-tuned tab and from its visibility count.
Extract the tab rules into source-tabs.ts.
* Studio: show local providers under Downloaded, Recommended first
Show LM Studio and other local provider models in the Downloaded
section in all modes (was chat-only). Put Recommended first and make it
the default section. Add a little more space below the search bar.
* Studio: make Recommended a sortable live Unsloth listing
Replace the static Recommended list (and its collapse chevron) with a
sort dropdown over Unsloth's own models: Recommended, Trending, Most
likes, Downloads, Recently updated. Recommended shows recently uploaded
GGUF/MLX models that fit the device (hidden if they do not); the other
sorts list all Unsloth models, badged but never hidden. Adds a sort
option to useHfModelSearch and a pure recommended-fit helper.
* Studio: size Recommended models from the repo name when metadata is missing
GGUF and MLX repos rarely expose safetensors metadata, so a large model
with no size could pass the Recommended fit check because unknown size was
treated as fitting. Parse the parameter count from the repo id, including
the Gemma E series, and hide anything we still cannot size.
* Studio: detect model capabilities and family from HF tags
Thread tags and the pipeline tag through the model search results and add a
pure helper that infers vision, reasoning and audio plus the architecture
family, falling back to repo-name keywords when tags are absent.
* Studio: add row details and inline section sorting to Select model
Give each model row more detail and make the Hub sections easier to scan:
- Show vision, reasoning and audio badges plus the architecture family tag
on each row, alongside the params, format and size.
- Drop the redundant unsloth/ prefix on the Recommended rows.
- Rename the Recommended section tab to Unsloth and enlarge the section tabs.
- Move the sort dropdown inline to the right of the tabs at a fixed width.
- Add Recent, Size and Downloaded sorting to the Downloaded and Custom tabs.
- Remove the header icons, pad the subheadings, and grow the list height.
* Studio: tune the Select model sort dropdown and trim row badges
- Recommended now lists the most recently created Unsloth repos.
- Narrow the sort dropdown, remove its border, and truncate long labels.
- Tighten the gap between the section tab icons and their labels.
- Remove the architecture family tag from rows since it repeats the name.
* Studio: extract the PillTabs toggle into a shared module
Move the segmented pill toggle out of the model selector into its own file so
the Hub picker can reuse it for a format filter without duplicating the markup.
* Studio: fix Recommended infinite scroll and add a format filter
- Re-attach the scroll observer on each loaded page so a filtered Recommended
list keeps paging until the viewport fills instead of spinning forever with
nothing new appearing.
- Add an All / GGUF / MLX / Safetensors toggle on the Unsloth listing that
filters every sort.
* Studio: default Recommended to Trending, rename Downloaded to On Device, and fade the scroll edge
Sort: default the Recommended view to Trending and add a Name option to
the On Device / Custom sort. Recent now orders by last load time while
Downloaded orders by file date, tracked in localStorage (model-usage.ts).
Formats: show the format filter on all three tabs (Unsloth, On Device,
Custom), exclude mobile GGUF builds from Recommended, and flag GGUF rows
that exceed the device with the same OOM badge as safetensors.
Polish: download-icon badge on already-downloaded Recommended rows, the
hugeicons view stroke-rounded vision badge, Search all models placeholder,
matched popover padding, and a top-edge mask fade once the list scrolls.
* Studio: size GGUF repos from gguf metadata so large ones flag OOM
Repos with no <n>B token in the name (Kimi, MiniMax) had no param count
and so never showed an OOM badge. Request the gguf expand field from
Hugging Face and read gguf.total, so those repos get a param chip and an
OOM badge when they exceed the device budget.
Keep the row name full contrast when over budget (the OOM badge already
signals the fit), shorten the format and sort dropdowns, narrow the
popover, and rename Recently updated to Recent and All formats to All.
* Studio: address selector review feedback
Add WAI-ARIA roving tabindex and Arrow Left/Right navigation to the pill
toggle so only the active tab is in the tab order. Keep the chat-only
GGUF/MLX filter for every Recommended sort, not just Recommended, so
chat-only users do not see unrunnable checkpoints under Trending. Feed
both listings' GGUF hints into repo detection so a tag-only GGUF in
Recommended expands variants instead of loading as a checkpoint.
* Studio: scope Select model search per tab and add an MLX tag
Search is now per section. The Unsloth tab searches the Unsloth HF
listing only, On Device filters downloaded and LM Studio models by name,
and Custom filters custom-folder models, each with its own empty state.
MLX repos get an MLX pill mirroring the GGUF tag. Downloaded quants in
the Unsloth and search lists get the same delete action as On Device.
Also: revert the model name to normal weight, narrow the popover to
558px so the format and sort dropdowns sit one gap-2 from the tabs,
tighten the dropdown menus to match the Projects activity Select, and
make the empty On Device state name the active format filter.
* Studio: show local ./models on the On Device tab so they stay selectable
Models under the local models directory (source models_dir) flow in as local
models but were dropped from every list: filtered out of Fine-tuned and never
re-added by the Hub picker, which kept only LM Studio and custom-folder
sources. Capture them in the local refresh and render a Local models group on
the On Device tab, with the same format, search, and chat-only GGUF rules as
the other local groups.
* Studio: add a Hub button beside the Select model search bar
Adds a Hub button next to the search bar that opens the full Hub Discover
page to browse more models. Styled like the section tabs (rounded, no
border, soft shadow with a faint top layer) and darkens on hover. Also
nudges the format and sort dropdown chevrons a touch toward the edge.
* Studio: align Select model padding and tighten the format pills
Sizes the popover to the tab cluster so the left and right padding match,
and drops the top row below the rounded corner so the Hub button lines up
with the Trending dropdown. Gives the Hub button a fixed width, lets the
list scrollbar sit inside the box, and shrinks the format pill dot with a
tighter dot-to-label gap.
* Studio: label the Hub button Search Hub and match the dropdown width
Renames the button to Search Hub, sets its width to the format and sort
dropdown width so it lines up above them, and tightens the icon gap.
* Studio: drop the vision and reasoning row badges to declutter
Removes the vision and reasoning capability icons from the model rows so
they read cleaner. Audio is kept.
* Studio: add a safetensors pill, hide diffusion models, eye on Vision
Gives safetensors rows a format pill and size so their meta matches GGUF
and MLX, drops image and video diffusion models from the listing since they
cannot run in chat, and shows an eye icon next to the Vision tag. Also
removes the em dashes from the Projects export and import labels.
* Studio: gate recommended folders on real weights and polish the selector
Only show a Recommended chip once the well-known dir actually holds
weights, so an empty LM Studio or Ollama scaffold no longer suggests
itself. _dir_has_downloaded_model checks for a GGUF/safetensors file or
a non-empty Ollama manifests store, with a bounded walk.
Selector polish: round the popover and option menus a touch more,
lighten the OOM badge in dark mode, soften the inner dropdown shadow,
even out the padding, and lift the toggle track and field triggers so
their edges read against the popover.
Also catch CogVideoX in the diffusion name fallback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: align the dark Select model panel with the sidebar
Match the popover, fields, dropdowns, tab toggle and row states to the
sidebar surface and accent so the dropdown reads as one piece in dark
mode. The active tab pill and Search Hub button sit a touch lighter
than the track, and the inner option menus drop their drop shadow for a
flatter look. Light mode is unchanged.
* Studio: re-derive the Select model tab on open
The picker remounts each time the dropdown opens, but the source tab
state did not, so a persisted fine-tuned or connected selection that
only lands in its list after an async load would reopen on Hub. Reset
the active tab to the selection-derived default on the open edge, while
still letting the user switch tabs freely within a session.
* Studio: fold Custom into On Device and polish the picker
Merge the Custom tab into On Device so custom folders sit right below
the downloaded models, with a folder shortcut on the group header.
Rename the first Hub tab to Recommended, give the format dropdown
colored dots, even out the tab row spacing, and tighten the popover
width. Align the folder browser with the app dialogs (soft surface,
roomier padding, green confirm, grey hover).
* Studio: fix On Device controls and nudge the folder browser close
The Hub redesign merge dropped the old Search Hub button styling, so the
On Device search row rendered flat. Point the search input and Search
Hub button at the shared .field-soft surface so they match the rest of
the Hub controls, and lift the folder browser close button slightly.
* Studio: run the Select model search on the Hub search stack
Point the picker at the Hub's useHubModelSearch and useHubInfiniteScroll
instead of its own useHfModelSearch/useInfiniteScroll, scoped to unsloth
so the listing matches the old one. Both the search and the recommended
feed now share the Hub implementation, so there is one search path. The
Hub result folds GGUF params into totalParams, so the dead ggufParams
fallback is dropped.
* Studio: trim the recommended sort to Recommended, Trending, Recent
Drop Downloads and Most likes from the sort dropdown.
* Studio: give the section tabs room off the rounded edge
The fit-mode toggle wrapped the tabs with no inset, so On Device sat
tight against the rounded-full edge. Add a small horizontal inset and
widen the popover a touch to fit it.
* Studio: drop the legacy HF search hooks for the Hub ones
Migrate the training model and dataset sections, export page, onboarding
steps and recipe dataset combobox off useHfModelSearch, useHfDatasetSearch
and useInfiniteScroll onto the Hub equivalents, scoped to unsloth so the
listings match. The picker reads recommended param counts off the search
results it already has instead of a separate fetch. Removes the duplicate
search stack: use-hf-model-search, use-hf-dataset-search,
use-hf-paginated-search, use-infinite-scroll, use-recommended-model-vram
and the old lib/hf-cache.
* Fix model selector section toggle proportions
Remove the fit-mode track inset so the active pill sits flush to the
track edge, matching the Hub's segmented controls.
* Tighten model selector width and tab padding
Reduce the popover width so the right edge aligns with the row, and
widen the fit-mode tab padding so On Device clears the track edge.
* Refine Recommended formats, sort width and tab padding
Recommended now suggests GGUF anywhere and MLX only on Mac, never
safetensors. Size the sort dropdown to its label so Recommended no
longer truncates, and match the On Device trailing gap to the active
pill's leading inset.
* Flush section toggle and match dropdown font to Search Hub
Drop the trailing track pad so the active pill fits the track exactly
at either end. Size the sort and format dropdown text to text-xs like
the Search Hub button, and clip long labels without an ellipsis.
* Fix sort menu checkmark overlap and lock dropdown widths
Keep the option's right padding so the selected checkmark no longer
overlaps the label, and let the open menu expand to fit it. Set the
format and sort triggers to a fixed width matching the Search Hub
button so they always line up.
* Keep section toggle and dropdowns on one row
Drop the wrap and size the Search Hub button, format and sort dropdowns
to a shared 100px so they stay equal width and fit on one row without
widening the box.
* Studio: pre-load inference settings dialog with native context
Add a gear on downloaded GGUF quant rows that opens a settings dialog
to adjust inference parameters before loading a model:
- Context length, KV cache dtype, speculative decoding and tensor
parallelism, all written to the runtime store the load call reads.
- Settings can be remembered per model in localStorage.
- The context slider ceiling and "Model supports up to N tokens" come
from the model's native context, read from GGUF metadata and returned
by /api/models/gguf-variants once a variant is downloaded.
Also drop models Studio can't run for chat (diffusion, image, video)
from the recommended feed and Hub search, plus minor selector polish
on row hover padding, Search Hub and dropdown widths, and tab spacing.
* Studio: model selector polish and memory-aware load warning
Search and listing:
- Drop the "Recommended" and "Hugging Face" section labels while
searching so results read as one list; keep the format and sort
dropdowns visible so search results can still be sorted and filtered.
- Request gguf metadata in the Hub listing so GGUF repos report a
parameter count, restoring the OOM badge for repos without a size
token in the name (Kimi, MiniMax, GLM).
Load settings dialog:
- Warn when weights plus the KV cache at the chosen context exceed
available memory. The KV size is sized by the backend's
architecture-aware estimator via a new kv-cache-estimate endpoint;
the budget uses VRAM plus system RAM. Best-effort, no warning on
failure or on auto context.
- Context Length placeholder reads "auto"; dark background slightly
lighter.
Other:
- Clicking the Custom Folders header opens the folder browser; its
title now reads "Select folder to detect models".
- On Device sort lists Downloaded last.
- Smaller chat template editor font; rounded wrapper clips the prompt
and template editor scrollbars so the right corners stay round.
* Studio: fix load dialog memory warning budget and KV dropdown width
- The memory warning never fired without a discrete GPU. useGpuInfo
returned zero system RAM in that case, so the budget was always zero.
Surface system RAM even when no GPU is present (Mac unified memory),
and have the load dialog read memory directly instead of through props.
- Give the dialog fields shrink-0 so the KV Cache Dtype value (e.g.
q8_0) is not squeezed and clipped by the row.
* Studio: fold fine-tuned models into On Device tab
Remove the Hub models and Fine-tuned source tabs. Fine-tuned models now
show as a section in the Hub tab's On Device view, above Custom Folders,
with the Train icon and a collapse toggle. The section only appears when
the user has fine-tuned models. With no external providers the lone Hub
tab hides its own toggle.
Also: tick-circle Show hidden checkbox and drop the divider above Eject;
keep run settings load params (KV cache dtype, speculative, tensor
parallel) from being clobbered by a mid-load status poll.
* Studio: stage load settings in the sidebar with a Load on selection toggle
Replace the pre-load settings popup with a staging flow in the Run settings
sidebar. The gear on a downloaded quant row now stages the model and opens
Run settings with Load model and Cancel buttons, so options like context
length, KV cache, speculative decoding and tensor parallelism are set before
the model loads. A "Remember these settings" tick reuses them next time.
Add a global Load on selection toggle in Settings, Chat tab (default on).
On: Unsloth auto-picks the best settings for your hardware and loads on
selection. Off: picking a model stages it in Run settings to customize first.
The gear always stages, regardless of the toggle.
Other polish in this change:
- Fine-tuned models live under the On Device tab, with a train icon on the
header that jumps to the Fine-tuned section.
- Default to the On Device tab when downloads exist, otherwise the last used
section.
- Standard Unsloth tooltips on the train, folder and gear icons.
- Request the gguf param count on every Hub listing fetch so Kimi, MiniMax
and GLM show a size badge.
- Search Hub hover state, scrollbar position and minor spacing fixes.
Remove the old inference load settings dialog.
* Studio: always show the fine-tuned shortcut and smooth out the picker
- Fine-tuned section and its train shortcut now always show on On Device,
with an empty state when no fine-tuned models exist yet.
- Folder icon on the header jumps to Custom Folders instead of opening the
browse popup, matching the train shortcut.
- Folder browser keeps the list mounted and dims it while refetching, so
toggling Show hidden or changing folders no longer flashes.
- Drop the tooltip hover grace area in the picker so moving between the
train, folder and gear icons switches the tooltip at once.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add quantization display options and drop the fine-tuned empty text
- Settings, Chat: 'Expand quantizations' toggle. On expands every On Device
GGUF model's quantizations by default; off keeps them behind a click
(default).
- Settings, Chat: 'Show all quantizations' toggle. On lists every quant
including ones not downloaded (default); off shows downloaded only.
- Remove the empty-state line under the Fine-tuned header; the header still
shows on its own.
* Studio: let expanded quantizations collapse on click and split the On/Off help
- With Expand quantizations on, clicking an On Device model now collapses or
re-expands its quantizations. The collapse state is in memory only, so it
resets on reload and when the setting is toggled.
- Put the Off sentence on its own line in the quantization setting descriptions.
* Studio: reorder chat settings and rename the model section
- Rename the Models section to Select model settings and move it above the
Chat menu section.
- Trim the section and Load on selection descriptions.
* Studio: tighten the On/Off lines in the model setting descriptions
Use a line break instead of separate spans so the On and Off lines sit on
consecutive lines without the extra paragraph gap.
* Studio: top-align the Load on selection toggle
Add an alignTop option to SettingsRow and use it so the toggle sits at the top
of the row next to the label, not centered against the tall description.
* Studio: put the gear hint and example chip on one line
Move the gear example chip inline with its label so it reads as a single line
instead of wrapping onto its own row.
* Studio: move the New badge from API keys to Chat settings
Add the New badge to the Chat settings tab and drop it from API keys.
* Studio: line the Load on selection toggle up with the first description line
Offset the top-aligned control past the label row so it sits next to the On
line instead of the label.
* Studio: label the chat menu item Chat with Files (RAG)
Rename the Chat with Files entry in the chat menu settings to clarify it is RAG.
* Studio: drop the pill around the gear example so it fits on one line
Remove the background and padding from the gear example chip so it sits inline
with its label at a lower height.
* Studio: fold the gear example into the description line spacing
Render the gear example inline in the same text block so its line spacing
matches the On and Off lines instead of an extra flex gap.
* Studio: scope Show all quantizations to On Device only
Gate the downloaded-only filter on an onDevice flag so Recommended and other
browse lists always show every quant, and note On Device in the setting copy.
* Studio: tidy On Device GGUF rows
- Drop the redundant Quantizations subheading under On Device models.
- Relay GGUF vision support up to the model name as a Vision badge instead.
- Drop the repo size from On Device GGUF model rows since the quants already
show their size.
* Studio: pin the eject button and tidy General settings
- Move Eject loaded model out of the scrollable list into a centered footer so
it stays in view no matter how far the list is scrolled.
- Space out and center the gear example in the Load on selection description.
- General: drop the duplicate Unsloth version section, move llama.cpp
notifications above Helper LLM, and note new models in its description.
* Studio: add left padding before the gear example
Nudge the gear example away from its label with a small left margin.
* Studio: make the eject footer a sticky bar over the list
Pin Eject loaded model to the bottom of the scroll area with the menu
background so rows scroll under it, and drop the divider line.
* Studio: drop the eject footer background, keep it a sticky button
Make the sticky eject a centered transparent button so it coexists with the
rows scrolling behind it. The wrapper ignores pointer events so only the button
is clickable.
* Studio: give the eject button a solid background
Add the menu background, a border and a soft shadow to the sticky eject button
so it reads as a floating button over the list.
* Studio: restore the eject footer block, keep hover on the button only
Bring back the full-width menu background behind the sticky eject footer, but
keep the button compact and centered so the hover stays on the button.
* Studio: show the vision badge on On Device rows without expanding
- cached-gguf listing reports has_vision (mmproj present), so the badge shows
on the model name without opening the quantizations.
- Make the vision badge icon-only with a tooltip: "This model can process
image inputs". Falls back to the expander-reported value on older backends.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make LM Studio and Local models sections collapsible
* Fade the eject footer instead of a solid block
* Wrap the vision badge in a bordered pill
* Taller model list with the eject footer pinned to the bottom
* Use purple for the vision badge to set it apart from GGUF
* Reduce the model list height
* Make the eject button inline with no background block
* Match the vision badge color to the Hub indigo tone
* Shorten the model list and square off the format tags
* Pin the eject button so it floats at the bottom of the list
* Give the floating eject button a tinted background
* Add bottom clearance so the list ends on white space under the eject button
* Match eject button to the menu background and unify the settings gear icon
* Move eject below the list and match its shadow and dark background
* Drop the min height so short model lists leave no white space
* Remove the eject button fill so it never covers the list
* Nest dropdown hover radius inside the menu corners
* Float the eject pill again and fix sort dropdown hover radius
* Make the eject button opaque in both themes on hover and dark
* Trim the model menu bottom padding so it stops clipping the last row
* Match dark eject background to the Search Hub button and pad row indicators
* Fade the model list bottom edge while rows sit below the fold
* Lift the eject button and trim the section toggle right padding
* Nudge the model list taller and run the bottom fade to the box edge
* Nudge the model list slightly taller
* Remove the eject button shadow
* Align the eject button to the right
* Widen the Search Hub and dropdowns and right-align them
* Seat the eject button at the base and restore On Device right padding
* Reduce the Search Hub and dropdown width by 4px
* Widen the model menu so the section toggle keeps its padding
* Make the eject button an icon-only button with shadow
* Tighten section tab padding to cut the grey between tabs
* Revert section tab padding back to px-3
* Remove the section toggle trailing padding
* Add an eject button beside the model selector trigger
* Shrink the in-list eject button to a smaller proportional size
* Raise the in-list eject button
* Make the trigger eject a bare icon next to the dropdown arrow
* Revert eject back to the labeled button on the right
* Place the format and sort dropdowns next to the section toggle
* Raise the eject button and shorten its label to Eject model
* Widen the gap between the toggle and dropdowns slightly
* Align Search Hub with the last dropdown via a shared-width grid
* Narrow the model menu for symmetric padding
* Stretch the search row so Search Hub lines up with the last dropdown
* Inset the list so the right padding matches the left
* Right-align dropdowns and full-width search so Search Hub meets the last dropdown
* Pack section toggle and dropdowns with a uniform gap
* Inset search row so Search Hub aligns with the Trending dropdown
* Trim model menu right padding to match the left
* Nudge model list scrollbar inward
* Move eject button to the bottom left with a light shadow
* Shorten show all quantizations description
* Keep eject button right-aligned, nudged in from the edge
* Move Connected into the section toggle as a cloud-icon tab
* Align eject button with the format tag edge
* Right-align Connected layout so Search Hub meets Trending
* Download selected models through the Hub download manager
* Add Other models section for non-Unsloth downloads
* Add directions icon and shortcut for Other models section
* Space out subheadings and gate Other models on non-Unsloth downloads
* Use direction-right icon for Other models
* Use flag icon for Other models
* Widen Connected menu so dropdowns align with Search Hub
* Model selector: truncate long quant labels and tidy layout
- Hub GGUF card: truncate long file-path quant labels with an ellipsis
instead of overflowing the row.
- Connected layout: left-pack the dropdowns and size the box so the last
dropdown's right gap matches the pill's left gap, with Search Hub on its edge.
- On Device: show MLX/Safetensors with the size on non-GGUF rows.
- Connected list rows use the same grey hover as the tabs; the selected
section tab no longer shows a hover change.
* Model selector: drop stale custom section on restore
A persisted custom section value no longer maps to a tab, so restoring it
opened the picker to an empty view. Fall back to recommended instead.
* Model selector: align the non-connected search bar with the All dropdown
Nudge the non-connected box width so the search bar's right edge meets the
All dropdown, which lands Search Hub on the last dropdown's edge.
* Studio chat model selector: remember last tab, route non-GGUF downloads through Hub, stack overlays
- Restore the last Hub section (Recommended / On Device) on every open instead of always snapping to On Device when downloads exist.
- Route uncached non-GGUF repos (safetensors / MLX) through the Hub download manager via a snapshot download, so every model download shows in the bottom-right indicator and follows Load on selection like GGUF.
- Allow safetensors in Recommended on Mac (they run locally there now), and honor the Safetensors format filter instead of dropping it via the recommendation default.
- Stack bottom-right overlays in one column so the download panel and banners never overlap.
- Add evenly spaced divider lines between the On Device subheadings.
- Pad the bottom of the list so the floating Eject pill never covers the last row.
* Studio downloads panel: widen left padding on header and rows
Bump the left inset to pl-4 while keeping pr-3 so the collapse and cancel buttons stay put.
* Studio: update cached-gguf route tests for the has_vision field
list_cached_gguf now returns has_vision per row (vision badge on On Device);
the expected dicts were missing it. True for the mmproj vision repo, False elsewhere.
* Studio: keep MLX/safetensors selectable in chat-only Mac search
The empty Recommended view allows GGUF plus MLX/safetensors on Mac, but the
curated and HF search lists dropped non-GGUF in chat-only via a GGUF-only filter,
so typing a query hid runnable Mac models. Reuse isRecommendableFormat in both
lists so search matches the empty view (chat-only non-Mac stays GGUF-only).
* Model selector: restore global model search and fix GGUF/device-fit regressions
- Search: training, export and onboarding pickers searched only the unsloth org
on a typed query. Restore the prior behavior (global Hub search with unsloth
floated first when a query is typed, curated unsloth listing when empty).
- Recommended browse: the GGUF/MLX-only gate ran before the format filter, so
the Safetensors filter and the Trending/Recent sorts always came back empty.
Apply that gate only for the Recommended sort and chat-only mode.
- GGUF metadata: request the gguf expand field through listModels so repos with
no size token in the name (Kimi, MiniMax, GLM) report a param count for the
size and OOM badge.
- Local GGUF: custom-folder and standalone ./models/*.gguf files now load
directly with the GGUF marker instead of dead-ending in the variant expander,
and scanned GGUF folders are classified via a backend model_format hint.
- Device fit: use system RAM in the budget on unified-memory hosts, and keep MLX
rows selectable on chat-only Macs.
- kv-cache-estimate: resolve the quant from the snapshot-relative path, skip MTP
drafter files, and prefer the most complete snapshot (mirrors the variant
scanner). Bound the Ollama manifest walk.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Model selector: classify suffixless local GGUF folders consistently
Complete the model_format plumbing so a GGUF folder is detected and loaded
through the same GGUF path that the format filter already uses:
- _scan_models_dir: a config.json no longer disqualifies a folder whose only
weights are .gguf, so HF GGUF repos shipping a config still classify as GGUF.
- _scan_lmstudio_dir: emit model_format for every GGUF row (LM Studio dirs
rarely carry a -GGUF suffix), via a shared _dir_model_format helper.
- Custom Folders and LM Studio rows: use localModelIsGguf (the same helper the
filter uses) so the row label, expand-vs-direct-load, and isGguf flag agree;
a suffixless GGUF folder no longer filters as GGUF but loads as non-GGUF.
Adds tests/test_local_model_format.py covering the classification rule.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio model selector: tighten section spacing
Trim each subheading's gap to its rows (pb-1.5 to pb-1) and pull the On Device
heading block tight to the controls while Recommended keeps a little top room.
* Hub: format filter fix, sort defaults, avatar and layout polish
- Format dropdown now filters the feed's Latest list too, so the default
GGUF hides fp8/safetensors and picking a format changes the rows.
- Latest Unsloth Models sorts by newest created, not recently updated.
- Sort dropdown order: Newest, Trending, Most downloads, Recently
updated, Most likes.
- Unsloth uploads with no upstream provider logo show the Unsloth avatar
instead of a colored initial.
- Owner scope pill gets a little more room before the chevron.
- README detail column lines up with the top bar (both-edges gutter).
- Long file-path quant labels truncate instead of overflowing the row.
- Model list keyboard nav no longer clips the focus ring.
- Run settings sheet: restore the Remember settings toggle and larger
Load/Cancel buttons on the staged load flow.
* Hub: hide the RAG embedding model from browse previews
The Hub discover feed and chat model selector pull from the Hugging Face
listing on the client, which the backend _is_hidden_model filter never
touches, so the RAG embedder (unsloth/bge-small-en-v1.5-GGUF) and the
llama.cpp validation probe leaked into the lists.
Added isHiddenModelId mirroring the backend needles and filtered it out of
the discover rows, the trending feed, and the selector's recommended and
Hugging Face search lists. Per-repo file and download views are untouched,
so the model is never deleted and a reinstall still shows it as already
downloaded.
* Studio: skip hidden dirs when checking a folder for downloaded models
_dir_has_downloaded_model walked the tree with rglob("*") bounded by
max_entries. rglob yields entries in arbitrary order and counts every one, so a
model directory that also holds a large hidden subtree (.git/.cache/venv) could
exhaust the budget before reaching the real weights and falsely report no model,
hiding a valid Recommended-folder chip. Replace the generic-weights pass with a
bounded BFS that skips hidden directories so their entries can't starve the walk.
Adds a regression test (50-entry .git beside the weights, max_entries=10).
* Fix/adjust model selector handling for PR #6364
* Studio: address codex review on the staging/recommended-folder paths
- chat-page auto-load: selectModel only clears pendingSelection on success, so a
failed auto-load left the hidden stage (and its edited load knobs) behind.
Abandon the stage when it still matches the failed pick.
- model picker: count fine-tuned rows in the On Device empty check so a
fine-tuned-only tab no longer shows a false 'No models on device' message
above the Fine-tuned section.
- general settings: add the remembered per-model load settings key to PREFS_KEYS
so 'Reset all local preferences' actually clears it.
- recommended-folders: recognize PyTorch .bin weights (gated by the scanner's
weight-name prefixes) so a .bin-only model folder still earns a chip; add tests.
* Studio: name-gate .bin weight detection and complete selector preference reset
Follow-up to the codex review on the model_format/recommended-folder paths:
- _dir_model_format and _scan_models_dir treated any .bin (incl. tokenizer.bin)
as a non-GGUF weight, so a suffixless GGUF folder shipping a companion .bin was
misclassified as a plain checkpoint and routed through the wrong load path.
Factor the scanner's weight-name gating into shared _is_weight_bin /
_has_non_gguf_weights helpers and use them everywhere (also in
_dir_has_downloaded_model).
- PREFS_KEYS was missing the new 'Select model settings' keys (load on selection,
expand/show-all quantizations), so 'Reset all local preferences' left them set.
- On Device cached search dropped the active format filter while a query was
typed; keep matchesFormatFilter applied so the format dropdown stays consistent.
Adds tests for the tokenizer.bin vs weight-.bin classification.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: validate Ollama blobs, gate staged context, honor RAM budget on no-GPU hosts
- recommended-folders: only count an Ollama dir once its manifest resolves to an
on-disk model blob, so a failed/pruned pull no longer surfaces an empty chip
- GGUF variant click: only seed the staged contextLength for already-downloaded
picks, so choosing an undownloaded quant from a partially cached repo still
starts its download (the staging effect short-circuits on a known context)
- device fit: classify GGUF variants against the system-RAM budget on no-GPU /
unified-memory hosts instead of reporting everything as fits, and pass
systemRamGb to every variant expander regardless of gpu.available
* Studio: scope Hub search to Recommended, fix staged non-GGUF settings, keep local MLX on Mac
- model picker: only run the Hub search hooks on the Recommended section. On
Device / Connected render local data, so typing there no longer fires HF
requests or a spinner and the local/offline flow is preserved
- chat settings: when a pick is staged, decide the GGUF-only controls from the
staged model's type, not the currently loaded model's. A staged non-GGUF Hub
repo no longer inherits a loaded GGUF's context/KV/speculative controls
- On Device: keep local MLX builds in ./models selectable on Mac (chat-only ran
GGUF/MLX only, but the filter dropped MLX before the format toggle)
---------
Co-authored-by: shimmyshimmer <info@unsloth.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Recognise the Gemma 4 separate-drafter MTP family, auto-download the drafter with retry, fall back to n-gram with a clear reason when it cannot be resolved, and retry the download on reload. Gemma 3n (ships no drafter) and embedded-MTP models (Qwen) are unaffected.
Fixes#6406
* Studio: Auto disables MTP for MLA models (GLM-5.2 et al.); UNSLOTH_MLA_MTP_ENABLED to re-enable
Studio's Auto speculative mode promotes any embedded-MTP model >=3B to
--spec-type draft-mtp. For MLA models (GLM-5.2/DeepSeek/Kimi) that is a
regression: llama.cpp's MLA/DSA MTP path keeps a duplicated full target-KV
context and recomputes the sparse-attention indexer every draft step, so it
runs ~2x slower than no speculation (GLM-5.2 UD-IQ1_S bench: 27 vs 45 tok/s,
flat across draft depth 1..6 and 96-100% acceptance, on both prose and code).
vLLM/SGLang get a speedup from the same model, so this is a llama.cpp
implementation gap, not a model property.
Auto now drops embedded MTP for MLA models and falls back to ngram-mod (or
spec-off when the binary lacks ngram-mod), mirroring the existing sub-3B
fallback. The metadata separator is kv_lora_rank: it is present on MLA models
and absent on non-MLA embedded-MTP models (Qwen3.x-MTP), whose MTP module is
structurally identical but fast, so a "full layer" heuristic cannot tell them
apart. Qwen MTP, separate drafters (Gemma, --model-draft), and non-MTP models
are unchanged.
Explicit overrides still engage the slower MTP route: choosing MTP / MTP+Ngram
in Settings, or passing --spec-type in extra args. UNSLOTH_MLA_MTP_ENABLED=1
re-enables Auto promotion for MLA once the upstream path is optimized.
A new spec_fallback_reason value "mla_mtp_disabled" surfaces this as an
Auto-mode policy downgrade (not a binary/update problem), with a settings
banner that points users at the MTP override. It is deliberately kept out of
the "Update llama.cpp" affordance since updating does not help.
Tests: resolver-matrix rows for MLA->ngram-mod / MLA-no-ngram->off /
non-MLA-Qwen->draft-mtp / MLA-separate-drafter->draft-mtp /
non-MTP-MLA->default / forced mtp|mtp+ngram on MLA->draft-mtp / env flag;
kv_lora_rank metadata fixtures; and reload-skip coverage (Auto ngram-mod is
idempotent, forced mtp bounces a reload).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: fix Bypass Permissions menu freeze and show decimal GB for model sizes
Bypass Permissions freeze: the warning dialog lived inside the composer
"+"/More dropdown and kept the menu mounted via onSelect preventDefault,
so confirming or cancelling the dialog left both popovers frozen open.
Lift the dialog out of the menu into a store-driven
BypassPermissionsConfirmDialog mounted at a stable spot in the composer.
The menu item now closes normally on select and just toggles a new
bypassConfirmOpen store flag, so the popovers dismiss as expected.
Model search sizes: formatBytes divided bytes by 1024 but labelled the
result "GB", so unsloth/GLM-5.2-GGUF:UD-IQ1_S showed 201.8 GB where
Hugging Face reports 217 GB. Switch the search display to decimal
(base-1000) units to match what Hugging Face reports. The GPU-fit math
stays base-1024 since VRAM capacity is binary.
* Studio: address review feedback and add GLM-5.2 high/max/disabled thinking
Review feedback on the Bypass Permissions and size-format changes:
- Mount the Bypass Permissions warning dialog once at the chat-page root
instead of inside each Composer. It is driven by global store state, so
the per-composer mount meant Compare mode (multiple composers) rendered
duplicate dialogs and the shared-composer menu had none. A single root
mount fixes both.
- Defer opening the dialog past Radix's menu-close focus restoration with
setTimeout(0), so the dropdown does not steal focus back and break the
dialog's focus trap.
- Clamp the unit index in formatBytes so units[i] cannot go out of bounds
past TB (and to absorb log() float error at exact powers of 1000).
GLM-5.2 reasoning levels:
GLM-5.2's template gates thinking with enable_thinking and also reads a
reasoning_effort level ('high' or 'max'), so it needs high / max /
disabled rather than the binary toggle it got before (its style was
detected as enable_thinking, which made 'high' unreachable). Add a new
reasoning style 'enable_thinking_effort' that reuses the effort dropdown
but, unlike gpt-oss, can be fully disabled:
- detect_reasoning_flags classifies a template that has both
enable_thinking and reasoning_effort, extracting the discrete levels
from the quoted effort literals it branches on. Templates with only one
of the two (gpt-oss, Qwen3, DeepSeek, GLM-4.6) are unchanged.
- _request_reasoning_kwargs maps the new style to enable_thinking plus an
in-range reasoning_effort; disabling sends enable_thinking=false. The
gpt-oss reasoning_effort path is left untouched.
- The backend reports reasoning_effort_levels on the load/status response;
the frontend carries them through to the effort dropdown and sends
enable_thinking + reasoning_effort for this style.
Verified: backend reasoning kwargs render the real GLM-5.2 template to
"Reasoning Effort: High/Max" (thinking) and an empty <think></think>
(disabled); tsc, eslint, i18n parity and the production build all pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review feedback on reasoning effort and formatBytes
- chat-adapter localReasoningEffort: accept 'minimal' so a template that
branches on it (extracted into reasoning_effort_levels) is sent through
instead of being coerced to 'low' and then dropped by the backend.
- formatBytes: return '0 B' for non-finite / non-positive sizes (missing
metadata -> NaN, Infinity, negatives) and clamp the unit index lower
bound to 0, so sub-1-byte values can't produce a negative index.
* Studio: hybrid reasoning none gate and decimal GB in load progress
- _request_reasoning_kwargs: for enable_thinking_effort models, treat a
raw reasoning_effort='none' (OpenAI 'no reasoning' sentinel) as the
enable_thinking=false off gate, so a direct API caller can disable
thinking even without passing enable_thinking. The frontend already
sends enable_thinking=false; this only affects raw API callers.
- use-chat-model-runtime: the download / 'X of Y GB in memory' load
progress divided bytes by 1024**3 but labelled GB, so it disagreed with
the model picker and Hugging Face. Use decimal GB (1e9) to match.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry hybrid reasoning levels on all load paths and harden formatBytes
Review follow-ups on the enable_thinking_effort work:
- Every model-load path now copies reasoning_effort_levels and derives
supportsReasoningOff, via a shared reasoningCapsFromLoad() helper. The
shared/Compare composer load and the three chat-adapter auto-load paths
previously set only reasoningStyle, so a GLM-style hybrid model loaded
through Compare or first-chat auto-load fell back to the default
low|medium|high and lost its Max / Off controls.
- The local send path clamps the effort to the loaded model's advertised
levels (clampReasoningEffortToLevels) instead of a hard-coded list. A
stale "max" carried over from an external provider no longer reaches a
pure reasoning_effort (gpt-oss) model that only accepts none|low|medium|
high, where the backend would have dropped it.
- formatBytes divides iteratively instead of via Math.log, which has float
error at exact powers of 1000 (log(1e12)/log(1000) = 3.9999... would
label 1 TB as "1000 GB"). Keeps the non-finite/non-positive guard.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: free chat model VRAM at training start only when the GPU is tight
The training start route unconditionally tore down the transformers/MLX
inference subprocess before training, and never stopped the llama.cpp GGUF
server at all, so a loaded GGUF chat model kept holding VRAM for the whole
run. Conversely the HF model was always unloaded even when there was plenty
of room to keep it.
Make the unload VRAM aware and cover every inference backend:
- Add routes/training_vram.py with summarize_resident_chat(),
can_keep_chat_during_training() and free_chat_models_for_training(). The
keep/unload decision reuses the same estimator and live per device free
VRAM reader the training GPU selection already uses (auto_select_gpu_ids,
estimate_required_model_memory_gb, get_visible_gpu_utilization), so the
probe agrees with the placement computed later in start_training.
- When a chat model is resident and training fits alongside it with a
conservative margin (required_gb * 1.15 + 4 GB), keep it loaded so the
user can train and chat at the same time; on a multi GPU box training
lands on a different GPU and both coexist. Otherwise unload the HF/MLX
orchestrator and the llama.cpp GGUF server before training starts.
- The export subprocess shutdown stays unconditional and now runs first so
its freed VRAM is reflected in the decision.
Default deny: non CUDA backends, unestimable models, or any probe error
fall back to the previous always unload behavior.
Adds tests/test_training_vram_coexistence.py and updates two existing route
tests in test_gpu_selection.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: per-GPU floor for explicit GPU lists + don't unload chat on invalid gpu_ids
Address review feedback on the chat coexistence probe:
- Explicit gpu_ids mode now enforces a per-GPU floor in addition to the
aggregate free-VRAM check, mirroring auto_select_gpu_ids' min_per_gpu_N.
Without it, an uneven split such as free [45, 10] for a 40 GB job passed
the aggregate threshold and kept chat loaded even though the 10 GB GPU
could not hold its training shard, risking an OOM.
- Invalid explicit gpu_ids (ids outside the visible set, or a UUID/MIG
mask) make resolve_requested_gpu_ids raise. That request is rejected with
a 400 before training starts, so leave the resident chat model untouched
instead of unloading it.
- Tighten the target_modules / gpu_ids type hints to List[str] / List[int].
Adds tests for the per-GPU floor (uneven split unloads, even split keeps)
and for invalid gpu_ids keeping the chat model loaded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: only free chat VRAM once training will start; handle in-flight and CPU-only chat
Address the second review pass on the chat-coexistence path:
- Run the chat/export VRAM teardown as a before_spawn hook inside
TrainingBackend.start_training, fired only after the start guards pass.
Previously the route freed chat VRAM before calling start_training, so a
refused start (e.g. a lingering pump thread) would tear down the resident
chat model even though no training job began.
- Treat an in-flight HF chat load (loading_models set, no active model yet)
as not safely sizeable: free it rather than risk both OOMing as the load
keeps allocating after training starts.
- Do not count or tear down a GGUF llama-server confirmed to run entirely on
CPU (_gpu_offload_active is False): it holds no VRAM, so killing it cannot
help training fit.
Adds tests for the before_spawn hook (runs on start, skipped when a
subprocess is alive or a pump thread will not die, survives a hook error),
the in-flight load flag, and the CPU-only GGUF exclusion in both the resident
summary and the unload path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: treat any in-flight chat load (HF swap / mid-start GGUF) as unsafe to keep
Tighten the in-flight detection in summarize_resident_chat so the keep check
never sizes a load that is still allocating:
- Flag loading on ANY non-empty loading_models, not only when active_model_name
is empty. load_model adds the new model to loading_models before clearing the
old active_model_name, so a replacement load during a swap was previously
sized as a normal resident and could OOM as the new model finishes loading.
- Flag a GGUF server that is active but not yet healthy (is_loaded False) as
in-flight: it is still mmaping/offloading layers, so its final VRAM footprint
is unknown.
Consolidates the signal into a single resident["loading"] flag; the route frees
the chat model whenever it is set. Adds tests for the replacement HF load and
the mid-start GGUF cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten comments in chat/training VRAM coexistence (comments only)
* Studio: run before_spawn VRAM hook only after GPU-selection validation
Reviewers found the before_spawn hook fired before prepare_gpu_selection
validated gpu_ids (and before config build), so a refused start (invalid
gpu_ids -> 400, or a bad grad-clip value) could still tear down chat/export
VRAM. Move the hook to immediately before proc.start(), once all synchronous
validation and process construction have passed. This also fixes the route's
in-flight-chat loading branch, since that teardown runs inside the same hook.
Add test_hook_skipped_when_gpu_selection_rejects.
* Studio: recompute GPU auto-selection after the before_spawn VRAM hook
Codex P2: with before_spawn moved after prepare_gpu_selection, placement was
frozen against the pre-teardown VRAM state while the hook freed export/chat
afterward. Auto-selection could pin training onto a GPU the hook then cleared
(or onto a kept chat model). Split validation from placement: explicit gpu_ids
are still validated before the hook (raise -> 400, no teardown; explicit
placement is VRAM-independent), but VRAM-dependent auto-selection now runs
after the hook so it sees the freed memory.
Add test_auto_placement_runs_after_hook and test_explicit_placement_validated_before_hook.
* Studio: allow chatting during training (lift sidebar gate + VRAM-aware load guard) (#6335)
* Studio: allow chatting during training (lift sidebar gate + VRAM-aware load guard)
The sidebar disabled New Chat, project, and home navigation while a training
run was active, so users could not chat during training even though the backend
serves inference fine alongside a run. This removes that gate and adds a backend
guard so the one genuinely risky operation, loading a new local chat model
mid-training, is refused with a clear 409 when it would not fit beside the run.
Frontend (app-sidebar.tsx): drop the chatDisabled = isTrainingRunning gate and
its consumers. Navigation triggers no model load on its own, so chat stays
usable during training.
Backend (routes/training_vram.py, routes/inference.py): add
can_load_chat_during_training plus a load/validate guard that sizes the same
effective load the loader performs (LoRA 4-bit to 16-bit resolved first, HF auto
placement via auto_select_gpu_ids, explicit multi-GPU per-GPU floor, GGUF sized
from on-disk shards and companions or the selected remote variant). It is a
no-op when training is inactive, never blocks external providers or
already-resident models, and default-denies only on a CUDA sizing failure so a
load can never OOM the run. Validate refuses early with the real settings so the
frontend does not unload the resident chat model for a load that would be
rejected.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review feedback for chat-during-training load guard
- Run the load/validate VRAM guard via asyncio.to_thread so the sync
nvidia-smi + HF metadata work never blocks the event loop.
- Size the GGUF KV cache at the requested context (_estimate_gguf_kv_gb)
and add it to the local GGUF estimate so large-context picks are not
under-counted.
- Keep the requested quantization when adapter_config.json is malformed
(not a JSON object) instead of raising in _effective_load_in_4bit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: size the training load guard at the launcher's effective GGUF context
The GGUF KV-cache estimate used max_seq_length only, but the llama.cpp
launcher honors a user --ctx-size/-c in llama_extra_args. A load such as
max_seq_length=4096 with --ctx-size 131072 was sized against a 4k cache
while the server allocates 131k, so the guard could approve a long-context
GGUF load that then OOMs training. Size the guard's KV at the larger of
max_seq_length and the parsed --ctx-size (reusing the launcher's own
parse_ctx_override), keeping the conservative f16 cache so the estimate is
never smaller than what the server allocates.
The chat model picker also validated with the raw max_seq_length while
/load sizes with resolveLoadMaxSeqLength, so validate could pass, unload
the current model, then have /load reject the native-context load. Validate
now uses the same effective context; the load path is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: size the GGUF training guard at the server parallel-slot count
The KV-cache estimate assumed a single slot, but llama-server allocates the
cache across --parallel slots (app.state.llama_parallel_slots). On a Studio
launched with --parallel N>1 the guard under-sized the cache N-fold and could
approve a GGUF chat load that then OOMs training. Thread the same slot count
the loader uses into the guard's KV estimate; default 1 leaves single-slot
setups unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments for chat-during-training guard
* Studio: keep chat generation alive across navigation; Train spinner + Return to Chat
Hoist the base chat runtime above the routed outlet so navigating to Train (or any tab) no longer aborts an in-flight generation; only an explicit Stop cancels. Add a Train sidebar spinner and swap New Chat to Return to Chat while a run is active, with a lightweight completion watch so the spinner clears from any tab. Also respawn a chat llama-server killed mid-session and guard unreadable HF cache dirs that 500'd the hub model list.
* Studio: show Return to Chat on the Train tab whenever a chat is live
Previously the top sidebar item only swapped to Return to Chat while training was running; on the Train tab with an idle/just-finished run it stayed New Chat, which started a fresh thread and cancelled an in-flight generation. Show Return to Chat (and navigate back, preserving the run) whenever a generation is running or its thread is still active, or training is in progress.
* Studio: keep a running chat alive when starting a New Chat
Starting a New Chat (or switching threads) while a generation was in flight
remounted the single-chat runtime provider, which detached the in-flight run
and cut the previous chat off (it showed up frozen / empty when reopened).
Key the single-chat view by project instead of by thread or new-chat nonce so
the provider stays mounted and assistant-ui switches to a fresh thread in place.
The previous generation keeps streaming in the background and autosaves on
completion, and returning to that thread reattaches the live run instead of
reloading a half-saved one.
Also:
- "Return to Chat" now lands on the thread that is still generating rather than
the empty new chat that became active after New Chat.
- Skip the explicit /inference/cancel POST when an abort comes from a runtime
detach (navigation / background switch) rather than an explicit Stop, so a
backgrounded generation is never cancelled behind the scenes.
* Studio: make model export non-blocking and inline
The Export tab opened a full-screen modal that trapped focus, could not be
closed or cancelled while running, and showed no progress. It also stopped
training and unloaded the chat model before loading, so export could not run
alongside them.
Export now mirrors the training runtime pattern:
- Inline panel embedded where the Export Model button was, with no modal or
backdrop, so the rest of the UI stays usable during an export.
- Global export runtime store plus an app-root lifecycle hook, so a run keeps
going and streaming across navigation and is reflected on the Export nav item
from any tab.
- The worker log stream now stays connected across the load to export phase
boundary instead of stranding on "Waiting for worker output".
- Progress bar driven by phase and quant index (quant N of M for GGUF), with
elapsed time and a working Cancel.
- load-checkpoint no longer stops training or unloads inference; export loads in
its own subprocess in parallel and surfaces out-of-memory as a clear error.
- Add POST /api/export/cancel and is_export_active on /api/export/status.
* Studio: show Return to Chat on the Export tab too
Extend the New Chat to Return to Chat swap to the Export route so leaving a
running chat for Export offers a way back to the live generation, matching the
Train tab.
* Studio: smooth out Export animations and polish the panel
- Drop the height-based reveal animations (source switch, run panel, quant
picker, hub fields) that caused flashing and reflow; use instant swaps and
quick opacity fades instead.
- Method and quant cards now transition colors only, with no transition-all or
hover lift, so selecting a method or quant is crisp instead of jumpy.
- Auto-scroll the export panel into view when it opens and add a scroll-to-bottom
button when its output is below the fold, like Chat.
- Show Return to Chat on the Export tab while an export is running, matching how
training drives it on the Train tab.
- Surface the current phase or stage in the live output before the first worker
line arrives so the panel never looks stuck while progress is advancing.
* Studio: show Return to Chat on every non-chat tab
Generalize the Return to Chat swap from just Train/Export to any non-chat route
(Recipes, Projects, Hub, ...) so a running or active chat is always one click
away, instead of showing New Chat there.
* Studio: stream export logs over the Cloudflare tunnel; drop janky export animations
Exporting over a --secure Cloudflare quick tunnel showed "connecting..." with no
logs while the progress bar advanced. Cloudflare buffers text/event-stream and
only flushes when the stream closes, so the SSE log stream never reached the
browser during the run (direct localhost is unaffected, which is why this only
showed up over the tunnel).
Add a tunnel-safe JSON poll fallback (GET /api/export/logs?since=) that the
runtime lifecycle hook polls while a run is active. Short JSON responses are not
buffered by the proxy, so logs show up in near real time over the tunnel. It
shares the orchestrator's monotonic seq cursor with the SSE stream and the store
de-dupes by seq, so the two transports run together (SSE on localhost, poll over
the tunnel) without double-printing. A successful poll marks the panel
"streaming" instead of leaving it stuck on "connecting...".
Also remove the framer-motion AnimatePresence reveals from the export config and
run panel (quant picker, hub fields, the inline run panel, and the live log
section). The expand/slide animations flashed and felt clunky; the sections now
render in place.
* Studio: recover export over the Cloudflare tunnel when the blocking POST times out (524)
A model export over a --secure Cloudflare quick tunnel showed "Request failed
(524)" even though the export succeeded on the backend (the GGUF was written).
Cloudflare returns 524 when a single request takes longer than ~100s to respond,
and a GGUF conversion routinely runs for minutes, so the blocking per-method
export POST is cut off while the backend keeps going.
Confirm completion via short status polls instead of relying on the long POST
response (the same approach that fixed log streaming):
- The orchestrator records each finished op's outcome (status / output_path /
error) with a monotonic seq, exposed on GET /api/export/status.
- parseJson now preserves the HTTP status; a 524/520/522/523/502/503 or a
status-less network drop is classified as a recoverable transport error.
- runExport wraps each phase (load, every export method, each GGUF quant): on a
recoverable failure it keeps the run alive (logs keep streaming, the panel
shows "reconnecting...") and polls status until the still-running op finishes,
then settles from the recorded result, recovering the output path for the
success banner. A real 4xx still fails immediately; localhost still uses the
fast POST response. applyBackendStatus also settles a reloaded run from the
last-op record.
Verified over the tunnel: a 3m14s gemma-4-E4B-it GGUF export now ends on the
success banner with the output path instead of 524.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep the export method + logs visible after navigating away mid-export
While an export was running, navigating to another tab and back to Export
remounted the page and reset the local form state (exportMethod, quant levels),
so the method card showed unselected and the run panel's log area was hidden
until the card was re-clicked. The run itself lives in the global store and was
unaffected.
Seed exportMethod / quantLevels from the active run's summary via lazy useState
initializers on (re)mount, and gate the panel's log area on the live run
(isExporting / logLines / the run's method) rather than only the local form
selection. The card stays selected and the logs/progress stay visible across
navigation; nothing changes when no run is active.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: address export/training review findings
- Export: guard Start against an empty GGUF quant selection so an inline-panel
run with no quant can't settle as success with no file produced.
- Export: thread the source HF token into the background load so gated/private
HF source exports (and gated bases) authenticate, matching the consent path.
- Export: only settle a recovered (non-owned) run as a finished export when the
last backend op was an export, not a standalone load_checkpoint.
- Training: free the export subprocess whenever an export is active, not only
once a checkpoint is loaded, so an in-flight export load can't race training
for VRAM (current_checkpoint is unset during the load phase).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: scale export GGUF size estimates from the real model size
The Export page showed hardcoded, model-independent GGUF quant size
labels (Q8_0 ~8.2 GB, BF16 ~14.2 GB, ...) calibrated for an ~8B model.
For a 35B MoE model like Qwen3.6-35B-A3B (67 GiB bf16, Q8 ~34 GiB) the
picker wrongly reported Q8 ~8.2 GB. Only the displayed estimate was
wrong; the actual export via save_pretrained_gguf was always correct.
Add GET /api/models/export-size, which returns a model's MoE-aware
fp16/bf16-equivalent size and total params using the existing
estimate_fp16_model_size_bytes (safetensors -> config -> local -> vllm).
The result is memoized and degrades to nulls so a size hint can never
break the Export page.
The Export picker now scales each quant from that size
(bytes ~= fp16_bytes * bits_per_weight / 16, GiB units to match the
model selector), and renders no size when it is unknown rather than a
misleading fixed number. The Est. size summary in the page and dialog
is restored now that the value comes from the backend.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio export-size: address review feedback
- Run the size estimate off the event loop with asyncio.to_thread so a slow
Hugging Face request cannot stall other API or SSE endpoints.
- Cache only successful estimates; a transient failure (offline, gated before
credentials) is no longer pinned as unavailable until restart.
- Forward the HF token so private and gated models can be sized, and refetch
when the token changes.
- Clamp the size formatter index so sub-1-byte values cannot pick an
out-of-range unit.
* Studio export-size: address second review pass
- Send the HF token in an X-HF-Token header instead of the query string, so
it never lands in URLs, logs, or browser history.
- Key the estimate cache by model id only (the fp16 size is token independent),
so HF tokens are never retained in the cache.
- Restrict local-path sizing to known Studio roots (outputs/exports/cache/home)
so an authenticated caller cannot trigger a scan of an arbitrary directory.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio export-size: fix CI (import-hoist + isolated-load test stubs)
- Import ExportSizeResponse from models.models in routes/models.py instead of
re-exporting it through models/__init__.py, so the import-hoist lint does not
flag a newly added but un-loaded re-export (models/__init__.py is unchanged).
- Add Header and ExportSizeResponse to the stubbed fastapi / models.models in
test_export_absolute_paths.py, which loads routes/models.py in isolation.
* Studio: validate export-size local path before filesystem access
CodeQL flagged the export-size local-path guard as path injection: the
user-provided model path was resolved and stat-ed before it was checked
for containment under a Studio data root. Decide containment by lexical
normalization (normpath/abspath/expanduser, no filesystem access) and
only touch the filesystem once the path is proven to sit under a trusted
root, so an unvalidated value never reaches a filesystem call. Add a
direct containment unit test (under-root, root itself, missing, /etc,
and '..' traversal).
* Studio: trim export-size comments to be more concise
Shorten docstrings and comments on the export-size endpoint, helpers, tests,
and frontend size utilities; drop comments that just restate the code. Verified
code-identical (comments only) via AST/TS-compiler check. No behavior change.
* Studio: harden export-size local-path handling
Address review feedback on the export-size endpoint's local sizing:
- Resolve symlinks and re-verify containment in _is_sizable_local_path so a
symlink inside a Studio root can't point the sizer outside it.
- Re-validate the resolved LoRA base before sizing, so a crafted adapter
whose base_model points outside the roots can't redirect the scan.
- Skip nested checkpoint-*/global_step* snapshots when summing local weight
sizes so a run dir's intermediate checkpoints don't inflate the estimate.
- Size the checkpoint directory for full fine-tune checkpoint exports (whose
base may be a local/custom path), keeping base-model sizing for adapters.
Adds tests for the adapter-base escape, symlink escape, and nested-checkpoint
exclusion.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Harden model fetching: consent gate for trust_remote_code
Add a load-path consent gate that scans a model's auto_map repository code
before it executes and blocks CRITICAL/HIGH findings unless the user pins
approval of that exact code version. Capability detection stays code-free,
reading raw config.json instead of AutoConfig.
- Scan config.json and tokenizer_config.json auto_map, nested local helpers,
and external owner/name--module repos; fail closed on partial downloads.
- Gate inference, training, and export workers, including the MLX path and a
LoRA's base model, and report requires_trust_remote_code from the raw config
so chat and auto-load surface the dialog.
- Verify trusted-org auto-enable against the Hub with the request token and key
the verdict cache by token; reject local-path and spoofed names.
- Add a consent dialog showing the flagged file, line, and surrounding code.
- Thread hf_token through the scan and load paths for gated repos.
* Address review: token handling, tokenizer/LoRA scan coverage, rollback
- Send the HF token for remote-code scans in the POST body, not the URL, so it
never lands in a log or browser history.
- Collect tokenizer_config.json auto_map files directly instead of relying only
on the repo file listing.
- Resolve a LoRA's base model for the validate flag and the scan endpoint so the
dialog scans the code the workers actually gate.
- Pass the request token to the training YAML trusted-org auto-enable.
- Resend a previously approved fingerprint when rolling back to a custom-code
model after a failed switch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Consent UX: drop legacy chat toggle, fix decline copy, purge declined downloads
The per-model consent dialog is now the single approval path for custom
(auto_map) code in chat, so three leftovers from before it existed are removed:
- Remove the "Enable custom code" switch from Chat Settings and stop persisting
trust_remote_code, so a previously saved blanket-on cannot linger and load a
model without going through per-version review. The flag stays as an internal
YAML/preset default (e.g. first-party auto-enable); the load path still gates
every custom-code load on a fingerprint only the dialog produces.
- Reword the decline message and the auto-load toast to describe approving the
model's code from the dialog, not a missing settings toggle.
- On decline, purge the repo the scan downloaded so untrusted code is not left
on disk. A new /api/models/discard-remote-code endpoint deletes only a
metadata-only cache entry the scan created; it refuses local paths, loaded
models, and any repo with weight files cached, so a model the user already had
or pre-downloaded is always left untouched. The frontend only calls it when
the scan reported created_by_scan.
Adds discard-endpoint tests (delete metadata-only, refuse on weights/gguf,
refuse local, no-op when not cached) and a created_by_scan payload assertion.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Export: remove the user-facing trust remote code toggle
The Export page kept a "Trust remote code" switch (default on) next to the HF
token field. Like chat, custom (auto_map) code should be approved per model
through the load-time review dialog, not a persistent blanket switch, so the
toggle is removed. The export load path already routes through the same consent
dialog: an HF source now starts with trust_remote_code off and only enables it
when the user approves the scanned code in the dialog (a local checkpoint the
user exported stays trusted by default). With the dialog unreachable and no
approval, an HF source loads with trust_remote_code off, which fails closed
rather than running unreviewed code.
* Block loads of repos with unsafe files using Hugging Face's security scan
The trust_remote_code consent gate covers one load-time RCE vector (a repo's
auto_map Python). It does not cover the other: a malicious pickle inside a weight
file (pytorch_model.bin, *.pkl, *.dat) deserializes during from_pretrained even
with trust_remote_code False, so a repo with a normal config plus a poisoned
pickle slips past the existing gate.
Add a metadata-only malware gate that uses Hugging Face's own scan (picklescan +
ClamAV), read via model_info(securityStatus=True).security_repo_status. It never
downloads, opens, or unpickles the flagged files; it only reads the Hub's verdict
and surfaces the flagged file names. New evaluate_file_security runs
unconditionally (independent of trust_remote_code) in every load path (inference,
training SFT/MLX, export), blocking the load when a file is flagged
unsafe/suspicious/malicious. The /remote-code-scan preflight and the validate
endpoint also report the result so the consent dialog opens as a hard block (no
override) listing the flagged files, even for a repo with no custom code.
Policy: hard block with no user override; fail open when the scan is unavailable
(offline/unscanned) so legitimate loads are not broken; no first-party exemption
(a poisoned pickle in a compromised trusted repo still blocks); local paths and
GGUF are skipped (no Hub scan, non-pickle format). Blocking does not gate on
scansDone, since that is often false for clean repos and a file already flagged
unsafe is unsafe regardless.
Adds test_file_security.py covering the block/allow/fail-open/skip matrix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: scan list-form tokenizer auto_map, gate unsafe files on all load paths
Fixes from a 10-reviewer pass on the model-fetching hardening:
- The remote-code scanner skipped tokenizer auto_map encoded as a [slow, fast]
list (transformers' standard tokenizer shape, e.g.
{"AutoTokenizer": ["owner/repo--tokenization_x.Slow", null]}). External
tokenizer code in that form was never fetched, scanned, or fingerprinted, so an
AutoTokenizer(trust_remote_code=True) load could run it. _auto_map_refs now
flattens string, list, and nested values. Adds a regression test.
- Compare-mode chat loads and background auto-load only gated on
requires_trust_remote_code, so a repo flagged unsafe by the Hub scan but with no
custom code skipped the hard-block dialog. Both now also gate on
requires_security_review, matching the main chat path.
- The /remote-code-scan and /validate routes collapsed a LoRA adapter to its base
before the malware scan, so unsafe files in the adapter repo itself were missed
in the pre-load review (the workers already scan both). Both routes now run the
file-security scan over the adapter and the base.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Require approval for all HIGH remote code, fail closed when unscannable
Tighten the load-time security gates based on review:
Consent gate
- HIGH-severity auto_map code now requires explicit, per-version approval for
every repo, including first-party unsloth/nvidia. The org is no longer a
blanket bypass: a compromised first-party repo with HIGH code still warrants
review. CRITICAL stays a hard block; clean code still loads after the consent
prompt.
- Fail closed when auto_map code is present but cannot be fully fetched or
listed to scan (gated, offline, transient, or a repo-listing failure that
could hide an imported helper). We cannot fingerprint code we cannot see, so
this is a non-approvable block, retryable once the repo is reachable.
- Scan auto_map from every config that can carry one (model, tokenizer, image
and feature processor, processor, video processor), not just config.json and
tokenizer_config.json, so a custom-processor model is not missed. The file
list is the single source of truth in remote_code_scan and is pinned to the
transformers filename constants by a guard test.
- Distinguish a genuine 404 (config truly absent) from a transient error: only
the latter forces a scan, so a repo with no config is correctly a no-op.
Malware gate
- Scan a remote repo even when its name ends in .gguf; only local paths skip the
Hub scan, so a repo cannot dodge the scan by naming itself "*.gguf".
- Correct the docstring: a file already flagged unsafe blocks regardless of
scansDone; the only fail-open path is an unavailable scan.
Coverage
- Resolve a remote LoRA adapter's base model (not just local directories) so the
base, where the code and weights actually execute, is scanned in validate,
the scan route, and the training and export workers.
- Gate the embedding training path (FastSentenceTransformer) with the malware
and consent checks, matching the other load paths.
Tests updated and added for each change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope malware gate to the load-path vector; stop false-blocking first-party models
Follow-up hardening from a second review pass + a broad live model matrix
(unsloth/* , nvidia/* , third-party, and the eicar malware repo).
Malware / unsafe-file gate
- Scope the block to the actual RCE vector: a root-level file in a code-executing
format. from_pretrained deserializes weight files at the repo ROOT, so a flag is
only a load-path pickle vector there. Two exclusions, because neither is loaded:
inert formats (safetensors is tensor-only, gguf is non-pickle, configs/text/
images) and files in subdirectories. This keeps eicar blocked (its *.pkl/*.dat/
eicar_test_file sit at the repo root) while no longer false-blocking legitimate
first-party repos: nvidia/Nemotron-H-8B-Base-8K ships root safetensors plus NeMo
pickle checkpoints under nemo/ that the loader never touches, and the Hub flags
both; the gate previously hard-blocked it.
- Unknown / future non-"safe" levels now fail closed (block) instead of being
silently allowed, so Hub schema drift cannot introduce a bypass; in-progress
("pending"/"scanning"/"error") levels stay non-blocking to avoid false blocks.
Consent gate
- Ignore a STALE own-repo auto_map target that is absent from the repo listing (an
older config pointing at a file the repo no longer ships) instead of failing the
whole repo closed as unscannable. The present .py are still fully scanned, which
is the stronger coverage, and a file that is not there cannot execute. This
unblocks first-party models like unsloth/PaddleOCR-VL (its tokenizer_config.json
names processing_ppocrvl.py while the repo ships processing_paddleocr_vl.py). A
referenced .py that IS present but cannot be fetched, and a repo-listing failure,
still fail closed.
Remote LoRA base resolution
- Distinguish a genuine 404 (not a LoRA / repo absent -> None) from a transient
error: the transient case is retried once, then logged as a WARNING (a missed
base is scanned by neither gate) rather than silently skipped.
Discard endpoint
- Treat .onnx and .ckpt as weights so a repo whose only heavy artifact is one of
those is never eligible for the declined-download purge.
Tests added for each: load-path scoping (safetensors/subdir/Nemotron-H shapes,
unknown-level fail-closed, pending non-block), stale own-repo auto_map ref, remote
LoRA transient retry, and the empty-config-list (all-404 -> []) semantics.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make LoRA-base transient-warning test robust to logging backend
Assert on the logger object directly instead of capsys, so the test does not
depend on whether the real structlog logger or the module-stub logger is active
(which varies with test collection order).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Allow a repo with auto_map but no executable code (e.g. GGUF) instead of blocking
A config can declare an auto_map yet the repo ship NO executable .py -- most
commonly a GGUF repo whose config.json carries an auto_map copied from the original
model (e.g. unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF references
modeling_decilm.py, which the GGUF-only repo does not contain). A GGUF model loads
through llama.cpp, which never executes auto_map, and transformers cannot run a file
that is not present, so there is nothing to scan and trust_remote_code is a no-op.
The fail-closed change treated this empty result the same as "code is present but we
could not fetch it" and hard-blocked the load. Distinguish the two: repo_remote_code_files
now RAISES RemoteCodeUnscannable when code is present but cannot be fully fetched or
listed (offline / gated / transient / a present .py that 404s / a listing failure),
and returns an empty dict only when the listing succeeded and the repo genuinely ships
no executable .py. The consent gate blocks on the exception (fail closed) and allows the
empty case as a no-op. Real unscannable code still hard-blocks; eicar and CRITICAL/HIGH
custom code are unaffected.
Verified against all 37 unsloth/*Nemotron* models (two GGUF repos were false-blocked,
now load) and the existing matrix (eicar still blocks; DeepSeek-OCR / NVLM-D-72B still
prompt approvable consent). Tests updated to expect the raise for unscannable cases and
added for the no-executable-code no-op.
* Ignore vestigial auto_map in GGUF repos (llama.cpp never runs it)
A GGUF repo's config.json is often copied verbatim from the original
transformers model, auto_map and all, but a GGUF load goes through
llama.cpp which never executes auto_map, so the config is inert. Treat
a direct .gguf reference, and a repo that ships .gguf weights with no
.safetensors, as having no remote code so the consent flow is never
triggered. A mixed repo with both .gguf and .safetensors is still gated,
since the safetensors variant would load through transformers where
auto_map does run. The check sits behind the existing auto_map-present
gate so normal models pay no extra repo listing.
* Add scanner-result copy to the remote-code consent dialog
Make the consent dialog state the scan outcome in plain language for
every model. When the static scan finds nothing, reassure the user with
'Our automatic scanner did not flag any worrying files, but please
double check.' (shown only for the clean, approvable case). When the
scan flags custom code or unsafe files, label the list with 'Our
automatic scanner flagged issues including:'. The Hugging Face
attribution for unsafe files stays in the dialog description.
* Close GGUF-suffix consent bypass for repo ids ending in .gguf
The .gguf short-circuit in _config_has_auto_map skipped the scan for any
model name ending in .gguf, including a bare two-segment repo id like
'evil/model.gguf'. Such a repo can still ship safetensors plus auto_map
Python that transformers would execute, so skipping the scan was an
asymmetric bypass (file_security already scans those repos). Restrict the
short-circuit to genuine direct GGUF file references via
_is_direct_gguf_file_ref: a local .gguf path, or a remote repo_id plus
filename (three or more segments). A two-segment repo id named *.gguf now
falls through to the config scan and _is_gguf_repo file inspection, so it
only skips consent when it actually ships .gguf weights and no safetensors.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align consent dialog body with the title and fix narrow-width overflow
The scan results (the 'Our automatic scanner...' label, finding/unsafe
cards, and the clean-scan reassurance) sat at the dialog's left padding
while the title and description were indented past the status icon, so
the body did not line up under the description. Move the title,
description and results into one column to the right of the icon so they
share a left edge, and let that column fill its width so the description
no longer wraps early.
Also stop a wide code snippet from pushing the dialog off-screen on
narrow viewports: AlertDialogHeader is a grid with place-items-center,
which sized the content row to its content; give the row w-full so it
fills the track, and add min-w-0 down the results chain so the snippet
scrolls inside its card instead of widening the dialog. Verified aligned
and contained from mobile portrait through ultrawide.
* Treat a repo as GGUF-only only when it ships no transformers weights
_is_gguf_repo excluded only .safetensors, so a repo with a .gguf and a
pytorch_model.bin (or .pt/.pth/.h5/.msgpack/.onnx/.ckpt) and no
safetensors was treated as GGUF-only and skipped the consent scan, even
though transformers can load that weight set and execute the repo's
auto_map code. Require the absence of ANY transformers-loadable weight
before treating the repo as a llama.cpp-only GGUF load. A genuine
GGUF-only repo (only .gguf) is still inert; a mixed repo with any pickle
or safetensors weight is gated. Adds a regression test across all the
non-safetensors weight formats.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Block flagged subdir weight shards referenced by a root index
The malware gate treated every subdirectory file as non-loadable, but
from_pretrained deserializes a subdir shard a root index references
(pytorch_model.bin.index.json -> shards/...-00001-of-00002.bin). Read the
root weight indexes and block a flagged subdir pickle the weight_map
points at; a flagged subdir pickle no index lists (NeMo nemo/*.distcp)
stays non-blocking, and an inconclusive index lookup fails closed.
* Pass hf_token to the export checkpoint load
ExportBackend.load_checkpoint scanned with hf_token in the worker but
loaded the weights unauthenticated, so a gated/private checkpoint passed
preflight then 401'd at from_pretrained. Add hf_token to load_checkpoint
and forward token to every from_pretrained branch; the worker passes the
command's hf_token.
* Scope created_by_scan to every HF cache the discard searches
created_by_scan used get_cache_path (active HF_HUB_CACHE only) while
/discard-remote-code deletes across active, legacy, and default caches. A
repo the user already had in a legacy/default cache was marked
scan-created and deleted on decline. Check all three caches for the repo
dir before declaring the scan created it.
* Scan the full .py closure of external auto_map repos
An auto_map cross-repo ref (owner/name--module.Class) only had its entry
file downloaded, but transformers also fetches that file's relative
imports from the same repo, so a dangerous helper.py was left outside the
scanned fingerprint. List each external repo's .py and scan the whole set
(plus the referenced entry files); fail closed if the repo cannot be
listed or fetched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fail closed when a weight index cannot be fully read
_indexed_shard_paths treated a partial result as definitive: if one weight
index read cleanly but another failed transiently, it returned the shard
paths it did see. A flagged subdirectory pickle listed only by the index we
could not read would then be classed as "not a load input" and skipped,
re-opening the very fail-open this guard was added to close.
Return None whenever any index read is inconclusive, even if another read
cleanly, so the caller blocks the already-flagged subdir pickle. A repo that
ships no index files raises EntryNotFoundError for each (never inconclusive)
and still returns an empty set.
* Match cached repos case-insensitively in the created_by_scan guard
_repo_in_any_hf_cache resolved casing only against the active cache and then
probed every cache with an exact directory name. A case-variant already
present in a legacy or default cache (models--Unsloth--Foo for a scan of
unsloth/foo) was missed, so the repo was marked created_by_scan and deleted
on decline -- but discard_remote_code_download deletes case-insensitively,
so that delete would hit the user's pre-existing cache entry. Detect
case-insensitively too, mirroring the deletion path.
* Skip remote-code and security review for selected GGUF variants
validate_model ran the trust_remote_code and Hugging Face security-scan
preflight against the repo even when the selected artifact is a .gguf. A
GGUF loads through llama.cpp, which never executes the repo's auto_map
Python and never deserializes root pickle weights, so repo-level Transformers
artifacts (a config.json with auto_map, or an unsafe pytorch_model.bin next
to the .gguf in a mixed repo) are inert for that load. Gating the GGUF on
them is a false positive. Run both preflights only for non-GGUF loads.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope the malware gate to actual load roots and serialized files
Two fixes to evaluate_file_security so it neither misses a load-path pickle nor
false-blocks an inert file:
- Honor subdirectory load roots. Spark-TTS / BiCodec call from_pretrained on the
snapshot's LLM subdirectory, so a flagged pickle directly under it is a
root-level load artifact there. A new load_subdirs parameter (set from the
model's audio type via security_load_subdirs) reclassifies those files relative
to the load root and looks for weight indexes under it, so a flagged shard in
that subdir is no longer skipped as "not root-level".
- Exempt source files. A root .py is never deserialized by from_pretrained;
executable repo code runs only through auto_map, which the remote-code consent
gate scans. Flagging a Python helper here would false-block a repo that merely
ships a build or train script.
* Scan a LoRA adapter and base as one consent unit, and gate MEDIUM code
A LoRA load runs both the adapter's and the base's repo code. The consent gate
scanned them separately and pinned one fingerprint per repo, so an adapter that
shipped its own auto_map code was either never shown in the dialog (which only
saw the base) or impossible to approve with the base's fingerprint.
evaluate_remote_code_consent_for_targets now scans all of a load's repos as a
single combined unit and pins ONE fingerprint over the union of their code, so
approving the load approves every repo's code together. evaluate_remote_code_consent
becomes a thin single-target wrapper, and an unscannable target fails the whole
load closed.
Also gate MEDIUM findings: like HIGH they now block pending pinned approval, so a
direct API caller cannot run flagged code by setting trust_remote_code=True
without consenting. Only a clean scan loads without a fingerprint.
* Preflight a LoRA load's adapter and base as one combined consent scan
scan_model_remote_code rewrote a LoRA adapter to its base and scanned only the
base for remote code, so the dialog never surfaced an adapter's own auto_map
code. Scan the adapter and base together through
preflight_remote_code_consent_for_targets, which pins one combined fingerprint
the worker gate accepts. The malware preflight is also scoped to each target's
load subdirectories.
* Apply combined consent and subdir-aware malware scan in load workers
Each load worker (inference, export, training) evaluated remote-code consent
once per target with a single shared fingerprint, so a LoRA adapter that ships
its own auto_map code could not be approved by the base's fingerprint. They now
scan the adapter and base together via evaluate_remote_code_consent_for_targets,
which pins one combined fingerprint over the union of their code. The malware
scan in each worker is also scoped to the model's load subdirectories so a
flagged pickle under a from_pretrained load subdir is not missed.
* Report a consistent trust_remote_code requirement after a model loads
validate_model reports requires_trust_remote_code from the YAML default OR the
raw auto_map, but the load, already-loaded, and status responses reported only
the YAML default. A custom-code model approved and loaded via auto_map was then
reported as not requiring trust_remote_code, so the frontend stored false and a
later retry or rollback sent trust_remote_code=false and failed.
A shared resolver reports the same requirement for a loaded model (a value
stored at load time, else the trust_remote_code the load used, else the YAML
default, else the raw auto_map check), and the load response persists it so the
status and already-loaded paths stay consistent. The selected-GGUF security
review is also scoped to the model's load subdirectories.
* Run the consent gate on training resume and for YAML-only trust_remote_code
Three frontend gaps left a model loading without the trust_remote_code it needs:
- The shared consent helper returned early when the scan found no auto_map and no
unsafe files, dropping a requirement that comes from a model's Studio YAML
default (e.g. GLM-4.7-Flash). It now grants the caller's requirement with an
empty pin instead of sending trust_remote_code=false.
- Resume-from-history called startTraining directly with no consent gate, so a
resumed run whose model needs custom code (or an old run with no approved
fingerprint) hit the worker block with no dialog. It now runs the same gate as
a fresh start.
- HF export passed requiresTrustRemoteCode=false for every HF source, so a
YAML-only model could not flip the flag before export. It now signals the
requirement for HF sources.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cover both LoRA repos in validate, report GGUF as inert, purge all declined repos
Three follow-on gaps from the combined adapter+base consent work:
- validate_model resolved requires_trust_remote_code from the base alone, so a
LoRA adapter that ships its OWN auto_map code (with a plain base) was reported
as not needing trust_remote_code and the consent dialog never opened. It now
checks the [adapter, base] target set, matching the scan route and the workers
(which already gate both) and the security review already running over both.
- The already-loaded, loaded, and status responses for a selected GGUF reported
requires_trust_remote_code from the model's YAML default. A GGUF loads through
llama.cpp, which never executes the repo's auto_map Python, so the requirement
is inert for that load. They now report False, matching validate_model (which
already skips both gates for GGUF) so a status refresh cannot flip the flag
back on.
- The remote-code scan downloads both the adapter's and the base's config, but
created_by_scan tracked only the primary, so a base the scan was first to pull
into the cache was left on disk when the user declined. The scan now reports
scan_created_repos (every repo it newly cached) and the decline cleanup purges
each; created_by_scan stays for older clients. The frontend falls back to the
primary flag when the list is absent.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scan the repo the load fetches, purge external code on decline, harden consent pins
Six follow-on hardening fixes from a fresh review pass over the gate:
- The malware gate scanned the literal "Spark-TTS-0.5B/LLM" alias, but the trainer
downloads it as unsloth/Spark-TTS-0.5B and loads LLM/, so the alias 404'd and
failed open, missing a flagged LLM/ pickle. evaluate_file_security now resolves
the alias to the repo the loader fetches and scans LLM/ as a load root.
- security_load_subdirs relied only on tokenizer detection, which fails on an
unresolved alias or offline; it now also honors the Studio YAML audio_type
default, so a BiCodec LLM/ load root is not missed.
- The remote-code scan downloads external auto_map repos (owner/name--module.Class),
but the decline cleanup tracked only the model/adapter/base, leaving the external
untrusted code cached. The scan now enumerates external auto_map repos and reports
the ones it created in scan_created_repos, so a decline purges them too.
- External auto_map refs failed the whole load closed on a stale or mis-derived
dotted ref (sub.mod.py vs the real sub/mod.py) even though the actual file was
present and scanned. They now drop such refs when the repo listing is real, exactly
like the own-repo path; an empty/incomplete listing still fetches and fails closed.
- The combined consent fingerprint keyed code by the raw target string, so the scan
endpoint's canonicalized casing and a worker's raw user input produced different
pins for identical code, rejecting a valid approval. Hub repo ids are now folded to
lowercase in the key (local paths stay case-sensitive), so the pin tracks the code.
- Export threaded hf_token into the weight load but not into detect_audio_type /
is_vision_model, so a gated multimodal base 404'd in detection and fell through to
the text loader. Both probes now use the same token.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Thread the token through check-vision and guard the gate's parallel sites
The /check-vision endpoint classified a model without the hf_token, so a gated or
private vision model 404'd in the probe and was reported as a plain text model --
the same dropped-token shape as the export probes, at a sibling site. It now passes
the token like the neighboring /check-embedding endpoint.
Add deterministic consistency guards (tests/test_security_gate_consistency.py) that
enumerate the gate's parallel sites mechanically instead of relying on a review to
spot a missed sibling: every is_vision_model / is_embedding_model / detect_audio_type
caller under routes/ and core/ must thread the token, every GGUF response must report
trust_remote_code via the resolver or False (never the raw YAML default), and every
load worker that runs the malware or consent gate must resolve the LoRA base. A new
site that drops the token or mis-reports the requirement now fails CI directly.
* Narrow the LLM alias rewrite and make audio detection token-aware
Three fixes from the confirmatory review, one a regression from the previous round:
- _load_scan_target rewrote EVERY remote repo ending in "/LLM" to unsloth/<parent>,
so a real third-party repo named "<owner>/LLM" was scanned as unsloth/<owner>
while the loader still fetched the real repo -- a fail-open hole introduced when
the Spark-TTS alias handling was added. It now rewrites only a registry-known
bicodec alias; every other "/LLM" repo is scanned as itself.
- detect_audio_type cached results under the bare model name, so an unauthenticated
probe of a gated/private repo cached None and poisoned a later authenticated call
with the token. The cache is now keyed by (normalized_name, token_fingerprint),
matching the vision cache.
- The training fallback /check-vision call dropped the hf_token, misclassifying a
gated/private VLM when the config endpoint failed. It now passes the token, like
the getModelConfig call it falls back from; checkEmbeddingModel takes the token too.
Extend the consistency guards: every capability cache must be keyed by a tuple
including the token, so a cache re-declared as Dict[str, ...] fails CI.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Document the broad .py scan as deliberate and enforce it with a test
The remote-code scanner scans every .py in a repo once an auto_map exists, not
just the auto_map entry's static import closure. This is intentional: the entry
module can reach a sibling via an absolute import, importlib, or exec, none of
which a static relative-import closure follows, so closure-only scanning would be
a real bypass of a load-time RCE gate. The broad scan never under-scans; the cost
is that an unrelated benign script can over-block, which is the safe failure
direction (HIGH stays approvable; only CRITICAL hard-blocks).
Spell this out at both the local and remote scan sites so the choice reads as
deliberate, and add a test asserting an unrelated, never-imported .py is still
scanned -- so a future narrowing to the static closure fails CI.
* Purge a declined remote LoRA adapter the scan downloaded
scan_model_remote_code probed the created-by-scan state AFTER resolving the base,
but get_base_model_from_lora_identifier downloads a remote adapter's own
adapter_config.json, so the adapter looked already-cached and was dropped from
scan_created_repos. On decline the adapter -- including the auto_map .py the
preflight fetched -- was left on disk, defeating the "untrusted code is not left
on disk" guarantee for the adapter itself.
Snapshot the primary's cache state BEFORE base resolution and use it when marking
the adapter scan-created; on any probe error treat it as pre-existing so a decline
never deletes it. The base and external repos are unaffected (their configs are not
downloaded before their own probe). Add a test that models the mid-scan download
side effect, which the prior static-stub tests did not.
* Clear remote-code approval when the training model changes
Switching the training model from an approved custom-code model to a clean one
kept the previous model's trust_remote_code=true and approved fingerprint in the
store: setSelectedModel reset visionImageSize on a true switch but not the
remote-code approval. The clean model then trained with trust_remote_code=true,
which bypasses the compiler and disables fused cross-entropy.
Reset trustRemoteCode and approvedRemoteCodeFingerprint on a true model switch.
The new model's own YAML default is re-applied by loadAndApplyModelDefaults, and a
custom-code model still re-opens the consent dialog before training starts, so the
only change is that a clean model no longer inherits a stale approval.
* Trim verbose comments across the model-fetching hardening changes
Condense the explanatory comments and docstrings introduced across the
trust_remote_code consent gate, the malware/unsafe-file gate, the remote-code
scanner, the load workers, the model routes, and the security frontend into
fewer, tighter lines while preserving every security rationale (fail-open vs
fail-closed direction, the deliberate broad-scan anti-bypass note, the
empty-vs-unscannable distinction, stale-ref handling, and the alias-rewrite
spoof guard).
Comments and docstrings only. No code, logic, identifiers, or test behaviour
changed; verified comment-only via the AST/TypeScript checker (40/40), with the
backend test suite and frontend tsc green.
* Do not cache transient audio-detection failures
detect_audio_type cached _detect_audio_from_tokenizer's result
unconditionally, so a transient read failure (network error or 5xx,
returned as None) poisoned the cache and the later successful probe never
ran. Mirror the vision cache: _detect_audio_from_tokenizer now returns
(audio_type, definitive) and the caller caches only definitive results.
A read that succeeds with no audio tokens, or clean 404s for every
tokenizer path, stays a cacheable None; only a genuine transient failure
(connection error, timeout, 5xx, malformed body) skips the cache so the
next call retries.
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add 'Load on selection' toggle to configure load options before loading
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: seed staged speculative decoding from the standing default
* Studio: address PR review for load-on-selection staging
* Studio: handle direct GGUF staging and stale-stage edge cases from load-on-selection review
* Studio: cancel replaced staged downloads and keep staged pick on load failure
* Studio: centralize staged-download cancel and guard staged-load restore
* fix: address staged GGUF load review
* fix: honor staged GGUF load metadata
* fix: clarify load-on-selection tooltip
Keep the load-on-selection hint visually anchored to the control and make the on/off behavior explicit without changing the broader deferred-load flow.
* Studio: reset orphaned staged knobs on abandon and cap Max Tokens to staged context
* Studio: remove dead code and cancel staged download when loading a different model
* fix: surface staged model in run settings before deferred load
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Studio: Add inline confirmation (Allow/Always allow/Deny) for tool calls
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix race in tool-call confirmation gate
* Studio: gate built-in tool calls and harden the confirmation handshake
The Allow / Always allow / Deny controls only lived in the fallback tool
card, but the built-in tools (web search, python, terminal, code
execution, image generation) render with their own components and so
never showed the buttons. Those calls paused after tool_start with no way
to approve them, hanging until the 1 hour timeout. Only MCP tools, which
use the fallback renderer, actually worked.
Render the controls for every tool card by wrapping each registered tool
component (and the fallback) in thread.tsx with a shared
ToolConfirmationControls, so the gate applies uniformly.
Also make the handshake robust:
- The gate keys on a per-call approval_id minted by the backend and
echoed in tool_start, instead of session_id alone, so a stale or
concurrent confirmation can no longer resolve the wrong call.
- The approval slot is registered before tool_start is yielded, closing
the race where a fast click or an auto "Always allow" could reach the
backend before the waiter existed.
- The frontend resolves with the same session id the request was sent
with (plus the approval_id), fixing the new-thread mismatch where the
confirmation targeted a different session than the blocked stream.
- The confirm endpoint returns {resolved}; the UI keeps the buttons and
shows a retry hint until the backend confirms a match, instead of
hiding them on a failed or mistargeted post.
- The gate runs after the disabled-tool and duplicate-call checks, so a
call that will not execute is not put up for approval. A denied call is
still excluded from duplicate detection, so re-issuing and approving it
works.
- "Always allow" is scoped per session to match the backend gate.
Add backend tests for the approval registry, the SSE no-deadlock
handshake, and the loop integration (allow, deny, disabled, duplicate,
re-issue after deny).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Move "Confirm tool calls" to the Tools section
* Studio: add Bypass Permissions (skip confirmation, disable tool sandbox)
Adds an opt-in Bypass Permissions toggle next to Confirm tool calls. When on,
no tool call shows a confirmation prompt and the python/terminal sandbox is
disabled: safety checks, command blocklist, and resource limits are skipped.
Secret env vars are still stripped and HOME stays repointed at the session
workdir. Default off keeps current behavior, and it takes precedence over
Confirm tool calls. Enabling it requires accepting a warning each time.
* Studio: harden Bypass Permissions secret handling and fix Anthropic tool path
Follow-up to the Bypass Permissions feature. Addresses the review findings:
- Anthropic /v1/messages 500: declare bypass_permissions on
AnthropicMessagesRequest so tool requests that omit the field default to
False instead of raising AttributeError (extra='allow' does not set absent
attributes).
- /proc parent-env leak: stripping the child env did not stop a same-uid
bypassed child from reading /proc/<parent>/environ to recover the
tool-executing process's unfiltered secrets. Clear PR_SET_DUMPABLE on that
process before the first bypass exec so its /proc entries become root-owned.
Hardening is fail-closed: if prctl is denied, bypass execution is refused
rather than run with the parent environ still readable. Mitigation, not a
full boundary; documented in the code.
- Broker/capability vars: strip SSH_AUTH_SOCK, SSH_AGENT_PID, GPG_AGENT_INFO,
GNUPGHOME, KUBECONFIG, DOCKER_HOST so a bypassed tool cannot use the
operator's live agents.
- Credential-bearing URL values: drop any env var whose value embeds URL
userinfo (scheme://user:pass@ and token-only scheme://token@) regardless of
the variable name. Benign proxy/index URLs without credentials are kept, so
proxy-only and internal-index setups still work in bypass mode.
- Windows temp isolation: repoint TEMP and TMP (not just TMPDIR) at the
per-session sandbox dir.
- Frontend: stop persisting bypassPermissions; a reload now starts with the
sandbox/confirmation bypass off and requires re-accepting the warning dialog.
Adds regression tests for each finding in test_bypass_permissions.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip cred-location env vars (HF_HOME etc.) in Bypass Permissions
Repointing HOME did not stop SDKs auto-reading cached creds via vars that
point at the real home/cache/config: HF_HOME (startup always sets it; token
lives under $HF_HOME/token), HF/XDG cache roots, NETRC/BOTO_CONFIG/
PIP_CONFIG_FILE, and Windows HOMEDRIVE/HOMEPATH. Drop those, and repoint
USERPROFILE/APPDATA/LOCALAPPDATA at the per-session workdir. Adds regression
tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: lock in bypass HF token resolution with an end-to-end test
The drop-based fix relies on the whole HF_HOME/XDG fallback chain being
removed so huggingface_hub resolves under the repointed HOME. Add a test
that sets HF_HOME and XDG_CACHE_HOME at a real cache and asserts the
resolved token path lands under the workdir, not the operator's cache.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip npm _auth, MYSQL_PWD, and BASH_ENV from bypass env
Three more credential vectors dodged the bypass scrubber: NPM_CONFIG__AUTH
(npm _auth, base64 so no URL userinfo and no AUTH marker), MYSQL_PWD (markers
use PASSWD, not PWD, since PWD is the cwd var), and BASH_ENV (bash -c sources
it for non-interactive shells, so a startup file can re-export stripped
secrets). Add an AUTH marker, the exact MYSQL_PWD name, and drop BASH_ENV plus
PGPASSFILE. Adds regression tests incl. an end-to-end check that a bypass
terminal call does not source BASH_ENV.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: extend bypass env scrubber and enforce confirm precedence in loops
From a parallel review pass over the bypass changes:
- Drop more credential-location vars in _build_bypass_env: npm/yarn/git/cargo/
rclone config pointers (NPM_CONFIG_USERCONFIG, NPM_CONFIG_GLOBALCONFIG,
YARN_RC_FILENAME, GIT_CONFIG_GLOBAL, GIT_CONFIG_SYSTEM, CARGO_HOME,
RCLONE_CONFIG) and the GIT_ASKPASS/SSH_ASKPASS auth helpers.
- Enforce confirm_tool_calls AND NOT bypass_permissions inside the safetensors
and GGUF tool loops, not just at the route, so a direct internal caller
passing both flags never prompts.
- Soften the toggle hint: environment secrets are stripped, but bypassed code
can still read files and credentials on the machine (no overclaim that keys
stay hidden).
Adds regression tests for the new names and the loop-level precedence.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add GGUF loop test for bypass-over-confirm precedence
The safetensors loop precedence is covered behaviorally; the GGUF loop needs a
live llama-server so add an AST guard asserting its _needs_confirm gate
references both confirm_tool_calls and bypass_permissions, matching the other
llama_cpp source-inspection tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add red Bypass Permissions badge in the composer
When Bypass Permissions is on, show a persistent red pill in the composer
tool-pill row (like the Search/Code pills), matching Claude Code's always-
visible bypass indicator. Clicking it turns bypass off, mirroring the other
composer toggles. Enabling still goes through the settings toggle + warning
dialog. Adds a data-variant=danger style for the destructive-colored pill.
* Studio: show Bypass Permissions badge in the Thread composer too
The empty-state and active Thread render their own composer (thread.tsx),
not shared-composer, so the badge only appeared in the split layout. Mirror
the red dismissible pill in ComposerAction so it shows in every composer.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep the Bypass Permissions badge visible when the composer is collapsed
The Thread composer only renders the pill row when expanded, so the active-mode
badge vanished on the default (collapsed) empty state. Render it before the
expand gate (it returns null when bypass is off) so the red indicator always
shows while bypass is on.
* Studio: make the Bypass Permissions confirm button a solid red button
The destructive button variant is a subtle 10% tint that read as bare red text
next to the outlined Cancel. Force the solid destructive fill (the variant's
class loses to the tint through AlertDialogAction's Slot merge, so use the !
override the codebase already uses for this case) and shorten the label to
'I understand' so it fits the small dialog's two-column footer.
* Studio: add Bypass Permissions to the composer + More menu
Adds a 'Bypass Permissions' entry to the composer plus-menu (under More by
default) in both composers, so it can be toggled without opening Run settings.
Enabling routes through the same danger warning dialog; disabling is immediate.
A shared BypassPermissionsMenuItem keeps the two composers in sync.
* Studio: harden bypass env scrubber for IMDS opt-out and connection strings
Two gaps in the Bypass Permissions secret scrubber:
- The broad AWS_ prefix also dropped AWS_EC2_METADATA_DISABLED, a non-secret
opt-out. Removing it re-opens the IMDS instance-role credential path that the
operator explicitly disabled, so a bypassed boto/AWS-CLI call could recover
cloud creds. Keep that flag (and AWS_EC2_METADATA_V1_DISABLED) via a keep-list
while still stripping the real AWS credential vars.
- Azure App Service connection strings (SQLCONNSTR_/CUSTOMCONNSTR_/...,
WEBSITE_CONTENTAZUREFILECONNECTIONSTRING) and values like Password=/AccountKey=
/SharedAccessKey= slipped past the name and URL-only value classifiers. Add
CONNSTR/CONNECTIONSTRING name markers and a connection-string value matcher.
* Studio: let Bypass Permissions suppress the confirm-tool-calls guards
The confirm-vs-bypass precedence (confirm and not bypass) was applied at the
loop call sites but not at the earlier request guards, so a client sending
confirm_tool_calls + bypass_permissions together was rejected (stream=true
required / unsupported for external or Anthropic tools) before the precedence
took effect. Gate all four confirm guards on not bypass_permissions so both
flags together proceed with the gate suppressed, matching the documented rule.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: add Anthropic-compatible thinking parameter
Add `thinking` parameter using Anthropic's format ({type: 'disabled'} /
{type: 'enabled'}) alongside the existing `enable_thinking` boolean for
backward compatibility.
The new parameter is mapped internally to `enable_thinking` at the route
layer so all downstream templates and backends continue to work unchanged.
Changes:
- Add ThinkingConfig model and `thinking` field to ChatCompletionRequest
- Add mapping logic in routes: thinking.type -> enable_thinking
- Add `thinking` field to frontend TypeScript types
- Update frontend request building to send thinking parameter
- Add tests for new thinking parameter
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: move thinking→enable_thinking mapping to model_validator
The Gemini review correctly identified that the route-level mapping
bypasses normalization for external provider requests. Moving the
mapping into a @model_validator on ChatCompletionRequest ensures it
runs during Pydantic validation regardless of routing path.
* Document ThinkingConfig scope and thinking validation behavior
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Expose MLX grad value clipping in Studio
* update test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* dataset ordering + wd
* fix mlx smoke step expectations
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* cast norm activation output back to original input dtype
* address mlx studio review feedback
* Fix present-but-None seed override for PR #5656
studio/backend/core/training/worker.py
`config.get("model_random_state", random_seed)` only fills the
default when the key is absent. When a caller passes
`config["model_random_state"] = None` explicitly (which happens
any time a JSON payload sends an explicit `null`), the old code
forwarded `None` to FastMLXModel and disabled deterministic init
silently. Same for `lora_random_state`. Treat absent and explicit
None the same way: fall back to random_seed.
studio/backend/tests/test_training_raw_support.py
Update the source-string assertions to match the new lines.
* Guard optional MLXTrainingConfig fields and normalize random_seed for PR #5656
The MLX worker now passes `cast_norm_output_to_input_dtype` and
`dataset_order` only when the linked unsloth-zoo dataclass actually
declares them. Released zoo trees that predate the paired PR can still
construct `MLXTrainingConfig` without raising
`TypeError: unexpected keyword argument`. Once the dependency floor is
bumped to a release that contains both fields, the feature-detect
guards become no-ops.
`random_seed = config.get("random_seed", 3407)` was unguarded against
explicit `None` from raw / backend callers. The same value seeded the
trainer and was the fallback target for `model_random_state` /
`lora_random_state`. Normalize once at the top of the function and use
the normalized value everywhere so an explicit `None` cannot reach
FastMLXModel / get_peft_model / MLXTrainingConfig.
Existing seed source-pattern test updated to match the new normalize
helper. New test asserts the feature-detection guards exist and that
the unconditional kwargs do not include the gated fields.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Normalize seed / cast / max_grad_value at TrainingBackend for PR #5656
Round-3 review consensus: the per-field guards that landed in the MLX
worker only protect the MLX path. The same `TrainingBackend.start_training`
config still reaches the CUDA/text trainer at `worker.py:2267`, the
embedding LoRA init at `worker.py:2450`, and embedding TrainingArguments
at `worker.py:2624` with raw `None` values, so an explicit
`random_seed=None` from a raw / backend caller still breaks non-MLX
training even after the previous fix.
Move the normalization into `TrainingBackend.start_training` itself,
where it runs once for every training mode:
- `_coerce_seed(value)`: explicit `None`, non-int, or absent all become
3407. Every downstream worker now sees an int.
- `_coerce_optional_bool(value, default)`: explicit `None` falls back
to `default` instead of `bool(None) == False`. Also normalizes the
common raw-config / YAML string aliases ("true" / "false" / "0" /
"1"). Used for `cast_norm_output_to_input_dtype`.
- `_coerce_optional_nonneg_float(name, value)`: rejects negative
numerics from raw / backend callers, matching the Pydantic
`ge=0` constraint the HTTP route already enforces. Used for
`max_grad_value`.
worker.py MLX path: the existing `bool(config.get(key, True))` for
`cast_norm_output_to_input_dtype` was changed to also fall back on
explicit `None`, so direct worker callers (bypassing
`TrainingBackend.start_training`) are equally safe. `max_grad_value`
also raises on negative values inside the worker for the same reason.
TrainingStartRequest.random_seed default bumped from 42 to 3407 so
direct REST callers that omit the field receive the same default as
the Studio frontend and the MLX worker.
New regression test exercises the three new helpers across explicit
None, valid values, string aliases, and negative-value rejection.
* Tighten feature-detect test paren tracking for PR #5656
The block-extraction used , which stops at the
first inner closing paren (e.g. )
and would silently miss a future unconditional
/ added later in the same dict literal. Switched to
proper paren-depth tracking so the unconditional block is checked end-to-end.
* Shorten verbose comments in MLX Studio backend
* Handle MLX Studio EOS appending by mode
* Wire MLX leaf norm clipping through Studio
* Respect VLM layer filters for explicit LoRA targets
Rationale / guardrails for the local Studio/vision push:
When callers provide explicit VLM LoRA target_modules together with layer filters, FastVisionModel still needs to route the explicit targets through get_peft_regex. Otherwise the layer filters are ignored and adapters can be attached outside the requested language/vision scope.
Do not revert this to plain list(target_modules) for explicit module lists. The CUDA/Studio-facing contract is that explicit targets and layer filters compose: target_modules selects module names, while finetune_language_layers / finetune_vision_layers / finetune_attention_modules / finetune_mlp_modules constrain where those targets are allowed.
The regression test covers the language-only explicit q_proj case and source-checks that explicit targets are wrapped through get_peft_regex when filters are active.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Refresh MLX smoke clip-config note for leaf_norm default
Trim the 11-line comment block to 5 lines and correct the stale claim
that MLXTrainingConfig defaults to max_grad_value=1.0. The new default
is max_grad_leaf_norm=1.0 (same memory profile as elementwise but
direction-preserving). The smoke still pins max_grad_value=1.0
explicitly to keep the 13-seed pass-rate fixture stable.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Forward max_grad_leaf_norm through the training route and warn when layer filters constrain explicit target_modules for PR #5656
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han-Chen <info@unsloth.ai>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* feat(studio): add S3 dataset configuration foundation (#4539)
Add foundational types and configuration for S3 bucket dataset loading:
- Add S3Config type to frontend training types
- Add S3Config Pydantic model to backend training models
- Add "s3" as a DatasetSource option
- Add s3Config state and setS3Config action to training config store
- Add i18n translations for S3 configuration (English and Chinese)
This provides the type definitions and UI text for S3 integration.
Full implementation requires boto3 dependency and data loading logic.
Refs: #4539
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Wire S3 config into training pipeline and prevent secrets persistence
- Pass s3_config from request into training_kwargs so it flows to training subprocess
- Add s3Config to NON_PERSISTED_STATE_KEYS to prevent AWS secrets from being
saved to localStorage
Addresses code review feedback on PR #5951.
* Exclude S3 config from database persistence to protect secrets
Filter out s3_config (which contains secret_access_key) from the
config_json stored in training_runs table, preventing AWS credentials
from being persisted to disk.
Addresses P1 security feedback on PR #5951.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Re-raise HTTPException in start_training and defer s3 DatasetSource widening for PR #5951
* Redact s3_config from W&B run config and accept camelCase S3 credential aliases for PR #5951
* feat(studio): implement S3 dataset loading end-to-end
Builds the actual S3 loader on top of the hardened #5951 foundation,
turning the 501-gated scaffold into a working dataset source.
Backend:
- Add core/training/s3_dataset.py: lists and downloads supported dataset
files (parquet/json/jsonl/csv) from an S3 bucket to a temp dir, using
IAM-role or access-key credentials. boto3 is imported lazily (optional dep).
- Wire s3_config into UnslothTrainer.load_and_format_dataset (downloads then
reuses the existing local-file path) and thread it through worker.py.
- Replace the 501 "not implemented" gate with a boto3-availability guard so
S3 works when boto3 is present and fails clearly when it is not.
- Add boto3 to studio.txt requirements.
- Add tests/test_s3_dataset.py (8 tests) covering download/filtering,
collisions, missing-boto3, and S3Config camelCase/IAM validation.
Frontend:
- Widen DatasetSource to include "s3"; add s3_config to the training payload
type and mapper; add an S3 validation branch and selectS3Source store action.
- Add s3-config-form.tsx (bucket/region/prefix/keys/IAM toggle) reusing the
existing studio.dataset.s3.* i18n strings.
- Add a Hugging Face / Local / Amazon S3 source toggle in dataset-section;
the S3 config card replaces the dataset combobox when S3 is selected.
- Fix DatasetPreviewDialog to accept the widened DatasetSource type.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix S3 dataset loader for PR #6222
* Fix S3 dataset edge cases for PR #6222
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix S3 IAM payload handling for PR #6222
* Block multimodal S3 datasets for PR #6222
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Ash <ash@MacBook-Pro.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: serve DiffusionGemma GGUFs with the on-device visual decoder
* Studio: render the DiffusionGemma denoising canvas live in chat with honest stats
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden DiffusionGemma runner resolution (Windows .exe, build/bin lookup, clear stale audio flag, safe PYTHONPATH, Linux-only pdeathsig)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio/responses): forward chat_template_kwargs enable_thinking to chat request
The /v1/responses translation in _build_chat_request dropped
chat_template_kwargs (e.g. {"enable_thinking": true}) sent via the
Responses extra-body, so reasoning control was silently ignored.
Lift enable_thinking onto the typed ChatCompletionRequest field,
mirroring openai_chat_completions, so both the non-streaming and
streaming Responses pass-through paths honor it.
Fixes#6198
Signed-off-by: Tai An <antai12232931@outlook.com>
* Fix/adjust Responses reasoning for PR #6202
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust reasoning none for PR #6202
* Fix/adjust structured reasoning for PR #6202
* Fix/adjust responses reasoning review findings for PR #6202
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust responses reasoning follow-ups for PR #6202
* Fix/adjust think parsing gate for PR #6202
---------
Signed-off-by: Tai An <antai12232931@outlook.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: Add Tensor-Parallel llama.cpp support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden Tensor-Parallel fallback and GPU selection
* Studio: reconcile split-mode extras and harden tensor-split planning
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reconcile split-mode extras in backend duplicate-load guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: preserve inherited non-tensor split modes on reload
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor cancellation in tensor fallback, preserve tensor mode on rollback, and don't raise an explicit small context
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reconcile split-mode in reload check and strip it on tensor downgrade
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Strip --tensor-split alongside --split-mode so inherited ratios don't override the tensor planner
An inherited or stale --tensor-split in llama_extra_args was appended after
Studio's computed --tensor-split and won last in llama.cpp, re-introducing the
asymmetric-GPU OOM tensor mode is meant to prevent. Group -ts/--tensor-split
into the split-mode shadow set so it is stripped on inherit and on the layer
fallback; parse_split_mode_override still keys on the mode value only.
* Drop quantized KV for the tensor attempt and report native max context
Tensor mode aborts on a quantized KV cache, so a user with q8_0/q4_1 etc. who
enabled Tensor Parallelism silently fell back to layer split. Clear the cache
type (and strip inherited/explicit --cache-type) for the tensor attempt only;
the layer fallback re-runs with tensor off and keeps the user's choice.
Also report max_available_ctx from the native context, not an explicit small
-c, so the context slider no longer warns too early in tensor mode.
* Reconcile inherited split-mode extras in the already-loaded check
When a same-model load omitted llama_extra_args, the tensor comparison resolved
the raw (None) request and treated an inherited --split-mode tensor server as a
mismatch, forcing a needless reload. Compare using the stored extras stripped
the same way the reload strips them.
* Pass tensor_parallel through compare-mode loads
The generalized compare path loaded each GGUF without tensor_parallel, so
compare ran layer split even with the toggle on and left the settings sheet
stale. Send the toggle and hydrate the loaded state from the response, matching
the main chat and recipe load paths.
* Add --tensor-parallel flag to unsloth studio run
The headless one-liner could only reach tensor mode by passing --split-mode
tensor as a raw llama.cpp extra. Add a first-class --tensor-parallel/
--no-tensor-parallel option that sets the tensor_parallel field on the
/api/inference/load payload, forwarded through the studio-venv re-exec like the
other polarity flags. Matches the web UI toggle and the API field.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* fix: allow absolute save_directory in export paths to prevent cross-drive copy failures
The GGUF export pipeline (and all other export flows) forced every
save_directory through resolve_export_dir(), which always resolved
the path under exports_root() — typically ~/.unsloth/studio/exports/
on the system drive (C: on Windows).
When a user selected an output directory on a different drive (E:):
1. The absolute path was rejected at the Pydantic validator level.
2. Even if it got through, resolve_export_dir would re-resolve it
under C:\Users\.unsloth\studio\exports\.
3. After GGUF conversion completed on E:, the relocation step would
try to move/copy the finished files to C:, causing:
- WinError 17 (cross-drive move failure when shutil.move falls
through to a cross-filesystem copy)
- WinError 112 (disk full on C:)
Fix both layers:
- _validate_save_directory: accept absolute paths (they represent an
explicit user choice of output location).
- resolve_export_dir, resolve_output_dir, resolve_tensorboard_dir:
return absolute paths as-is instead of forcing them under the
default root. Keep the existing safety checks (null bytes, '..'
segments) and fall through to resolve_under_root for relative paths.
Fixes: https://github.com/unslothai/unsloth/issues/6082
* refactor: centralize user path validation into _resolve_user_path helper
Addresses code review feedback: the null-byte, '..', and absolute-path
checks were duplicated across resolve_output_dir, resolve_export_dir,
and resolve_tensorboard_dir. Extract a single _resolve_user_path helper
that all three delegate to.
No behavioral change — pure consolidation.
* fix: address code review — contain destructive cleanup and scope absolute paths
Address all review feedback from gemini-code-assist:
1. P1: destructive subdirectory cleanup (export_gguf)
The flattening loop in export_gguf previously rmtree'd every
subdirectory under abs_save_dir. When targeting an existing user
directory on a different drive (#6082), this could nuke unrelated
subdirectories. Now snapshot existing subdirectories before the
export and only clean up dirs created during this run.
2. P2: keep scan/read endpoints contained
Only resolve_export_dir accepts absolute paths (export is a write
path where user picks location). Reverted resolve_output_dir and
resolve_tensorboard_dir to use resolve_under_root directly — these
are used by scan/read/training endpoints that must stay contained
under their respective roots.
3. Centralization feedback
Removed the _resolve_user_path helper since it's no longer needed
with the narrowed scope. resolve_export_dir has the absolute path
logic inline with a clear docstring.
* fix: skip pre-existing subdirs in GGUF flatten loop and clean stale export intermediates
Two issues caught in code review (chatgpt-codex-connector):
1. The flattening loop moved ALL .gguf files from ALL subdirectories
into abs_save_dir, including pre-existing unrelated user subdirs.
Now skip pre-existing subdirs entirely unless they are known
export-owned intermediates (model/, model_gguf/).
2. After a failed export, known export-owned subdirectories (model/,
model_gguf/) were snapshotted as pre-existing on retry and never
cleaned up. These are now always cleaned up regardless, since they
are known intermediates created by the export pipeline.
* fix: separate write vs read export paths, guard same-dir rmtree
Three issues caught in code review (chatgpt-codex-connector):
1. P1: scan endpoint containment
resolve_export_dir was changed to accept absolute paths, but it's
also used by scan/read endpoints (routes/models.py) that must stay
contained under exports_root(). Split into:
- resolve_export_dir: contained, used by scans
- resolve_export_write_dir: accepts absolute paths, used by export
backend only
2. P1: same-directory rmtree
When a non-PEFT checkpoint's gguf_dir resolves to the same path as
abs_save_dir (user selected the checkpoint's gguf output as their
export directory), shutil.rmtree(gguf_dir) would delete the user's
chosen output directory. Now skip relocation when both paths resolve
to the same location.
3. P1: pre-existing subdir flatten loop
Reverted _EXPORT_OWNED_SUBDIRS logic — 'model/' and 'model_gguf/'
are common directory names in shared model folders and don't prove
export ownership. Now only clean up subdirs that didn't exist before
the export started.
* fix: remove dead _EXPORT_OWNED_SUBDIRS and fix _export_details for absolute paths
Two fixes from review comments:
1. Remove unused _EXPORT_OWNED_SUBDIRS declaration (leftover from
previous iteration that was intentionally removed).
2. _export_details now returns the full absolute path when the export
target is outside exports_root(), instead of truncating to basename.
Users who export to E:\ can now see the full destination path in
the success dialog.
* fix: use unique tmp dir for GGUF intermediates to avoid overwriting user dirs
When exporting to an absolute destination that already contains a
model/ subdirectory (e.g. a shared models folder), the hard-coded
model_save_path would overwrite files in that unrelated directory.
Use _tmp_model_<uuid> as the intermediate path instead, so user
directories are never touched. The tmp dir is created as a new subdir
of abs_save_dir and cleaned up by the flatten loop after GGUF files
are relocated.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix GGUF local export paths for PR #6088
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address GGUF export follow-ups for PR #6088
* Clean GGUF temp dirs on export failure for PR #6088
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust export path tests for PR #6088
* Fix/adjust export path review findings for PR #6088
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust home export path handling for PR #6088
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Daniel Han