* inference: add passthrough tool-call healing core (heal_gate, heal_openai_message, StreamToolCallHealer, nudge helpers)
Small GGUF models often emit tool calls as text (<tool_call>{...}</tool_call>,
Gemma <|tool_call>, <function=> XML) instead of structured tool_calls. Studio's
enable-tools loop already heals these, but the client-tool passthrough
(unsloth run --disable-tools, unsloth start agents) relays them verbatim, so
the agent sees prose and the turn dies.
This module is the shared response-side repair layer the passthrough routes
will call: promote parsed text-form calls to structured calls, but only for
function names the client actually declared; coerce arguments through the same
canonical-key healing as the tool loop; never touch the upstream request body
(llama-server KV/slot reuse stays byte-identical). StreamToolCallHealer is the
streaming buffer-and-repair state machine: prose forwards immediately, only a
partial-signal tail or a suspected tool block is held, false alarms flush
verbatim, and a 64 KiB bound caps memory. nudge_should_retry/nudge_messages
support an opt-in single-retry nudge for non-streaming routes (wired later).
Kill-switch: UNSLOTH_DISABLE_TOOL_CALL_HEALING=1. Reuses
core/tool_healing.parse_tool_calls_from_text, strip_tool_call_markup, and
tool_loop_controller.coerce_tool_arguments unchanged.
* inference: heal text-form tool calls on the OpenAI and Responses passthrough
Wire the passthrough healing core into /v1/chat/completions and /v1/responses,
default ON whenever the request declares client tools:
Non-streaming: heal_openai_message runs inside the existing response-mutation
loop; a promoted call flips finish_reason to tool_calls and nulls the content,
and the verbatim-bytes fast path still applies when nothing was healed.
/v1/responses non-streaming inherits this through openai_chat_completions.
Streaming: a StreamToolCallHealer per stream. Ordinary prose relays
byte-for-byte (a fast path keeps upstream bytes when the healer passes a chunk
through whole); once a tool signal appears, content is held, and at the
finish/[DONE] boundary either synthetic delta.tool_calls chunks replace the
markup (finish_reason rewritten to tool_calls, including the synthetic-finish
path) or a false alarm flushes the held text verbatim. Structured upstream
deltas put the healer to sleep after flushing anything held, so grammar-mode
responses stay byte-identical. The Responses stream feeds healed calls through
the same per-call state machinery as structured deltas (indexes live in a
disjoint range so a healed call can never merge into a structured call's
state), and the visible/reasoning split runs first so reasoning text is never
promoted. parallel_tool_calls=false caps healed calls on every path.
The upstream request body is never touched and healing issues no extra
generation, so llama-server slot/KV-cache reuse is unchanged. Opt-out per
request with auto_heal_tool_calls=false (Responses reads it from the
extra-body); requests without tools relay verbatim.
* inference: heal text-form tool calls on the Anthropic /v1/messages passthrough
Streaming: AnthropicPassthroughEmitter.enable_healing(allowed_tools) routes
content deltas through the shared StreamToolCallHealer. A promoted call closes
any open text block (only the safe prose prefix ever streamed into it), opens a
synthetic tool_use block with a fresh toolu_* id, carries one input_json_delta,
and closes; finish() then forces stop_reason to tool_use unless a truncation
(max_tokens) wins. Structured upstream deltas flush anything held and put the
healer to sleep, so grammar-mode responses are untouched, as is every stream
where enable_healing is never called (Studio's own loop, no-tools requests).
disable_parallel_tool_use caps healed calls too.
Non-streaming: the OpenAI message dict is healed BEFORE block building, so the
existing tool_use promotion loop and stop_reason line treat promoted calls
exactly like native ones (finish_reason length still maps to max_tokens). The
legacy tool-XML strip still runs on remaining text, so opted-out requests keep
today's cleanup behavior byte-for-byte.
auto_heal_tool_calls is now a typed field on AnthropicMessagesRequest
(default True, mirroring Chat Completions) and threads into both passthrough
calls. Healing never touches the upstream request body.
* inference: opt-in single-retry tool-call nudge on the non-streaming passthrough
When the model clearly tried to call a tool (a tool signal in the text) but
healing produced nothing usable, re-ask once: the retry body is the original
body plus an assistant turn (the model's own failed text) and a short user
nudge naming the declared tools. The prompt prefix stays byte-identical, so
llama-server reuses the slot's KV cache and only the two-message suffix is
prefilled. The retry replaces the original response only when it actually
yields a promotable or structured call; on any error or still-garbage output
the original response is returned unchanged. Exactly one retry, non-streaming
OpenAI and Anthropic passthroughs only (a stream has already emitted bytes).
OPT-IN per user decision: nudge_tool_calls=true per request (typed on both
ChatCompletionRequest and AnthropicMessagesRequest, lifted from the Responses
extra-body), or UNSLOTH_TOOL_CALL_NUDGE=1 to flip the process default.
auto_heal_tool_calls=false disables healing AND the nudge.
Also align the non-streaming heal on allow_incomplete=True: the response is
final, so a trailing unclosed tool block is a model failure worth repairing,
matching the enable-tools loop's drain semantics.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: never assume the upstream response shape in the nudge helpers
llama-server error bodies can carry message: null (or no choices at all), and
_last_assistant_text / response_has_promotable_calls / nudge_should_retry
called .get() on the message without a dict check, so a malformed upstream
response raised an AttributeError the surrounding except tuples did not catch,
failing the request instead of degrading to 'nothing to heal'. Route the shape
probing through one _first_choice_message helper that returns None for any
non-dict message, and add a parametrized test over the malformed shapes.
* inference: constrain healing by tool_choice, preserve length finish_reason, keep healed event order in Responses streams
Three review findings on the passthrough healer:
- heal_gate now honors the request's tool_choice: "none" disables healing
outright and a forced function narrows the promotion allowlist to that
one function, so healing can never contradict the request's tool-choice
constraint. Wired through the OpenAI chat (stream and non-stream),
Responses, and Anthropic (converted shape) passthroughs.
- The OpenAI non-streaming heal only upgrades finish_reason "stop" to
"tool_calls"; a truncated generation keeps "length" (the healed call
stays attached) matching the streaming and Anthropic paths.
- The Responses stream emits healer events in order instead of collapsing
all text ahead of the healed calls, so text after a healed call no longer
jumps ahead of the function_call item and output indexes are claimed in
the order the model produced them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: all-or-nothing promotion when a response mixes declared and undeclared text-form calls
Promoting a subset used to strip ALL tool markup from the content, which
silently deleted the text of any call naming an undeclared tool. The heal
now declines entirely when any parsed call is unpromotable, so the whole
message relays verbatim (pre-PR behavior) and no bytes are ever lost. In
streaming, a declared call that completed before an undeclared one arrived
is already emitted; the late undeclared markup still flushes as raw text.
The nudge helpers mirror the same contract via a shared predicate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests: wrap long lines in the Responses healing tests to the project style
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: span-exact healing, disjoint healed stream indexes, per-call Responses message items, allowlisted nudge acceptance
Four review findings on the passthrough healer:
- parse_tool_calls_from_text gains an optional with_spans return so healing
removes EXACTLY the promoted calls' markup. This supersedes the previous
all-or-nothing rule: declared calls promote and every unpromoted byte
(undeclared calls, unparseable closed blocks, suppressed alternate
formats such as a <function=...> block after a JSON call) relays as text.
The stream healer also processes one block per pass, so text between two
healed calls keeps its document position instead of trailing them.
- The OpenAI chat stream shifts native tool-call delta indexes past any
already-emitted healed calls; clients merge deltas by index, so a healed
call and a later native call can no longer merge into one.
- A healed call in the Responses stream closes the open message item and
trailing text opens a fresh one with a later output index, matching the
native stream shape; response.completed snapshots every message item
with its own text.
- The nudge retry only replaces the original response when the retry's
structured call names a DECLARED tool; a hallucinated undeclared call is
not an improvement.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop the heal path folding trailing prose into a closed function call
parse_tool_calls_from_text(allow_incomplete=True) cut a <function=...> body only
at an end-anchored </function>, so a fully closed call followed by trailing prose
(<function=..>..</parameter></function> words) folded </parameter></function> and
the prose into the tool argument and deleted the prose from visible content. The
strict path (allow_incomplete=False) already cut at the real </function> via rfind.
Do the same in both modes: trim the body at the real </function> when present and
end the removal span there, falling back to the end-anchored strip and body_end
only when the call is genuinely truncated. Add a regression test.
* inference: one shared single-call budget for healed and native calls
Codex round 5: the parallel-call caps counted healed and native calls
separately, so a healed text-form call followed by a native structured
delta double-emitted on all three streaming surfaces when the client
disabled parallel calls.
- OpenAI SSE: once a healed call went out with parallel_tool_calls
false, native tool_call deltas are dropped instead of index-shifted.
- Anthropic emitter: native deltas skip block allocation when the
healed-plus-native count already filled the single slot, and healed
emission counts open native states too.
- Responses stream: native deltas that survived the chunk-level cap are
skipped once a healed call claimed the slot.
Also adds a span assertion for the closed-</function> trailing-prose
parse fixed in the previous commit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: relay undeclared text-form calls as text on Anthropic non-streaming
heal_openai_message promotes only declared text-form tool calls and
span-trims just their markup, deliberately leaving every unpromoted byte
(undeclared text-form calls included) in the content to relay as text.
The Anthropic non-streaming builder then ran a blanket _TOOL_XML_RE strip
over that content unconditionally, deleting the undeclared block before
building the text part, so Anthropic clients silently lost a call the
OpenAI non-streaming path preserves. The strip was harmless when healing
was all-or-nothing but became data loss once healing turned span-exact.
Gate the legacy strip on whether healing promoted a call, matching the
OpenAI passthrough and the intent already stated in the comment above.
Add a route-level regression test for the mixed declared+undeclared case.
* inference: require fully declared nudge retries; keep unpromoted Anthropic text
Codex round 6, two findings:
- response_has_promotable_calls accepted a nudge retry when any one
structured call named a declared tool, so a mixed retry (hallucinated
undeclared call plus a declared one) replaced the original and the
caller forwarded the undeclared call, or with parallel_tool_calls
false could keep only it. All structured retry calls must be declared.
- The Anthropic non-streaming builder still ran the legacy _TOOL_XML_RE
strip after span-exact healing, deleting undeclared or malformed call
text that healing deliberately preserved. The legacy strip now runs
only when healing is off (no declared tools, or opted out), matching
the OpenAI passthrough.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: keep unpromoted Anthropic text whenever healing is active
The previous commit skipped the legacy strip only when a call was
actually promoted, so an undeclared-only (or malformed-only) response
was still silently emptied: exactly the dead-turn shape this path
exists to fix, and inconsistent with the OpenAI passthrough, which
relays those bytes verbatim. Gate the strip on healing being active
instead; opt-out and no-tools requests keep the legacy strip.
* Fix schema-aware tool healing for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix passthrough healing ordering for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix stream finish ordering for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>
* replaced connect with start
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix
* Studio: build the coding-agent command from the selected server
The API keys panel showed a hardcoded `unsloth start claude`. `unsloth start`
defaults to 127.0.0.1:8888 and only mints a key for a loopback server, so a
non-default port or a tunnel/remote base would target the wrong server or fail
to mint. Build the command from the panel base/key (and emit a key for
non-loopback), matching the other snippets in the panel.
* CLI: keep `unsloth connect` as a hidden alias for `unsloth start`
Avoids breaking existing scripts and docs that still call `unsloth connect`.
* Tests: stub _unstarted_cleanup in same-task disconnect test
The test builds _SameTaskStreamingResponse via __new__, so set the attribute
that __call__ now reads.
* Match coding-agent command loopback check to the CLI 127.0.0.0/8 rule (#6613)
* Keep unsloth_cli.commands.connect importable as a deprecated shim (#6613)
* Format the new coding-agents panel strings and import per biome (#6613)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop the unsloth connect alias and shim; unsloth start is the only command (#6613)
* Route unsloth connect to unsloth start as a hidden backward-compatible alias (#6613)
* Forward unsloth run model-load flags to unsloth start (gguf-variant, context-length, load-in-4bit, tensor-parallel) (#6613)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Session-scope coding agent config in unsloth start
Configure each agent for the current session instead of writing the Studio endpoint, key, and default model into the user's own config. Codex, OpenCode, OpenClaw, and Hermes get a private config relocated through their config-path env vars (CODEX_HOME, OPENCODE_CONFIG overlay, OPENCLAW_CONFIG_PATH plus OPENCLAW_STATE_DIR, HERMES_HOME). Claude Code suppresses the attribution header for the session via the CLAUDE_CODE_ATTRIBUTION_HEADER env var plus a --settings overlay, with no ~/.claude write. --launch uses an ephemeral temp dir removed after the agent exits; --no-launch uses a stable Unsloth-owned dir and prints the matching export lines.
* Read relocated agent session config in Local Agent Guides CI
The contract crosscheck and the openclaw/hermes patch helpers now read each agent's config from the relocated path printed by unsloth start --no-launch (CODEX_HOME, OPENCODE_CONFIG, OPENCLAW_CONFIG_PATH, HERMES_HOME) instead of fixed home paths. The Claude attribution A/B toggles the header for the session only (shipped-config HIT vs vanilla MISS) instead of editing ~/.claude/settings.json.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Skip the POSIX-only --no-launch parser test on Windows
test_no_launch_output_is_parseable mirrors the #6547 bash CI parser, which greps export/unset lines and only runs on Linux/macOS runners. On Windows --no-launch prints PowerShell ($env: / Remove-Item), so the export-line assertion does not apply there. Cross-OS staging CI surfaced this.
* Size Claude Code's auto-compact window to the loaded model's context
Claude Code auto-compacts against its native (~600k token) window, so against a smaller local model it overflows the server's context (silent truncation) long before it compacts. Set CLAUDE_CODE_AUTO_COMPACT_WINDOW to the loaded model's real context length (the value codex/openclaw already get via model_context_window / contextWindow). Omitted when the model reports no context length.
* Pin OpenCode/Hermes context window and set 90% compaction across agents
Feed every agent the server-determined sequence length (the value /v1/models reports from runtime_context_length) and a ~90% compaction threshold. OpenCode: a custom-provider model with no limit defaults to context 0, which silently disables auto-compaction, so set limit.context/output and scale the compaction buffer to 10% of the window. Hermes: pin model.context_length (it otherwise falls back to a 256k default when the server's /v1/models omits the field) and set compression.threshold 0.9. Claude: add CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=90 alongside the window. Codex (model_context_window) and OpenClaw (contextWindow) already carried the window and auto-manage off it.
* Add `unsloth start pi` recipe
Pi was the only agent without a built-in recipe, so the agent-guides CI
hand-wrote ~/.pi/agent/models.json. Add a first-class `pi` command mirroring
the others:
- write_pi_config writes the session-scoped OpenAI-compatible provider config
(key in the config, like openclaw/opencode).
- pi() launches `pi --provider unsloth --model <id>` (Pi defaults to the google
provider, so the provider/model are pinned on the command line) with HOME
relocated for the session. Pi has no config-dir env var and resolves ~/.pi off
$HOME, so HOME-scoping keeps the user's ~/.pi untouched.
Migrate the agent-guides CI off the hand-written config onto the
`unsloth start pi --no-launch` path (connection + file-edit), with a crosscheck
for the provider api, so the documented recipe is exercised.
* Harden unsloth start for Windows and WSL agent launches
Address the Codex review on PR 6613:
- write_pi_config now pins the loaded contextWindow and a sane maxTokens so Pi
compacts instead of overflowing a small Studio context (it otherwise assumes
its 128000 default), matching the other agents.
- pi() sets USERPROFILE (and HOMEDRIVE/HOMEPATH when present) alongside HOME on
native Windows, where Node resolves ~/.pi via USERPROFILE rather than HOME, so
the session no longer reads or writes the user's real ~/.pi.
- The WSLENV bridge flags path-valued vars with /p so a Windows npm shim under
/mnt receives translated paths, while scalar vars (the numeric context window)
pass through untranslated. WSLENV is deduped on the bare name.
- _print_env prints the launch command with PowerShell-safe quoting so the inline
--settings JSON survives copy-paste on native Windows --no-launch.
Add tests for the WSLENV path flagging, PowerShell quoting, the Pi context
window, and the Pi USERPROFILE relocation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Set CLAUDE_CODE_NO_FLICKER for the Claude session
A local server streams in bursts, so Claude Code's full-screen TUI redraw
flickers between tokens. Disable it for the session via CLAUDE_CODE_NO_FLICKER,
alongside the other CLAUDE_CODE_* session env knobs.
* Add a normalized --yolo flag routed to each agent's auto-approve mode
It is easy to forget which agent spells "run tools without prompting" which way,
so `unsloth start` now accepts all three spellings as one option (--yolo,
--dangerously-skip-permissions, --dangerously-bypass-approvals-and-sandbox) and
routes to the agent's own mechanism:
- claude: --dangerously-skip-permissions
- codex: --dangerously-bypass-approvals-and-sandbox
- hermes: --yolo
- pi: --approve (Pi's only approval gate is project trust)
- opencode: a permission allow block in opencode.json (no CLI flag exists)
- openclaw: tools.exec security=full / ask=off / host=gateway (no CLI flag exists)
Because the option is parsed by `unsloth start`, the "wrong" spelling for an
agent still routes correctly instead of leaking through to the agent and erroring.
IS_SANDBOX is deliberately left unset for Claude so its root/sandbox safety gate
still applies. Adds routing, cross-routing, and per-config tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix review findings: IPv6 loopback command, pi USERPROFILE under WSL, yolo guard
From a 10-reviewer pass over the PR:
- studio/frontend agent-command.ts: normalize bracketed IPv6 hosts. URL.hostname
returns "[::1]" for http://[::1]:8888, which never matched the "::1" loopback
checks, so the copied command embedded the placeholder API key for a local IPv6
server instead of the bare auto-minting command. Now [::1] is treated as loopback
like the CLI's is_loopback_url, so the command matches the CLI contract.
- pi(): also relocate USERPROFILE (and HOMEDRIVE/HOMEPATH) when running under WSL
against a /mnt Windows shim, not just on native Windows. Windows Node resolves
~/.pi via USERPROFILE, and the WSLENV bridge translates the path, so pi no longer
falls back to the user's real ~/.pi in that case.
- _yolo_command_flags: use .get so a config-based agent (or a typo) yields no flag
instead of a latent KeyError.
Adds tests for the WSL pi USERPROFILE relocation, the yolo unmapped-agent guard,
and that opencode/openclaw --yolo stays config-only (no argv flag).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix round-2 review findings: WSLENV /p upgrade, agent help text
- _merge_wslenv now upgrades a user's pre-existing unflagged WSLENV entry (e.g. a
bare HOME or USERPROFILE) to the path-translated form (HOME/p) instead of leaving
it as-is, so a Windows agent shim under WSL receives the translated session path
rather than the raw Linux path.
- Generalize the `unsloth start` registration help to list all six agents (was only
"Claude Code, Codex").
Adds a test for the WSLENV unflagged-entry upgrade.
* Fix round-3 review findings: complete openclaw --yolo, refresh stale copy
- openclaw --yolo now also writes the host approvals file (exec-approvals.json with
defaults security=full / ask=off / askFallback=full) alongside the tools.exec
config. OpenClaw gates tool execution on both layers (the stricter wins), so the
config alone could still leave it prompting or denying. Mirrors `openclaw
exec-policy preset yolo`. ask=off means nothing is ever prompted, so the runtime
socket block is unnecessary.
- Studio API panel copy: clarify that a local server auto-mints the key while a
remote one embeds it in the command, and add pi to the swap hint.
- Local Agent Guides CI: drop the stale "pi has no start.py recipe" note now that
all six agents are driven via `unsloth start <agent> --no-launch`.
Adds the openclaw approvals-file assertions and a no-yolo openclaw test.
* start: parse claude --version with a regex so a format change does not drop optimization flags
* start: offer to install a missing agent (prompt then run its install command)
* start: auto-start a Studio server for --model when none is running, and stop it on exit
* inference: surface an actionable message when llama-server cannot compile a tool grammar
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix review findings: kill the auto-started server tree on Windows; apply the tool-grammar message to the OpenAI passthrough too
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* start: split --model org/repo:variant so a running session is not evicted
`unsloth start <agent> --model org/repo:QUANT` failed against an already-running
Studio server and, worse, killed whatever model another session had loaded.
/v1/models lists a loaded GGUF under its bare repo id (e.g. unsloth/Qwen3-1.7B-GGUF),
so _resolve_model never matched the `:QUANT`-suffixed request. It then POSTed
/api/inference/load with model_path=org/repo:QUANT, which (a) Hugging Face rejects
("Repo id must use alphanumeric chars, '-', '_' or '.'") and (b) evicts the model the
other session was using, so a second 'unsloth start' in a new tmux/terminal tore down
the first. Re-running the command then attached to the now-empty server, which is why
it 'worked the second time'.
Mirror the org/repo:QUANT -> org/repo + --gguf-variant QUANT shorthand that
'unsloth run' and llama.cpp already accept, splitting it in _connect before we match or
serve. Matching now resolves against the loaded bare repo id (no spurious reload, no
eviction), and any real load uses a valid repo id plus gguf_variant. An explicit
--gguf-variant still wins; local paths and Windows drive letters pass through untouched.
The auto-serve path likewise spawns 'unsloth run --model org/repo --gguf-variant QUANT'.
* start: harden auth-key handling, codex teardown, and CI transcript redaction
Three review findings:
1. CI could leak a live key. agent-guides-drive.sh printed the raw
'unsloth start --no-launch' transcript (which carries export UNSLOTH_API_KEY /
ANTHROPIC_AUTH_TOKEN lines) to the Actions log on both the failure path and the
success path before redact() ran. Add cat_redacted() and use it for those two
prints, so the key is scrubbed on the way to the log while the on-disk file stays
intact for the env parsing that follows.
2. Outages masqueraded as bad keys. _key_accepted caught a broad Exception and
returned False, so a 5xx or timeout while checking a cached key looked like a
rejection: it discarded a good key and minted extra ones (local) or reported 'no
saved key' (remote). Only treat HTTP 401/403 as a rejection; let other errors
propagate so a real outage surfaces.
3. Codex preflight could leave the auto-started server up. _require_gguf_for_codex
runs after _connect may have auto-started Studio but before _run installs its
teardown finally, so a preflight rejection (e.g. a transformers-backend model) left
the server holding the port/GPU until the atexit backstop. Tear it down explicitly
at the point of failure.
Tests: a 5xx on a saved key surfaces without minting; a non-GGUF codex preflight
tears down the auto-served server.
* start: fix IPv6/portless studio URLs, Pi config-dir isolation, and Pi install recipe
Four review findings:
1. Pi ignored the session config when PI_CODING_AGENT_DIR was already set. Pi's
getAgentDir() reads process.env.PI_CODING_AGENT_DIR before falling back to
$HOME/.pi/agent, so a value inherited from the user's shell sent Pi to their real
config and skipped our provider/key (the HOME relocation alone was not enough). Pin
PI_CODING_AGENT_DIR at the session's .pi/agent dir; it is path-valued so the WSL
bridge translates it automatically.
2. Pi install hint dropped Pi's documented --ignore-scripts. Pi's README installs with
'npm install -g --ignore-scripts @earendil-works/pi-coding-agent' and notes it needs
no install scripts, so accepting the prompt now follows that safe recipe.
3. Auto-start ignored a portless UNSLOTH_STUDIO_URL. unsloth run binds to
'parsed.port or 8888', so http://127.0.0.1 launched the child on 8888 but the health
poll (and the returned base) still used port 80, stalling until the startup timeout.
Normalize the base to host:8888 (IPv6-safe) before starting and polling.
4. API-panel command mistook IPv6 loopback for the bare default. The bare 'unsloth
start' only probes 127.0.0.1:8888 on the IPv4 stack, so http://[::1]:8888 must carry
an explicit UNSLOTH_STUDIO_URL. Drop ::1 from the bare-default host set while keeping
it a loopback host (URL emitted, no key needed).
Tests: PI_CODING_AGENT_DIR is set to the session dir; _effective_base normalizes
portless/IPv6 bases; a portless UNSLOTH_STUDIO_URL auto-serves on :8888.
* start: apply fresh-review findings across CLI, CI, and the API-panel command
From a fresh multi-reviewer pass over the merged head plus the latest Codex bot review:
1. Load knobs now always consult the server. _resolve_model matched on model id alone,
so --gguf-variant / --context-length / --no-load-in-4bit / --tensor-parallel were
silently ignored whenever the id was already loaded (asking for UD-Q4_K_XL kept a
Q8_0 serving). With any explicit knob the CLI defers to /api/inference/load, whose
already-loaded dedup answers without reloading when variant and settings match, so a
second session running the same command still attaches without evicting the first.
2. OpenCode --yolo and the session model pin now ride in OPENCODE_CONFIG_CONTENT. A
project's own opencode.json outranks OPENCODE_CONFIG, so a repo config could silently
override the session model and the --yolo permission block; OPENCODE_CONFIG_CONTENT
outranks project config. The API key stays in the private file, never in printed env.
3. The --no-launch recipe's last line is a self-contained one-liner (inline VAR=value
assignments before the command, conflicting vars blanked). People copy just the last
line, and a bare codex/claude there ran against the user's real ~/.codex or Anthropic
credentials with zero isolation, e.g. inheriting a pre-existing damaged ~/.codex
state DB and blaming the recipe. The CI drive script scrubs the key from the one
'invoking:' echo this adds.
4. The auto-serve log is 0600 and the parent handle is closed. It sat world-readable in
the shared tempdir under a predictable name while carrying the minted sk-unsloth-
key from the unsloth run banner.
5. _key_accepted fails with a clean message on outages. Non-auth errors (5xx, network,
timeout) surfaced as a raw traceback; 401/403 still mean a rejected key.
6. _effective_base strips URL paths, and https loopback targets never auto-serve.
http://127.0.0.1:8888/studio polled /studio/api/health (404) and https://127.0.0.1
polled the wrong scheme, both spinning until the 15-minute startup timeout.
7. API-panel command: only literal 127.0.0.1:8888 earns the bare command. localhost can
resolve to ::1, which the bare CLI never probes, so it keeps UNSLOTH_STUDIO_URL.
8. CI artifact sweep covers redacted-configs/ and agent-workdir/, not just logs/.
Tests: 125 CLI tests pass (new coverage for each fix), 156 backend tests pass, ruff
clean. Adds an unsloth connect alias regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* start: hand Pi a clean screen at launch
Pi paints inline from wherever the cursor sits: its first render assumes a
clean screen instead of clearing or entering the alternate screen itself
(current Pi never emits a clear at startup). Launched under unsloth start,
that left the session starting mid-scroll beneath the connection output.
Clear the screen (click.clear, cross-platform, no-op without a TTY) right
before the Studio banner so Pi opens exactly one line down on a clean
viewport. Launch path only: --no-launch recipes and piped output are never
wiped, and alternate-screen agents are left alone.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* start: auto-override hermes' 64K context floor for small model windows
Hermes refuses to initialize when the served model's context window is
under 64,000 tokens, and a second copy of the same check rejects the
compression model mid-session. write_hermes_config previously pinned the
real window, so any small local model (e.g. 40,960) failed at startup
with manual config.yaml instructions.
For windows below the floor the recipe now claims 65,536 in
model.context_length, scales compression.threshold so compaction still
fires at 90% of the real window, and sets
auxiliary.compression.context_length to cover the mid-session check.
Windows at or above the floor keep the exact previous behavior.
* ci: install pi with --ignore-scripts, matching the start.py hint
The pi cell predates the pi recipe in start.py and still installed the
package with lifecycle scripts enabled, so CI stopped exercising the
exact command users are prompted to run. npm_retry now passes extra
flags through, the pi branch mirrors the install hint verbatim, and the
stale no-recipe comment is refreshed.
* ci: fail loudly when a relocation var is missing from connect output
The empty-string guards ran after appending /config.toml or /config.yaml,
so they could never fire: crosscheck_contract silently skipped its
contract checks and patch_hermes_tools died on the root path with a bare
traceback. Check the raw variable first and guide_fail with the real
cause.
* staging: 6613 round 6 (https elision, no-launch home reuse, auto-start key fallback)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: shimmyshimmer <107991372+shimmyshimmer@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* report a complete load once llama-server is healthy
load_progress() derived its fraction purely from the llama-server's VmRSS over the GGUF shard total. With layers offloaded to VRAM (-ngl) the process releases the mmap'd weight pages after upload, so VmRSS sinks back well below the shard total: the fraction climbs toward ~1.0 during mmap, then collapses to a small value (~8%) once the weights are on the GPU. A fraction-driven progress bar therefore restarts and sticks there indefinitely even though the model is loaded and serving, which reads as a hang at "Starting model...".
Once the server is healthy the load is complete by definition, so report
fraction 1.0 (and bytes_loaded == bytes_total) in the ready phase regardless of resident set size. The VmRSS read is factored into _read_rss_bytes() with its original semantics preserved (0 on a missing VmRSS line, None when /proc is unavailable) so it can be unit-tested off Linux.
Fixes#5740
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* stub heavy deps in the load-progress test and guard a valueless VmRSS
Two review fixes:
1. The new test imported core.inference.llama_cpp at module top, which pulls in
loggers/structlog/httpx and fails collection with ModuleNotFoundError in the
lightweight backend test env when the file is run on its own. Stub loggers,
structlog and httpx via sys.modules.setdefault before the import, mirroring
test_llama_cpp_load_progress_matrix.py; setdefault keeps the real modules when installed. Verified the file now collects and passes with only pytest present.
2. Catch IndexError in _read_rss_bytes: a "VmRSS:" line with no value column
would make line.split()[1] raise and crash a load-progress poll. Return None
instead, with a test for the valueless line.
* Hold load-progress high-water mark and explain a never-healthy load (#5740)
load_progress() now holds a per-process VmRSS high-water mark, so the bar
no longer regresses to ~8% when -ngl offloads the weights and frees the
mmap pages mid-load.
A live server that never returns 200 on /health now gets a specific error
(context/VRAM too large, or a local proxy/VPN intercepting the loopback
probe) instead of the generic invalid-GGUF/out-of-memory message.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Hakan Baysal <hakan.baysal@trmix.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* feat: add mlx public trainer api
* test: cover mlx public trainer api
* fix: preserve mlx epoch trainer configs
* fix: pass mlx warmup ratio through config
* fix: align mlx trainer dataset order
* fix: keep mlx chat templates import-light
* fix: infer mlx trainer context length
* fix: mirror cuda mlx context defaults
* fix: align mlx notebook trainer defaults
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: keep mlx public helpers import-light
* refactor: reuse mlx optimizer normalization
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: address mlx review feedback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: tighten mlx training argument parity
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: align mlx trainer eos default
* Fix MLX trainer to accept DataCollatorForSeq2Seq and handle TokenizerWrapper in get_chat_template
* Trim redundant docstrings on internal MLX helpers
* MLX review fixes: Studio optimizer import-safe on non-MLX hosts, preserve explicit max_length, skip MLX tests before import
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MLX review round 2: defer max_length to model context, optimizer alias fallback for older zoo, skip non-MLX test on missing GPU deps
* MLX review round 3: keep chat_templates importable without torch on MLX
* fix: preserve MLX trainer notebook shims
* fix: ignore CUDA tokenizer moves on MLX
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden MLX trainer shims
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: unwrap MLX scheduler enum args
* fix: coerce integral MLX epoch counts
* fix: spoof CUDA compatibility APIs on MLX
* fix: harden MLX notebook compatibility shims
* MLX: add torch.cuda.mem_get_info to the compatibility shim
Notebook memory cells call torch.cuda.mem_get_info()[0] directly (not gated by
is_available), so on MLX it raises without a shim. Return (free, total) bytes
from the MLX device stats, consistent with the other torch.cuda compat helpers,
and add a matching assertion to the compat-API test.
* MLX: use active memory for mem_get_info; fix BatchEncoding.to keyword device
Address review on the MLX compatibility shim:
- torch.cuda.mem_get_info() now derives free bytes from current active MLX
memory instead of the peak high-water mark, so a capacity check stays
accurate after a transient spike or a prior run.
- BatchEncoding.to(device=...) passed by keyword no longer forwards a positional
None alongside the keyword (which raised "multiple values for 'device'"), so
non-CUDA keyword moves like .to(device="cpu") delegate correctly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MLX: accept preserve_dataset_order; stub RL trainers with a clear error
Two fixes so unmigrated notebooks behave predictably on MLX (torch present):
- preserve_dataset_order is a real MLXTrainingConfig field but was missing from
the extra-argument allowlist, so passing it (as a config or trainer kwarg)
could be rejected as unknown on a zoo without the field. Add it to
_MLX_IMPLEMENTED_EXTRA_ARGUMENTS so the documented no-shuffle path is reachable.
- GRPO/DPO/ORPO (and KTO/PPO/Reward) have no MLX trainer yet. Retarget the ones
the installed trl exposes to a stub that raises a clear 'not supported on MLX'
error instead of importing the real torch/CUDA trainer and crashing deep
inside it. Only existing trainers are retargeted (no invented attributes),
idempotent across re-imports.
* MLX: make RL-trainer stubbing import-safe; back current-memory APIs with active memory
Address review on the MLX shims:
- The RL-trainer stub loop probed trl with getattr(_trl, name), which triggers
trl's lazy trainer import and pulls torch -- that can crash import unsloth on a
torch-free MLX install just to check existence. Decide what to stub from
trl.__all__ + already-materialized attrs (vars) instead; never resolve the real
trainer. All trl trainer names are in __all__, so they are still stubbed (even
torch-free), and the probe no longer imports torch.
- torch.cuda.memory_reserved / memory_allocated (the current, non-max APIs) were
aliased to peak max_memory_reserved. Back them with current active MLX memory so
cleanup / capacity checks see live usage; max_* keep the peak high-water mark.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MLX: keep TRL's SFTConfig epoch default under the trl.SFTConfig alias
Unmigrated notebooks import SFTConfig from trl, which the MLX build aliases to
the public training-args class. TRL/HF SFTConfig defaults to num_train_epochs=3
(max_steps=-1); the native MLX config defaults to max_steps=60. So an SFTConfig
built without an explicit length silently ran 60 MLX steps instead of TRL's 3
epochs under the alias. Alias trl.SFTConfig to a thin subclass that seeds the
TRL epoch default only when neither max_steps nor num_train_epochs is given;
explicit lengths pass through untouched, and the native public args class keeps
its MLX default. Epoch mode is supported by the MLX trainer.
* MLX CI: keep the GGUF reload smoke under the job timeout
The RELOAD-GGUF-via-llama-cli step timed out at 300s. BF16 GGUF decode is
CPU-bound on the macOS runner (~10s+/token), so generating 24 tokens landed
right on the 300s cliff and killed the process. This step is a save/reload
integrity smoke (it only needs a few chars of output), so the token count is
incidental: generate 8 tokens with explicit threads and a small headroom on the
subprocess timeout, all env-tunable (UNSLOTH_GGUF_RELOAD_N / _THREADS /
_TIMEOUT). Cuts the reload well under the 25 minute job budget.
* MLX: broaden trainer stubs, real peak-memory reset, fix shim tests
Address review on the MLX public API:
- The SFTConfig identity tests asserted trl.SFTConfig is UnslothTrainingArguments,
but the alias now points at the _MLXSFTConfig subclass that preserves TRL's
epoch default, so the MLX suite failed before testing the shim. Assert
issubclass instead.
- torch.cuda.reset_peak_memory_stats was a no-op, so max_memory_reserved kept
earlier model-load peaks across a scoped run. Wire it to mx.reset_peak_memory
with the same core/metal fallback used for the reads.
- The unsupported-trainer stubs were a fixed list, so trainers outside it (a
newer RLOOTrainer) still routed to the real torch trainer. Derive the set from
trl.__all__ (every non-SFT *Trainer) so all non-SFT surfaces fail with a clear
MLX message; names come from __all__ so trl is never resolved.
- The non-MLX export smoke skipped only on missing bitsandbytes/triton; other
absent GPU deps (numpy/torch/unsloth-zoo, or _gpu_init re-raising ImportError)
made it fail on CPU hosts. Skip on any ImportError.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: keep MLX notebook compatibility minimal
* MLX CI: force CPU + small context for the GGUF reload smoke
The RELOAD-GGUF-via-llama-cli step timed out even at 8 tokens (>420s), so it is a
fixed hang, not per-token cost: on the paravirtual macOS runner GPU llama.cpp's
Metal backend stalls, and the gemma3 GGUF advertises a 32768 context that llama-cli
would otherwise fully allocate. Run llama-cli CPU-only (-ngl 0) with a small context
(-c 256); keep generation short. All env-tunable (UNSLOTH_GGUF_RELOAD_NGL / _CTX /
_N / _THREADS / _TIMEOUT). Also print llama.cpp's partial stdout/stderr on timeout so
a future hang is diagnosable instead of an opaque TimeoutExpired.
* MLX CI: export the reload-smoke GGUF as q8_0, not bf16
The GGUF reload via llama-cli timed out on the runner even CPU-only with a tiny
context and 8 tokens. Root cause is the format, not the flags: the smoke exported
quantization_method='not_quantized', which maps to a bf16 GGUF, and llama.cpp's
bf16 CPU decode is unusably slow on the paravirtual macOS runner. Export q8_0
(fast_quantized, the exporter default and what users deploy) instead -- llama.cpp
has optimized q8_0 CPU kernels, so the fresh-process reload loads and generates in
seconds. The reload stays CPU-only (-ngl 0) with a small context.
* test: clear TRL shim before availability check
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Add --with-llama-cpp-dir flag to install.ps1 and install.sh
Users can now pass --with-llama-cpp-dir /path/to/llama.cpp to the
installer to skip downloading or building llama.cpp and use a local
directory instead. A junction (Windows) or symlink (Linux/macOS) is
created at the canonical install location, bypassing both the prebuilt
download (Phase 3) and source build (Phase 4) steps in setup.ps1/setup.sh.
The path is passed via UNSLOTH_LOCAL_LLAMA_CPP_DIR env var which
setup.ps1 and setup.sh read directly.
Ported from the idea in unslothai/unsloth#4384, reimplemented against
current Studio architecture.
* test: add static wiring test for --with-llama-cpp-dir flag
Cross-checks install.sh, install.ps1, studio/setup.sh and studio/setup.ps1
so the flag's contract (parse -> UNSLOTH_LOCAL_LLAMA_CPP_DIR env var -> link
local dir, skip prebuilt download and source build) can't silently regress.
Wired into studio-backend-ci.yml alongside the other tests/sh installer tests.
* Address review feedback on --with-llama-cpp-dir flag
- setup.ps1: delete an existing junction/symlink via DirectoryInfo.Delete()
instead of a recursive remove, which can traverse the link and wipe the
user's real llama.cpp directory on PowerShell 5.1.
- setup.ps1: short-circuit the build chain when a local dir is linked so CMake
never runs inside the user's checkout when it lacks a Windows-layout binary.
- install.sh / setup.sh: resolve paths with CDPATH= cd -P so a set CDPATH
cannot corrupt the resolved path.
- install.sh: seed _WITH_LLAMA_CPP_DIR from UNSLOTH_LOCAL_LLAMA_CPP_DIR so an
exported env var (piped-install style) is honored instead of being clobbered.
- setup.sh: create the root llama-quantize shim when linking a local source
build so GGUF export's check_llama_cpp() still finds it.
- setup.sh / setup.ps1: drop a stale link before the custom-home ownership
assert so re-runs with the flag stay idempotent.
- test: pin the new linked-dir build short-circuit.
* Harden --with-llama-cpp-dir against Codex/Gemini review findings
- install.sh: error when --with-llama-cpp-dir is the final arg with no path,
matching the existing --package/--python post-loop guards (was a silent
fallback to the normal prebuilt/source install).
- studio/setup.sh: canonicalize LLAMA_CPP_DIR before the self-link no-op
compare. _RESOLVED_LOCAL is fully resolved while LLAMA_CPP_DIR was textual,
so a symlinked $HOME made the guard miss and the rm -rf could wipe the
user's real llama.cpp tree.
- studio/setup.sh: make the llama-quantize shim non-fatal; it writes through
the link into the user's tree, which may be read-only (shared/CI cache),
and under set -e a failed ln aborted an otherwise-good reuse.
- studio/setup.ps1: detect a broken junction via Get-Item -Force instead of
Test-Path so a dangling link from a prior run is removed and mklink can
relink to a new valid directory.
- studio/setup.ps1: use Copy-Item -LiteralPath so a source path containing
[ ] isn't treated as a wildcard in the junction copy fallback.
- tests: update the wiring assertions for the LiteralPath copy and the
canonicalized compare.
* Validate/reuse local llama.cpp tree and guard the in-use case
Addresses the second Codex pass on the --with-llama-cpp-dir flag:
- Validate the linked tree before disabling installs (setup.sh + setup.ps1):
reusing a local dir skips BOTH the prebuilt download and the source build,
so the dir must already contain a runnable llama-server (build/bin on
Linux/macOS, build\bin\Release\llama-server.exe on Windows). Bail out with a
clear message instead of linking an unbuilt/wrong-platform checkout and
leaving Studio with no usable binary.
- Treat a canonical-path target as already linked when it holds a build
(setup.sh + setup.ps1): point the flag at ~/.unsloth/llama.cpp itself and an
existing build is reused (skip prebuilt + source) rather than clobbered by
the staged prebuilt installer (which uses os.replace()/replace). An empty
canonical dir still falls through to the normal in-place install.
- Abort when an in-use llama.cpp can't be removed on Windows (setup.ps1):
Remove-Item -ErrorAction SilentlyContinue can silently leave a locked tree
in place; detect that and stop with the same active-process message + exit 3
the prebuilt path uses, instead of junctioning over a half-present dir.
Left as follow-up (already tracked by the PR author as a non-blocker): the
in-app "Update llama.cpp" updater does not yet recognize a local-link install
as externally managed; that fix belongs in studio/backend/utils/llama_cpp_update.py.
* Accept all backend llama-server layouts in --with-llama-cpp-dir validation
The linked-tree validation only accepted build/bin[/Release]/llama-server, but
LlamaCppBackend._layout_candidates() resolves a root-level llama-server first,
then build/bin, then build/bin/Release on Windows. A `make` build or a flat
release extract (binary at the dir root) was therefore rejected with a hard
installer failure even though Studio would have run it.
Validate the same candidate set the backend uses in both setup scripts, and add
wiring-test assertions so the check can't silently narrow again.
* Treat --with-llama-cpp-dir local links as externally managed
A --with-llama-cpp-dir install junctions/symlinks the canonical llama.cpp dir to
the user's own checkout, but two backend paths still treated it as a Studio-owned
tree:
- The in-app updater (llama_cpp_update) offered and could apply an official
prebuilt over the link, writing through it into the user's checkout (or
failing) and silently dropping the link the flag created.
- Orphan cleanup (LlamaCppBackend._kill_orphaned_servers) resolved the linked
root into its kill allowlist, so a llama-server the user launched from the same
checkout was classified as ours and killed on startup.
Detect the canonical dir being a symlink/junction (reparse point) and treat the
install as unmanaged: get_update_status reports unsupported, start_update refuses
with reason "local_link", and the linked root is left out of the orphan
allowlist. Adds behavioral tests (link vs plain dir, updater refusal, and the
spared-vs-killed orphan control).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add behavioral shell test for --with-llama-cpp-dir linking
The existing tests/sh/test_with_llama_cpp_dir_flag.sh is a static grep of the
scripts. This adds a behavioral test that extracts the real link block from
studio/setup.sh (by content anchors, with a self-validating extraction) and runs
it against hermetic fake dirs, asserting the outcomes that matter:
- an external CMake build links and arms neither the prebuilt download nor the
source build
- a flat / make tree (root-level llama-server, no build/bin) is accepted too
- an unbuilt tree is rejected with a non-zero exit and no link left behind
- relinking over a stale link preserves the target's contents (no data loss)
- pointing at the canonical path is a no-op reuse, not a self-referential link
Symlink-identity checks run only where real symlinks exist (skipped on Windows
git-bash copy-mode); the link/skip/no-data-loss checks run everywhere. Wired into
studio-backend-ci.yml next to the static test.
* Install psutil in backend CI so orphan-cleanup tests run
The new orphan-cleanup tests import psutil for the process scan, but the Backend
CI deps step installed studio.txt plus a fixed extras list that omits it, so the
two tests failed with ModuleNotFoundError. Add psutil to both backend pytest dep
steps (kept in shared shape), and guard the import with pytest.importorskip so a
minimal env without psutil skips these tests instead of erroring.
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: Implementation of the Portuguese (Brazil) language and VRAM/RAM monitor.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/frontend/src/hooks/use-gpu-utilization.ts
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/main.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/main.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/backend/utils/hardware/hardware.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/utils/hardware/hardware.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/frontend/src/features/settings/components/usage-examples.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/backend/main.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/utils/hardware/hardware.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/frontend/src/features/studio/sections/progress-section.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix: resolve automated review feedback on API shape
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix review issues for PR #6509: Cpu icon, VRAM percent, system polling
- model-inspector: use the exported CpuIcon (Cpu is not a Hugeicons export)
- app-sidebar: guard the VRAM percent on totalVram to avoid Infinity, and
reset the system poll cache only after each request settles so a slow probe
is reused instead of stacking overlapping requests
- use-gpu-info: populate CPU/RAM on hosts without a GPU
- progress-section: label GPUs by visible_ordinal instead of array index
- hub-page: base the RAM label on systemRamTotalGb
- usage-examples: emit JS sampling and tool options at the top level instead
of nesting them under extra_body (the JS SDK does not unwrap extra_body)
- main: read torch and transformers versions from package metadata instead of
importing the libraries on every system poll, and guard the VRAM math
against null values
- hardware: translate a leftover comment to English
* Harden /api/system: guard psutil.boot_time for PR #6509
Simulating restricted containers and some VMs (where psutil.boot_time can raise)
showed the /api/system endpoint would 500 on the unguarded boot_time call, the
same failure class already handled for cpu_freq, disk_usage, and Process. Wrap
boot_time and return uptime_seconds as null when it is unavailable so the sidebar
monitor degrades gracefully instead of breaking. Widen the uptime_seconds type to
number | null to match.
* Studio: make the sidebar hardware monitor a toggle (default on) for PR #6509
Adds a "Show hardware monitor" switch under Settings > Appearance > Layout,
backed by a localStorage preference (default on), mirroring the existing
useSidebarPin pattern. When turned off, the sidebar hides the VRAM/RAM meters
and useSystemInfo stops the 3s /api/system poll entirely, so no nvidia-smi /
SMI probes run while the monitor is disabled. Adds the en and pt-BR strings.
* Studio: default the sidebar hardware monitor to off (opt-in) for PR #6509
* Studio pt-BR: fix three small translation defects for PR #6509
- learningRateDescription: "5e-5 for CPT" -> "5e-5 para CPT" (leftover English)
- exportScopeRecents: "Recents" -> "Recentes" (untranslated)
- relativeMonthsAgo/relativeYearsAgo: add the missing space ("há {count} meses"/
"há {count} anos") so they no longer render as "há 3meses"
* Studio pt-BR: translate the last 10 fallback keys for PR #6509
Adds the settings.general.storage block (Armazenamento) and the
settings.chat.modelDisclaimer pair, so pt-BR now covers all en keys
(679/679) with no English fallbacks.
* Studio: hide sidebar VRAM row on CPU-only hosts for PR #6509
* Studio: tighten and trim code comments for PR #6509
* fix: UI issue in the stop button dialog box (fine-tuning)
* Studio pt-BR: translate 18 new keys from main merge (password dialog, GGUF export, dataset streaming) for PR #6509
* Rounding to GB
* Fix/adjust System resources tab for PR #6509
* Fix/adjust GPU monitor review items for PR #6509
* Fix/adjust remaining GPU monitor review items for PR #6509
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust MLX resource fallback for PR #6509
* floating window implementation
* resize for floating window
* Fix resource monitor review items
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restore frontend optional dependency lock entries
* Make GPU selection tests hermetic
* Fix GPU monitor CI test failures
* Bound MLX GGUF reload smoke
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix MLX GGUF reload smoke exit
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: self-heal a pre-#6483-fix anyio>=4.14 stuck in existing installs
The <4.14 cap in constraints.txt/no-torch-runtime.txt only constrains new
anyio resolutions. An install made before that cap existed can already be
sitting on anyio 4.14+, and since it already satisfies mcp/fastmcp's
anyio>=4.5 floor, every later constrained install skips it as
already-satisfied -- so affected installs never recover and keep hitting
the cancel-scope RuntimeError on every request (#6797, a recurrence of
#6483). Force-reinstall anyio<4.14 whenever a stuck 4.14+ is detected.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: also repair anyio on the update fast path
setup.sh's _SKIP_PYTHON_DEPS and setup.ps1's $SkipPythonDeps skip
install_python_stack.py entirely once the installed package version already
matches PyPI latest, so an install stuck on anyio>=4.14 with an otherwise
up-to-date package never reaches the repair added in install_python_stack.py.
Probe anyio on that fast path too and fall through to the full dependency
pass when it's still >=4.14, mirroring the existing ROCm/CPU-torch override
right below it.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add customizable RAG embedding model setting and reorganize settings tabs
Chat with files, project sources, and knowledge bases previously always
embedded with unsloth/bge-small-en-v1.5. This adds a Settings option to
pick any Hugging Face embedding model (or local path), with HF search
autocomplete, server-side verification that the repo is actually an
embedding model, and a save anyway escape hatch for offline or local
models. The setting persists in app_settings and applies at runtime to
both the sentence-transformers and llama-server GGUF embedder backends
without a restart.
Also reorganizes the General settings tab: Documents & RAG sits above
Uploads, Helper LLM moved above the danger zone, and Model auto-switch
(OpenAI API) moved to the bottom of the API tab.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Support local model paths on the GGUF embedder and normalize default saves
Found by simulation testing of the embedding model setting:
Local paths saved as the embedding model now work on the llama-server
GGUF backend (the default backend on macOS and CPU). A path to a .gguf
file is used directly and a directory is scanned for a variant-matching
non-mmproj .gguf, with a clear error when none exists. Previously a
local path was sent to the HF hub API and failed with a repo lookup
error.
Saving the default model explicitly no longer stores an override, so
is_custom stays false and the UI does not show a reset button for the
default value.
* Address review: stale-vector handling, GGUF derivation, save-time guards
Review follow-ups, each verified by new tests:
Re-uploading a document after an embedding model change now re-indexes
instead of deduping by content hash. Documents record the embedder that
produced their vectors (lazy embedding_model column, NULL legacy rows
keep deduping) and a mismatch replaces the old document.
A vector width change no longer bricks the dense index. ensure_vec
drops and recreates chunks_vec when the dim changes (old vectors are in
a foreign space and only block inserts) and search_dense returns empty
on a width mismatch instead of surfacing a vec0 error, so lexical
search keeps working until documents are re-uploaded.
Saving a local sentence-transformers folder with no .gguf now returns
409 with a clear message when the install embeds via llama-server,
instead of failing at first index. force still saves.
A custom RAG_EMBEDDING_MODEL env without RAG_EMBED_GGUF_REPO now
derives the -GGUF companion repo instead of silently keeping the bge
GGUF on CPU and macOS installs.
The resolved GGUF path is tagged with the repo captured at entry, so a
setting change during a download cannot mark the old model as current.
GGUF repo detection matches gguf as a whole name segment rather than a
substring, hf_token is trimmed before verification, and the settings
combobox drops a redundant state mirror of its controlled value.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shrink embedding model font to 11px in the input and dropdown
The combobox wrapper applies className to the outer input group, so the
size utility must target the inner input element; the previous text-xs
never reached it and the field rendered at the browser default.
* Show curated unsloth embedding models when the search field is empty
The empty-query listing was the global top-downloads page, which holds
no unsloth mirrors for the unsloth-first float to reorder, so the
dropdown opened on third-party models. Match the model picker: curated
unsloth listing when empty, whole-Hub search once a query is typed.
* Address review: settings resilience and index consistency
Keep the last known embedding model on settings store errors, remove the
re-entrant dim lock in the llama-server backend, accept local GGUF saves
and verify GGUF availability for HF repos on that backend, match local
path embedders exactly in model list filters, drop same-width stale
vectors from dense search, pin the embedder per ingestion job, and only
replace completed documents after the re-index succeeds.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Consolidate the GGUF repo derivation tests
* Trim to a single core embedding-model test
* Address review: GGUF repo saves and cache race
Accept a GGUF-named HF repo on the llama-server backend by verifying GGUF
availability instead of the sentence-transformers metadata gate, and guard
the settings cache with a generation counter so a read overlapping a save
cannot repopulate it with the pre-save value.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio RAG: fix RTL/Indic PDF corruption and dropped DOCX tables
The RAG parser prefers pymupdf4llm.to_markdown for PDFs, but that rebuilds text from
positioned glyphs and mangles complex-shaping scripts (RTL Arabic/Hebrew come back as
shaped Presentation Forms, Indic matras drop to U+FFFD) and can silently drop most of a
heavy-RTL page. _pdf now compares the Markdown against PyMuPDF's logical-order
get_text() per page and falls back to it when the Markdown looks corrupted (shaped
Presentation Forms or U+FFFD above a small floor/ratio) or holds far fewer letters than
the raw layer. Latin PDFs are unaffected and keep their Markdown tables/headings.
_docx walked document.paragraphs, which excludes table cells, so DOCX tables were
dropped entirely. It now walks body content in document order via iter_inner_content,
emitting each table row as pipe-joined cells (deduped across merged cells); the preview
locator already anchors on pipes.
Adds parser tests for the corruption and incompleteness fallbacks and for DOCX table
extraction. These mirror the chat document-extractor guard raised in the unslothai/
unsloth#5351 review; the RAG parser is a separate module and needed its own fix.
* RAG DOCX: keep empty table cells and collapse in-cell newlines
Skipping empty cells shifted later cells left and broke column alignment across rows;
a cell with internal paragraphs (newlines) also broke the pipe-joined row. Keep every
cell (dropping the row only when all are empty) and normalize each cell with
" ".join(split()) so multi-paragraph cells stay on one row. Adds a test for both.
* RAG DOCX: dedup merged table cells on the <w:tc> element directly
Store the shared <w:tc> lxml element in the seen set instead of its id(); it is
hashable and compares by the underlying node, so it dedups spanned/merged cells the
same way without relying on id(). Adds a merged-cell test.
* RAG DOCX: align merged cells, pad skipped grid columns, flatten nested tables
* RAG DOCX: walk cells in document order so nested tables keep in-cell position
* RAG DOCX: dedup vertically merged cells so a spanning label is indexed once
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Add a shared fits-on-device filter to the model selects
The chat model selector gains an Only show models that fit on this
device tick under its filter row, and the Hub page gains a matching
Fits device pill next to the sort menu. Both read one persisted
preference (unsloth_models_fit_on_device_only), so toggling either
applies to both.
The filter reuses the Recommended sort's existing fit math, extracted
into hfModelFitsDevice: size from safetensors metadata, GGUF param
count, or the repo name, against the 0.7 GPU + 0.7 RAM budget, with
unsizable models hidden. In the chat selector it extends the fit
filtering to the Trending and Recent sorts and to search results;
downloaded models stay visible regardless. An unknown device budget
keeps everything. The preference is cleared by Reset all local
preferences like the other picker toggles.
* Move the device-fit toggle into the sort dropdowns
* Tighten sort menu footer spacing and shorten the label
* Align the footer checkbox with the option text
* Make the footer checkbox circular with a smaller tick
* Clear menu highlight when the pointer leaves the options
* Address review: fit filter coverage and sizing
Exempt on-disk models from the Hub fit filter, apply it to the feed
trending rows and curated search results, size safetensors and MLX rows
by the quantized load estimate instead of checkpoint bytes, and replace
the native title hint with the app Tooltip.
* Make the whole device-fit row toggle the filter
* Studio: opt-in OpenAI /v1 model auto-switch and idle keep-warm
The OpenAI-compatible endpoints serve whichever GGUF is loaded and ignore the
request model field, so an OpenAI client that changes model never reloads. Add
an opt-in setting that, when a /v1 request names a downloaded local GGUF
different from the loaded one, loads it before serving by reusing the existing
/load path (its dedup, tensor fallback, and threading apply). Unknown names
still serve the loaded model, so drop-in compatibility is preserved and no
remote download is triggered.
Also add an optional idle auto-unload (TTL keep-warm): a pure-ASGI middleware
tracks in-flight inference requests so a stream is never unloaded mid-response,
and a lifespan loop unloads the model after the configured idle seconds. Both
settings default off and live in the app_settings store, exposed via
GET/PUT /api/settings/openai-auto-switch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: variant-aware auto-switch, /v1/responses coverage, keep-warm load stamp
Follow-ups from review of the opt-in OpenAI auto-switch path:
1. Variant-aware dedup. _maybe_auto_switch_model compared only the repo id, so
requesting another quant of the loaded repo (e.g. Q4_K_M loaded, Q8_0 asked)
was served by the old quant. Compare hf_variant too, matching /load dedup.
2. Streaming /v1/responses now calls the auto-switch hook. It went straight into
_responses_stream and only checked is_loaded, so stream=True could serve the
old model or 400. Non-streaming already routed through chat completions; the
hook is idempotent once loaded.
3. resolve_local_gguf tries an exact id match before splitting a trailing
:VARIANT, so local ids that contain a colon (e.g. a Windows path) resolve
instead of being cut at the drive letter.
4. Idle keep-warm stamps activity on a load/swap transition. _last_active was
only refreshed by inference requests, so a model loaded after the server sat
idle past the TTL could be unloaded before its first request.
Tests cover each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make the /v1/responses auto-switch test order-independent
The new streaming-responses test passed in isolation but failed under the CI's
randomized collection order with "object has no attribute 'state'": it passed a
bare object() as the request and stubbed only one dispatcher, so an ordering
where the real dispatcher ran hit request.state. Give the request a state and
stub both dispatchers; the test still asserts the hook fires before dispatch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: assert /v1/responses auto-switch wiring on source, not at runtime
The behavioral version executed openai_responses and relied on stubbing its
callees, which a randomized collection order in CI could defeat (the real
dispatcher ran and hit request attributes). Assert on the function source that
the hook precedes both dispatchers instead; the hook's runtime behavior is
already covered by the direct _maybe_auto_switch_model tests.
* Studio: auto-switch on /v1/embeddings, GGUF-only targets, idle-unload race gate
Second-pass review follow-ups on the opt-in auto-switch path:
1. /v1/embeddings now calls the auto-switch hook before the loaded-state check,
matching the other model-bearing OpenAI endpoints (the keep-warm middleware
already treats embeddings as inference).
2. The resolver index is now GGUF-only. The local-model scanners also surface
Transformers/safetensors repos; without a filter, auto-switch could unload
the GGUF and route a request into the non-GGUF loader. _has_local_gguf checks
a direct .gguf, a models-dir folder, and the HF-cache snapshots layout.
3. Idle keep-warm now holds an asyncio gate across the idle check and the
unload, and a request bumps inflight under the same gate, so the loop can no
longer unload in the window between "looks idle" and the kill.
Tests cover each. Broader local-model source parity (LM Studio, Ollama, legacy
caches, custom scan folders) is a follow-up; missing one of those today just
falls through to the loaded model.
* Studio: variant-aware local resolver, count_tokens + audio auto-switch coverage
Third-pass review follow-ups on the opt-in auto-switch path:
1. The resolver is now variant-aware via list_local_gguf_variants. It indexes
only the quants actually on disk, recursing snapshots and quant subdirs such
as the nested per-quant folders, so a requested repo:VARIANT resolves only
when that quant is local and a bare repo resolves to a concrete local quant.
This fixes two gaps: the previous shallow glob rejected nested-variant GGUF
repos, and a request for an uncached quant could send /load down the remote
download path, breaking the local-only contract.
2. /v1/messages/count_tokens now auto-switches like its sibling /v1/messages, so
a count uses the requested model's tokenizer.
3. /api/inference/audio/generate (direct GGUF TTS) is now tracked as in-flight
inference, so the idle loop cannot unload the model mid-generation.
Tests cover each. Two reviewer items are left as follow-ups: indexing the
remaining local sources (LM Studio, Ollama, legacy/default caches, custom scan
folders), which fails safe today by falling through to the loaded model; and
fully serializing concurrent different-model requests, an inherent limit of the
single-slot llama backend that the opt-in feature is not designed around.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make local GGUF resolver fail-safe so a bad model name cannot 500
The auto-switch hook calls resolve_local_gguf without its own guard, and
/v1/completions and /v1/embeddings pass body.get("model") through unchanged.
A non-string model (e.g. {"model": 123}) or any internal scan failure would
then raise out of the resolver and turn a request that would otherwise be
served by the loaded model into a 500, breaking the drop-in compatibility the
feature is built on.
Guard the resolver at its boundary: reject non-string input up front and wrap
the lookup so any failure returns None (fall through to the loaded model).
Add regression tests for both paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: per-model launch flags for auto-switched GGUF models
* Studio: list switch-eligible GGUFs in /v1/models when auto-switch is on
* Studio: settings UI for OpenAI model auto-switch and idle auto-unload
* Studio: show save error over the disabled-idle hint in auto-switch settings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address gemini review (case-insensitive /v1/models retrieve, idle-input empty guard)
* Studio: address codex review (deterministic override args, exclude probe/embedding models from discovery)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep-warm count_tokens, gate idle on auto-switch, drop hidden models
Three hardening fixes to the opt-in auto-switch path surfaced while reviewing
the work that builds on it:
1. count_tokens keep-warm. /v1/messages/count_tokens counts via the loaded
tokenizer and already auto-switches, but the keep-warm middleware did not
track it, so idle auto-unload could free the model mid-count. It is now a
tracked in-flight path.
2. "Off means unchanged" for idle unload. get_auto_unload_idle_seconds now
reports 0 while auto-switch is disabled. Idle unload only makes sense with
auto-switch on (an unloaded model returns only via the next request's swap),
so a stray TTL can no longer trigger a destructive unload while the feature
is off, keeping the disabled state identical to pre-feature behavior.
3. Hidden models are not switch targets. The resolver index now skips what
Studio hides from its own pickers (the llama.cpp validation probe, RAG
embedding weights) via _is_hidden_model, so they can never be auto-switched
to by name.
Tests added for each.
* Studio: bare-id reuse, responses validation order, in-flight tracking
Review follow-ups after folding in the per-model overrides and discovery work:
1. A bare model id (no :VARIANT) is now satisfied by any loaded quant of that
repo. Previously a bare name resolved to the largest local quant, so it could
force a slow reload when a different quant of the same repo was already
serving. An explicit repo:VARIANT request still honors the quant.
2. /v1/responses now runs the auto-switch hook after the empty-input validation
so a request that 400s can no longer trigger a multi-minute model load before
being rejected. The hook still precedes both dispatchers, so streaming
requests switch.
3. The keep-warm middleware now tracks in-flight requests whenever auto-switch
is enabled rather than only when the idle TTL is already positive, so a stream
that starts with the TTL at 0 is still protected if idle-unload is enabled
mid-stream. Off still passes straight through.
Tests added for each.
* Studio: tighten auto-switch code comments
Comment/docstring-only pass over the OpenAI auto-switch feature: collapse
multi-line blocks, drop a comment that restated the gate it sits next to, and
trim verbose docstrings on internal helpers while keeping the load-bearing
rationale (concurrency, API behavior, drop-in compat, gotchas). No logic
change: verified comment-only with the AST/printer signature check.
* Studio: bind auto-switch locks per running loop
Review follow-up. The auto-switch swap lock and the keep-warm unload gate were
module-level asyncio.Lock objects. That is safe under the single uvicorn loop
and on Python 3.10+ (the Lock resolves the running loop lazily on acquire), but
a module-level Lock binds to one loop on pre-3.10, which can raise a loop
mismatch in multi-loop runners. Resolve each lock through a per-loop accessor
backed by a WeakKeyDictionary so every running loop gets its own Lock and stale
loops are collected. No behavior change under the server's single loop.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch re-review fixes (body codes, coverage, swap, alias, tracking)
Follow-ups from a second review pass over the opt-in OpenAI auto-switch feature:
1. OFF-state status codes: /v1/completions and /v1/embeddings moved the body
read ahead of the loaded-state check, so a malformed/empty body with no model
loaded returned 500 instead of the prior 503. A shared helper reads the body
defensively (an unparseable/non-dict body yields no model), and the handler
re-reads after the 503 gate to surface the original parse error exactly as
before. OFF behavior is unchanged.
2. Local-model coverage: the resolver index only scanned ./models and the active
HF cache, while the model picker also lists the legacy/default HF caches, LM
Studio dirs, and user scan folders. A request for one of those named models
silently served the loaded model instead. _build_index now scans the same
roots (Ollama's symlink-creating scanner is skipped on the request path), and
resolution is offloaded with asyncio.to_thread so the wider scan never blocks
the event loop.
3. Swap vs in-flight stream: a cross-model swap killed the llama-server while
another client was still streaming from it. The hook now tracks how many
requests are streaming on the loaded model (in-flight minus those still inside
the hook) and returns 409 instead of swapping while one is active. Concurrent
same-model requests never reach this path, so they are unaffected.
4. Idle-unload + alias: after idle-unload freed the model, an unknown/alias name
resolved to nothing and 503'd, though it served the active model before the
TTL. Idle-unload now remembers the freed id and an alias request reloads it
(only an already-local model, so no remote download), cleared once a model is
loaded again.
5. In-flight tracking: the keep-warm middleware tracked in-flight only while the
feature was on, so a stream started while off could be unloaded if idle-unload
was enabled mid-stream. It now tracks on every inference path; counting is
cheap and invisible to clients.
Tests added for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove stray async_task_outputs files committed by mistake
* Studio: auto-switch review round 3 (revert swap guard, hardening)
Addressing a third review pass:
- Revert the cross-model swap guard. It counted keep-warm in-flight (which
includes external-provider calls that never touch the local model) and so
could 409 a local swap spuriously, and it still left a same-model request able
to start streaming on the model a concurrent swap was unloading. A correct fix
needs a request-lifetime reader/writer barrier; a partial guard was worse than
the honest single-slot behavior, so concurrent different-model use is back to
being serialized (documented), like llama-swap's single slot.
- Non-string request model (e.g. {"model": 123} on a raw-body endpoint) is now
treated as absent, so it falls through instead of raising in the membership
checks once an idle-unload stash exists.
- Idle-unload now stashes and replays the freed quant: an alias reload restores
the exact (id, variant) that was freed rather than the largest local quant.
- Anthropic /v1/messages validates max_tokens before the auto-switch hook, so a
request that 400s never triggers a model load.
- Keep-warm tracks a pending count for requests waiting on the unload gate, so
the idle loop cannot unload the model out from under a request that is blocked
on the gate but not yet counted as in-flight.
- The idle-unload task is awaited after cancel on shutdown to avoid pending-task
warnings.
- The resolver's HF cache scan is None-safe and logs at debug instead of letting
a bad root abort the whole index build.
- upsert_app_setting_map_entry rolls back explicitly on error.
Tests updated/added for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep saved idle-unload seconds when auto-switch is toggled off
* Studio: auto-switch hardening (thread-safe lock maps, body validation)
Defensive fixes from review:
- Guard the per-loop WeakKeyDictionary get-or-create for both the unload gate and
the auto-switch lock with a threading lock, since WeakKeyDictionary mutation is
not thread-safe when two event loops run on different threads.
- Build the resolver index under the cache lock so concurrent callers with an
expired cache don't all run the multi-dir scan at once.
- /v1/completions and /v1/embeddings return a clean 400 for a valid JSON body
that is not an object (e.g. a list), instead of a 500 from body.get(...).
- The keep-warm middleware only tracks POST requests (inference is always POST),
so CORS preflight (OPTIONS) is not counted, and tolerates a None path.
Tests added for the list-body 400 and the non-POST skip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch review round 4 (local-path load, swap guard, idle fixes)
From a 10-reviewer pass:
- HF-cache entries now load by a concrete local path, not the bare repo id. The
resolver records a load_path (the snapshot dir for a models--* cache repo, the
file/dir otherwise) so /load takes the local branch and can never trigger a
download to satisfy a partial cache. The advertised loader_id (repo id) is kept
as the launch-override key. resolve_local_gguf now returns
(load_path, variant, loader_id).
- Re-add a single-slot swap guard: a cross-model swap returns 409 model_switch_busy
while another inference request is active rather than killing its stream (the
caller is excluded from the count), and holds the keep-warm gate across the load
so no new inference starts mid-swap. Concurrent same-model requests never reach
this path. A residual spurious 409 is possible while a concurrent or external-
provider request is active; that is the documented single-slot tradeoff.
- Idle keep-warm tracks (model_identifier, hf_variant): reloading the same repo at
a different quant counts as a fresh model, so it is not unloaded before one TTL.
- Track Studio's own /api/inference/generate/stream so the idle loop can't unload
the model mid-stream on that route.
- A successful manual /load clears the idle-unload reload stash synchronously, not
only on the next idle poll.
Also merged origin/main (the branch had fallen behind, which would have reverted
unrelated files on merge). Tests added/updated for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch review round 5 (concurrency, identity, load gate)
From a 10-reviewer pass (9 request-changes, 1 approve):
- Concurrent same-target requests load once instead of each returning 409. The
count-based busy guard could not tell "another request wants the same model"
(safe, load once) from "another request is using the loaded model" (refuse).
Track in-flight auto-switch requests per (target, variant) and subtract
same-target waiters from the busy count; a cross-model swap still 409s while a
genuinely different request is active.
- Fix the identity confusion introduced when round 4 began loading by concrete
local path: the backend identifier became a filesystem path. Record the
advertised repo id on the backend after an auto-switch load and use it so
(a) a model loaded manually by repo id is recognized as already serving
(no spurious reswap/409), (b) /v1/models reports the repo id, never a host
path or a duplicate, and (c) the idle-unload stash keeps the override keyed by
the repo id, so an alias reload after TTL keeps the user's saved launch flags.
- Gate the manual /load route with the keep-warm lifecycle gate so idle
auto-unload can't unload a model mid-load. load_model now wraps _load_model_impl
in the gate; auto-switch calls _load_model_impl directly since it already holds
the gate.
- Restore default-off parity on Anthropic /v1/messages: an unloaded backend with
auto-switch disabled 503s before the max_tokens 400 check, as it did pre-feature.
When the feature is on, request-shape validation still runs before any load.
Tests added for each; full backend suite diff vs baseline is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch review round 6 (concurrency ordering, leaks, unload gate)
From a second 10-reviewer pass (8 request-changes, 2 approve):
- Same-target concurrency: register a waiter by the raw requested model before
the (slow) resolve, and exclude pending requests from the swap busy count. The
middleware counts a concurrent same-model request as in-flight before it
resolves and joins the resolved-target waiter map, so the prior fix could still
409 it. The guard now subtracts max(same resolved-target, same raw-request)
waiters and ignores pending (a pending request is blocked in the middleware,
not generating, so a swap can't interrupt it).
- External-provider requests no longer block a local swap. The keep-warm
middleware counts every inference-path POST, but external-provider chat returns
before the auto-switch hook and never touches the local GGUF. The chat handler
now untracks itself before proxying, so its in-flight stream can't trip
model_switch_busy on a concurrent local auto-switch. The middleware skips its
own end-decrement for an untracked request.
- Manual /unload is gated like load and idle-unload: it holds the lifecycle gate
and returns 409 rather than tearing down llama-server while an inference request
is in flight.
- Response model id no longer leaks the load path. /v1/models already advertised
the repo id; chat, completions, embeddings, Anthropic messages, and audio
response bodies now use the same _llama_public_model_id helper instead of the
concrete on-disk model_identifier.
- Chat completions validates the non-system-message requirement before the
auto-switch hook (as /responses and /messages already do), so an invalid
request can't swap the resident model before returning 400.
Tests added for each; full backend suite diff vs baseline is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch review round 7 (teardown policy, Unsloth-active swap, training)
From a third 10-reviewer pass (9 request-changes, 1 approve), all on the same
asymmetric-teardown theme. Resolved per the intended policy that only automatic
paths defer to an active stream; deliberate user actions stay interrupting:
- Revert the manual /unload in-flight guard added last round. A manual /load or
/unload is a deliberate action and tears down immediately, as before; only the
automatic idle-unload loop and auto-switch defer to an active request. This
removes the asymmetry the reviewers flagged (manual /load, the /unload Unsloth
branch, and the opposite-backend swaps inside _load_model_impl) by not
extending the guard to deliberate paths, rather than spreading it.
- Auto-switch now refuses a swap whenever another inference request is in flight,
not only when a GGUF is already loaded. _load_model_impl also unloads an active
Unsloth/transformers backend before loading a GGUF, so the busy guard must cover
that case too; otherwise an Unsloth stream could be killed by an auto-switch.
- Refuse API-initiated training while inference is active. When Studio is driven
as an inference API (sk-unsloth key auth), POST /api/training/start returns 409
if a request is in flight, since training frees VRAM by unloading the chat
model and would kill the stream. The Studio UI (session auth) still starts
training and coexists/frees VRAM as before. A mixed UI+API session is not yet
special-cased. Adds auth.authentication.authenticated_via_api_key.
Tests added/updated for each; full backend suite diff vs baseline is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add UNSLOTH_MODEL_IDLE_TTL env override for idle-unload
Borrowed from PR 6517: a startup env var that sets the idle-unload TTL without
the settings UI. Unlike the stored setting (gated on auto-switch), the env value
is a standalone default that enables idle-unload even with auto-switch off, for
headless/container deploys. An explicit UI/API value still overrides it and stays
gated. The settings GET reflects the env default when nothing is stored.
* Studio: auto-switch fixes from review (paths, embeddings input, env idle reload)
- /v1/models advertises a client-facing alias instead of a filesystem path:
the ./models and LM Studio scanners report the on-disk path as the model id,
so the index now prefers model_id/display_name as the advertised/override id
and keeps the concrete path internal as load_path, still resolvable by path.
- /v1/embeddings validates input before auto-switch: a request with a model but
no input now 400s before the hook (like chat/responses/messages), so an
invalid embeddings request cannot unload or swap the resident model.
- Standalone UNSLOTH_MODEL_IDLE_TTL reloads the freed model: the hook now runs
when auto-switch or idle-unload is active, and with auto-switch off it skips
the resolver and only restores the idle-unloaded model, so the first idle
timeout no longer leaves later /v1 requests with nothing loaded.
- Do not resurrect a stale GGUF over an active Unsloth model: the reload-stash
path bails when a non-GGUF backend is loaded, so an unknown /v1 name cannot
tear down a live Transformers/Unsloth model.
- Defensive HF cache scan: each cache root's resolve/dedup is wrapped so a
missing or malformed root skips that root rather than aborting the index.
- Single-model retrieve checks the id is a string before lowercasing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix automatic-load asymmetry, audio reload, preview, idle timer
The standalone UNSLOTH_MODEL_IDLE_TTL reload is a second automatic-load
trigger, but several validate-before-switch guards and reload hooks only
checked the auto-switch toggle. Add a shared _automatic_model_load_may_run()
(auto-switch on, or idle TTL > 0) and route every guard through it.
- /v1/completions validates prompt before any automatic load (it was the one
model-bearing route with no pre-check).
- /v1/chat/completions and /v1/embeddings pre-checks gate on the shared
predicate so a standalone idle TTL cannot reload then reject.
- /v1/messages no longer 503s before the reload hook can restore an idle-freed
model when auto-switch is off.
- Raw completions/embeddings with no model field pass a non-empty sentinel so
the idle-stash reload runs, restoring the legacy "omit model, use loaded" path.
- /api/inference/audio/generate gains the reload hook (after message validation)
so an idle-freed audio GGUF is restored.
- Public preview opts out of auto-switch via a request-scope flag, so a caller's
model field cannot swap away from the pinned checkpoint; preview chat streams
are now matched by _is_inference_path so idle-unload cannot kill them.
- Keep-warm no longer stamps activity on request start, and external-provider
untracking decrements without restamping, so periodic external traffic can no
longer keep the local GGUF warm forever.
Merges origin/main (the branch had fallen behind, which also brought in the
preview route the review flagged).
* Studio: surface model auto-switch in the API tab and demo it in examples
The OpenAI auto-switch toggle previously lived only in Settings -> General.
Add the same toggle to the API tab's usage-examples panel (it shares the
settings cache), and make the examples reflect it: when on, the Python
examples append a second call naming a different downloaded GGUF (so the
model field visibly selects which model serves), and the curl examples gain
a one-line note. Reuses the existing settings API client and i18n keys.
* Studio: harden OpenAI auto-switch reload-only path and Anthropic tool validation
- Omitted-model raw-body requests pass a reload-only sentinel so the idle-stash
reload still restores an idle-freed model, but the resolver never matches a
downloaded GGUF literally named "default".
- Reject malformed Anthropic client tools before _maybe_auto_switch_model so an
invalid request can no longer evict the loaded model.
* Studio: extend auto-switch reload-only and tool validation to schema endpoints
- Schema-backed endpoints (chat completions, responses, count_tokens, messages,
audio) defaulted an omitted model to "default" and passed it to the switch
hook, so a downloaded GGUF named "default" could be swapped to. Route the hook
through a helper that switches only on an explicitly set model, else reload-only.
- Propagate the explicit-set status when building the chat request from a
Responses request, so the non-streaming chat re-check stays reload-only too.
- Validate Responses function tools before the switch hook so a malformed tool
returns 400 without evicting the loaded model.
* Studio: serialize auto-switch swaps across event loops with a process-wide gate
The auto-switch lock is a per-event-loop asyncio.Lock, so two /v1 swaps on
different loops in one process could both pass it and race the single model slot
(the backend and _load_model_impl are process-wide). Add a process-wide
threading gate around the swap, acquired off the loop so a cross-loop wait never
blocks it, layered with the existing per-loop lock. Add a cross-loop test that
fails without the gate (two slow loads overlap) and passes with it.
* Studio: make the auto-switch swap gate wait cancellation-safe
_acquire_swap_gate awaited asyncio.to_thread(lock.acquire) when another loop held
the process-wide gate. to_thread cancellation doesn't stop the worker thread, so a
/v1 request cancelled mid-wait (client disconnect during a cross-loop swap) would
have its thread acquire the gate after the fact, while the finally that releases it
never runs -- permanently deadlocking later auto-switch swaps.
Poll a non-blocking acquire off a short asyncio.sleep instead: it still keeps the
wait off the loop and serializes across loops, but a cancel now lands during the
sleep, when the gate is not held, so nothing leaks. Add a test that deadlocks the
to_thread variant (it times out) and passes with the poll.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: validate modality and tool-confirmation before auto-switch
Two more request shapes could load a named GGUF and only then 400, evicting the
resident model:
- An image request naming a different text-only GGUF. The switch hook now takes
require_vision and rejects a swap to a non-vision target before loading it; a
GGUF's vision capability is its companion mmproj, knowable without a load, and
matches the post-load guard. Only the resolver branch is checked, never the
reload-stash restore.
- confirm_tool_calls=true with stream=false and local tools. /v1/chat/completions
now rejects that shape before the hook, mirroring the local tool path's
bypass_permissions exemption and intent signal.
The vision probe threads the ambient HF token to keep the capability-probe
invariant. Reload-only and idle-reload paths are unaffected.
* Studio: extend validate-before-switch and make the lifecycle gate process-wide
- /v1/messages/count_tokens now rejects malformed client tools before the switch
hook, like /messages (shared _validate_anthropic_client_tools helper), so a
count request can't evict the loaded model.
- /v1/chat/completions rejects a malformed tool_choice forcing object (a
{"type":"function","function":{}} with no name) before the switch hook.
- The inference lifecycle gate that blocks new inference during a swap is now
process-wide (a poll-acquired threading lock, cancellation-safe), not a
per-loop asyncio lock, so a request on another event loop can't start inference
while a swap tears the single backend down.
- Usage examples no longer hard-code a switch-demo repo most users lack; the
model is an explicit placeholder the user replaces.
* Studio: extend the auto-switch modality guard to /v1/responses and /v1/messages
The pre-load vision check that guards /v1/chat/completions now also runs on
/v1/responses and /v1/messages, so an image request naming a text-only GGUF is
rejected before the swap and never evicts the resident vision model. Run the
vision capability probe off the event loop. Make the /v1/models retrieve
loaded fast-path case-insensitive, and never advertise a host path from the
resolver. Remove the dead list_switch_eligible_ids helper, superseded by the
/v1/models catalog.
* Studio: filter /v1/models to GGUF, per-loop catalog lock, reject system-only Responses
Address review findings on the auto-switch path:
- /v1/models advertises only GGUF models the API can actually switch to; a
safetensors/LoRA entry would be selectable but never loadable via llama.cpp.
- The /v1/models catalog cache uses a per-loop lock (like the auto-switch path)
so a second event loop awaiting it can't hang in a multi-loop process.
- /v1/responses rejects system/developer-only input before the switch, mirroring
chat, so an invalid request can't evict the resident model.
- _build_index guards each scan source on its own so one bad root drops only
that source; the vision probe logs a real detection failure instead of
swallowing it.
* Studio: list cached GGUFs in /v1/models by inspecting files, not model_format
The HF-cache scanner leaves model_format unset for GGUF snapshots, so the
previous model_format == "gguf" filter dropped every downloaded HF-cache GGUF
from /v1/models and the retrieve fallback. Decide GGUF-ness from the on-disk
files via the resolver (info_has_local_gguf) instead, run off the event loop, so
the catalog advertises exactly what /v1 can serve.
* Studio: fix /v1/messages/count_tokens route binding plus auto-switch review fixes
The @router.post decorator for /messages/count_tokens had been separated from
anthropic_count_tokens by the _validate_anthropic_client_tools helper, so the
route bound to the validator and dropped its auth dependency. Move the decorator
back onto the handler. Add route-binding tests asserting each /v1 endpoint maps
to its handler with the auth dependency, so a decorator/handler split is caught
at the route level (the direct-call tests missed it).
Also from review:
- update_openai_auto_switch writes both settings keys in one transaction so a PUT
can't leave one updated and the other stale (drop the now-unused single setters).
- max_seq_length override rejects 0 at the boundary (ge=1) instead of accepting
then silently dropping it.
- Document that embeddings auto-switch is best-effort: GGUF pooling has no cheap
pre-load probe like vision's mmproj, so a guard would false-reject GGUF embedders.
- Add a positive idle-unload test (loop frees the model and stashes it for reload).
* Studio: validate Responses tool_choice + Anthropic mixed tools before switch, filter Ollama from catalog
More auto-switch review findings:
- /v1/responses rejects a forcing-function tool_choice with no name before the
switch, mirroring chat, so a malformed request can't evict the resident model.
- /v1/messages rejects mixing Anthropic server tools with custom client tools
before the switch (the check depends only on the payload, so it moves up cleanly).
- /v1/models no longer advertises Ollama-link models: info_has_local_gguf excludes
.studio_links / ollama_links entries, which the resolver skips and can't switch
to, so an advertised id never silently falls through.
* Studio: guard chat audio input before switch; surface env-backed idle unload in settings UI
A chat request carrying audio_base64 rides the same companion mmproj
projector as a vision request, so a text-only target cannot serve it
either. Flag require_vision for audio input as well so the multimodal
probe runs before the switch and a rejected request never evicts the
working model. Generalize the reject message to cover image and audio.
The settings response now reports idle_unload_active (effective TTL > 0)
so the UI can distinguish idle-unload that is active via the
UNSLOTH_MODEL_IDLE_TTL env var from the case where it needs the toggle
enabled.
* Studio: harden auto-switch eviction guards (count_tokens vision, TTS reload-only, mmproj/stash)
Four eviction/correctness fixes on the opt-in /v1 auto-switch path:
- /v1/messages/count_tokens now carries the same require_vision guard as
/messages, so an image count naming a text-only GGUF can't evict a loaded
vision model for a swap that can't serve the request.
- /audio/generate is now reload-only. A local GGUF's audio-input capability
is not a cheap pre-load probe (the companion mmproj signal can't tell an
audio projector from a vision one, and codec TTS ships no projector), so
resolving the client model could load a text/vision-only target and evict
the working audio model before the audio check fails. Only the idle-stash
restore runs here; switching TTS models is an explicit /load.
- The resolver no longer treats a standalone mmproj .gguf as a servable
model. _scan_models_dir's standalone-file pass does not filter mmproj the
way its directory scan does, so /v1/models could advertise a projector and
a switch could load it over the real weights.
- A non-GGUF (Transformers/Unsloth) load and a deliberate /unload now clear
the idle reload stash, so a manual load/unload is never superseded by a
stale idle-freed GGUF that the next /v1 request resurrects.
* Studio: report advertised repo id consistently after an auto-switch
Two model-id reporting fixes so an auto-switched cached HF GGUF is named by
its repo id everywhere, not its snapshot path:
- Streamed /v1/responses envelopes now derive the model id from
_llama_public_model_id (which prefers _openai_advertised_id) instead of the
raw model_identifier. After an auto-switch the identifier is the snapshot
path while the repo id lives in _openai_advertised_id, so the stream used to
report a snapshot basename while /v1/models, chat completions, and
non-streaming Responses all reported the repo id.
- When an advertised alias already resolves to the loaded model (a model
loaded by local path, requested by its repo or LM Studio id), the
already-serving early return now records the alias as the advertised id, so
/v1/models and responses report the alias and mark it loaded instead of the
path-derived basename. Resolver branch only; safe lock-free because an
in-flight request blocks any concurrent swap via the single-slot busy guard.
* Studio: validate request shapes before auto-switch (prompt/input/audio/mcp confirm)
Four more validate-before-switch guards so a deterministic client error never
evicts the resident model on the opt-in /v1 auto-switch path:
- /v1/completions rejects an object/number prompt (only a string or array is
valid) before the switch, instead of loading the named GGUF and letting
llama-server reject the shape afterward.
- /v1/embeddings rejects an object/number input the same way.
- Chat rejects an oversized audio_base64 upload (413) before the switch. The
size cap is a cheap, target-independent length check; the decode itself
stays post-switch to avoid decoding a valid upload twice.
- The chat confirm-without-stream pre-switch guard now mirrors the tool loop's
actual enablement: _effective_enable_tools (honoring a CLI --enable-tools
policy) and mcp_enabled (which opens the tool loop on its own but defers to a
CLI --disable-tools policy). Previously a confirm+no-stream request with only
mcp_enabled slipped past and 400'd after the swap.
* Studio: fix model-id retrieval, streaming n>1, resolver cache TTL, keep-warm auth
Four fixes from review:
- GET /v1/models/{id} legacy raw-path fallback now maps the raw identifier to
the same public id its /v1/models entry uses. After an auto-switch load the
identifier is the snapshot path while the entry is keyed by the advertised
repo id, so a client that cached the old absolute path no longer 404s on a
model that is in fact loaded.
- stream=true with n>1 is now rejected before the switch. Only the
non-streaming GGUF path returns multiple choices, so streaming n>1 is invalid
on every local serving path; both fields are known pre-switch, so it must not
load model B only to 400 and evict model A. Non-streaming n>1 stays
post-switch where the serving path decides.
- The resolver index cache is stamped after _build_index, not with the pre-scan
timestamp. On installs with enough local models for the multi-root scan to
exceed the 5s TTL, the cache was stored already expired and every request
rebuilt it.
- The keep-warm middleware no longer stamps model activity for 401/403
responses. It runs before FastAPI auth, so unauthenticated probes used to
refresh the idle timer without touching llama.cpp; they now decrement the
in-flight count without keeping the model warm.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
Extracted and narrowed from unslothai/unsloth#6543 by @TheJagStudio.
This keeps the startup/banner and text file encoding hardening separate from the already-merged Python code-exec UTF-8 fix in #6548.
Co-authored-by: Jagrat Patel <81472856+TheJagStudio@users.noreply.github.com>
* Studio RAG: disable trust_env on loopback llama-server httpx clients
The RAG embedder health probe (embed_llama_server.py), its pooled httpx.Client, and the vision captioner (captioner.py) call the local 127.0.0.1 llama-server with httpx's default trust_env=True, so an ambient HTTP(S)_PROXY that returns 503 for loopback breaks embedder startup and captioning. Set trust_env=False on these loopback clients, matching the existing fix on the main llama_cpp and inference clients. External provider calls are untouched.
Follow-up to the loopback trust_env fix; covers the remaining local llama-server clients in the RAG path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio RAG tests: accept trust_env kwarg in captioner httpx.post mocks
The loopback captioner now passes trust_env=False; update the _vision_complete
fake_post stubs to accept it and assert it is False.
* Trim comments in Studio RAG trust_env fix (comment-only)
* RAG trust_env test: explicit UTF-8 read + scan all package .py files
Addresses review: utf-8 open avoids a Windows decode error, and scanning every
.py in core/rag catches any future file that adds an httpx call.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Condense the verbose comments and docstrings added by the recent
chat template, GPT-OSS detection, PEFT tensor-parallel, and Studio
inference proxy fixes. Comments and whitespace only; no code changes.
* add models for /update endpoint
* add logic for identifying out of date hf models
* add endpoint for updating hf models
* add relevant field to GgufVariantDetail
* make exception handling better
* add update_available flag for cached_models, and moved /update endpoint from inference -> models
* hook up /update endpoint on the frontend
* implement update scenarios for the model picker
* fix bug where downloaded flag for an older revision was being wrongly set to false
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix import and make hf calls async
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* remove has_vision from UpdateRequest
* fix ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* clear cancel event before updating gguf variant
* set _cancel_event back if it was set initially
* add hf_token to get_paths_info
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: harden model update endpoint and update checks
- update_hf_model: pass snapshot_download local_dir (local_path is not a
valid kwarg and 500s when updating bicodec audio models)
- get_gguf_variants: wrap the remote update check so a network, rate-limit,
gated, or offline failure degrades to "no update info" instead of failing
the whole variant listing, matching list_cached_models
- add regression tests for both paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: HF model update detection and Update action for cached models
Surface an "Update available" cue and a managed Update action for cached
on-device models. /api/hub/update-status compares each cached main GGUF
file's local blobs against the remote main revision using set membership
across all cached revisions, so a repo that was already updated (and still
holds the old snapshot alongside the new one) is not falsely flagged.
The Update action re-downloads through the download manager so it shows in
the Downloads panel with progress and cancel. The frontend wires the Update
button into the GGUF, on-device, and model-selector cards and keeps the
quant label fully visible when the action buttons crowd the row.
Adds regression tests for the multi-revision update check.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: accept force_download kwarg in hf_xet_fallback test double
The download seam now passes force_download to the attempt callable; the _FakeAttempt mock did not accept it, failing 6 tests with TypeError. Add the keyword (default False) so the scripted-results double matches the seam.
* Fix Studio model update regressions
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address Studio update review feedback
* Address Studio update edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Share GGUF update status helper
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix GGUF update detection and cache cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix cached GGUF update badges
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: shimmyshimmer <107991372+shimmyshimmer@users.noreply.github.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* i18n: register Japanese language support in messages
* i18n: add Japanese locale support
* Update studio/frontend/src/i18n/locales/ja.ts
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/frontend/src/i18n/locales/ja.ts
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* test(i18n): add ja locale to parity check
* i18n: fill remaining missing keys for Japanese locale
* i18n: fix terminal string localization in ja locale
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* fix(studio/llama_cpp): disable trust_env on the loopback health probe
_wait_for_health() polls http://127.0.0.1:<port>/health with the default
httpx trust_env=True, so an ambient HTTP(S)_PROXY in the environment is
applied to the loopback request. A proxy that returns 503 for 127.0.0.1
makes every probe fail, so the loop runs until timeout and Studio load
hangs (trust_env=False returns 200 immediately).
Pass trust_env=False so the local readiness probe never goes through a
proxy. This mirrors the existing trust_env=False handling in the sibling
llama_http / external_provider HTTP clients.
* test(offline_gguf_cache): accept trust_env kwarg in fake_get mock
_wait_for_health now calls httpx.get(..., trust_env=False); update the retry test's fake_get to accept the kwarg so it doesn't raise TypeError.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/llama_cpp): bypass proxies for loopback clients
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/routes): bypass proxies for llama streams
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* studio: announce Cloudflare tunnel state and warn about public exposure on startup
The startup banner only printed a line when a tunnel URL was up, so a plain
`unsloth studio -H 0.0.0.0` launch silently created a public trycloudflare.com
URL with no indication that Studio had become reachable from the internet. The
only hint at the tunnel was the CLI help, shown when an invalid command was typed.
Make the banner always state the tunnel state for wildcard binds:
- ON: the public URL plus a warning that anyone with it can reach Studio from
outside the network, and that --no-cloudflare keeps it local-only.
- FAILED: requested but did not start (local network only).
- OFF: --no-cloudflare was passed (local network only).
Secure mode keeps its existing wording (the authenticated tunnel is intended and
--no-cloudflare is not valid there). Clarify the --cloudflare help text in both
the argparse and typer definitions. Default behavior is unchanged.
Also surface the state on the `unsloth studio run` banner, which runs the server
with silent=True and prints its own banner: it now calls _print_cloudflare_line
too, so the ON/OFF/FAILED notice and public-exposure warning are no longer
skipped on that path (previously it only echoed the URL when a tunnel was up).
For the OFF and FAILED notices, do not claim "local network only" when the
reachability probe just confirmed the raw port is reachable from the public
internet: --no-cloudflare and a failed tunnel disable only the Cloudflare link,
not the wildcard bind, so the message is reworded to flag the public raw port.
* Fix/adjust Cloudflare banner warnings for PR #6515
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust Cloudflare banner comments for PR #6515
* Fix/adjust IPv6 Cloudflare tunnel gate for PR #6515
* Fix/adjust Cloudflare review comments for PR #6515
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix silent run Cloudflare notice
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: quick eject from the model selector
Add a one-click eject shortcut to the loaded-model pill so users do not
have to open the picker to unload a model.
- The loaded-status indicator shows a green checkmark at rest and swaps to
a red eject icon on pill hover, with an "Eject model" tooltip. Clicking
it ejects without opening the picker.
- On Device tab now uses the placeholder "Search local models" instead of
"Search Unsloth models".
- The picker's "Eject model" button uses medium font weight.
* Studio: drop unused group/eject marker class on the eject control
* Studio: make the inline eject control valid HTML
The eject shortcut was a focusable span (role/tabIndex) nested inside the
trigger button. A button's content model forbids focusable descendants, so
make it a plain decorative span (aria-hidden, no role/tabIndex) that keeps
the mouse shortcut. Keyboard and screen-reader users eject via the picker's
"Eject model" button.
* Studio: disable the inline eject shortcut on touch devices
On touch (no hover) the red eject icon and title tooltip never reveal, so
tapping the loaded pill could unload the model with no visible affordance.
Add [@media(hover:none)]:pointer-events-none so taps fall through to the
trigger and open the picker; touch users eject from the picker instead.
---------
Co-authored-by: shimmyshimmer <shimmyshimmer@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* Add whole-document context mode to RAG chat attachments
Thread-attached files are injected in full when they fit a token budget,
instead of only top-K retrieved chunks, so the model reads the entire file
for summarize/reason-over-document requests. Oversized files fall back to
top-K retrieval so the context window is never blown. KB and project
corpora are unchanged (still retrieval).
- core/rag/store.py: all_chunks_for_scope returns every completed-document
chunk for a scope, ordered document-then-index, joined with filename.
- core/rag/tool.py: whole_document_context renders the chunks as the same
<chunk> blocks + citation source-map retrieval produces, returns None
when empty or over budget.
- core/inference/tools.py: build_rag_autoinject tries whole-document first
for thread scopes, falls through to search_for_autoinject otherwise.
- core/rag/config.py: THREAD_WHOLE_DOC + WHOLE_DOC_MAX_TOKENS (env-tunable).
- tests/test_rag_whole_document.py: store ordering, whole-doc render +
budget cutoff, auto-inject whole-doc vs top-K fallback, KB never whole-doc.
* Add scanned-PDF OCR fallback to RAG ingestion
A PDF page with no extractable text layer (a scanned or image-only page)
previously ingested as empty, so image PDFs were invisible to retrieval and
whole-document context. Such pages are now rendered and transcribed by the
loaded vision model during ingestion, so they become searchable and readable
like any other page. This restores OCR for the RAG document flow without a
separate extraction pipeline.
- core/rag/parsers.py: render_pdf_pages renders whole pages (1-based) to PNG.
- core/rag/captioner.py: factor the shared vision call into _vision_complete;
add _ocr_one + ocr_pages (transcribe rendered pages, OCR_MAX_PAGES bound).
- core/rag/ingestion.py: _ocr_scanned_pages runs right after parse, replacing
text on near-empty PDF pages. No-op when OCR is off, no page is scanned, or
no vision model is loaded (degrades like figure captioning).
- core/rag/config.py: OCR_SCANNED, OCR_MIN_CHARS, OCR_MAX_PAGES, OCR_DPI,
OCR_TIMEOUT_S, OCR_MAX_TOKENS (env-tunable).
- tests/test_rag_ocr_fallback.py: page render, ocr_pages gating + cap, scanned
PDF end-to-end OCR into chunks + whole-doc, born-digital skips OCR, disabled
leaves the page empty.
* Broaden OCR prompt to figures/tables and guard against repetition runaway
The OCR prompt now asks the vision model to also transcribe text inside figures,
diagrams, charts and tables, so labels and table cells on scanned pages are
indexed rather than skipped. Verified on real documents that this does not
regress plain-text transcription.
Some vision models loop on sparse images (e.g. a title-only cover) and emit the
same line hundreds of times. _collapse_runaway caps any run of identical
consecutive lines so a pathological page cannot flood the index; legitimate
short repeats (a label appearing a few times) survive. Applied in ocr_pages.
* Restrict whole-document injection to thread attachments only
whole_document_context resolved the combined project+thread scope, so a project
chat (the frontend sends both thread_id and project_id) injected the entire
project corpus in full, contradicting the design that project and KB corpora stay
retrieval-only. A large project corpus could also push the total over budget and
drop a small thread attachment back to top-K.
Resolve the thread scope alone in whole_document_context, and in
build_rag_autoinject only enter whole-doc mode when a thread attachment is present
and no KB is selected (a KB pick is exclusive: search that corpus). Project
sources and KBs keep top-K retrieval. Adds regression tests for the mixed
project+thread payload, the budget isolation, and KB precedence.
* Address review: keep project retrieval, harden budget + OCR guards
Follow-up to the 8-reviewer pass on the whole-document + OCR work.
- Preserve project grounding in project chats. The thread-scope-only fix made
whole-doc exclusive of retrieval, so a thread attachment silently dropped the
project corpus for that turn. build_rag_autoinject now whole-docs the thread
attachment AND retrieves the project sources top-K, merged under one citation
numbering via tool.render_sources. KB selection stays exclusive.
- Budget: a NULL/zero token_count no longer bypasses the cap (length-based
fallback in _row_token_count), so a malformed huge doc can't inject in full.
- OCR runaway guard: _collapse_runaway now also caps each distinct line at a
generous total across the page (not just consecutive), bounding the
interleaved/alternating loops weak models emit; blank-line floods collapse too.
- OCR: warn when a scanned PDF exceeds OCR_MAX_PAGES (pages past the cap stay
untranscribed) instead of silently dropping them.
- Document the known limits: OCR'd pages have no PDF highlight regions; vision
models need a micro-batch >= image tokens (Gemma-family) or the server aborts.
- Tests for project-retrieval composition, NULL-token budget, and interleaved
runaway; drop the now-superseded exclude-project test.
* Add OCR toggle to RAG retrieval settings
Make scanned-PDF OCR user-controllable per upload instead of only via the
RAG_OCR_SCANNED config default. The retrieval settings panel gains an OCR
scanned pages switch (persisted in localStorage, on by default); the chosen
value is read fresh at upload time and sent with each document upload.
Backend: the three upload routes accept an optional ocr form field and pass it
through start_ingestion to _ocr_scanned_pages, which now treats None as use the
config default and an explicit bool as an override. The on/off policy lives only
in _ocr_scanned_pages now, so ocr_pages no longer re-checks the config (that
double gate would have blocked a per-upload ocr=True while the default was off).
Tests cover both override directions (force on while config off, force off while
config on).
* Add "Describe figures & charts" toggle with chart-aware captions
Surface RAG figure captioning as a user control and make it actually useful for
graphs and plots. The figure detection already clustered vector drawings and
raster images into regions and rendered them, but captioning was off by default,
had no UI, and used a thin generic prompt.
Accuracy: the caption prompt now asks for chart type, axis titles and units,
legend or series, salient trends and readable values, and table columns, while
forbidding invented numbers. The token budget is configurable (CAPTION_MAX_TOKENS)
and captions pass through the same runaway guard as OCR so a looping vision model
cannot flood the index.
Control: a per-upload caption override threads from the three upload routes through
start_ingestion and _run, with the on/off policy single-sourced in _run (caption
self-gating removed from caption_images, mirroring the OCR change) so a force-on
override works when the config default is off. The frontend adds a "Describe
figures & charts" switch in the retrieval settings, persisted in localStorage and
sent with each upload. Default on; it is a no-op without a vision model and bounded
to CAPTION_MAX_IMAGES figures per document.
Tests cover the new caption_images contract, the runaway guard on captions, the
chart-aware prompt and token budget (and that OCR keeps its own prompt and budget),
and both override directions end to end through ingestion.
* Generalize figure understanding: transcribe-first prompt + high-DPI tiling
Make figure/chart description work across any visual and any model strength, not
just a strong VLM on simple figures. Two changes, validated by a recall benchmark
on authoritative documents (ResNet/Attention papers, USDA, UN UDHR).
1. Transcribe-first caption prompt. The caption now asks the model to transcribe
every visible label verbatim (titles, axis labels and units, legends, every
box/node/arrow label, table cells, equations) and then add a one-line summary,
instead of only describing the figure. Transcription is the most model-robust
visual task, so weak models that cannot reason about a chart still recover its
labels.
2. High-DPI tiling of figure pages. Figure-bearing pages are rendered as an
overlapping grid of high-DPI tiles (plus a full-page pass for context); each
tile is transcribed, then merged and de-duplicated. This keeps small diagram
labels legible and covers every sub-figure without relying on exact region
detection, which previously missed sub-figures and small labels.
Supporting changes: figure render DPI 130 -> 200 with a clip margin so edge labels
are not lost; vision calls are deterministic (temperature 0) so transcription does
not randomly drop labels; the repetition guard now applies to captions too. New
config knobs: FIGURE_DPI, FIGURE_MARGIN_FRAC, FIGURE_TILE_ROWS/COLS, FIGURE_TILE_
OVERLAP, FIGURE_FULLPAGE, CAPTION_MAX_PAGES, larger CAPTION_MAX_TOKENS, and
CAPTION_MAX_IMAGES as a per-document tile budget.
Measured figure context recall (per-label, dense academic figures):
Qwen2.5-VL: 0.50 -> 0.83 (overall 0.81 -> 0.94)
Gemma-4-E2B (weak): ~0 with loops -> 0.83 (overall 0.91)
Born-digital text and scanned-page recall are unchanged (no regression).
parsers gains _figure_boxes (shared detection), pages_with_figures, and
render_pdf_figure_tiles; captioner gains merge_page_captions and a temperature
parameter; ingestion routes figure captioning through the tiled path.
* Fix RAG review issues: whole-doc budget pre-check, figure gating, empty re-ingest, vision auth
Whole-document context now runs a cheap token-sum pre-check (store.scope_token_estimate)
before hydrating every chunk's text, so an attachment that cannot fit the budget is
rejected without loading the whole corpus into memory. The estimate mirrors
all_chunks_for_scope's filter and the per-row token-count fallback exactly.
Ingestion skips all figure work (PDF rasterization and detection, not just the caption
call) unless a vision model is loaded, so a text-only deployment pays nothing. When OCR
is enabled, scanned/image-only pages are excluded from figure tiling since OCR already
transcribes them whole, avoiding double vision work and overlapping index entries; a
scanned figure page is still tiled when OCR is off.
start_ingestion no longer dedupes forever to a prior ingest that produced zero chunks
(e.g. a scanned PDF uploaded before a vision model was loaded): the empty record is
dropped and the content is re-ingested.
Vision OCR and caption requests now send the backend Authorization header, so they
match the chat endpoint and do not 401 under direct-stream (--api-key) mode.
Adds tests for the budget estimate, scanned-page exclusion, the vision-model gate, the
empty re-ingest path, and the auth-header passthrough.
* Trim RAG vision-ingestion comments and docstrings
Tighten the verbose multi-line docstrings and comments added across the RAG vision
ingestion work (captioner, config, parsers, ingestion, store, tool, build_rag_autoinject,
the RAG tests, and the chat-store/upload-hook frontend toggles) to one or two lines while
keeping their intent. No code changed: verified comment/docstring-only against the prior
commit, and the RAG test suite still passes.
* Fix figure-tiling exclusion and client dedupe for re-ingestable docs
Figure tiling now excludes only the pages OCR actually transcribed, not every
text-less page. _ocr_scanned_pages returns the set of pages it OCR'd, and _run passes
that to pages_with_figures as exclude_pages (replacing the ocr_on-keyed min_text_chars
heuristic). A scanned page that OCR skipped (past OCR_MAX_PAGES, or whose OCR returned
empty) is no longer dropped from captioning, so a chart on such a page still gets a
caption.
The document panel's upload dedupe no longer skips re-selecting a file whose only
matching doc completed with zero chunks. Such a doc is re-ingestable (e.g. a scan
attached before a vision model loaded), and the backend re-ingests on the same content
hash, so the client must let it reach the backend; healthy or still-indexing docs are
still skipped. The SSE complete frame's chunk count is recorded on the doc so the
check is exact.
Adds a regression test for the un-OCR'd scanned figure page and updates the
pages_with_figures test to the exclude_pages interface.
* Address review findings: whole-doc budget guard, job numChunks, dead code, upload cap
whole_document_context now treats a non-positive max_tokens as "never inject" instead
of injecting the whole corpus unbounded, so RAG_WHOLE_DOC_MAX_TOKENS=0 tightens rather
than disables the budget (the real off switch stays RAG_THREAD_WHOLE_DOC=0).
The job-status endpoint and get_job_status now expose num_chunks (joined from the
document), and the upload hook threads it through the SSE-fallback completion paths
(reconcile + poll). Previously a document that finished via the connection-cap fallback
had no chunk count client-side, so the re-ingest dedupe wrongly treated it as empty and
re-uploaded it. IndexJob/JobEvent gain the field and the untyped cast is dropped.
Removes the dead render_pdf_figures function (superseded by the tiling path), its test,
and the unused FIGURE_MARGIN_FRAC config knob.
Adds an upload size cap (RAG_MAX_UPLOAD_BYTES, default 200 MB; 413 on exceed with the
partial file cleaned up) so a pathological file can't drive unbounded parse + vision
work. render_pdf_figure_tiles clamps rows/cols to >= 1 (no ZeroDivisionError on a
misconfigured grid). Captioning progress is reported after OCR so the bar is monotonic.
sqlite connections set busy_timeout=5000 so a long figure/scan ingest holding its
connection doesn't make a concurrent ingest/read fail with "database is locked".
Adds tests for the non-positive budget, the zero-grid clamp, job-status num_chunks, and
the oversize-upload rejection.
* Extract PDF text as layout-aware Markdown via pymupdf4llm
parsers._pdf now extracts each PDF page as Markdown with pymupdf4llm.to_markdown
(page_chunks=True) instead of flat page.get_text("text"), so tables, headings and lists
keep their structure in the indexed chunks and retrieve far better (a table's cells stay
associated with their row instead of flattening into a token stream). Gated by
RAG_PDF_MARKDOWN (default on); falls back to plain PyMuPDF text when the toggle is off,
pymupdf4llm is missing, extraction fails, or a page yields no Markdown. The scanned-page
OCR and figure-tiling passes operate on rendered pixels and are unaffected; docx/html/txt
keep their existing extractors.
The preview-highlight locator already strips Markdown punctuation when building anchors;
it now also splits anchor tokens on pipes so a Markdown table row still anchors to the
raw PDF word stream.
Declares pymupdf4llm as a studio/RAG dependency (was only transitively present via the
data-designer plugin). Adds parser tests (Markdown table reaches the page text, the
plain-text fallback, the missing-lib fallback) and a locator test for table-pipe anchoring.
* Pin pymupdf4llm to 0.3.4 so the package scan does not pull onnxruntime
The lockstep pymupdf4llm 1.27.x line makes pymupdf-layout a hard dependency,
which in turn pulls onnxruntime (plus numpy/networkx/protobuf). The security-audit
pip scan-packages job resolves requirements --with-deps, so adding pymupdf4llm to
no-torch-runtime.txt and studio.txt surfaced onnxruntime's un-baselined CRITICAL
finding and flipped the hf-stack shard from pass to fail.
pymupdf4llm 0.3.x keeps pymupdf-layout behind an optional [layout] extra, so a plain
install resolves to pymupdf + tabulate only and never touches onnxruntime. 0.3.4
requires pymupdf>=1.27.1, satisfied by our pinned pymupdf==1.27.2.3, and to_markdown
(page_chunks=True) produces equivalent layout-aware Markdown on real PDFs (verified on
the Attention, ResNet and USDA documents). Production already installs these files
--no-deps, so onnxruntime was never shipped at runtime; this only fixes the scanner.
The parser test now asserts Markdown markup (heading or table pipes) rather than table
pipes specifically, since 0.3.4 emits a heading but not a pipe table on the tiny
borderless synthetic fixture; both markers are absent from the plain-text fallback.
* Fix RAG whole-doc review findings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address RAG whole-doc review follow-ups
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address RAG review follow-up edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reserve image budget for whole-document RAG
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire imatrix GGUF option and FP8/NVFP4 compressed export into the export UI
GGUF export gains an importance-matrix toggle. When enabled it auto-downloads the
upstream Unsloth imatrix for the base model (or uses a custom path), which unlocks
the IQ low-bit quants iq2_xxs, iq2_m, iq3_xxs and iq4_xs. Merged export gains an
FP8 / NVFP4 compressed-tensors precision selector that runs llm-compressor for vLLM.
Backend threads imatrix_file through routes -> orchestrator -> worker -> export_gguf
(both the local save and the hub push), and maps the new compressed format_type
values onto the fp8/nvfp4 save_method, reporting the "<dir>-<suffix>" sibling output
directory. Frontend adds the imatrix Switch on the GGUF card and a merged precision
picker on the merged card, threaded through the export runtime store.
Depends on unslothai/unsloth#6706 (save.py imatrix_file and compressed-tensors
export) and unslothai/unsloth-zoo#839 (quantize_gguf imatrix flag).
* Studio export: guard imatrix/compressed against older unsloth builds and force imatrix for IQ quants
Addresses review feedback on the export wiring:
- GGUF: pass imatrix_file only when set, so a plain no-imatrix export (e.g. Q4_K_M) no
longer fails with an unexpected-keyword error against an unsloth build that predates the
imatrix_file parameter. When imatrix is requested but unsupported, return a clear
upgrade message instead of a TypeError.
- Merged: gate FP8/NVFP4 compressed-tensors export on the installed unsloth actually
supporting it, returning a clear message rather than a cryptic save_method failure.
- Frontend: IQ quants (iq2_xxs, iq2_m, iq3_xxs, iq4_xs) are imatrix-only, so force the
imatrix on when one is selected and lock the toggle, instead of submitting an IQ quant
with no imatrix that llama.cpp would reject.
Extends the backend tests for the new capability guards and the conditional kwarg wiring.
* Studio: upload compressed merged models to the Hub without recompressing
For an FP8/NVFP4 Hub export the model is already produced locally in the "<dir>-<suffix>"
output. Uploading it directly with HfApi.upload_folder (mirroring export_base_model) avoids
re-running the expensive compressed-tensors quantization a second time inside
push_to_hub_merged, which for NVFP4 also re-runs calibration and risks OOM. Falls back to
push_to_hub_merged when there is no local compressed output to reuse.
* Studio: name the missing extractor when a Recipes upload fails
A missing optional dependency (pymupdf4llm for PDF, mammoth for DOCX) was
reported as a generic "Text extraction failed", which gives the user nothing to
act on. Catch ImportError and surface the package name instead.
* Studio: narrow missing extractor error handling
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* (feat) Add project names to studio training runs to avoid models being overwritten when doing similar training runs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/frontend/src/features/export/export-page.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/frontend/src/features/export/export-page.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/frontend/src/features/export/export-page.tsx
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* better project name sanitization, removed duplicated project name normalization
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* implement checkpoint scanning utilities and tests for base model inference
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard project_name against null and use leading important modifiers
* Fix/adjust training project names for PR #6512
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust training project names for PR #6512
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address project-name review feedback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Show project names in training recents
* Keep GGUF export directories source-specific
---------
Co-authored-by: NZ-Linix <nz-linix@outlook.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: NZ-Linix <linus.ordowski@outlook.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Speed up Studio desktop startup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address Studio startup review findings
* Keep orphaned run cleanup before readiness
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Local models in the Studio Hub tab (Custom folders, LM Studio, and
Local models sections) did not reveal their on-disk path on hover,
unlike the Fine-tuned rows which already do. Each of these rows maps
over a LocalModelInfo with a required path, so pass tooltipText built
from the model name and path via a small shared localPathTooltip
helper, matching the existing FT-row tooltip format.
Refs #6382
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* cascade user message deletion to include assistant reply
* Fix comment typo in delete-thread-message
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: stop handing CI/user secrets to downloaded llama.cpp binaries
The macOS prebuilt path installs llama.cpp from the unslothai/llama.cpp
fork's latest (unpinned, mutable) release and then executes the
downloaded llama-server / llama-quantize binaries during install-time
validation. binary_env() built that child environment from a full
os.environ.copy(), so a compromised or tampered prebuilt would inherit
every secret in the process: HF_TOKEN and the workflow GitHub tokens in
CI, and HF / cloud credentials for end users running install.sh /
setup.sh.
We publish prebuilts daily, so pinning a release tag is not workable.
Instead, neutralise the impact: these binaries have no reason to read any
token, so strip secret-bearing variables (exact names plus
TOKEN/SECRET/PASSWORD/CREDENTIAL/PRIVATE_KEY/API_KEY markers) before
handing the env to a downloaded binary. The installer's own GitHub and
Hugging Face API calls read os.environ directly, so authentication and
release-API rate limiting are unaffected; PATH, LD_LIBRARY_PATH,
DYLD_LIBRARY_PATH and CUDA/ROCm vars are preserved. One change covers the
install-time validation path for all six macOS workflows and end users.
Follow-up (separate, sequenced): publish build-provenance attestations
from the fork's prebuilt workflows and verify them in CI, so a forged
release is rejected rather than merely starved of secrets.
* Strip KUBECONFIG, SSH_AUTH_SOCK, and PASSPHRASE-marked vars from binary env
Extend the deny-list per PR review: KUBECONFIG and SSH_AUTH_SOCK are
credential pointers/capabilities a downloaded binary never needs, and a
PASSPHRASE marker catches SSH_PASSPHRASE / GPG_PASSPHRASE. Tests updated.
* Studio: also scrub proxy/index env vars and URL-embedded credentials before running prebuilt binaries
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope mlx-ci secrets to the install + download commands for PR #6696
Drop the ambient step-level env block and pass GH/GITHUB/HF tokens only
on the installer and GGUF-download commands, so the directly invoked
llama-quantize / llama-server smoke runs see no secrets. The installer
still reads tokens from os.environ for the releases API and probe fetch.
* Trim verbose comments around the secret-env scrubber for PR #6696
Comment-only: condense the block comments added across this PR. Logic
unchanged (comment_tools.py check confirms code-only signature equal).
* Redirect HOME / cache pointers to an empty dir for prebuilt binaries (PR #6696)
Address Codex P2: stripping token env vars still let a tampered binary
read on-disk token stores (~/.cache/huggingface/token, ~/.aws/credentials,
~/.config/gh) through $HOME and the cache/config pointers. Point HOME plus
the HF / XDG / Windows home pointers at a single empty throwaway dir for
the downloaded-binary env. Defense in depth: a binary resolving the real
home via getpwuid is out of scope and needs OS sandboxing.
* Close residual credential-probe gaps for PR #6696
Address the latest Codex review:
- Strip token-only URL userinfo too (scheme://ghp_token@host), not just
the user:pass form.
- Redirect HOMEDRIVE/HOMEPATH alongside USERPROFILE so a Windows binary
cannot reconstruct the real profile from %HOMEDRIVE%%HOMEPATH%.
- Drop explicit credential-file pointers (NETRC, PIP_CONFIG_FILE,
DOCKER_CONFIG, GIT_CONFIG_GLOBAL) that live outside HOME.
- Probe ldd with a secret-free env: linux_runtime_dirs ran ldd on the
untrusted prebuilt with the inherited os.environ, and ldd may execute
the binary, so it could observe HF_TOKEN/GITHUB_TOKEN during the probe.
Factored the shared scrub into secret_free_environ().
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Separate token-bearing install from binary smoke; drop CI command files (PR #6696)
Address the two P1s in the latest review:
- mlx-ci: GitHub bakes secrets into the run-script text, so inline token
assignments in a step that later runs the prebuilt let a tampered binary
read them from the script. Split into a token-bearing install + download
step that never launches a binary, and a secret-free smoke step that runs
llama-quantize / llama-server.
- secret_free_environ now drops the GitHub Actions command files
(GITHUB_ENV, GITHUB_PATH, GITHUB_OUTPUT, GITHUB_STEP_SUMMARY, BASH_ENV) and
the smoke step unsets them, so a tampered prebuilt cannot inject PATH/env
into the later token-bearing MLX steps.
* Run the prebuilt smoke last, after all token-bearing steps (PR #6696)
Address the P1 workspace-poisoning vector: even with no secrets in its env,
a tampered prebuilt could edit the checkout or installed modules, and the
later HF_TOKEN MLX steps would then execute that poisoned code on push
builds. Move the prebuilt install + smoke to the end of the job so the
untrusted binary runs after every token-bearing step, leaving nothing for it
to corrupt. The MLX GGUF reload uses a source-built llama-cli, not this
prebuilt, so nothing depends on the earlier position.
* Trim comments around the secret-env scrubber and prebuilt CI steps (PR #6696)
Comment-only: condense the security-rationale block comments and merge the
duplicated prebuilt-step description in mlx-ci. Logic unchanged
(comment_tools.py check confirms the code-only signature is equal; install
suite still passes).
* Authenticate the GGUF export release-API lookup with the read-only GITHUB_TOKEN (PR #6696)
* Rename env scrubber off the secret-named identifier CodeQL flags as a clear-text sink (PR #6696)
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: restore tensor parallelism for vision/mmproj GGUFs
#6416 disabled --split-mode tensor for any GGUF that ships an mmproj projector to
dodge a GGML_ASSERT crash (#6415) seen on an older llama.cpp build with consumer
Blackwell (sm_120). The blanket skip silently dropped tensor_parallel=true for
every multimodal/MTP GGUF (e.g. Qwen3.6-35B-A3B-MTP); on hardware where the model
fits on one GPU the load then collapsed to a single GPU. mmproj + --split-mode
tensor works on current builds (verified end to end on B200/sm_100), so the skip
was disabling a working configuration.
Make the vision skip self-healing per binary:
- attempt tensor for vision models by default
- skip upfront only on a binary already seen to abort on tensor + mmproj this
session (_vision_tensor_split_aborts), recorded when such a launch crashes at
startup (_record_vision_tensor_split_abort). Process scoped, so a studio update
re-probes the new build. The route-level layer-split fallback stays the net.
- add _select_gpus(min_gpus=...) so a downgraded tensor request can keep multiple
GPUs instead of collapsing to one (default 1, no behavior change).
Add tests/test_tp_vision_regression.py: an AST allowlist guard over the
tensor_parallel drop sites (which would have flagged #6416), plus cache and
_select_gpus coverage. No GPU required.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review on vision tensor-parallel self-healing
Three fixes from the PR review:
- Record a vision-tensor abort only after every startup retry fails. The first
version cached the binary on the first spawn crash, which on every build
(including capable ones) is the benign --fit step abort that the existing
--fit off retry resolves. That poisoned the cache so the next vision load in
the same process skipped tensor. Recording now happens at the post-retry
failure block (after fit-off, flash-attn-off and MTP-drop), so a binary that
actually works is never cached.
- Gate the record on the tensor/mmproj crash signature: a hard signal fault
(_is_signal_crash) with no non-tensor cause (_output_has_nonprojector_diagnostic
excludes OOM and unknown-arch), so an OOM, bad extra args, or MTP/flash-attn
crash no longer marks an otherwise capable binary incompatible.
- Preserve the multi-GPU request on the cached downgrade. The vision gate now
raises _layer_min_gpus to the visible GPU count and threads it through the
layer-split GPU selection (_select_gpus min_gpus and the subset loops), so a
downgraded tensor request still spreads across GPUs instead of collapsing to a
single card the model happens to fit.
Verified two vision+tensor loads in one backend process both tensor-split across
4 GPUs (the benign fit abort no longer poisons the cache). Tests updated.
* Studio: harden vision tensor-parallel self-healing (review round 2)
Address the second review round on the vision/mmproj tensor-parallel fix:
- Preserve vision on the first load: a --split-mode tensor + --mmproj
GGML_ASSERT now raises so the route-level tensor->layer fallback retries
layer split with the projector intact, instead of stripping --mmproj and
silently loading text-only (which returned success and skipped the fallback,
losing vision on the first load until the next cached load).
- Symmetric multi-GPU preservation: the pooled-VRAM tensor downgrade now raises
_layer_min_gpus from the usable tensor GPUs like the vision downgrade, so it
no longer collapses a multi-GPU request to a single card.
- Base the layer fallback minimum on usable GPUs: _select_gpus caps min_gpus to
the count of cards with usable VRAM, so a downgrade never forces a nearly-full
card in (or trips --fit) just to hit the count.
- Re-probe after in-app updates: key the per-binary abort cache on (path, mtime)
like _capability_cache, so POST /api/llama/update swapping the binary in place
(no backend restart) re-probes the new build instead of inheriting the old
build's abort.
- Bump _layer_min_gpus for a known-bad vision binary independent of the tensor
drop, so the route fallback's layer retry (tensor already off) still spreads
across GPUs.
Adds deterministic non-GPU regression tests for each.
* Studio: gate cached-vision layer minimum on the current tensor request
The cached-vision _layer_min_gpus bump fired for every later vision load on a
binary recorded as tensor+mmproj-incompatible, including loads that did not
request tensor parallelism. A plain non-tensor vision load that fits on one card
would then grab every GPU just because an earlier TP attempt aborted in the same
backend process.
Re-tie the bump to the current tensor request (back inside the tensor-drop
guard), so only a downgraded tensor request preserves the multi-GPU spread; a
non-tensor vision load minimizes device count as before.
* Studio: preserve GPU count + confirm assert on vision tensor fallback
Third review round on the vision/mmproj tensor-parallel fix:
- Preserve multi-GPU on the first tensor->layer fallback. The route-level retry
runs tensor-off, so the in-function downgrades can't see the original tensor
request and a fits-on-one-card model loaded the first successful fallback on a
single GPU. The GGUF load closure now passes preserve_multi_gpu_on_layer (the
toggle asked for tensor, this attempt is layer) and load_model raises
_layer_min_gpus for it, so the downgrade still spreads across GPUs.
- Cap the auto-context layer loops to usable GPUs. They bypass _select_gpus, so a
raised _layer_min_gpus could force a nearly-full card into the subset (or trip
--fit). They now start from _auto_min_gpus, capped to the GPUs with usable VRAM.
- Confirm the tensor/mmproj assert before caching. Recording (and the layer-retry
raise) now require the ggml assert marker via _is_tensor_split_assert, not the
bare-signal predicate shared with the projector-incompat branch, so a corrupt
or too-new projector that SIGSEGVs independent of split mode is no longer cached
as tensor/mmproj-incompatible.
Adds deterministic non-GPU regression tests for each.
* Studio: extend multi-GPU fallback to extra/env tensor + overhead-aware cap
Fourth review round on the vision/mmproj tensor-parallel fix:
- Preserve multi-GPU fallback for all tensor requests, not just the UI toggle.
Tensor can also be requested via --split-mode tensor in extra args or an
inherited LLAMA_ARG_SPLIT_MODE=tensor env; the fallback retries those too, so
the preserve_multi_gpu_on_layer hint now keys off _effective_tensor_parallel
(the same check the fallback uses), comparing the overall request against the
current attempt instead of only request.tensor_parallel.
- Cap the auto-context layer fallback to GPUs that can pay the per-device layer
overhead. The cap counted any card with positive usable VRAM, so a nearly-full
GPU with a few MiB free stayed eligible and could be exposed to llama.cpp and
OOM. It now mirrors _select_gpus: a card counts only if usable VRAM exceeds the
per-device pipeline overhead.
Adds deterministic non-GPU regression tests for both.
* Studio: match the #6415 split-axis assert + replay layer-preserve hint
Fifth review round on the vision/mmproj tensor-parallel fix:
- Narrow the tensor/mmproj crash signature. _is_tensor_split_assert matched any
GGML_ASSERT/GGML_ABORT, so an unrelated invariant a corrupt GGUF or projector
trips with --mmproj present could be cached as tensor/mmproj-incompatible. It
now matches the specific #6415 warmup assertion
(GGML_ASSERT(src_ss[0].axis != GGML_BACKEND_SPLIT_AXIS_0) in ggml-backend-meta),
whose split-axis signature is inherent to tensor splitting. A reworded future
assert just re-crashes-then-falls-back (vision preserved via layer split)
instead of poisoning the cache for other models.
- Persist the layer-preserve hint for respawns. A successful tensor->layer
fallback committed _last_load_kwargs without preserve_multi_gpu_on_layer, so
_respawn_if_dead replayed only --split-mode layer + tensor_parallel=False and a
mid-session respawn of a fits-on-one-card model came back single-GPU. The hint
is now in the replay snapshot, so recovery keeps the multi-GPU placement.
Adds deterministic non-GPU regression tests for both.
* Studio: tighten comments on the vision tensor-parallel fix
Make the comments and docstrings added by this PR succinct: collapse the
multi-line block comments in llama_cpp.py / inference.py to one or two lines,
trim the verbose test docstrings (the names and assert messages already carry the
intent), and shorten the module docstring. No code changes; verified comment-only
with scripts/comment_tools.py check --strip-docstrings.
* Studio: cache vision tensor abort only on the split-axis token
_is_tensor_split_assert also accepted any GGML_ASSERT/GGML_ABORT from
ggml-backend-meta, but that file holds many asserts, so an unrelated
scheduler/projector/model invariant on an --mmproj launch could cache the binary
as tensor/mmproj-incompatible and make later compatible vision models skip tensor
parallelism. Match the GGML_BACKEND_SPLIT_AXIS_* token itself (unique to the
#6415 warmup assert), not the source file name.
* Studio: don't leak the httpx test stub into later tests
The regression module stubbed httpx via sys.modules.setdefault, which installs
the lightweight stub even when real httpx is present but not yet imported. The
stub then persists for the whole pytest process, so provider/HF tests collected
later (importing httpx or huggingface_hub.errors) got a module missing
HTTPError/Response. Mirror the neighboring llama_cpp helper tests: import real
httpx first and only fall back to a stub on ImportError.
* Studio: latch the #6415 tensor-split abort on the first spawn, key it per model
The self-heal recorded the --split-mode tensor abort only in the post-retry
failure block, after the flash-attn-off retry. But SPLIT_MODE_TENSOR requires
flash_attn, so the flash-off retry can't run tensor and its output no longer
carries the warmup split-axis assert (ggml-backend-meta :541). The record
therefore never fired on the real reproducer and the crash loop repeated on
every load (reported by oobabooga on #6659).
Latch instead on the first spawn that shows the signal crash + split-axis
marker: record it, kill the process, and raise straight to the route's layer
fallback, skipping the futile flash-attn/MTP retry ladder for this crash.
The crash is a tensor-split geometry limit (e.g. MQA n_head_kv=1 splitting to
GGML_BACKEND_SPLIT_AXIS_0), not a vision/mmproj property: it reproduces without
--mmproj and even single-GPU tensor. So drop the vision/mmproj scoping, rename
_vision_tensor_* -> _tensor_split_*, and key the session cache on
(binary, mtime, model) rather than (binary, mtime) so one model's abort no
longer skips tensor for every other model on the same build.
Regression tests updated to pin the early-spawn record, the per-model cache,
and that an unrelated ggml-backend-meta assert is not treated as the marker.
* Studio: reload on explicit tensor-off after a multi-GPU layer fallback
When a tensor load is downgraded to layer but kept multi-GPU to honor the
tensor request (preserve_multi_gpu_on_layer, the geometry-cache gate, or the
budget downgrade), the server reports tensor_parallel=False with --split-mode
layer stored. A later Apply that explicitly turns the tensor toggle off then
matched the loaded state and deduped to already_loaded, so Studio kept the
fallback's all-GPU CUDA_VISIBLE_DEVICES placement instead of re-selecting
normal placement (a single GPU for a model that fits on one card).
Latch a _layer_preserves_tensor_intent flag in load_model whenever a tensor
request is downgraded to layer with the multi-GPU floor raised
(_layer_min_gpus > 1), clear it when tensor stays on or on unload, and force a
reload in _request_matches_loaded_settings when the user explicitly turns the
tensor toggle off while that flag is set. An Apply that does not touch the
toggle still dedupes, so a working multi-GPU layer server is not churned.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address reviewer.py findings on the tensor-split self-heal
P1 (dedup): tensor intent can be dropped via extras, not only the toggle. An
explicit llama_extra_args=["--split-mode", "layer"] matches the stored fallback
extras, so _request_matches_loaded_settings deduped to the preserved all-GPU
placement instead of reloading. Now reload when layer_preserves_tensor_intent
and the user explicitly drops tensor via the toggle OR via extras
(_effective_tensor_parallel of the explicit extras is false).
P1 (downgrade symmetry): the len(tp_gpus) < 2 compute-buffer downgrade cleared
tensor_parallel without raising _layer_min_gpus, unlike the budget and geometry
downgrades. GPUs below tensor's replicated compute-buffer reserve can still take
layer split's lower overhead, so keep the multi-GPU request (len(gpus) >= 2) and
let _select_gpus cap unusable cards.
P2 (cache key): key the tensor-split abort cache on st_mtime_ns, so a binary
replaced in place within the same second after an abort is re-probed instead of
inheriting the stale entry.
P2 (test hygiene): load routes/inference.py via importlib in the regression
tests instead of importing the routes package, which runs routes/__init__.py and
pulls in every router (e.g. python-multipart). Added regression coverage for the
extras-off reload, the compute-buffer multi-GPU preservation, and the same-second
nanosecond cache invalidation.
* Studio: record the tensor-split abort on the Windows CRT abort exit too
The first-spawn split-axis latch only recorded when _is_signal_crash matched
(POSIX signal or 0xC0000000+ NTSTATUS). On MSVC builds GGML_ASSERT terminates
through the CRT abort() path with exit code 3, which is neither, so the cache
never filled on Windows and every later load of the same bad binary/model
repeated the tensor crash before falling back to layer.
The split-axis marker is definitive, so accept either a signal crash or the
Windows abort() exit (3) when the marker is present. Add _is_abort_exit and a
unit test, and assert the early latch honors it.
* Studio: fix UnboundLocalError on --fit-on fallback, reload backend fast path
Two follow-ups from review on the tensor-split self-heal:
UnboundLocalError: _layer_min_gpus was initialized inside the GPU-selection try.
If NVML probing or GGUF/mmproj sizing raised, the except path logged "using
--fit on" and fell through to the command builder, where the new
self._layer_preserves_tensor_intent = _layer_min_gpus > 1 then raised, turning a
safe --fit-on layer fallback into a hard load failure. Bind _layer_min_gpus
before the try so the except path always has it.
Backend fast path: _request_matches_loaded_settings forces a reload when a
preserved tensor->layer fallback gets an explicit tensor-off request, but
load_model's own _already_in_target_state still matched the tensor-off/layer
settings and short-circuited, so the placement re-selection never ran. Mirror
the guard there: reload when layer_preserves_tensor_intent and the request drops
tensor intent. The flag clears on that reload, so there's no loop.
Added regression coverage for both.
* Studio: testable tensor-split record decision; skip futile fit-off retry
Follow-ups from a deeper review of the tensor-split self-heal:
Extract the record decision into _should_record_tensor_split_abort(rc, output)
(marker AND (signal crash OR Windows abort)) and call it from the early latch.
The combined boolean was only covered by source-inspection substring checks, so
an or->and typo would silently stop recording on Windows (CRT abort exit 3 is
not a signal) with every test still green. Add a behavioral test over the
POSIX / Windows / NTSTATUS / clean-exit / SIGKILL / no-marker matrix.
Skip the --fit off retry inside _spawn_and_wait when the crash already shows the
split-axis marker: that abort is fit-independent, so the retry just warms up and
crashes a second time before the latch records it. Skipping it lets the caller
latch immediately and corrects the latch comment.
Also clarify the dedup-guard comments (toggle read from model_fields_set vs
extras via _effective_tensor_parallel without env; the backend fast path is
intentionally broader and only ever forces a reload).
* Studio: don't reload-loop tensor-off requests under env tensor
The preserved-fallback reload guard fired on the raw tensor toggle, ignoring
LLAMA_ARG_SPLIT_MODE=tensor. For an env-driven tensor user, an explicit
tensor_parallel=false request then forced a reload that re-engaged tensor via
the env and re-created the same preserved layer fallback, so every /load
reloaded -- bypassing the env-downgrade matching that exists to avoid exactly
this loop.
Gate the guard on the env-aware effective tensor state: reload only when an
explicit toggle/extras change leaves _effective_tensor_parallel (which consults
the env) off. If the env still forces tensor, fall through to the existing
env-downgrade match, which dedupes instead of looping. Added a regression test
with LLAMA_ARG_SPLIT_MODE=tensor set.
* Studio: tighten comments and test docstrings on the TP self-heal
Condense the verbose comments and test docstrings added across the review rounds
into fewer, succinct lines without changing their intent: the early-latch and
downgrade-site rationale, the cache/key and helper docstrings, the dedup-guard
comments, and the per-test docstrings. No code changes (AST-verified comments
and docstrings only); tests and lint unchanged.
* Studio: clear preserved tensor flag on diffusion; carry it across non-drop reloads
Two follow-ups on the preserved-fallback machinery:
Diffusion: the DiffusionGemma path early-returns from load_model before the
command builder that sets/clears _layer_preserves_tensor_intent, so the flag
from a prior tensor->layer fallback leaked onto a later diffusion load and
forced needless reloads of the diffusion server on tensor-off/extra Applies.
Clear it when starting diffusion.
Settings reload: the preserve hint was recomputed only from the new request, so
a reload for an unrelated setting (e.g. max_seq_length) with the tensor toggle
omitted dropped a preserved multi-GPU layer placement back to one GPU. Carry
llama_backend.layer_preserves_tensor_intent into the hint when the request is
not an explicit tensor-off/extras-off drop, so a fitting model stays multi-GPU.
Added regression tests for the diffusion clear, the carry-forward, and the
updated tensor-intent computation.
* Studio: gate the preserve carry-forward on the same model being loaded
The tensor-intent carry-forward read llama_backend.layer_preserves_tensor_intent
without checking it belonged to the model being loaded. On a direct model switch
(load B without an explicit /unload of A), the flag is still set from A's
downgrade (it isn't reset until B's load_model reaches the command builder, after
the route reads it), so a plain load of B got preserve_multi_gpu_on_layer=True
and was spread across all GPUs even though it fits on one and the user never
requested tensor for it. The backend dedup doesn't have this leak (it checks
model_identifier first); the leak was only in the route hint.
Extract the decision into _carry_preserved_tensor_intent(preserved, same_model,
explicit_drop) and gate it on the backend still holding the same model. Add a
behavioral truth-table test (catches a `not` inversion and a missing same-model
guard) and tighten the compute-buffer downgrade test to bound its source window.
* Studio: match the HF quant too when carrying preserved tensor intent
The same-model guard on the preserve carry-forward compared only model_identifier,
which is variant-agnostic for HF repos. A later load of the same repo with a
different gguf_variant (which already bypassed dedupe on the variant mismatch)
was treated as the same model, so a request that omits tensor settings inherited
the prior variant's preserved intent and forced multi-GPU layer placement for a
quant that never requested tensor. Also require the loaded hf_variant to match for
HF repos (local direct-file loads already differ by model_identifier path). Added
a regression test for the variant guard.
* Studio: match the loaded GGUF by path too when carrying preserved tensor intent
A local directory holding multiple GGUF variants keeps one variant-agnostic
model_identifier (the directory) while config.gguf_file selects the file, so the
same-model guard let variant B inherit variant A's preserved tensor->layer
fallback and forced B onto multi-GPU. Mirror _already_in_target_state's identity
logic: match by resolved path when both sides have a local file, else by HF
variant. #6659
* Studio: let implicit same-settings reloads dedupe after a preserved fallback
The backend _already_in_target_state mirror forced a reload on ANY effective
tensor-off request once a tensor->layer fallback was preserved. In the HF
auto-pick / local-directory flows the route-level dedup is skipped, so an
identical /load with tensor omitted reached this guard and reloaded every time
even without an explicit drop. Thread the route's preserve_multi_gpu_on_layer
decision in so only an explicit drop reloads; implicit carry-forward dedupes. #6659
* Studio: only an explicit tensor/split-mode change drops preserved intent
The explicit-drop test treated request.llama_extra_args is not None as a drop,
so a same-model reload that merely added an unrelated pass-through arg (e.g.
--top-k 20) without touching the tensor field or --split-mode disabled the
carry-forward and collapsed a fitting model back to one GPU. A drop now requires
an explicit tensor_parallel field change or a non-tensor --split-mode override,
via a shared _is_explicit_tensor_drop helper used by both the already-loaded
dedup and the load carry-forward so the two readers agree. #6659
* Studio: treat an explicit clear of extras as a tensor drop
When tensor intent was extras-driven (--split-mode tensor) and fell back to a
preserved layer split, a later request that explicitly clears extras
(llama_extra_args=[]) but omits tensor_parallel left the empty list with no
split-mode override, so the carry-forward kept the model pinned multi-GPU instead
of returning to normal layer selection. _is_explicit_tensor_drop now also counts
an explicit empty-list clear as a drop, while an unrelated extra (--top-k) or
inherit (None) still carries the preserved intent. #6659
* Studio: don't treat the UI's tensor_parallel echo as a tensor drop
The Studio frontend always sends tensor_parallel and copies the /load response's
resolved value back into its state, so after a tensor->layer fallback every
ctx/settings reload carries tensor_parallel=false even though the user never
changed it. Keying the drop on the field (or on an empty extras clear) collapsed
the preserved multi-GPU placement on the next reload. A fallback also always
stores --split-mode layer, never a tensor split mode, so a clear never wipes
tensor intent. _is_explicit_tensor_drop now drops only on an explicit non-tensor
--split-mode override; the bare field echo, an empty clear, an unrelated extra,
and inherit all keep the preserved placement, and --split-mode tensor /
tensor_parallel=true re-engage tensor. #6659
* Studio: match the resolved config.identifier when carrying tensor intent
The same-model guard for the carry-forward compared the raw request id, but
ModelConfig.from_identifier normalizes it (adds the unsloth/ prefix for a
shorthand, fixes repo-id case) before load_model stores config.identifier. So a
ctx/settings reload using the shorthand id missed the match, dropped
_carry_preserved_tensor_intent, and could collapse a preserved multi-GPU layer
placement to one GPU. Compare against config.identifier (what the backend stores),
keeping it symmetric with _already_in_target_state. #6659
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: harden the data-recipe and inference consumer loops against pump death
Follow-up to #6643. The same single-unsupervised-consumer pattern the training
pump had lives in two sibling loops, with the same failure mode: one bad event
kills the only thread that updates the in-memory state every UI surface reads,
while the worker subprocess keeps running.
- data_recipe JobManager._pump_loop: a malformed worker log line that makes
parse_log_message raise no longer kills the pump. Guard _handle_event, the
queue read, and the worker-exit finalize, and broaden _drain_queue so a drain
error still finalizes the job instead of leaving it wedged "active" (which also
leaked the workflow-scoped API key until its 24h expiry).
- inference InferenceOrchestrator._dispatcher_loop: guard the routing body so a
malformed response or a mailbox put error can't kill the dispatcher and hang
every in-flight generation (callers key liveness on the subprocess, not on
this thread).
Adds regression tests for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: extend consumer-loop hardening to RAG, hub, auth, and stream-reader paths
Continuation of the data-recipe and inference pump hardening: the same
"background producer updates in-memory state that a single unsupervised
consumer surfaces to the UI" pattern shows up in several more Studio paths,
each able to silently freeze a UI surface while the worker keeps running.
RAG ingestion SSE (core/rag/ingestion.py):
- job_events polled the queue with a blocking get and never noticed client
disconnect or a dead worker, so a closed tab or a producer that died
without emitting a terminal event left the stream hanging. It now polls
with a timeout, emits heartbeats, ends on terminal job status, caps idle
time, and always pops the job registry in finally.
- Added _reap_finished_jobs() and call it from start_ingestion so finished
job state does not accumulate.
Startup reconcile (storage/rag_db.py, main.py):
- reconcile_orphaned_ingestion_jobs() marks ingestion jobs (and their
documents) that were left non-terminal by a previous crash as failed, so
the UI does not show jobs stuck "running" forever after a restart. Wired
in at startup next to cleanup_orphaned_runs().
Hub download watcher (hub/services/download_lifecycle.py):
- _watch() could leave a job pinned "running" if finalize raised. Body is
now guarded: on failure it logs and sets the job to error, and always
invalidates the hf cache scan in finally.
External provider stream (core/inference/external_provider.py):
- read timeout was None (no stall ceiling); set to 300s so a wedged
upstream surfaces as an error instead of an indefinitely hung stream.
Auth store (auth/storage.py):
- Enable WAL + busy_timeout on the auth DB so token validation (read on
every request) and login writes stop serialising on the rollback journal.
Matches studio_db / rag_db / providers_db.
Login rate limiter (routes/auth.py):
- _LOGIN_IP_BUCKETS could grow unbounded under spoofed-IP traffic; cap it
and prune stale buckets, mirroring the per-account bucket handling.
Training progress SSE (routes/training.py):
- Break promptly on client disconnect instead of waiting for the next
yield to fail on a closed socket, matching the export / data-recipe SSE
routes.
llama-server stdout drain (core/inference/llama_cpp.py):
- Broaden the drain guard so an unexpected decode/read error logs at debug
and stops the drainer cleanly instead of escaping the thread.
Frontend stream readers (chat-api.ts, rag-api.ts):
- Wrap the SSE read loops in try/finally + reader.cancel() so early return
([DONE]), thrown errors, and consumer aborts release the reader lock
instead of holding it until GC.
Tests:
- test_training_progress_stream_nan: fake request now implements the async
is_disconnected() the route polls, matching the other SSE route fakes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address Codex review feedback on the consumer-loop hardening
Four follow-ups from the automated review, all on code this PR introduced:
- Data-recipe pump (manager.py): a queue read that keeps raising an error
outside the read's narrow catch set (e.g. a broken queue pipe after the
child died) hit the `continue` guard and skipped the dead-worker finalize
below, spinning forever and leaving the job wedged "active" with its
workflow key unretired. On a read failure, fall through to finalize when
the worker is no longer alive. Added a regression test.
- RAG ingestion SSE (ingestion.py): the 5-minute idle cap could end the
stream while the job was still pending/running (a large document spends
minutes in embedding/storing with no per-batch progress event). The route
then sends [DONE], and the client treats a no-terminal-frame end as
completion, marking the document indexed mid-ingestion. Drop the idle cap:
while the worker is alive and non-terminal we keep heartbeating; the stream
ends only on terminal DB status, the None sentinel, or client disconnect.
- Login rate limiter (auth.py): the per-IP path pruned but then added the
new IP unconditionally, so a spoofed-source-IP spray kept _LOGIN_IP_BUCKETS
unbounded and made every new IP pay a full-dict prune scan. Gate the add on
the cap, mirroring the account path.
- Hub download watcher (download_lifecycle.py): if finalize raised before it
reaped (proc.wait) and dropped the worker (e.g. an I/O error draining
stderr), the crash path published a terminal state while the live Popen
stayed registered and kept writing the cache, and the terminal set_job let
claim() admit a retry on the same repo. Terminate + drop the worker before
setting the terminal state.
* Studio: keep login throttling working when the per-IP bucket dict saturates
Review follow-up. The previous cap fix skipped creating a bucket for a new IP
once _LOGIN_IP_BUCKETS was full, returning ip_fails=0. Under a sustained spray
that also fills the account dict, every failure from such an IP then looked
first-seen and _login_blocked had no bucket to enforce, so the cap effectively
disabled throttling once saturated.
Bound the dict with a FIFO eviction instead: if the IP is new and the dict is
full, reclaim expired buckets (rate-limited so a burst of distinct IPs can't
make each failure an O(n) sweep) and, if still full, evict the oldest-inserted
IP. The new IP always gets a real bucket, so a saturating (e.g. spoofed
X-Forwarded-For) spray stays throttled while memory stays bounded. Added a
regression test that saturates the dict and asserts a later IP is still blocked.
* Studio: address Codex review (RAG queue lifecycle, stream error, orphan chunks)
Three follow-ups on the Phase 6 changes:
- RAG ingestion SSE (ingestion.py): job_events removed the per-job queue in its
finally on ANY exit, including an early client disconnect while the worker is
still running. That dropped the worker's later events (the queue is the only
one _emit writes to) and made a reconnect find no queue and receive only
[DONE], which the client treats as completion. Only drop the queue on a
terminal exit (None sentinel / terminal DB status); leftover terminal queues
are still swept by _reap_finished_jobs. Added queue-lifecycle tests.
- External provider stream (routes/inference.py): once the 300s read timeout can
fire, the stream's except path failed the monitor but ended without an error
frame or [DONE], so the chat client saw a bare EOF and saved the timed-out
answer as a successful partial with no error. Emit an SSE error frame (and
[DONE]) on stream failure so the client surfaces it.
- RAG startup reconcile (storage/rag_db.py): marking a half-ingested document
failed left its chunks/fts/vec rows intact, and retrieval filters by scope not
status, so a failed document could still be retrieved and cited. Purge the
document's chunks when reconciling it to failed (the doc row stays for
re-ingest).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: release the remaining SSE stream readers (training, data-recipe, export)
reviewer.py follow-up. The chat and RAG SSE readers were wrapped in
try/finally + reader.cancel(), but the other three readers built on the same
response.body.getReader() pattern were left without it: streamTrainingProgress,
streamRecipeJobEvents, and streamExportLogs leak the ReadableStreamDefaultReader
lock (held until GC) when the consumer aborts, returns early, or a parse/callback
throws. Wrap each in try/finally + reader.cancel() (export already had a
try/catch, so it only needed the finally). All five frontend SSE readers now
release the reader symmetrically.
* Tighten resilience comments and docstrings
Condense the verbose explanatory comments and internal-helper docstrings added
in this branch to shorter, clearer forms. Comment/whitespace only; verified no
code changed via AST diff. No behaviour change.
* Studio: keep chunks for completed docs during ingestion reconcile
Startup reconciliation flips orphaned (non-terminal) ingestion jobs to failed and
purges the document's chunks so a failed source can't be retrieved. But it dropped
the chunks unconditionally, so a document the worker had already committed as
'completed' before the crash (only its job row left non-terminal) lost every chunk
while still reporting 'completed'. That leaves an empty source that retrieval can't
return and dedup (status != 'failed') blocks from re-ingest.
Only purge chunks when the document UPDATE actually transitions it to failed; an
already-completed document keeps its chunks. Adds reconcile regression tests for
both the completed-doc and genuine in-flight-orphan cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: drop a finished RAG job's queue when the client disconnects
job_events kept the per-job queue until it consumed the None sentinel, so a UI
that stops on the terminal event (its reader.cancel aborts the stream before
[DONE]) left the queue registered until the next _reap_finished_jobs sweep; a
batch of uploads followed by idling retained them all.
_run writes the terminal DB status before emitting the terminal event, so on
generator exit, drop the queue when the job's DB row is already terminal (worker
done, nothing to resume) and keep it only while the worker is still running. Adds
a disconnect-after-terminal-event regression test.
* Remove stray async task output files committed by mistake
* Studio: harden login IP throttle and end progress stream on disconnect
Two Codex review items:
Login per-IP throttle: when the per-IP bucket dict saturated, FIFO eviction could
drop a still-hot (blocked) bucket, so an IP could flood the dict with distinct
(or spoofed) source IPs to push out its own bucket and retry as first-seen. Stop
evicting hot buckets; a new IP that can't fit now shares a bounded overflow
counter that still trips the per-IP threshold, so a saturating spray stays
throttled and no live counter is reset.
Progress SSE: on client disconnect the polling loop only broke and fell through
to the unconditional final 'complete' frame, so a buffered or proxying consumer
could read a still-active run as completed. Return from the generator instead.
Adds regression tests for both (spray cannot reset a hot bucket; disconnect while
active emits no complete frame).
* Studio: shard the login overflow counter and stop cancelling chat stream after [DONE]
Two Codex review items:
Login throttle overflow: the single shared overflow counter meant that once a
saturating spray pushed it past the per-IP threshold, _login_blocked returned 429
for every new unbucketed source IP, before credentials were checked -- a global
login denial. Shard the overflow into a fixed array of counters keyed by hash(ip),
so a hot shard only throttles the IPs that map to it while a single source's
repeated failures still concentrate in one shard and stay throttled. Memory stays
bounded and no live bucket is evicted. Adds a regression test that a hot overflow
shard does not block an unrelated IP.
Chat stream: the reader.cancel() in the SSE finally fired even after a natural
[DONE]/EOF. The backend finalizes its api-monitor entry right after yielding the
sentinel (the local pass-through finishes after the last yield), so a client
cancel there can be observed as a disconnect and mark a completed request as
cancelled. Track natural completion and only cancel on an early/abnormal exit.
(No frontend unit test: the Studio frontend has no test harness.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: give prep-timeout test fakes an is_disconnected method
The progress stream now ends on client disconnect (await request.is_disconnected()
before falling through to the terminal frame). After merging that into the
prep-timeout tests added later on main, their _FakeRequest/_ReconnectRequest must
provide is_disconnected or the generator raises AttributeError under CI.
* Studio: keep the login overflow throttle when bucket capacity frees up
_login_blocked only consulted the per-IP overflow shard while the bucket dict was
still at capacity. If a slot freed before the 60s window expired (e.g. another
IP's successful login calls _clear_login_bucket), a source counted in a hot shard
stopped being blocked and its next failure got a fresh per-IP bucket, resetting
the throttle the overflow path exists to preserve. Always max in the IP's shard
(shards are empty outside saturation, so it is a no-op in the common case). Adds a
regression test that a hot source stays throttled after a bucket frees.
* Studio: clear a login IP's overflow throttle on successful login
_clear_login_bucket reset the per-IP and per-account buckets on a successful
login but not the overflow shard, so after the dict saturated and an IP was
counted in overflow, a later successful login left those entries behind and the
next failed attempt could immediately return 429.
Store overflow entries as (timestamp, ip) so a source is throttled by its own
count within the shard (also removing cross-IP collateral within a shard), and
drop just that IP's entries in _clear_login_bucket. Adds a regression test that a
successful login clears the overflow throttle.
* Studio: bound the login overflow shard memory under high-cardinality spray
The per-IP overflow tracked failures in a time-pruned deque of (timestamp, ip)
tuples, so a spoofed-X-Forwarded-For spray of distinct one-off IPs grew memory and
the per-check scan with request cardinality for the whole window -- undermining
the bucket cap that exists to bound memory. Replace each shard with a fixed-
capacity dict (ip -> [count, window_start]): O(1) lookups, and when a shard is
full a one-off IP evicts the lowest-count entry (Space-Saving) so memory is hard-
bounded while a persistent attacker keeps a high count and is never evicted. Adds
a regression test that shards stay within the per-shard cap under a 5000-IP spray.
* Studio: purge chunks for already-failed docs during ingestion reconcile
The reconcile chunk-purge was gated on the documents UPDATE actually flipping a
non-terminal doc to failed. A doc the worker had already marked 'failed' before
the crash (job row left non-terminal) was not re-flipped, so its committed chunks
were kept and stayed retrievable/citable, since retrieval filters by scope not
status. Purge chunks whenever the document is not 'completed' (failed, in-flight,
or gone), preserving the completed-doc carve-out. Adds a regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: don't inherit an evicted IP's count onto a new overflow source
When a full overflow shard evicted the lowest-count entry, the new source
inherited that count (Space-Saving base + 1). If a shard was saturated with hot
entries, an unrelated new IP could land at/over the threshold and be 429'd after a
single attempt -- cross-IP collateral despite the per-source-isolation intent.
New entries now start clean at count 1; the only cost is that a heavy hitter that
is the lowest-count entry in a fully saturated shard can briefly reset, which is
preferable to blocking a bystander. Adds a regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry overflow failures into a new IP bucket on transition
_login_blocked took max(per-IP bucket, overflow shard) rather than combining them,
so a source could log (threshold-1) failures in overflow during saturation and,
once a bucket slot freed, another (threshold-1) in a fresh bucket within the same
window -- roughly doubling the per-IP limit. When a saturated-era IP first gets a
real bucket, migrate its windowed overflow count into that bucket (and drop the
overflow entry) so the combined failures throttle at the intended limit. Adds a
regression test.
* Studio: reconcile a completed doc's orphaned job to completed, not failed
When a crash left an ingestion job non-terminal after its document was already
committed as completed, reconcile marked the job failed. After restart the upload
UI has no in-memory SSE queue and falls back to getJob(), which treats a failed
job as an indexing failure and removes/toasts a document that is actually
searchable. Mark the job completed (keeping its chunks) when its document is
completed. Extends the completed-doc reconcile test to assert the job status.
* Studio: clamp the overflow failure count migrated into a login bucket
A saturated source could accrue an unbounded overflow count, then materialize
one deque entry per recorded failure when a bucket slot freed, allocating an
arbitrarily large deque under the login lock. Only at-or-above the per-IP
threshold matters for blocking, so cap the count there at the record and take
sites; the migration is now bounded without weakening the limit.
* Studio: keep the RAG job stream alive on a transient status read
The heartbeat poll read the job row unguarded; a momentarily-locked DB would
raise out of job_events, which the SSE route turns into a terminal error frame,
and the UI drops a document whose worker is still running. Treat a failed status
read as non-terminal: heartbeat and retry, and keep the queue so a reconnect can
resume.
* Studio: set busy_timeout before journal_mode on the auth DB
Switching journal_mode needs a lock, so if a refresh-token write already holds
one, journal_mode=WAL raises SQLITE_BUSY and the shared try leaves the
connection on SQLite's default zero lock wait. Set busy_timeout first so the
switch waits instead of failing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: improve Unsloth Studio chat title generation quality
* fix: address self-review (guard echoed role labels before punctuation stripping)
* Address title generation review feedback
Consolidate the echo guard into a single leading-label check (now also
covering base and lora) and drop the post-punctuation duplicate that
could never match a colon once punctuation is stripped. Swap the
slice-based first-assistant lookup for an indexed find to avoid copying
the messages array, and note the brace counter's assumptions in the
test helper.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Daniel Han <unslothai@gmail.com>
* Studio: set the admin password before exposing it on the network
On first run Studio seeds the default `unsloth` admin with a random
bootstrap password and embeds it into index.html (window.__UNSLOTH_BOOTSTRAP__)
so the local user can change it without typing it. A request with no Origin
header counts as same-origin, which is what a normal top-level GET sends, so
the page hands out the password to whoever loads it. That is harmless on the
default 127.0.0.1 bind, but `--secure` (public Cloudflare tunnel) and
`--host 0.0.0.0` (raw port reachable on the network) would serve the plaintext
admin password to remote visitors during the bootstrap window.
Fix this at the source: when launching a network-exposed web UI, prompt the
operator in the terminal for a real admin password (with confirmation) before
the socket binds or the tunnel opens, and persist it via update_password (which
clears must_change_password and deletes the .bootstrap_password file). After
that there is no bootstrap secret to leak. Non-interactive launches can supply
it via UNSLOTH_STUDIO_ADMIN_PASSWORD. The masked reader echoes '*' per
character and works on Linux, macOS, and Windows (PowerShell/cmd). Loopback
binds, --api-only (no web UI), and Colab are unaffected.
As defense in depth, the index handler now embeds the bootstrap object only for
a direct local navigation: same-origin AND a loopback TCP peer with no
proxy/tunnel forwarding headers (cf-ray, cf-connecting-ip, x-forwarded-for,
x-forwarded-host, x-real-ip, forwarded). Colab stays exempt. This keeps the
password off the wire even when the prompt is skipped (no TTY and no env var).
Adds unit coverage for the prompt/confirm/decision logic, an integration test
that provisioning clears the bootstrap state, and regression tests for the
local-direct gate (loopback/IPv6/mapped/localhost peers, LAN/public peers,
missing client, each forwarding header, spoofed XFF, and the Colab exemption).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fail fast on an explicitly empty admin-password env var
resolve_admin_password_source treated UNSLOTH_STUDIO_ADMIN_PASSWORD="" like
the var was unset and fell back to the bootstrap backstop. Treat any set value
(including empty) as the env source so it reaches the minimum-length guard and
refuses to expose the server instead of silently keeping the seeded password.
* Studio: apply repo kwarg-spacing format to the secure-admin-password files
* Studio: drop the pre-exposure password prompt; keep the local-direct gate
Per review, the blocking prompt added friction for --secure / 0.0.0.0 first-run
launches without extra security: the local-direct injection gate in main.py
already keeps the bootstrap password off the network for any remote request.
Remove the prompt module and its tests; the gate plus the existing
must_change_password first-login flow are the fix.
* Studio: shut down an exposed first-run instance if the admin password is never changed
The local-direct gate keeps the seeded bootstrap password off the network, but
it stays a valid credential until first login changes it. For an exposed web UI
(--secure / 0.0.0.0, not --api-only, not Colab), arm a daemon timer: if the
password is still the seeded one after the deadline (UNSLOTH_STUDIO_BOOTSTRAP_TIMEOUT,
default 3600s, 0 disables), print a message and shut Studio down via the existing
graceful-shutdown path; if it was changed, leave Studio running.
* Studio: revert the local-direct injection gate; keep the 1-hour auto-shutdown
Per maintainer decision, keep the first-run auto-fill behavior unchanged (the
bootstrap password still seeds the login form for convenience) and rely on the
exposed-instance auto-shutdown to bound the window: an exposed web UI that never
changes the seeded admin password is torn down after UNSLOTH_STUDIO_BOOTSTRAP_TIMEOUT
(default 1h). Restores studio/backend/main.py and its origin test to upstream.
* Studio: render the bootstrap-timeout shutdown message with a human duration
The message hardcoded 'minute(s)' via timeout//60, so a sub-minute timeout
(e.g. a 30s test value) printed 'within 1 minute(s)'. Add _format_duration so
it reads '30 seconds' / '1 minute 30 seconds' / '60 minutes' as appropriate.
The default 3600s still renders '60 minutes'.
* Studio: drop stale local-direct gate reference from bootstrap_timeout docstring
The gate was reverted (timer-only), so the module docstring should not describe
a main.py gate that no longer exists.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio Colab: add opt-in shareable Cloudflare tunnel link
colab.start(cloudflare=True) opts in to a free Cloudflare quick tunnel and
shows a trycloudflare.com link above the proxy iframe, reachable from any
device. Default OFF: bare start() keeps the in-tab Colab-proxy behavior.
run_server suppresses the tunnel on Colab by design, so colab.py starts it
directly via cloudflare_tunnel.start_studio_tunnel(); failures degrade to
the Colab proxy only.
* Studio Colab notebook: surface opt-in cloudflare=True in start cell
* Studio Colab: reskin shareable Cloudflare link to match the proxy banner
Retrofit _shareable_link_html to reuse the original Colab proxy banner skin
from show_link (white card, black border, Unsloth gem, black Open button)
instead of the plain dark box, so the shareable Cloudflare link gets the same
prominent 'Ready!' treatment.
* Studio Colab: address review feedback on Cloudflare tunnel
- try/finally around tunnel start + embed + keepalive so a KeyboardInterrupt
while the tunnel is starting or the iframe is rendering tears it down instead
of orphaning the cloudflared process (Gemini review).
- Publish the directly-started tunnel URL onto app.state.cloudflare_url via a new
_publish_cloudflare_url helper so /api/health advertises it; otherwise the
frontend's API examples fall back to the unreachable raw server_url (Codex P2).
_stop_cloudflare_tunnel now also clears it so health stops showing a dead tunnel.
- Notebook: make cloudflare=True a replacement for start(), not an addition, since
start() blocks and the second call would never run if both are left in (Codex P2).
* Studio Colab: gate Cloudflare tunnel on auth + honor opt-out in run_server
- Refuse to open the Cloudflare tunnel while the admin still holds its seeded
bootstrap password. While requires_password_change is true the server injects
that password into same-origin index GETs, and a public tunnel request counts
as same-origin, so sharing the link would leak admin access. New
_bootstrap_password_pending() gate (fails safe) blocks the tunnel and tells the
user to change the password first, then re-run start(cloudflare=True) (P1).
- Pass cloudflare=False into run_server so the opt-out holds even when Colab
detection fails; this helper is now the sole owner of the tunnel decision,
preventing run_server from opening a tunnel on the 0.0.0.0 bind by default (P2).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio Colab: drop duplicate tunnel link log and simplify start cell guidance
* Studio Colab: validate /api/health identity before reusing or tunneling a port
* Studio Colab: condense verbose docstrings and comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Fix offline checkpoint load/export failing with "tokenizer is weirdly not loaded"
Loading a fine-tuned checkpoint with no internet (e.g. a Studio export) crashed
with "Unsloth: The tokenizer is weirdly not loaded? Please check if there is one."
For a LoRA adapter the loader reassigns model_name to the base model repo id and
only keeps the local checkpoint dir as tokenizer_name when it contains
tokenizer_config.json, tokenizer.json AND special_tokens_map.json. Modern
tokenizers (e.g. Gemma) store special tokens inside tokenizer_config.json and
omit special_tokens_map.json, so tokenizer_name fell back to the base repo id.
The tokenizer/processor loads in vision.py then hit the Hub with no
local_files_only, so with no network they failed (AutoProcessor) or hung for
minutes (AutoTokenizer) even though every file was already cached.
loader.py: keep the local checkpoint dir as tokenizer_name when it has a
tokenizer config plus the actual tokenizer files (tokenizer.json / tokenizer.model
/ vocab files); special_tokens_map.json is no longer required.
vision.py: compute an effective local_files_only (explicit kwarg plus the
HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE env vars, mirroring loader.py and
diffusion.py) and thread it through every AutoConfig, AutoProcessor,
AutoTokenizer and the manual VLM processor fallback, including the
hf_hub_download in that fallback (which now prefers a local file). When a load
fails and no offline env var is set, retry against the local cache. The retry
forces HF offline mode because local_files_only alone does not stop
AutoProcessor / AutoTokenizer from issuing a /api/models request during class
resolution. The final error now explains the offline/cache cause instead of the
misleading "weirdly not loaded" message.
studio export: probe Hub reachability once per checkpoint load and pass
local_files_only when offline so exports use the local checkpoint dir / cache
instead of hanging or crashing with no internet.
Online behavior is unchanged: the new flags default to off and the retry only
runs after a network related failure.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: safer offline forcing, cached fallback config, proxy-aware probe
Follow-up to the offline checkpoint load fix, addressing review feedback:
- vision.py: only flip the process-wide HF offline flag when offline is actually
requested (local_files_only / env) or after a real network failure, never
pre-emptively while we might be online. The flip is now guarded by a lock +
depth counter so nested or concurrent windows restore the flag correctly
(no stale value).
- vision.py: guard the get_auto_processor fallback so a network error there
returns None and the local-cache retry still runs instead of escaping.
- vision.py: in the manual VLM processor fallback, read tokenizer_config.json
via hf_hub_download(..., local_files_only=...) so a cached repo-id config is
still resolved offline and the model-specific image/video tokens are restored.
- studio export: make the reachability probe proxy aware (probe the configured
HTTP(S) proxy egress, honour NO_PROXY, use the endpoint port) so a proxy-only
setup is not wrongly marked offline; allow UNSLOTH_OFFLINE_PROBE=0 to disable.
- studio export: run the audio/vision type-detection probes inside the
forced-offline window when offline, so their config/tokenizer reads hit the
local cache instead of waiting out connection timeouts.
Online behavior remains unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate offline retry, safer tokenizer_name pop, skip audio net probe offline
- vision.py: only force the process-wide HF offline flag on the tokenizer
retry when offline was requested or the captured primary error is actually
network related, so a permanent tokenizer error no longer toggles global
offline mode for other concurrent loads.
- loader.py: always pop tokenizer_name out of kwargs and let a caller-supplied
value win, avoiding a "multiple values for keyword argument 'tokenizer_name'"
TypeError when it is also passed explicitly downstream.
- model_config.py / export.py: add local_files_only to detect_audio_type so the
raw requests.get tokenizer_config fetch is skipped offline (it ignores the HF
offline flag), and pass it from the export probe.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: classify LocalEntryNotFoundError as offline-related
huggingface_hub's LocalEntryNotFoundError subclasses FileNotFoundError, so the
"not isinstance(cur, FileNotFoundError)" guard in _is_offline_related_error was
swallowing it and it could never be recognised as offline, despite being listed
in the network error types. It means "not in cache and the Hub is unreachable",
which is genuinely offline. Capture the class into an isinstance-checkable tuple
(empty, hence a no-op, if the import is unavailable) and exclude it from the
FileNotFoundError guard, so a real offline failure now triggers the local-cache
retry while a plain missing-file error still propagates.
* Address review: require merges.txt for BPE, status-gate HTTP errors, isolate local-only audio cache
- loader.py: a local dir with vocab.json but no merges.txt (and no tokenizer.json)
is not a loadable BPE tokenizer, so do not treat it as self-sufficient; require
merges.txt alongside vocab.json in both gate blocks, otherwise fall back to the
base model tokenizer as before.
- vision.py: _is_offline_related_error no longer buckets every HfHubHTTPError /
requests HTTPError as offline. HTTP errors are judged by status code: only a
transient 5xx triggers the forced local-cache retry, while 401/403 (auth/gated)
and 404 (missing) propagate as the real error instead of being masked. Hard
signals (connection/timeout/OfflineModeIsEnabled/LocalEntryNotFoundError) still
classify as offline.
- model_config.py: include local_files_only in the audio-detection cache key so a
local-only (offline) negative result cannot be reused by a later online probe,
which would otherwise route an audio model through the text loader until restart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address re-review: fix studio test stubs, force offline env in probe window, drop redundant retry
- studio/backend/tests/test_vision_cache.py: the three _detect_audio_from_tokenizer
stubs were called with the new local_files_only kwarg and raised TypeError, failing
Backend CI. Add local_files_only to the stub signatures and add a test that a
local-only negative does not poison a later online audio probe.
- export.py: the type-detection probe window now also sets HF_HUB_OFFLINE /
TRANSFORMERS_OFFLINE env vars (saved/restored), not just the in-process flag.
transformers_version._load_config_json / _check_tokenizer_config_needs_v5 gate
their urllib fetches on the env vars, and is_vision_model may spawn a subprocess
that inherits os.environ but not the in-process flag; without the env vars a
probe-detected offline export could still block on a network timeout.
- vision.py: only retry the processor load when the first attempt was online and
failed with a network error. When local_files_only was already requested the first
attempt was forced offline, so the previous retry just repeated identical failing
work before the last-resort path.
- model_config.py: correct the _audio_detection_cache type annotation to the 3-tuple
key (name, token_fingerprint, local_files_only).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: thread-safe probe-offline env window, clear error for local dir without config
- export.py: guard the HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE mutation in
_force_offline_probe_window with a lock + depth counter (mirrors _force_hf_offline),
so concurrent / nested export probes only flip on first entry and restore on last
exit. This prevents overlapping export requests from permanently poisoning those
env vars or restoring a stale value.
- vision.py: in the VLM processor fallback, when tokenizer_name is a local directory,
read its tokenizer_config.json directly and raise a clear FileNotFoundError if it is
absent, instead of handing the local path to hf_hub_download (which would treat it as
a repo id and raise a confusing HFValidationError / RepositoryNotFoundError).
hf_hub_download is now only used for actual repo ids.
* Address review: classify raw socket.gaierror DNS failures as offline
Add the platform-specific getaddrinfo / DNS-resolution wording to the offline
detection list in _is_offline_related_error so a bare socket.gaierror (an OSError
subclass) is recovered from the local cache: "Name or service not known" and
"Temporary failure in name resolution" (Linux) and "nodename nor servname
provided" (macOS). Genuine non-network OSErrors (disk full, permission denied)
and plain FileNotFoundError still propagate.
* Address review: retry degraded VLM offline, force offline for text export + patch-tokenizer fallback
- vision.py: a degraded VLM processor (text-only, no image_processor) whose manual
fallback fails offline used to be kept, so image inputs broke even with cached
files. _construct_vlm_processor_fallback now returns its failure error;
_acquire_processor surfaces it, and the caller retries forced-offline when the
result is None OR a degraded VLM and the failure was network related, keeping the
original result if the retry is not strictly better (never regress). The retry is
still gated on an online first attempt + offline-related error so a permanent
error never flips the global offline flag.
- vision.py: wrap the patch_tokenizer except-branch AutoTokenizer.from_pretrained in
the same forced-offline-on-network-error pattern as the primary / last-resort
loads, so an offline export where patch_tokenizer raises does not hang or fail.
- export.py: force HF offline around the two FastLanguageModel loads (text and SNAC)
when the probe detected offline. Their text tokenizer path (load_correct_tokenizer
-> AutoTokenizer) does not forward local_files_only, so without this a text export
could still contact the Hub. Added a small _offline_window_if helper reused by the
probe and load windows.
* Consolidate offline loading into one entry-point decision
Decide offline once per entry point instead of at every HF call site. The
prior approach threaded local_files_only into ~15 scattered config / tokenizer
/ processor / weight loads, each wrapped in its own try-online, classify-error,
retry-forced-offline dance, which is what kept surfacing "another call site you
missed", "another error shape misclassified", and global-flag thread-safety in
review.
FastLanguageModel / FastModel / FastBaseModel.from_pretrained now share an
@_offline_aware_load decorator: when offline (explicit local_files_only kwarg or
HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE env) it sets local_files_only and runs the
whole load inside one _force_hf_offline() window so every nested HF call inherits
it; when online it runs normally and, only if the load fails with a genuinely
network-related error, retries once forced-offline. The online path is unchanged
(no probe added) and 401 / 403 / 404 / permanent errors still propagate.
Centralise the offline helpers in loader_utils.py as the single source of truth
(shared by loader.py, re-exported from vision.py, and reused by the Studio
exporter):
- _force_hf_offline now sets the HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE env vars
AND the in-process huggingface_hub / transformers flags, refcounted under one
lock so nested / concurrent windows restore correctly. Setting the env vars
covers env-gated urllib probes and spawned subprocesses too.
- _get_effective_local_files_only, _is_offline_related_error (unchanged
classifier, retains the 5xx-vs-4xx, LocalEntryNotFound and gaierror handling),
_offline_aware_load, and _resolve_checkpoint_tokenizer_name.
loader.py: wrap both entry points; drop the two duplicated env-var fallback
blocks and the two byte-identical local-tokenizer-gate blocks (now
_resolve_checkpoint_tokenizer_name).
vision.py: drop the per-site force_offline params and the three retry gates
(processor, patch_tokenizer fallback, last-resort). They now just surface the
underlying error so the single entry-point safety net retries forced-offline. A
network fallback error now takes precedence over a permanent primary error so the
offline retry still fires when the manual VLM fallback needs cached repo files.
studio/backend export.py: reuse the unified core _force_hf_offline (env + flags)
and drop the duplicate probe-window primitive; the snac / text branches no longer
need their own window. model_config.py: also gate the raw requests.get audio
fallback on the HF offline env vars so it is covered even without the kwarg.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address 10-reviewer P1 findings: vision cache split, PEFT offline, retry OOM
Split the Studio vision-detection cache by local_files_only, mirroring the audio
cache fix. is_vision_model / _is_vision_model_uncached / _raw_config_has_vision_config
/ load_model_config now thread local_files_only, the cache key includes it, and the
exporter passes it. Offline detection also skips the transformers-5 network
subprocess and stays on the local cache, so an offline negative can no longer be
keyed under the online entry and poison a later online probe. Adds a regression
test mirroring the audio poison test.
Forward local_files_only to both PeftModel.from_pretrained adapter-attach sites in
loader.py so a cached remote LoRA adapter resolves from the local cache under
explicit local-only / offline loads (defence-in-depth alongside the forced-offline
window).
_offline_aware_load: run the forced-offline retry OUTSIDE the except block and
collect + empty the device cache first. An except-scoped exception keeps its
__traceback__, which pins the failed attempt's frame locals (a partially loaded
model) until the block exits; loading the model again while that copy is still
alive could OOM a large VLM. Letting the except block close drops the traceback so
the partial load is freed before the retry reallocates.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address Codex review: env-offline cache key + rebuild HF sessions in offline window
Key the Studio audio and vision detection caches on the EFFECTIVE offline state
(local_files_only OR the HF offline env vars), not just the kwarg. detect_audio_type
and is_vision_model both skip the remote fetch / network subprocess when
HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE is set even with the default
local_files_only=False, so the result reflects offline; storing it under the online
(False) key let an env-offline negative poison a later online lookup once the env var
was cleared. Both now compute effective_offline once and use it for the cache key and
the downstream call. Adds a regression test for the env-offline dimension.
_force_hf_offline now rebuilds huggingface_hub's cached sessions on enter and exit
(best-effort _reset_hf_sessions). On hub 0.x the offline adapter is baked into the
per-thread requests.Session at creation, so flipping the constant alone leaves an
already-cached online session able to hit the network inside the window (and an
offline one stuck offline after restore); resetting forces the next get_session() to
match the current flag. On hub 1.x offline is checked dynamically per request, so
reset_sessions does not exist and the helper is a safe no-op.
The third review point (release the failed load before retrying) was already fixed in
af0f58a: the forced-offline retry now runs outside the except block and frees the
device cache first, so the failed attempt's traceback-pinned partial model is
released before the retry reallocates.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align Studio _env_offline parsing with the canonical offline helper
model_config._env_offline gates the raw requests.get tokenizer-config fallback in
detect_audio_type and the audio/vision detection cache keys, but it only accepted
unstripped "1"/"true"/"yes". unsloth's offline helpers (loader_utils._env_says_offline
and the from_pretrained env fallback) accept the canonical set {1,true,yes,on} after
strip + lowercase, so HF_HUB_OFFLINE=on or HF_HUB_OFFLINE=" 1 " was treated as offline
by the loaders but online here, leaving the raw network fetch reachable while
"offline". Use the same strip + lowercase {1,true,yes,on} set. Adds parsing tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix lint: drop dead offline-helper re-exports from vision.py
The import-hoist verifier (scripts/verify_import_hoist.py) flagged vision.py's
re-export block as HOISTED-IMPORT-UNUSED blockers: it imported eight offline
helpers from loader_utils but only used three internally
(_get_effective_local_files_only, _is_offline_related_error, _offline_aware_load).
The other five were imported purely to preserve `from unsloth.models.vision import
X`, but nothing imports four of them from vision, and loader.py already imports
_resolve_checkpoint_tokenizer_name straight from loader_utils.
Import only the three names vision.py actually uses, and point the Studio exporter
at the canonical source (from unsloth.models.loader_utils import _force_hf_offline)
instead of re-exporting it through vision. loader_utils stays the single source of
truth; no behaviour change.
* Address Opus review: chain probe errors, unify env-offline, status-less HTTP
Chain the original AutoConfig/PeftConfig probe exception into the combined
RuntimeError in both FastLanguageModel.from_pretrained and FastModel.from_pretrained
(`raise RuntimeError(combined_error) from (autoconfig_exc or peft_exc)`). The probes
caught every Exception and stringified it, so the re-raised RuntimeError had no
__cause__/__context__ and _is_offline_related_error could not classify it -- the
network-down-but-cached auto-retry never fired for these entry points. With the
cause chained, the decorator sees a ConnectionError/LocalEntryNotFoundError/5xx and
retries forced-offline from cache; a permanent cause (404 / bad config) is still not
offline-classified and propagates without a wasted retry.
Unify the third offline-env parser: studio/backend/utils/transformers_version._env_offline
now uses the canonical {1,true,yes,on} + strip + lowercase set (matching
loader_utils._env_says_offline and model_config._env_offline), so HF_HUB_OFFLINE=on
or " 1 " no longer leaks the direct urllib metadata fetches to the network.
_is_offline_related_error: a status-less HTTP error (no response / unparseable code)
now falls back to the network-wording check instead of being dropped, so a transient
HTTP failure with clear "couldn't connect" wording is treated as offline. HTTP errors
with a real status code still decide by code (4xx propagates, 5xx is offline).
* Condense offline-loading code comments, drop dead helper, dedupe import for PR #6554
* Add unit tests for offline-loading helpers for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard load cleanup with try/finally and add retry-contract tests for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add gc.collect retry-step test for PR #6554
* Tighten offline-loading comments and docstrings for PR #6554
* Raise the both-config-failed error before model-type lookup so offline retry fires for PR #6554
* Prefer offline cause for retry and bound export reachability probe for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Skip remote mapper while offline, harden text-load cleanup, and stop stacked offline retries for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Surface VLM fallback offline errors, probe offline before export version activation, and restore progress bars across retries for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restore offline env after export version activation so the persistent worker re-decides per load for PR #6554
* Classify socket.gaierror and urllib URLError as offline by type for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Probe offline around export load preflights and never offline-retry TLS failures for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Force in-process offline for export preflights, verify proxy egress in probe, and skip caching offline version negatives for PR #6554
* Snapshot offline constants before forcing env and require local processor files for VLM checkpoints for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: require signed capability tokens for /p preview links
The public /p preview routes added in #6486 run model load and chat
generation as the admin user with no authentication. The only gate is the
preview ref, a deterministic outputs-root path (run or run/checkpoint) that
is guessable rather than secret. On a network-reachable Studio (--secure
tunnel or -H 0.0.0.0), an unauthenticated caller who guesses a ref can
consume GPU and probe a private fine-tuned checkpoint.
Make the share link an unguessable, revocable capability:
- Sign the canonical ref with a dedicated server-side secret (HMAC-SHA256,
stored in app_secrets, independent of the JWT/login secret).
- Require a valid token on every /p chat, models, and page request before
resolving a checkpoint or loading a model; missing or invalid tokens get a
generic 404 so the surface never confirms a ref exists.
- Accept the token via ?k= (browser link and preview page) or
Authorization: Bearer (OpenAI-compatible clients).
- Rotate the secret to revoke every outstanding link
(POST /api/settings/preview-links/rotate).
- Clamp preview generation (max_tokens/max_completion_tokens <= 1024, n = 1)
and set Referrer-Policy: no-referrer on the page so the token is not
leaked via Referer.
Training history hands the authenticated owner the signed token, and the
copy-link button builds /p/{ref}?k={sig}.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor a lower caller token limit in the preview clamp
Codex review: when only the legacy max_tokens was sent, the clamp left
max_completion_tokens at the 1024 default, and _effective_max_tokens prefers
max_completion_tokens, so a request like max_tokens=16 could still generate up
to 1024 tokens. Derive one effective limit (max_completion_tokens wins, else the
legacy max_tokens) and pin both fields to it so a caller's lower limit is kept.
* Studio: add preview kill switch, rate limit, and revoke-links UI
Follow-ups to the /p preview capability work:
- Public-sharing kill switch: a persisted setting (default on) gates the public
/p surface. When off, every preview request 404s even with a valid token, and
the owner UI stops offering share links. GET/PUT /api/settings/preview-sharing;
enforced in _verify_or_404.
- Per-IP rate limit on the preview chat route: a coarse in-process sliding-window
limiter (20 req/min/IP) returns 429 + Retry-After before the GPU lock is taken.
Client IP honors X-Forwarded-For only when UNSLOTH_STUDIO_TRUST_FORWARDED is
set, matching the login limiter's trust model.
- Settings UI: a "Preview sharing" section with the public-sharing toggle and a
"Revoke all preview links" button (confirm dialog) that rotates the secret.
Tests cover the kill switch (404 when off), the 429 path, the sliding window,
client-IP trust behavior, and the setting default.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix preview-fields sharing arg and refresh sigs after revoke
Codex review:
- P1: get_training_run_detail and update_training_run called _preview_fields
with only output_dir after it gained a required sharing_on parameter, raising
a 500 TypeError once get_run succeeded. Pass get_preview_sharing_enabled() at
both sites; add a detail-endpoint regression test.
- P2: after rotating the preview secret from settings, the history grid still
held stale preview_sig values, so a freshly copied link would 404. Emit
emitTrainingRunsChanged() after a successful revoke so the grid refetches
freshly signed refs.
* Studio: harden preview sharing controls (Codex review)
- Fail closed: a read failure on the preview-sharing kill switch now returns
False instead of defaulting to enabled, so an unavailable settings DB can't
reopen the public surface. A missing key still defaults to enabled.
- Per-IP rate limit behind the managed Cloudflare tunnel: client_ip now honors
CF-Connecting-IP when the socket peer is loopback, so tunneled visitors are
keyed by their real IP instead of collapsing onto the local cloudflared peer.
- GET /p no longer mints key/share_url when sharing is disabled; it returns
sharing_enabled=false so clients don't distribute links that 404.
- Settings UI: toggling public sharing emits the training-runs-changed event so
the history grid shows/hides Copy preview link without a manual refresh.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden preview rate limiter and IP keying (Opus review)
From a two-agent review of the PR:
- Rate limiter no longer evicts an active bucket when the table is full: a flood
of distinct keys could otherwise cycle out a throttled bucket and reset its
counter. Evict only aged-out buckets; if the table is full of live clients,
fail closed (deny the new key) instead.
- client_ip keys on the rightmost (proxy-appended) X-Forwarded-For hop when the
trust env is set; the leftmost is client-spoofable. Documented the
append/overwrite-proxy assumption.
- _verify_or_404 checks the capability token before the kill-switch DB read, so
unauthenticated /p spam can't be used as an unbounded settings-DB sink and the
response is identical regardless of the sharing on/off state.
Tests: nested run/checkpoint happy path + wrong-ref rejection, the eviction
fail-closed behavior, and route-level coverage for the rotate / preview-sharing
settings endpoints.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: add GPU-aware model filtering and For You section- Add fit filter toggle (All / Fits GPU / Comfortable) to Hub discover tab- Add For You section showing only hardware-compatible models- Fix MoE active parameter extraction (Qwen3.5-35B-A3B now correctly reads as 3B active, not 35B)- Add gpu-fit-filter.ts with instant VRAM estimation from HF metadata without fetching model configs- Add fit badges to model cards and table rows- No backend changes- Closes#6556
* fix: handle unified memory systems in GPU fit classification
* fix: tighten GPU model fit filtering
---------
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: stop leaking the auth token through HTML canvas preview frames
The artifact preview frame placed the Studio bearer token in the iframe URL
(?token=) whenever canvas network access was enabled. Untrusted canvas HTML
runs in that frame and can read its own window.location.href, and the
network-mode CSP allows outbound http/https, so the token could be
exfiltrated and replayed against authenticated Studio APIs. The auto-render
HTML cards widened the reach: ordinary or prompt-injected assistant html
fences become a Preview card that opens this same frame, and the render_html
tool path auto-opens it without a click.
Root cause: never put the token in the frame URL. The preview shell is a
static document that only renders HTML posted to it by its embedder, and
frame-ancestors plus the no-same-origin sandbox already constrain it, so the
endpoint no longer accepts or validates the token and selects the network
CSP from allow_network alone. No credential ever reaches the frame.
Defense in depth: only tool-rendered canvases may opt into network mode;
fences auto-extracted from assistant text never do.
* Studio: stop strict canvas frames from self-upgrading to network mode
Network mode is selected from the allow_network query param alone, so untrusted
canvas code in a strict frame could navigate its own iframe to
?allow_network=1; the frame's onLoad handler then reposted the same untrusted
HTML into the now network-enabled frame, giving a no-network or fenced canvas
unauthorized network egress.
Only inject the artifact for loads we initiated (mount or a src change), tracked
by a pending flag set when src changes. A self-navigation also fires onLoad but
is no longer fed, so the upgraded frame stays the inert shell. The strict CSP
default-src 'none' already blocks the child-iframe variant.
* Studio: trim comments in the canvas artifact security fix
Condense the added explanatory comments and the artifact-preview-frame docstring
to one line each while keeping the security rationale. No code change (verified
comment-only).
* Studio: keep the training event pump alive so progress can't silently freeze
The parent-side event pump is the only writer of the in-memory progress state
that SSE /progress, /status, /metrics and the DB history all read. It ran in a
single unsupervised daemon thread with no guard around event handling, so one
malformed event or a transient queue/DB error would terminate it permanently.
The worker subprocess keeps training regardless (mp.Queue puts never block on an
unbounded queue), so a run kept burning GPU for hours while every progress
surface froze on the last step the pump saw.
- Guard each pump iteration: a bad event or queue-read error is logged and
skipped instead of ending the loop. _read_queue now reads any error as
"no event", not just Empty/EOFError/OSError/ValueError.
- Add a _pump_running flag and an _ensure_pump_alive watchdog wired into
is_training_active, so a pump that dies while the worker is alive is restarted
on the next status poll and the UI catches up from the still-open queue.
- Start respawned and restarted pumps under the lock so the watchdog can never
spawn a duplicate during the brief start window.
Adds tests/test_training_pump_resilience.py covering both guarantees.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio training pump: address review (drain guard, start race, read backoff, respawn flag)
Follow-up to the event-pump resilience change, closing four edge cases a
review surfaced in the same pump/queue surface:
- _drain_queue now tolerates any error during the worker-exit drain and
finalizes with whatever it drained, instead of skipping finalization and
leaving the run wedged "active" with a dead worker.
- start_training clears a stale _pump_running flag during reset and assigns
the subprocess handles plus starts the pump under the lock, so a concurrent
status/SSE poll can't spawn a duplicate pump during setup.
- _read_queue goes back to the narrow EOFError/OSError/ValueError catch;
truly unexpected errors are left to _pump_loop's guarded read, which logs
and backs off so a persistently raising queue can't spin a hot loop.
- The xet respawn-failure path clears _pump_running so a later run can't
inherit a stale flag.
Adds regression tests for all four.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: revive a crashed pump after worker exit + stop test module pollution
Two review follow-ups on the training event pump:
- _ensure_pump_alive refused to restart once the worker had exited
(not self._proc.is_alive()), so a pump that crashed just before the worker
finished never drained the terminal complete/error events still sitting in
the queue. progress.is_training stayed True and is_training_active() returned
True forever, leaving the run stuck "running" behind a dead pump. A True
_pump_running flag with a dead thread is an unambiguous crash regardless of
worker state, so restart there too: the fresh pump drains the backlog and
finalizes. Updated the watchdog test to assert the revive-and-finalize.
- The resilience test imports core.training.training while heavy module-level
deps are stubbed, then restores the stubs -- but the cached training module
kept the stubs bound in its globals, so a later test in the same session
could exercise the fakes (e.g. prepare_gpu_selection) instead of the real
code. Evict the training module (and its package) after import when this file
created it, so subsequent tests re-import it cleanly.
* Studio: finalize training run when queue reads keep failing on a dead worker
reviewer.py follow-up. _read_queue only swallows EOFError/OSError/ValueError;
an unexpected error escapes to the pump's outer guard, which logged, slept and
`continue`d. If those reads keep raising after the worker has already exited
(e.g. a broken queue pipe), the loop never reaches the dead-worker finalize
block, so the pump spins on with _pump_running True and progress.is_training
stuck True -- the run looks like it is still training forever. On a read failure
now fall through to finalize when the worker is gone, only backing off and
retrying while it is still alive. Mirrors the data-recipe pump fix; added a
regression test.
* Tighten training pump resilience comments and docstrings
Condense the verbose explanatory comments and docstrings on the training event
pump and its tests to shorter, clearer forms. Comment/whitespace only; verified
no code changed via AST diff. No behaviour change.
* Studio: create the training DB run before starting the event pump
start_training started the event pump before the eager _ensure_db_run_created()
call, so for a worker that completes or fails immediately the pump could race the
main thread into creating and finalizing the same run row (duplicate INSERT, or a
finalize skipped while _db_run_created was still false). Create the run first; the
pump then only ever finalizes. Adds a regression test asserting the pump observes
an already-created run.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: don't time out the live progress stream during pre-first-step prep
The live progress SSE counts every 1s poll without a step update toward a
30-minute stall timeout, after which it emits an error event and ends the
stream. But that counter also runs during the pre-first-step phase (model
load + tokenizing the dataset), which is never reset because no step has
happened yet. On a large dataset that prep can take well over 30 minutes, so
the live view is torn down with an error while the run is perfectly healthy
and still preparing -- the run then trains on in the background with the UI
showing nothing, exactly the "no progress for hours" decoupling.
Apply the stall timeout only once the stream has actually seen a live step.
Before the first step the run is preparing and may legitimately emit no step
for a long time; heartbeats still flow so the client stays connected and the
worker's liveness still ends the loop when training finishes. A genuine
post-step stall still times out. Extracted the threshold to a module constant
so it can be tuned/tested.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: seed seen_live_step from the resume point on reconnect
Review follow-up: seen_live_step reset to False on every SSE request, so a
client reconnecting past the first step (Last-Event-ID set, or the run already
has step history) only receives heartbeats and never flips it true. A worker
that hangs after step N would then never trip the stall timeout for that
reconnected client. Initialize it from resume_from_step / existing step
history so reconnects keep the post-step timeout behavior, while a genuine
pre-first-step run still stays exempt. Added a reconnect regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten prep-phase progress timeout comments
Condense the verbose explanatory comments and docstring on the prep-phase stall
timeout exemption to shorter, clearer forms. Comment/whitespace only; verified no
code changed via AST diff. No behaviour change.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Daniel Han