* Studio: warn when a GPU model silently loaded on CPU
llama-server can serve HTTP 200 while running a model entirely on CPU when its GPU backend fails to init, so Studio could run a GGUF on CPU without saying so (#5807 / #5106 / #5830). The silent-CPU warning already exists but stopped firing on current llama.cpp because _classify_gpu_offload keyed only on the dropped 'model buffer size' lines. Add a shared classify_gpu_offload_lines (offloaded N/M counts, GPU model-buffer markers excluding _Host, device_info disconfirm-only) and delegate to it so the warning fires again. Log-only: no install or load behavior changes.
Pure classification of already-captured startup log lines, run once after load; no new subprocess, no slowdown.
* Studio: key the CPU-offload warning on the main model, not a draft
With MTP/speculative decoding llama-server logs 'offloaded N/M layers to GPU' twice: once for the main model and once for the small draft model. The old scan returned True on any non-zero count, so a drafter that fits on GPU while the main GGUF runs on CPU suppressed the warning (the Qwen3.6-27B-MTP case). Decide on the line with the most layers (the main model) instead, so a drafter cannot mask a main model on CPU.
* Studio: make lifespan shutdown resilient to a dead default executor
On an abrupt shutdown (closing the Windows console window, or interpreter
teardown racing uvicorn's graceful stop) the event loop's default thread-pool
executor can already be shut down by the time the FastAPI lifespan shutdown
runs. The first post-yield statement was an unguarded
`await asyncio.to_thread(terminate_hub_downloads)`, so executor.submit raised
`RuntimeError: cannot schedule new futures after shutdown`. That raise
propagated up through every nested merged_lifespan __aexit__, aborted the rest
of the cleanup (DEVICE reset, compiled-cache clear), and surfaced as
"Application shutdown failed. Exiting."
Extract the post-yield cleanup into utils/lifespan_shutdown.run_lifespan_shutdown
and guard each step independently. On the to_thread RuntimeError, fall back to
running the (already best-effort, quick) terminate inline on the loop thread so
shutdown still completes cleanly. The helper is dependency-injected and free of
the heavy backend import graph, so it is unit-tested in isolation.
Add tests/test_lifespan_shutdown.py (4 cases: dead-executor survival, normal
path, terminate error, clear error). Validated on windows-latest and
ubuntu-latest runners on Python 3.13.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: only retry terminate inline when scheduling fails, not when the body raises
Address review feedback on the shutdown helper: the previous
`except RuntimeError` after `asyncio.to_thread(terminate_downloads)` could
not tell a dead-executor scheduling failure from a RuntimeError raised by
terminate_downloads itself, so a body-side RuntimeError on a healthy executor
ran the cleanup a second time inline.
Schedule via loop.run_in_executor and await separately: a dead default executor
raises synchronously at submit time (inline fallback), while a body exception
only surfaces when the future is awaited (logged, never retried). Add a
regression test that a body RuntimeError runs terminate exactly once.
* Studio: run terminate cleanup with a copied context (parity with asyncio.to_thread)
Simulation across the executor-state x exception x DEVICE matrix surfaced the one
behavioural difference from the original implementation: asyncio.to_thread copies
the caller's contextvars into the worker thread, while a bare
run_in_executor(None, fn) does not. Restore exact parity by scheduling
ctx.run(terminate_downloads) from a contextvars.copy_context(), so the refactor
is a behavioural no-op apart from the intended dead-executor recovery. Add a
regression test asserting the copied context is visible to terminate_downloads.
* Studio: tighten comments in lifespan_shutdown helper and tests
Condense the verbose docstrings/comments to the non-obvious rationale and drop
the self-evident ones. Verified comment-only with comment_tools.py check
--strip-docstrings (code signature unchanged); tests and sims still green.
* Studio: address review nits on lifespan shutdown helper
Annotate hw_module as types.ModuleType, reword the schedule/await comment
(inline fallback runs, it does not retry), and add a test for the public
loop.shutdown_default_executor() path (the 'Executor shutdown has been called'
RuntimeError that real uvicorn shutdown takes, distinct from the submit-time
'cannot schedule new futures after shutdown').
---------
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: implement thread forking functionality with associated database updates and UI components
* fix(studio/chat): register fork-count listener even when thread unsaved
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Polish thread fork action menu
* fix-studio-fork-project-test-order
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Fix the libaray path for probe_server_capabilities()
even when running something as simple as `./llama-server --help`,
the binary still requires correct LD_LIBRARY_PATH to work - or it
returns merely an "error while loading shared libraries":
"libllama-server-impl.so: cannot open shared object file: No such file or directory"
For a local installation with no LD_LIBRARY_PATH specifically set,
the probe_server_capabilities() run of `./llana-server --help` should
share the same libaray resolution logic as start_llama_server().
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Adjust and readd comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* feat(studio): expose provider_type selector in model provider dialog
Use the configured provider_type in the model provider payload instead of hardcoding openai. Add a provider type selector to the model provider dialog for the provider types supported by Data Designer.
* feat(studio): add MiniMax-M2.7 inference defaults
Add MiniMax-M2.7 model family entry to inference_defaults.json
with recommended parameters (temperature=1.0, top_p=0.95, top_k=40).
Pattern placed before minimax-m2.5 for correct longest-match-first
ordering.
* Fix/adjust provider type support for PR #4277
* Clarify provider type validation for PR #4277
* Fix provider type default validation for PR #4277
---------
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: PR Bot <pr-bot@minimaxi.com>
* Studio: guard the training import against a namespace-package shadow of unsloth
A directory named unsloth without an __init__.py on sys.path (a stray checkout,
a partial clone, or a polluted PYTHONPATH) makes the path finder return a
namespace package, so the worker's 'from unsloth import FastLanguageModel' fails
with 'cannot import name ... (unknown location)'. A normal site-packages install
always wins this race, so only source/editable installs are exposed. Before the
import, drop the offending sys.path entries, bind the real packages, then
restore sys.path so other modules on those entries keep importing. It is a
no-op when unsloth already resolves to a real package.
* Studio: import unsloth before unsloth_zoo in the namespace-shadow guard
_ensure_real_packages imported the requested names in argument order, so a
unsloth_zoo namespace shadow made it import unsloth_zoo directly before
unsloth. That skips unsloth.__init__ -> _gpu_init, which runs its ROCm and
Windows bitsandbytes fixes before its own import unsloth_zoo, so the recovery
path could import zoo with those guards skipped and fail on the ROCm/Windows
cases those fixes handle.
Import parent-first via reversed(names) so unsloth is imported first and pulls
in the real unsloth_zoo after _gpu_init has run; the later cached import is a
no-op. This also covers the case where only unsloth_zoo is shadowed: the bad
sys.path entry is still dropped and unsloth owns the zoo import. Detection,
sys.path pruning, shadow-cache clearing, and restoration are unchanged.
Add tests/test_namespace_shadow_guard_pr6269.py: CPU-only subprocess scenarios
(only zoo shadowed, both shadowed, only unsloth shadowed, healthy no-op, real
package absent, multiple shadow entries) that assert unsloth imports before
unsloth_zoo and that sys.path is restored.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: restore sys.path even if invalidate_caches fails in the shadow guard
Move importlib.invalidate_caches() inside the try/finally so a failure there
still restores sys.path, and tighten the import-order comment. Add a test that
forces invalidate_caches to raise and asserts sys.path is restored.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: show llama.cpp version and GPU specs in the About panel
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: run the hardware endpoint off the event loop
* Studio: show the full llama.cpp release tag (incl -mix-<sha>) in the About panel
* Studio: order About-panel GPUs by visible ordinal and skip the llama.cpp probe during updates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix hardware info refresh and endpoint scope
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
When an empty assistant turn is dropped (a 0-token model reply, or a Stop-button
sentinel handled by _drop_empty_assistant_sentinels) the GGUF chat history can be
left with two user turns back to back. Strict Jinja chat templates (Gemma 3, some
Mistral variants) call raise_exception("Conversation roles must alternate ...") on
the first role-parity break, so llama-server returns a 400 and the thread becomes
unsendable.
Add _coalesce_consecutive_user_turns (merging only adjacent user turns, never
assistant/tool turns, preserving multimodal parts) and apply it in
_openai_messages_for_gguf_chat. The tool path inherits the fix for free because it
rebuilds from this same normalized history via _set_or_prepend_system_message. The
passthrough path is left untouched (it forwards messages verbatim). No-op for
already-alternating histories. Adds unit and end-to-end coverage.
On a restart the constructor reaps the previous run's orphaned llama-server, but the driver does not reclaim that VRAM synchronously. _kill_orphaned_servers now returns the number of processes it killed, and __init__ arms _last_kill_monotonic when that count is positive, so the first load_model waits for VRAM to settle before ranking GPUs by free memory instead of pinning the model onto the smaller card. The compute-graph / auto-fit reserve is handled separately in #6312.
* Studio: add --secure Cloudflare-only mode and revamp API usage examples
--secure / --not-secure on `unsloth studio` and `unsloth studio run`:
- --secure binds 127.0.0.1, requires the Cloudflare tunnel, and advertises only
the Cloudflare link. cloudflared reaches the server over localhost, so the raw
port is never exposed on a public interface.
- If the tunnel cannot start, fail closed with a clear message instead of
silently leaving a raw 0.0.0.0 link.
- Default stays not-secure (no behavior change); coexists with the existing
--cloudflare/--no-cloudflare flag. Host defaults are unchanged.
- /api/health (authed) now reports the live tunnel URL.
API usage examples (Profile > API):
- Example tabs for curl, Python, curl + tools, Python + tools, plus an OS row
(Linux/macOS/WSL vs Windows) auto-detected from the platform.
- Windows curl passes the JSON body via a file so PowerShell does not strip the
quotes when calling curl.exe.
- Python + tools forwards enable_tools/enabled_tools through extra_body and
guards chunk.choices, since tool-lifecycle events carry no choices.
- Shows the loaded model name and the real API key while it is still revealed.
- A Cloudflare Tunnel toggle (default on) shows the public tunnel URL and uses
it as the base_url in the examples when a tunnel is running.
Tests cover the tunnel start gate and the --secure flag on both commands.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate --secure tools on public exposure and harden API examples
In secure mode the server binds loopback but is reachable via the public
Cloudflare tunnel, so resolve the tool policy against the public exposure
(0.0.0.0) rather than the loopback bind. This keeps server-side tools off by
default and prompts before enabling them, instead of inheriting the loopback
default of on. The startup tool notice now names the public surface.
Also reject --secure with --no-cloudflare directly in run_server and the
run.py argparse (not only the CLI), JSON-encode interpolated model names so
Windows paths and quotes cannot produce invalid JSON or broken snippets, and
force-refresh /api/health on the API panel so a tunnel that starts after the
first health read still surfaces its URL.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: API examples show direct host when tunnel toggle is off; move Copy onto code
The Cloudflare Tunnel toggle had no visible effect when Studio was opened
through the tunnel: the off state fell back to window.location.origin, which
equals the tunnel URL in that case. /api/health now reports the direct
host:port (server_url), and the API panel uses it for the off state so it shows
the real non-tunnel base. Also move the Copy button out of the tab row and onto
the code block.
* Studio: highlight API examples, add advanced tabs, fix tunnel toggle row
Syntax-highlight the curl/PowerShell/Python snippets with the app's shared
shiki plugin (bash/powershell/python). Add 'curl + advanced' and
'Python + advanced' tabs that set temperature/top_p/top_k/min_p/
repetition_penalty/max_tokens, enable thinking, and turn on all tools.
The Cloudflare Tunnel row no longer shifts the code block: the tunnel URL is
always rendered (dimmed when off) so toggling keeps the row height constant.
Key the highlighted block on its content so it remounts when only the base URL
changes (the renderer's block memo otherwise kept a stale URL).
* Studio: rename API tunnel toggle to Secure HTTPS, hint --secure when exposed
Rename the API examples toggle from Cloudflare Tunnel to Secure HTTPS. When the
server was not launched with --secure, show an info tooltip noting the raw
0.0.0.0 port is still globally reachable and pointing at --secure. /api/health
now reports whether --secure was used so the hint is hidden in secure mode.
* Studio: force tools off for plain network/secure launches
The plain 'unsloth studio --secure' (and '-H 0.0.0.0') launcher re-execs run.py
and never installed a tool policy, so the process default (honor per-request
enable_tools) let any API-key holder run Python/terminal tools over the public
endpoint. Force the policy off at the run.py entrypoint when network-reachable
(0.0.0.0 or --secure); 'unsloth studio run' still installs its own resolved
policy and does not go through this path.
* Studio: apply default tool policy in run_server, not the run.py entrypoint
The plain launcher runs from the studio venv and calls run_server directly, so
it never hit the run.py __main__ guard. Move the network/secure default-off tool
policy into run_server so every launch path (plain, --secure, direct run.py)
gets it; the run subcommand still overrides it with its resolved policy.
* Studio: clarify --secure help text on the network exposure tradeoff
Spell out in --help (both unsloth studio and unsloth studio run, plus the
run.py argparse) that --not-secure also serves the raw 0.0.0.0 port reachable
from anywhere on the network, matching the API panel's Secure HTTPS hint.
* Studio: cache API-key PBKDF2 derivation to cut per-request /v1 auth overhead
validate_api_key re-ran the 100k-round PBKDF2 on every authenticated
request, adding ~15ms to each /v1 call made with an sk-unsloth- key.
Benchmarked against the bare llama-server it proxies to, API-key requests
carried ~22ms of fixed overhead vs ~7ms for the JWT path; the gap was
entirely this redundant key derivation (Pydantic validation measured
0.005ms, so it is not a factor).
The raw-key to hash mapping is a pure deterministic function of the fixed
server salt, so memoize it per process, keyed by a salted HMAC of the key
(never the key or a recoverable digest). The cached value equals what is
already stored at rest. Revocation and expiry remain enforced by the
SQLite read on every call, so a cache hit only skips the KDF, never the
active or expiry checks. Only keys that exist in the DB are cached, so
unknown-key spam cannot grow it.
After the change the API-key /v1 overhead drops to ~8ms, at parity with
JWT, while the at-rest PBKDF2 hashing is unchanged.
Adds test_api_key_expiry.py covering API-key and JWT expiry enforcement
and the new cache: it skips the KDF on repeat and still rejects revoked
or expired keys.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten comments across the secure-tunnel and API-key changes
Condense multi-line comments and docstrings to one or two lines, drop the
ones that restate obvious code, and remove an orphaned test section header.
Comment-only: verified with comment_tools.py check (9/9 code unchanged), the
auth/secure-tunnel/CLI test suites, and a clean frontend typecheck and build.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: Add inline confirmation (Allow/Always allow/Deny) for tool calls
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix race in tool-call confirmation gate
* Studio: gate built-in tool calls and harden the confirmation handshake
The Allow / Always allow / Deny controls only lived in the fallback tool
card, but the built-in tools (web search, python, terminal, code
execution, image generation) render with their own components and so
never showed the buttons. Those calls paused after tool_start with no way
to approve them, hanging until the 1 hour timeout. Only MCP tools, which
use the fallback renderer, actually worked.
Render the controls for every tool card by wrapping each registered tool
component (and the fallback) in thread.tsx with a shared
ToolConfirmationControls, so the gate applies uniformly.
Also make the handshake robust:
- The gate keys on a per-call approval_id minted by the backend and
echoed in tool_start, instead of session_id alone, so a stale or
concurrent confirmation can no longer resolve the wrong call.
- The approval slot is registered before tool_start is yielded, closing
the race where a fast click or an auto "Always allow" could reach the
backend before the waiter existed.
- The frontend resolves with the same session id the request was sent
with (plus the approval_id), fixing the new-thread mismatch where the
confirmation targeted a different session than the blocked stream.
- The confirm endpoint returns {resolved}; the UI keeps the buttons and
shows a retry hint until the backend confirms a match, instead of
hiding them on a failed or mistargeted post.
- The gate runs after the disabled-tool and duplicate-call checks, so a
call that will not execute is not put up for approval. A denied call is
still excluded from duplicate detection, so re-issuing and approving it
works.
- "Always allow" is scoped per session to match the backend gate.
Add backend tests for the approval registry, the SSE no-deadlock
handshake, and the loop integration (allow, deny, disabled, duplicate,
re-issue after deny).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Move "Confirm tool calls" to the Tools section
* Studio: add Bypass Permissions (skip confirmation, disable tool sandbox)
Adds an opt-in Bypass Permissions toggle next to Confirm tool calls. When on,
no tool call shows a confirmation prompt and the python/terminal sandbox is
disabled: safety checks, command blocklist, and resource limits are skipped.
Secret env vars are still stripped and HOME stays repointed at the session
workdir. Default off keeps current behavior, and it takes precedence over
Confirm tool calls. Enabling it requires accepting a warning each time.
* Studio: harden Bypass Permissions secret handling and fix Anthropic tool path
Follow-up to the Bypass Permissions feature. Addresses the review findings:
- Anthropic /v1/messages 500: declare bypass_permissions on
AnthropicMessagesRequest so tool requests that omit the field default to
False instead of raising AttributeError (extra='allow' does not set absent
attributes).
- /proc parent-env leak: stripping the child env did not stop a same-uid
bypassed child from reading /proc/<parent>/environ to recover the
tool-executing process's unfiltered secrets. Clear PR_SET_DUMPABLE on that
process before the first bypass exec so its /proc entries become root-owned.
Hardening is fail-closed: if prctl is denied, bypass execution is refused
rather than run with the parent environ still readable. Mitigation, not a
full boundary; documented in the code.
- Broker/capability vars: strip SSH_AUTH_SOCK, SSH_AGENT_PID, GPG_AGENT_INFO,
GNUPGHOME, KUBECONFIG, DOCKER_HOST so a bypassed tool cannot use the
operator's live agents.
- Credential-bearing URL values: drop any env var whose value embeds URL
userinfo (scheme://user:pass@ and token-only scheme://token@) regardless of
the variable name. Benign proxy/index URLs without credentials are kept, so
proxy-only and internal-index setups still work in bypass mode.
- Windows temp isolation: repoint TEMP and TMP (not just TMPDIR) at the
per-session sandbox dir.
- Frontend: stop persisting bypassPermissions; a reload now starts with the
sandbox/confirmation bypass off and requires re-accepting the warning dialog.
Adds regression tests for each finding in test_bypass_permissions.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip cred-location env vars (HF_HOME etc.) in Bypass Permissions
Repointing HOME did not stop SDKs auto-reading cached creds via vars that
point at the real home/cache/config: HF_HOME (startup always sets it; token
lives under $HF_HOME/token), HF/XDG cache roots, NETRC/BOTO_CONFIG/
PIP_CONFIG_FILE, and Windows HOMEDRIVE/HOMEPATH. Drop those, and repoint
USERPROFILE/APPDATA/LOCALAPPDATA at the per-session workdir. Adds regression
tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: lock in bypass HF token resolution with an end-to-end test
The drop-based fix relies on the whole HF_HOME/XDG fallback chain being
removed so huggingface_hub resolves under the repointed HOME. Add a test
that sets HF_HOME and XDG_CACHE_HOME at a real cache and asserts the
resolved token path lands under the workdir, not the operator's cache.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip npm _auth, MYSQL_PWD, and BASH_ENV from bypass env
Three more credential vectors dodged the bypass scrubber: NPM_CONFIG__AUTH
(npm _auth, base64 so no URL userinfo and no AUTH marker), MYSQL_PWD (markers
use PASSWD, not PWD, since PWD is the cwd var), and BASH_ENV (bash -c sources
it for non-interactive shells, so a startup file can re-export stripped
secrets). Add an AUTH marker, the exact MYSQL_PWD name, and drop BASH_ENV plus
PGPASSFILE. Adds regression tests incl. an end-to-end check that a bypass
terminal call does not source BASH_ENV.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: extend bypass env scrubber and enforce confirm precedence in loops
From a parallel review pass over the bypass changes:
- Drop more credential-location vars in _build_bypass_env: npm/yarn/git/cargo/
rclone config pointers (NPM_CONFIG_USERCONFIG, NPM_CONFIG_GLOBALCONFIG,
YARN_RC_FILENAME, GIT_CONFIG_GLOBAL, GIT_CONFIG_SYSTEM, CARGO_HOME,
RCLONE_CONFIG) and the GIT_ASKPASS/SSH_ASKPASS auth helpers.
- Enforce confirm_tool_calls AND NOT bypass_permissions inside the safetensors
and GGUF tool loops, not just at the route, so a direct internal caller
passing both flags never prompts.
- Soften the toggle hint: environment secrets are stripped, but bypassed code
can still read files and credentials on the machine (no overclaim that keys
stay hidden).
Adds regression tests for the new names and the loop-level precedence.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add GGUF loop test for bypass-over-confirm precedence
The safetensors loop precedence is covered behaviorally; the GGUF loop needs a
live llama-server so add an AST guard asserting its _needs_confirm gate
references both confirm_tool_calls and bypass_permissions, matching the other
llama_cpp source-inspection tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add red Bypass Permissions badge in the composer
When Bypass Permissions is on, show a persistent red pill in the composer
tool-pill row (like the Search/Code pills), matching Claude Code's always-
visible bypass indicator. Clicking it turns bypass off, mirroring the other
composer toggles. Enabling still goes through the settings toggle + warning
dialog. Adds a data-variant=danger style for the destructive-colored pill.
* Studio: show Bypass Permissions badge in the Thread composer too
The empty-state and active Thread render their own composer (thread.tsx),
not shared-composer, so the badge only appeared in the split layout. Mirror
the red dismissible pill in ComposerAction so it shows in every composer.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep the Bypass Permissions badge visible when the composer is collapsed
The Thread composer only renders the pill row when expanded, so the active-mode
badge vanished on the default (collapsed) empty state. Render it before the
expand gate (it returns null when bypass is off) so the red indicator always
shows while bypass is on.
* Studio: make the Bypass Permissions confirm button a solid red button
The destructive button variant is a subtle 10% tint that read as bare red text
next to the outlined Cancel. Force the solid destructive fill (the variant's
class loses to the tint through AlertDialogAction's Slot merge, so use the !
override the codebase already uses for this case) and shorten the label to
'I understand' so it fits the small dialog's two-column footer.
* Studio: add Bypass Permissions to the composer + More menu
Adds a 'Bypass Permissions' entry to the composer plus-menu (under More by
default) in both composers, so it can be toggled without opening Run settings.
Enabling routes through the same danger warning dialog; disabling is immediate.
A shared BypassPermissionsMenuItem keeps the two composers in sync.
* Studio: harden bypass env scrubber for IMDS opt-out and connection strings
Two gaps in the Bypass Permissions secret scrubber:
- The broad AWS_ prefix also dropped AWS_EC2_METADATA_DISABLED, a non-secret
opt-out. Removing it re-opens the IMDS instance-role credential path that the
operator explicitly disabled, so a bypassed boto/AWS-CLI call could recover
cloud creds. Keep that flag (and AWS_EC2_METADATA_V1_DISABLED) via a keep-list
while still stripping the real AWS credential vars.
- Azure App Service connection strings (SQLCONNSTR_/CUSTOMCONNSTR_/...,
WEBSITE_CONTENTAZUREFILECONNECTIONSTRING) and values like Password=/AccountKey=
/SharedAccessKey= slipped past the name and URL-only value classifiers. Add
CONNSTR/CONNECTIONSTRING name markers and a connection-string value matcher.
* Studio: let Bypass Permissions suppress the confirm-tool-calls guards
The confirm-vs-bypass precedence (confirm and not bypass) was applied at the
loop call sites but not at the earlier request guards, so a client sending
confirm_tool_calls + bypass_permissions together was rejected (stream=true
required / unsupported for external or Anthropic tools) before the precedence
took effect. Gate all four confirm guards on not bypass_permissions so both
flags together proceed with the gate suppressed, matching the documented rule.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: add Anthropic-compatible thinking parameter
Add `thinking` parameter using Anthropic's format ({type: 'disabled'} /
{type: 'enabled'}) alongside the existing `enable_thinking` boolean for
backward compatibility.
The new parameter is mapped internally to `enable_thinking` at the route
layer so all downstream templates and backends continue to work unchanged.
Changes:
- Add ThinkingConfig model and `thinking` field to ChatCompletionRequest
- Add mapping logic in routes: thinking.type -> enable_thinking
- Add `thinking` field to frontend TypeScript types
- Update frontend request building to send thinking parameter
- Add tests for new thinking parameter
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: move thinking→enable_thinking mapping to model_validator
The Gemini review correctly identified that the route-level mapping
bypasses normalization for external provider requests. Moving the
mapping into a @model_validator on ChatCompletionRequest ensures it
runs during Pydantic validation regardless of routing path.
* Document ThinkingConfig scope and thinking validation behavior
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* fix(studio): Windows GGUF cancel hang + CPU spinlock overhead (#5692)
Two fixes for Windows-native GGUF inference via llama-server:
**Issue 1 — GPU/CUDA Hang on Stream Cancellation:**
- Add `Connection: close` header to all httpx requests proxying to
llama-server, preventing Keep-Alive from masking downstream socket
closure.
- Introduce `_await_disconnect_then_close` background watcher that
polls `request.is_disconnected()` every 100ms and calls
`resp.aclose()` immediately when the client disconnects. This runs
alongside the existing cancel-POST watcher and covers client aborts
that never reach the /cancel endpoint (tab close, proxy aborts,
Colab, mobile navigation, etc.).
- Change all StreamingResponse `Connection: keep-alive` headers to
`Connection: close`.
**Issue 2 — High CPU Spinlock & KV Cache Backup Overhead:**
- Set OMP_WAIT_POLICY=PASSIVE and OMP_NUM_THREADS=2 in the
llama-server subprocess environment on Windows to prevent OpenMP
from spin-waiting on all logical cores while the GPU decodes.
- Limit `--threads` to 2 on Windows when the model is fully
GPU-offloaded (`-ngl -1`). Auto-detect otherwise.
- Pass `--cache-ram 0 --ctx-checkpoints 0 --no-cache-prompt
--checkpoint-every-n-tokens -1` on Windows to disable prompt-cache
snapshots that copy KV cache to system RAM over the WDDM/PCI-E bus.
Closes#5692.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: use local import to avoid ruff F823 (sys used before assignment)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* review: address gemini review feedback
- Simplify _fully_gpu_offloaded init: default to False, only set True
in the gpu_indices branch, drop redundant else.
- Log exceptions in _await_disconnect_then_close at debug level instead
of silent pass, per review suggestion.
* Adjust review feedback for PR #5749
- _await_disconnect_then_close: set cancel_event before resp.aclose() so
the streamer's RemoteProtocolError handler treats the watcher-driven
close as cancellation, not an upstream error. Both call sites pass
cancel_event through.
- Windows --cache-ram / --no-cache-prompt / --ctx-checkpoints block: gate
on _fully_gpu_offloaded so CPU and partial-offload Windows runs keep
prompt-cache reuse across turns.
- Windows OMP_WAIT_POLICY / OMP_NUM_THREADS env: same gate so CPU and
partial-offload Windows runs keep default OpenMP parallelism.
* Shorten code comments touched by PR #5749
* Clean up local imports and rename underscore locals in PR #5749
- Drop the function-local `import sys as _sys` introduced as an F823
workaround; remove the redundant in-function `import os`/`import sys`
block so module-level imports resolve sys/os instead. F823 no longer
triggers because no shadowing import remains inside load_model.
- Rename `_fully_gpu_offloaded` and `_t` to `fully_gpu_offloaded` and
`threads_arg`. Underscore-prefixed names usually mean private/module-
level; plain locals match Python style for in-function temporaries.
No behavior change. ruff clean, py_compile clean, 35 studio cancel-
infra tests + 13 launch-gating AST locks + 6 disconnect-watcher locks
+ 4 spoof live-import tests all pass.
* Fix Windows GGUF follow-ups for PR #5749
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix cache flag gating for PR #5749
* Fix Python 3.9 annotations for PR #5749
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Anmol Mishra <anmolx.work@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: enable stdio MCP servers on a loopback bind
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address codex review on stdio MCP loopback gate
* Studio: fix banner URL and preserve stdio MCP env opt-in on network binds
* Studio: scope loopback to exact aliases and honor force-disable on run_server reuse
* Studio: cover force-disable across a public re-bind and fix a stale test comment
* Studio: keep stdio MCP off on Colab loopback launches
* Studio: set tool policy before server startup
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Expose MLX grad value clipping in Studio
* update test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* dataset ordering + wd
* fix mlx smoke step expectations
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* cast norm activation output back to original input dtype
* address mlx studio review feedback
* Fix present-but-None seed override for PR #5656
studio/backend/core/training/worker.py
`config.get("model_random_state", random_seed)` only fills the
default when the key is absent. When a caller passes
`config["model_random_state"] = None` explicitly (which happens
any time a JSON payload sends an explicit `null`), the old code
forwarded `None` to FastMLXModel and disabled deterministic init
silently. Same for `lora_random_state`. Treat absent and explicit
None the same way: fall back to random_seed.
studio/backend/tests/test_training_raw_support.py
Update the source-string assertions to match the new lines.
* Guard optional MLXTrainingConfig fields and normalize random_seed for PR #5656
The MLX worker now passes `cast_norm_output_to_input_dtype` and
`dataset_order` only when the linked unsloth-zoo dataclass actually
declares them. Released zoo trees that predate the paired PR can still
construct `MLXTrainingConfig` without raising
`TypeError: unexpected keyword argument`. Once the dependency floor is
bumped to a release that contains both fields, the feature-detect
guards become no-ops.
`random_seed = config.get("random_seed", 3407)` was unguarded against
explicit `None` from raw / backend callers. The same value seeded the
trainer and was the fallback target for `model_random_state` /
`lora_random_state`. Normalize once at the top of the function and use
the normalized value everywhere so an explicit `None` cannot reach
FastMLXModel / get_peft_model / MLXTrainingConfig.
Existing seed source-pattern test updated to match the new normalize
helper. New test asserts the feature-detection guards exist and that
the unconditional kwargs do not include the gated fields.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Normalize seed / cast / max_grad_value at TrainingBackend for PR #5656
Round-3 review consensus: the per-field guards that landed in the MLX
worker only protect the MLX path. The same `TrainingBackend.start_training`
config still reaches the CUDA/text trainer at `worker.py:2267`, the
embedding LoRA init at `worker.py:2450`, and embedding TrainingArguments
at `worker.py:2624` with raw `None` values, so an explicit
`random_seed=None` from a raw / backend caller still breaks non-MLX
training even after the previous fix.
Move the normalization into `TrainingBackend.start_training` itself,
where it runs once for every training mode:
- `_coerce_seed(value)`: explicit `None`, non-int, or absent all become
3407. Every downstream worker now sees an int.
- `_coerce_optional_bool(value, default)`: explicit `None` falls back
to `default` instead of `bool(None) == False`. Also normalizes the
common raw-config / YAML string aliases ("true" / "false" / "0" /
"1"). Used for `cast_norm_output_to_input_dtype`.
- `_coerce_optional_nonneg_float(name, value)`: rejects negative
numerics from raw / backend callers, matching the Pydantic
`ge=0` constraint the HTTP route already enforces. Used for
`max_grad_value`.
worker.py MLX path: the existing `bool(config.get(key, True))` for
`cast_norm_output_to_input_dtype` was changed to also fall back on
explicit `None`, so direct worker callers (bypassing
`TrainingBackend.start_training`) are equally safe. `max_grad_value`
also raises on negative values inside the worker for the same reason.
TrainingStartRequest.random_seed default bumped from 42 to 3407 so
direct REST callers that omit the field receive the same default as
the Studio frontend and the MLX worker.
New regression test exercises the three new helpers across explicit
None, valid values, string aliases, and negative-value rejection.
* Tighten feature-detect test paren tracking for PR #5656
The block-extraction used , which stops at the
first inner closing paren (e.g. )
and would silently miss a future unconditional
/ added later in the same dict literal. Switched to
proper paren-depth tracking so the unconditional block is checked end-to-end.
* Shorten verbose comments in MLX Studio backend
* Handle MLX Studio EOS appending by mode
* Wire MLX leaf norm clipping through Studio
* Respect VLM layer filters for explicit LoRA targets
Rationale / guardrails for the local Studio/vision push:
When callers provide explicit VLM LoRA target_modules together with layer filters, FastVisionModel still needs to route the explicit targets through get_peft_regex. Otherwise the layer filters are ignored and adapters can be attached outside the requested language/vision scope.
Do not revert this to plain list(target_modules) for explicit module lists. The CUDA/Studio-facing contract is that explicit targets and layer filters compose: target_modules selects module names, while finetune_language_layers / finetune_vision_layers / finetune_attention_modules / finetune_mlp_modules constrain where those targets are allowed.
The regression test covers the language-only explicit q_proj case and source-checks that explicit targets are wrapped through get_peft_regex when filters are active.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Refresh MLX smoke clip-config note for leaf_norm default
Trim the 11-line comment block to 5 lines and correct the stale claim
that MLXTrainingConfig defaults to max_grad_value=1.0. The new default
is max_grad_leaf_norm=1.0 (same memory profile as elementwise but
direction-preserving). The smoke still pins max_grad_value=1.0
explicitly to keep the 13-seed pass-rate fixture stable.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Forward max_grad_leaf_norm through the training route and warn when layer filters constrain explicit target_modules for PR #5656
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han-Chen <info@unsloth.ai>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: decide diffusion routing before the SWA resolver
Loading a DiffusionGemma GGUF could fail with "llama-server failed to start. Check that the GGUF file is valid" while llama-diffusion-cli ran the same model fine.
_read_gguf_metadata set self._is_diffusion after calling _resolve_swa_pattern, both inside one try/except. The resolver reaches into transformers/HF, which can raise for an architecture transformers does not know (diffusion-gemma); the shared except then swallowed it and left _is_diffusion False, so the model was routed to plain llama-server instead of the diffusion runner. It only reproduced where the SWA pattern was not already cached/inline.
Set _is_diffusion right after the KV parse loop (before the resolver) so a resolver error can no longer drop the routing, and skip the resolver for diffusion models, which do not use Studio's SWA pattern.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The update banner is driven by update_available from
/api/llama/update-status. Prebuilt (marker) installs decide this via
freshness.is_behind, which is mix-aware: a release at the same upstream
base build but with a new -mix-<sha> suffix counts as behind.
Source-build installs went through _source_build_status, which compared
only the numeric base build (installed_build < latest_build). When a new
prebuilt shared the installed upstream base (our usual mix re-tag at the
same base), it returned update_available=False and the banner never
showed. This is the common macOS case, where a failed prebuilt fetch
falls back to a source build that lacks the mix patches.
Make the source-build path mix-aware to match the marker path: same base
plus a -mix-<sha> tag now offers the update (and displays the mix tag),
a bare same-base rebuild does not, and the downgrade guard and unknown
-version fallback are unchanged.
* Studio: derive training output dir from model basename for local-drive models
A LoRA/QLoRA run started from a model loaded by absolute path (common when
models live on a non-system drive, e.g. G:\modelsAI\...\gemma-4-12B-it) seeded
the default output dir with that full path. resolve_output_dir then raised
"path escapes root ... is not under the studio outputs folder", so training
could not start from a model stored off the system drive.
Add default_run_dir_name(): Hugging Face repo ids keep their namespace
(org/model becomes org_model), while local paths collapse to their final
component so an absolute source path can no longer leak into the output dir.
Use it at the three worker derivation sites and add a regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: cap run dir name length and drop redundant resolve
Apply PR review feedback: length-cap the auto-generated output dir component so an unusually long model name stays under the filesystem name limit, and drop the redundant double resolve_output_dir at the embedding site so all three derivation sites match.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: keep llama-server discovery from crashing on an access-denied candidate
_find_llama_server_binary probed candidates with Path.is_file(), which raises
PermissionError (WinError 5) when a path exists but is momentarily inaccessible
(antivirus lock, an install replace in flight, an elevated-install ACL),
aborting model validation. Treat a denied-but-present path as the real binary
so discovery returns it; absent paths still skip.
* Retry a transiently locked binary instead of returning a denied path
Returning a still-denied path only moved the PermissionError to the next
is_file() (probe_server_capabilities). Retry briefly so a transient lock
clears and discovery returns an accessible path; on a persistent lock return
nothing rather than a path downstream cannot stat.
* Studio: do not fall back to another llama-server when a pinned one is locked
A denied LLAMA_SERVER_PATH made discovery skip the explicit pin and run a
lower-priority managed or PATH binary, so a load could silently use a stale or
incompatible server. Split the probe into a file/absent/denied status: when the
pinned path exists but stays access-denied, warn and stop rather than falling
back to a different executable.
* Studio: never downgrade past a denied pinned or managed llama-server
Extend the no-fallback rule beyond LLAMA_SERVER_PATH: a present-but-denied
UNSLOTH_LLAMA_CPP_PATH or managed ($STUDIO_HOME/llama.cpp, ~/.unsloth/llama.cpp)
binary now reports temporarily-unavailable instead of silently launching a
lower-priority legacy or PATH server. Shared _scan_pinned/_unavailable helpers;
legacy in-tree and PATH stay genuine fallbacks (a denied candidate there just
continues).
* Studio: let diffusion asset lookup use a locked llama-server path for its dir
DiffusionGemma does not run llama-server; _find_diffusion_assets only needs the
install dir to find the adjacent llama-diffusion-gemma-visual-server. The
no-fallback rule returning None on a transiently locked llama-server therefore
hid an available visual-server and raised 'runner not found'. Add an
include_denied option so diffusion lookup gets the locked path (its dir is all
it needs), while inference keeps the no-denied-path, no-downgrade behavior.
* Studio: report a locked llama-server as temporarily unavailable, not missing
When the pinned/managed binary stays access-denied through the retries, discovery
returns None and load_model raised 'binary not found', a terminal error that
points users at reinstalling rather than retrying a transient AV/install lock.
Reuse include_denied to detect the locked path and raise a distinct
temporarily-unavailable, retry message instead.
* Studio: GGUF preflight treats a locked llama-server as present
The pre-download preflight (and so /api/inference/validate) used the default
discovery, which returns None for a transiently access-denied binary, so it
raised 'binary not found' for a binary that merely needs the lock to clear. Use
include_denied so the existence check counts a locked binary as present; the
load itself still reports a still-locked binary as temporarily unavailable.
* fix: extend llama.cpp first-token timeout
* fix: timeout label pluralization
* studio: distinguish llama stream timeout phases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust timeout handling for PR #5841
* Fix lint failure for PR #5841
* Fix/adjust stream timeout handling for PR #5841
* Fix/adjust first token timeout for PR #5841
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust passthrough timeouts for PR #5841
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust preheader stream cancellation for PR #5841
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust timeout PR diff for PR #5841
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust Python 3.9 stream iteration for PR #5841
* Fix first body timeout for PR #5841
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix first token timeout deadlines for PR #5841
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
The studio backend lazily imports unsloth_zoo submodules (export's
llama_cpp, hardware's vllm_utils, mlx, chat_templates). Each imports the
top-level unsloth_zoo, which raises 'Please install Unsloth via pip
install unsloth' unless UNSLOTH_IS_PRESENT is set (normally by import
unsloth). Only hardware.py set it per-site; the rest crash on a clean
install, and it surfaced on Windows. Set it once at backend startup (as
unsloth does on import) so every site, and the DiffusionGemma runner,
works across platforms. Follow-up to #6259 (which covered only the shim
subprocess env).
list_gguf_variants() keys files by _extract_quant_label(), which only captured the base quant token. Repos that ship the same base quant at multiple bits-per-weight (e.g. byteshape/Qwen3.6-35B-A3B-MTP-GGUF with three IQ4_XS files at 3.53/3.97/4.19 bpw) collapsed into a single row and Studio summed their sizes (~48 GB).
Extend the regex to capture an optional trailing -<N>(.<N>)?bpw modifier so each flavor produces a unique label. Round-trips through _find_local_gguf_by_variant and _download_gguf since both sides use the same extractor.
Fixes#5728.
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
A clean install could not run DiffusionGemma: the runner spawns
python -m unsloth_zoo.diffusion_studio.shim, and unsloth_zoo refuses to
import unless UNSLOTH_IS_PRESENT is set (normally by import unsloth). The
shim never imports unsloth, so the subprocess died with
'Please install Unsloth via pip install unsloth!' and the model load
failed with a 500. Set the flag in the runner child env, as unsloth does
on import.
* fix(studio): keep local GGUF vision on llama-server
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): lower local GGUF vision log level
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): find GGUF companions from variant dirs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Studio: account for mmproj VRAM in GGUF fit budget (#5825)
Vision GGUFs load the mmproj projector onto the GPU via --mmproj
alongside the weights, but the context auto-sizing / GPU-selection
budget sized off _get_gguf_size_bytes(model_path), which counts only
the weight file(s). The projector was never added, so the budget was
too optimistic: context got mis-estimated and tight vision loads
spilled to system RAM / OOM'd.
Resolve the launch projector once before GPU selection and fold its
size into the fit budget. The same resolved path feeds both the budget
and the --mmproj launch flag, so the two cannot disagree. The summary
log now reports the projector size separately, keeping "GGUF size"
accurate.
Adds _mmproj_vram_bytes() + unit tests (no GPU / network / subprocess).
* Studio: simplify mmproj summary-log concatenation (#5825)
Address review: the summary log mixed explicit `+` with implicit
f-string concatenation. Extract the optional projector fragment into
`mmproj_note` so the logger.info uses uniform implicit concatenation.
No behavioral change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: trim mmproj VRAM comments
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
DiffusionGemma serves via the visual runner, which streams per-step
canvas frames so the answer resolves live in the bubble. The agentic
tool loop (generate_chat_completion_with_tools) does not forward those
frames, so whenever a tool pill (Search/Code) was on the live canvas
silently vanished while text still streamed. DiffusionGemma is not a
tool-calling target anyway, so report supports_tools=False for it: the
chat always takes the frame-forwarding path, and the Search/Code pills
disable themselves (a local model has no builtin web search either).
Also turn the artifacts canvas on by default for DiffusionGemma so a
full-HTML answer (e.g. a playable game) renders as an interactive
sandboxed card without the user flipping the global artifacts toggle.
* Studio: make project sources work with RAG and polish project UI
Projects had a disabled Sources tab with an Add sources placeholder.
This wires it up end to end on top of the RAG engine:
- Add a project scope to the RAG store, ingestion and retrieval
- New endpoints: POST/GET /api/rag/projects/{id}/documents
- search_knowledge_base resolves kb, project and thread scopes; an
explicit KB stays exclusive, project and thread scopes combine
- Multi-scope search: FTS uses scope IN (...), vec0 KNN runs per
scope and merges by cosine score
- Lazy ALTER TABLE adds documents.project_id on existing databases
- Deleting a project also removes its indexed sources
- Sources tab now uploads with progress chips and drag and drop
- Chats inside a project auto-enable retrieval over project sources
when the project has indexed documents (cached probe, no Docs pill
needed); external providers still never receive rag_scope
UI polish:
- Rounder project cards with folder icon chip and softer shadow
- Project header icon in a rounded chip
- Chats/Sources pills and Add sources button without borders
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: match Add sources button shadow to the chat composer in light mode
* Studio: round project switcher hover pill and pad the folder icon
* Studio: remove border from project sources box
* Studio: grey hover on project cards and menu, move search into header, widen page spacing
* Studio: shorten sources copy, white header pills with composer shadow, fixed-width search, hub-size page headings
* Studio: align project landing blocks to the composer width
* Studio: restore muted background and flat look on projects header controls
* Studio: darker grey hover on project cards in light mode
* Studio: soften project card hover grey
* Studio: keep project card menu button visible while its menu is open
* Studio: drop focus outlines and rings on buttons and clickable icons, keep input focus styles
* Studio: address review feedback on project sources
- Remove uploaded files from disk when a project is deleted, confined
to the uploads root
- 404 project uploads when the project does not exist, matching the KB
endpoint
- Guard lexical search against an empty scope list
- Re-invalidate the project sources probe after uploads and removals
settle so a chat sent mid-upload cannot cache a stale negative
- Keep keyboard focus rings: only mouse focus drops the Tailwind ring,
the browser default outline stays removed
* Studio: add a green New badge to the project Sources tab
* Studio: unify New pills, fully round with soft emerald fill and no border
* Studio: a touch more vertical padding on New pills
* Fix project RAG source edge cases for PR #6205
* Fix duplicate RAG upload cleanup for PR #6205
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: cache MCP tool discovery instead of re-probing every chat send
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Stop re-probing offline/down MCP servers every time
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add tests for mid-probe delete and OAuth cool-off paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Don't cool-off a server edited or deleted mid-probe
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard MCP refresh cache writes
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* feat(studio): add S3 dataset configuration foundation (#4539)
Add foundational types and configuration for S3 bucket dataset loading:
- Add S3Config type to frontend training types
- Add S3Config Pydantic model to backend training models
- Add "s3" as a DatasetSource option
- Add s3Config state and setS3Config action to training config store
- Add i18n translations for S3 configuration (English and Chinese)
This provides the type definitions and UI text for S3 integration.
Full implementation requires boto3 dependency and data loading logic.
Refs: #4539
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Wire S3 config into training pipeline and prevent secrets persistence
- Pass s3_config from request into training_kwargs so it flows to training subprocess
- Add s3Config to NON_PERSISTED_STATE_KEYS to prevent AWS secrets from being
saved to localStorage
Addresses code review feedback on PR #5951.
* Exclude S3 config from database persistence to protect secrets
Filter out s3_config (which contains secret_access_key) from the
config_json stored in training_runs table, preventing AWS credentials
from being persisted to disk.
Addresses P1 security feedback on PR #5951.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Re-raise HTTPException in start_training and defer s3 DatasetSource widening for PR #5951
* Redact s3_config from W&B run config and accept camelCase S3 credential aliases for PR #5951
* feat(studio): implement S3 dataset loading end-to-end
Builds the actual S3 loader on top of the hardened #5951 foundation,
turning the 501-gated scaffold into a working dataset source.
Backend:
- Add core/training/s3_dataset.py: lists and downloads supported dataset
files (parquet/json/jsonl/csv) from an S3 bucket to a temp dir, using
IAM-role or access-key credentials. boto3 is imported lazily (optional dep).
- Wire s3_config into UnslothTrainer.load_and_format_dataset (downloads then
reuses the existing local-file path) and thread it through worker.py.
- Replace the 501 "not implemented" gate with a boto3-availability guard so
S3 works when boto3 is present and fails clearly when it is not.
- Add boto3 to studio.txt requirements.
- Add tests/test_s3_dataset.py (8 tests) covering download/filtering,
collisions, missing-boto3, and S3Config camelCase/IAM validation.
Frontend:
- Widen DatasetSource to include "s3"; add s3_config to the training payload
type and mapper; add an S3 validation branch and selectS3Source store action.
- Add s3-config-form.tsx (bucket/region/prefix/keys/IAM toggle) reusing the
existing studio.dataset.s3.* i18n strings.
- Add a Hugging Face / Local / Amazon S3 source toggle in dataset-section;
the S3 config card replaces the dataset combobox when S3 is selected.
- Fix DatasetPreviewDialog to accept the widened DatasetSource type.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix S3 dataset loader for PR #6222
* Fix S3 dataset edge cases for PR #6222
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix S3 IAM payload handling for PR #6222
* Block multimodal S3 datasets for PR #6222
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Ash <ash@MacBook-Pro.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: override chat template for unsloth/gemma-4-*-GGUF with bundled gemma-4.jinja
The chat templates baked into the shipped unsloth/gemma-4-*-GGUF quants predate
Google's gemma-4 chat-template PR #118 and lack the preserve_thinking flag, so
Studio cannot surface the "Preserve thinking" toggle for Gemma 4. Bundle the updated
template and override the embedded one at llama-server launch via --chat-template-file,
scoped to the gemma-4 GGUF family, so users do not need to re-download any quant.
- Add studio/backend/assets/chat_templates/gemma-4.jinja (PR #118 based;
preserve_thinking defaults false, the one deliberate divergence from upstream).
- Add core/inference/chat_templates.py: gemma-4 GGUF matcher plus an
effective-override resolver (explicit user template still wins).
- Wire the resolver into routes/inference.py ahead of the reload-dedup check and
both load_model calls so the live backend and the incoming request compare against
the same template text (no spurious reloads).
- Default preserve_thinking off in the launch-time chat_template_kwargs so direct
API callers match the UI default.
- Ship the asset via package-data and add unit tests.
* Studio: ship E2B/E4B edge variant of the bundled Gemma 4 template
Google ships two distinct gemma-4 chat templates: E2B and E4B omit the empty
"<|channel>thought<channel|>" block on enable_thinking=false, while the
12b/26B-A4B/31B family emits it (confirmed against google/gemma-4-E2B-it,
-E4B-it, -12b-it, -26B-A4B-it, -31B-it; the two families differ only in that
one block). The single PR #118 based template followed the larger-model
behavior, which is wrong for the E2B/E4B GGUFs this feature most targets.
- Add studio/backend/assets/chat_templates/gemma-4-edge.jinja: identical to
gemma-4.jinja minus the empty-thought-block, matching E2B/E4B behavior.
- Route unsloth/gemma-4-E2B-it-GGUF and -E4B-it-GGUF to the edge template;
12b/26B-A4B/31B keep gemma-4.jinja.
- Extend tests for the edge matcher, per-family routing, and the empty-thought
block difference (off for edge, on for standard).
* Studio: address review feedback on the gemma-4 template override
- Normalize owner-less shorthand model ids in the template matcher: a bare
"gemma-4-E2B-it-GGUF" is canonicalized to "unsloth/" the same way
ModelConfig.from_identifier does, so shorthand loads still get the override
(and the preserve_thinking capability) instead of falling back to the
embedded template.
- Scope the test's module stubs with unittest.mock.patch.dict instead of
sys.modules.setdefault, and only stub deps that are missing, so the global
module registry is not polluted for tests that run afterwards.
- Guard the Jinja render tests with pytest.importorskip("jinja2") so the suite
stays runnable in minimal Studio environments where jinja2 is not present.
- Add tests for shorthand resolution.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address 10-reviewer P1 findings on the gemma-4 template override
- /status no longer surfaces Studio's auto-applied bundled template as a
user-authored chat_template_override. The frontend adopts that field as
editable state and would otherwise re-send the gemma-4 template as an explicit
override for a later, unrelated model. /status now reports None when the live
override equals the model's auto-resolved bundled template.
- When a bundled family template is in effect, strip an inherited
--chat-template-file from llama_extra_args too (not only when the raw request
set chat_template_override). Otherwise a stale inherited template, appended
last, shadows the bundled one while Studio reports the bundled template's
capabilities.
- Write the temp chat-template file as UTF-8 explicitly, and keep the bundled
templates ASCII (replaced em dashes), so non-UTF-8 Windows locales cannot raise
UnicodeEncodeError or emit a mis-encoded template. Added an ASCII guard test.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: serve DiffusionGemma GGUFs with the on-device visual decoder
* Studio: render the DiffusionGemma denoising canvas live in chat with honest stats
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden DiffusionGemma runner resolution (Windows .exe, build/bin lookup, clear stale audio flag, safe PYTHONPATH, Linux-only pdeathsig)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: fix Downloaded model list disappearing and order it by last download
The chat model picker scan for cached GGUF and safetensors models aborted
whenever an auxiliary Hugging Face cache dir (such as ~/.cache/huggingface/hub)
was unreadable, returning an empty list. That hid the Downloaded section and
let already downloaded models appear under Recommended. Isolate each cache
probe so an inaccessible directory is skipped instead of failing the scan.
Also order Downloaded newest-first using cached blob mtimes (multi-quant repos
group by their most recent quant), keep the section visible while searching,
and make the per-quant downloaded check per-snapshot and mmproj aware so a
Recommended quant is never falsely marked downloaded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden gguf-variants scan and dedupe by newest timestamp
Guard f.stat() per file so a broken symlink or unreadable file in a
snapshot no longer aborts the downloaded check early, and match quant
labels case-insensitively. When the same repo is present in multiple
caches with equal size, keep the newest last_modified so Downloaded
ordering reflects the most recent copy.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: apply cache-scan guards to sibling endpoints found in review
Extend the inaccessible-cache guard and mmproj/stat hardening to the
parallel HF cache code paths flagged in review:
- list_local_models and the Hub inventory scan now skip an unreadable
auxiliary cache instead of returning 500.
- The GGUF download-progress endpoint excludes mmproj adapters and
guards f.stat() so one bad file does not zero a repo's progress.
- The offline snapshot scanner guards its is_dir() probes.
- The chat-only picker no longer renders a blank list when a search
matches only cached non-GGUF models.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* install.sh: persist ROCm-on-WSL drop-in even when rocminfo already works
_maybe_bootstrap_rocm_wsl calls _ensure_rocm_probe_env (which exports a
transient HSA_ENABLE_DXG_DETECTION + adds /opt/rocm/bin to PATH on the
installer process) right before the "rocminfo enumerates gfx1151 -> already
set up, return early" gate. On any reinstall over an existing /opt/rocm --
the common case, since the uninstaller keeps shared ROCm userspace but
removes /etc/profile.d/unsloth-rocm-wsl.sh -- that probe env makes rocminfo
succeed, so the gate returns 0 WITHOUT ever persisting the drop-in. The
transient env dies with the installer, so the next login shell (Studio,
llama-server) sees no GPU: torch cuda_avail=False, rocminfo finds nothing,
the llama.cpp ROCm prebuilt segfaults on a GPU it can't reach.
Factor the drop-in writer into _persist_rocm_wsl_dropin() and call it before
the early return so the persistent env is restored whenever librocdxg is
present. Idempotent (only writes when the drop-in is missing), gated on
librocdxg so it never fires on non-WSL/non-ROCDXG hosts, root-writes or
sudo-tees like before. The fast-path branch now reuses the same helper.
Reproduced on gfx1151 (Radeon 8060S) under dash (the curl|sh shell):
before the fix a reinstall left the drop-in absent and torch cuda_avail
False; after, the drop-in is persisted and a fresh login shell reports
cuda_avail True. Verified under both dash and bash, and idempotent on
re-run.
* Studio WSL: load system HIP before a prebuilt's bundled runtime (gfx1151)
The lemonade / published llama.cpp ROCm prebuilts bundle their own HIP
runtime (libamdhip64) built for bare-metal Linux. In WSL the GPU is reached
through the system ROCm's librocdxg bridge over /dev/dxg, which the bundled
runtime cannot drive -- it segfaults on the first GPU call. So:
- install_llama_prebuilt.py: the prebuilt's llama-quantize/llama-server
validation runs with the bundle dir first on LD_LIBRARY_PATH, segfaults
(empty stderr), and the install silently falls back to a CPU source build
(which on this host can't even build for GPU -- hipcc absent). The Strix
Halo WSL user ends up on CPU despite a working GPU.
- llama_cpp.py: even if a GPU prebuilt were kept, the serve-time launcher
put the bundle dir first too, so it would crash at load.
Fix: on a ROCDXG WSL host (gated on /dev/dxg + "microsoft" /proc/version +
a librocdxg-providing /opt/rocm), prepend the system ROCm lib dir to
LD_LIBRARY_PATH so the WSL-capable libamdhip64 + librocdxg load first, while
the bundle still supplies libggml-hip / librocblas with the gfx1151 kernels.
Set HSA_ENABLE_DXG_DETECTION=1 alongside. Added _wsl_system_rocm_lib_dirs()
to both modules (kept identical so a prebuilt that passed install validation
runs the same way at serve time). Strict no-op on bare-metal Linux, NVIDIA,
macOS, and Windows.
Verified on gfx1151 (Radeon 8060S) in WSL (ROCm 7.2.1 + librocdxg, Adrenalin
ROCDXG): before, the lemonade gfx1151 prebuilt segfaulted and the install
fell back to a broken CPU build; after, install_llama_prebuilt validates and
keeps the GPU prebuilt (source=published, prebuilt_fallback_used=False), and
Studio serves Qwen3-1.7B-GGUF at 53 tok/s with the model resident in GPU
memory (llama-server device_info: ROCm0 = AMD Radeon 8060S).
* tests: cover the WSL ROCDXG drop-in + system-HIP-ordering fixes
- _wsl_system_rocm_lib_dirs: no-op without /dev/dxg, on bare-metal Linux,
and on WSL without librocdxg; returns the system lib dir on a ROCDXG WSL
host.
- binary_env: prepends the system ROCm lib dir ahead of the bundle and sets
HSA_ENABLE_DXG_DETECTION on WSL; unchanged on bare-metal Linux.
- install.sh: _persist_rocm_wsl_dropin exists, is gated on librocdxg, and the
rocminfo-already-works early return calls it before returning.
- llama_cpp.py: the serve-time launcher prepends the WSL rocm dirs before the
bundle dir (mirrors binary_env).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten WSL ROCDXG fix comments (no logic change)
Condense the drop-in / system-HIP-ordering comments and docstrings added in
this PR. Verified comment-only via AST parse + py_compile + sh/bash -n, the
308-test rocm_support suite, and a dash functional re-run of the bootstrap
(drop-in still persisted, env still set).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio/responses): forward chat_template_kwargs enable_thinking to chat request
The /v1/responses translation in _build_chat_request dropped
chat_template_kwargs (e.g. {"enable_thinking": true}) sent via the
Responses extra-body, so reasoning control was silently ignored.
Lift enable_thinking onto the typed ChatCompletionRequest field,
mirroring openai_chat_completions, so both the non-streaming and
streaming Responses pass-through paths honor it.
Fixes#6198
Signed-off-by: Tai An <antai12232931@outlook.com>
* Fix/adjust Responses reasoning for PR #6202
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust reasoning none for PR #6202
* Fix/adjust structured reasoning for PR #6202
* Fix/adjust responses reasoning review findings for PR #6202
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust responses reasoning follow-ups for PR #6202
* Fix/adjust think parsing gate for PR #6202
---------
Signed-off-by: Tai An <antai12232931@outlook.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: Add Tensor-Parallel llama.cpp support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden Tensor-Parallel fallback and GPU selection
* Studio: reconcile split-mode extras and harden tensor-split planning
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reconcile split-mode extras in backend duplicate-load guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: preserve inherited non-tensor split modes on reload
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor cancellation in tensor fallback, preserve tensor mode on rollback, and don't raise an explicit small context
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reconcile split-mode in reload check and strip it on tensor downgrade
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Strip --tensor-split alongside --split-mode so inherited ratios don't override the tensor planner
An inherited or stale --tensor-split in llama_extra_args was appended after
Studio's computed --tensor-split and won last in llama.cpp, re-introducing the
asymmetric-GPU OOM tensor mode is meant to prevent. Group -ts/--tensor-split
into the split-mode shadow set so it is stripped on inherit and on the layer
fallback; parse_split_mode_override still keys on the mode value only.
* Drop quantized KV for the tensor attempt and report native max context
Tensor mode aborts on a quantized KV cache, so a user with q8_0/q4_1 etc. who
enabled Tensor Parallelism silently fell back to layer split. Clear the cache
type (and strip inherited/explicit --cache-type) for the tensor attempt only;
the layer fallback re-runs with tensor off and keeps the user's choice.
Also report max_available_ctx from the native context, not an explicit small
-c, so the context slider no longer warns too early in tensor mode.
* Reconcile inherited split-mode extras in the already-loaded check
When a same-model load omitted llama_extra_args, the tensor comparison resolved
the raw (None) request and treated an inherited --split-mode tensor server as a
mismatch, forcing a needless reload. Compare using the stored extras stripped
the same way the reload strips them.
* Pass tensor_parallel through compare-mode loads
The generalized compare path loaded each GGUF without tensor_parallel, so
compare ran layer split even with the toggle on and left the settings sheet
stale. Send the toggle and hydrate the loaded state from the response, matching
the main chat and recipe load paths.
* Add --tensor-parallel flag to unsloth studio run
The headless one-liner could only reach tensor mode by passing --split-mode
tensor as a raw llama.cpp extra. Add a first-class --tensor-parallel/
--no-tensor-parallel option that sets the tensor_parallel field on the
/api/inference/load payload, forwarded through the studio-venv re-exec like the
other polarity flags. Matches the web UI toggle and the API field.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* fix: allow absolute save_directory in export paths to prevent cross-drive copy failures
The GGUF export pipeline (and all other export flows) forced every
save_directory through resolve_export_dir(), which always resolved
the path under exports_root() — typically ~/.unsloth/studio/exports/
on the system drive (C: on Windows).
When a user selected an output directory on a different drive (E:):
1. The absolute path was rejected at the Pydantic validator level.
2. Even if it got through, resolve_export_dir would re-resolve it
under C:\Users\.unsloth\studio\exports\.
3. After GGUF conversion completed on E:, the relocation step would
try to move/copy the finished files to C:, causing:
- WinError 17 (cross-drive move failure when shutil.move falls
through to a cross-filesystem copy)
- WinError 112 (disk full on C:)
Fix both layers:
- _validate_save_directory: accept absolute paths (they represent an
explicit user choice of output location).
- resolve_export_dir, resolve_output_dir, resolve_tensorboard_dir:
return absolute paths as-is instead of forcing them under the
default root. Keep the existing safety checks (null bytes, '..'
segments) and fall through to resolve_under_root for relative paths.
Fixes: https://github.com/unslothai/unsloth/issues/6082
* refactor: centralize user path validation into _resolve_user_path helper
Addresses code review feedback: the null-byte, '..', and absolute-path
checks were duplicated across resolve_output_dir, resolve_export_dir,
and resolve_tensorboard_dir. Extract a single _resolve_user_path helper
that all three delegate to.
No behavioral change — pure consolidation.
* fix: address code review — contain destructive cleanup and scope absolute paths
Address all review feedback from gemini-code-assist:
1. P1: destructive subdirectory cleanup (export_gguf)
The flattening loop in export_gguf previously rmtree'd every
subdirectory under abs_save_dir. When targeting an existing user
directory on a different drive (#6082), this could nuke unrelated
subdirectories. Now snapshot existing subdirectories before the
export and only clean up dirs created during this run.
2. P2: keep scan/read endpoints contained
Only resolve_export_dir accepts absolute paths (export is a write
path where user picks location). Reverted resolve_output_dir and
resolve_tensorboard_dir to use resolve_under_root directly — these
are used by scan/read/training endpoints that must stay contained
under their respective roots.
3. Centralization feedback
Removed the _resolve_user_path helper since it's no longer needed
with the narrowed scope. resolve_export_dir has the absolute path
logic inline with a clear docstring.
* fix: skip pre-existing subdirs in GGUF flatten loop and clean stale export intermediates
Two issues caught in code review (chatgpt-codex-connector):
1. The flattening loop moved ALL .gguf files from ALL subdirectories
into abs_save_dir, including pre-existing unrelated user subdirs.
Now skip pre-existing subdirs entirely unless they are known
export-owned intermediates (model/, model_gguf/).
2. After a failed export, known export-owned subdirectories (model/,
model_gguf/) were snapshotted as pre-existing on retry and never
cleaned up. These are now always cleaned up regardless, since they
are known intermediates created by the export pipeline.
* fix: separate write vs read export paths, guard same-dir rmtree
Three issues caught in code review (chatgpt-codex-connector):
1. P1: scan endpoint containment
resolve_export_dir was changed to accept absolute paths, but it's
also used by scan/read endpoints (routes/models.py) that must stay
contained under exports_root(). Split into:
- resolve_export_dir: contained, used by scans
- resolve_export_write_dir: accepts absolute paths, used by export
backend only
2. P1: same-directory rmtree
When a non-PEFT checkpoint's gguf_dir resolves to the same path as
abs_save_dir (user selected the checkpoint's gguf output as their
export directory), shutil.rmtree(gguf_dir) would delete the user's
chosen output directory. Now skip relocation when both paths resolve
to the same location.
3. P1: pre-existing subdir flatten loop
Reverted _EXPORT_OWNED_SUBDIRS logic — 'model/' and 'model_gguf/'
are common directory names in shared model folders and don't prove
export ownership. Now only clean up subdirs that didn't exist before
the export started.
* fix: remove dead _EXPORT_OWNED_SUBDIRS and fix _export_details for absolute paths
Two fixes from review comments:
1. Remove unused _EXPORT_OWNED_SUBDIRS declaration (leftover from
previous iteration that was intentionally removed).
2. _export_details now returns the full absolute path when the export
target is outside exports_root(), instead of truncating to basename.
Users who export to E:\ can now see the full destination path in
the success dialog.
* fix: use unique tmp dir for GGUF intermediates to avoid overwriting user dirs
When exporting to an absolute destination that already contains a
model/ subdirectory (e.g. a shared models folder), the hard-coded
model_save_path would overwrite files in that unrelated directory.
Use _tmp_model_<uuid> as the intermediate path instead, so user
directories are never touched. The tmp dir is created as a new subdir
of abs_save_dir and cleaned up by the flatten loop after GGUF files
are relocated.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix GGUF local export paths for PR #6088
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address GGUF export follow-ups for PR #6088
* Clean GGUF temp dirs on export failure for PR #6088
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust export path tests for PR #6088
* Fix/adjust export path review findings for PR #6088
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust home export path handling for PR #6088
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
The --local / GGUF-only install resolves its Python deps from
no-torch-runtime.txt, installed with --no-deps. That file was missing the
RAG group that studio.txt declares (sqlite-vec, pymupdf, python-docx), so a
fresh `unsloth studio` came up with RAG disabled: rag_db.py cannot import
sqlite_vec and logs "RAG unavailable: sqlite-vec extension could not be
loaded", and the knowledge-base routes return 503. python-docx was also
absent, so DOCX ingestion failed.
Add the three RAG store and document-parsing deps with the same pins as
studio.txt so knowledge bases work out of the box on the no-torch path.
sentence-transformers (dense embeddings) was already present.
The post-install path cleared only the in-memory freshness caches and then
re-primed the 24h disk cache with a forced GitHub refresh. When that refresh
cannot reach GitHub, latest_published_release falls back to the last-good disk
value, so a still-fresh same-base mix tag cached before the swap (b9596-mix-aaa
vs the just-installed b9596-mix-bbb) is replayed and the prebuilt reads as
behind, surfacing a false update banner that points back at the build that was
just replaced.
Give reset_caches a drop_disk option and use it on the update path: with the
disk cache gone, an offline post-install refresh leaves latest as None and the
banner fails open (off) instead of lingering on the stale same-base value. The
no-arg form stays in-memory only. Adds regression coverage for the drop, the
default no-op, and the fail-open vs stale-replay contrast.
* Studio: tune llama.cpp env for data-center GPUs
Detect datacenter/professional NVIDIA GPUs at llama-server launch and set
the llama.cpp env flags that help them, gated so consumer GeForce, AMD/ROCm,
CPU and macOS are never touched.
- GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F=1 for any DC GPU (FP32 cuBLAS
accumulation). On a B200 this is ~0% throughput cost with identical
perplexity (7.3230 wikitext-2-raw, baseline and on), where on GeForce the
same flag costs real throughput, hence the gate.
- GGML_CUDA_P2P=1 and CUDA_SCALE_LAUNCH_QUEUES=4x for multi-GPU DC boxes.
Benchmarked on 6x B200: +33-51% prompt processing on tensor (row) split and
+8-16% on the default pipeline (layer) split, with no regression on the
other split or on token generation.
Detection uses torch device names (A100/A30/H100/H200/H800/GH200/B200/GB200/
GB300/L40/L4/RTX PRO 6000/RTX 6000 Ada). A mixed box with one consumer GPU in
the selection is treated as non-DC. All writes are setdefault so a user value
always wins, and UNSLOTH_DISABLE_DC_TUNING=1 turns the whole thing off.
37 unit tests cover detection, multi-GPU gating, user-override precedence, the
disable flag and fail-open on error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix data-center GPU detection false positives and physical-id mapping
Two issues in the data-center llama.cpp env tuning gate:
- _is_datacenter_gpu matched the marker allowlist as unbounded substrings, so
workstation/laptop parts "NVIDIA RTX A1000" and "NVIDIA RTX A3000" matched
"a100"/"a30" and were wrongly tuned as data-center GPUs (forcing FP32 cuBLAS
accumulation and the multi-GPU env, which carry a real cost on those cards).
Switch to a word-boundary regex.
- gpu_indices carries physical GPU ids (translated from torch ordinals by
_get_gpu_free_memory via CUDA_VISIBLE_DEVICES), but they were passed straight
into torch.cuda.get_device_properties, which expects mask-relative ordinals.
On a masked host (e.g. CUDA_VISIBLE_DEVICES=4,5,6,7) a selection like [4,5]
fell out of range and silently dropped the tuning, and on a mixed mask it could
probe the wrong GPU class. Build a physical-id to device-name map mirroring
_get_gpu_free_memory, then look up the selection by physical id.
Add regression tests for the A1000/A3000 false positives and for masked-host
physical-id selection (reordered and mixed-class masks included).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten data-center GPU tuning comments
Comment-only pass over the DC tuning block and its tests: shorten verbose
docstrings/comments, drop ones that restate the code, collapse multi-line
blocks. Keep the load-bearing rationale (physical-id vs ordinal mapping, the
word-boundary reason, the B200 benchmark numbers). No code change: verified
with comment_tools.py check --strip-docstrings (code unchanged, comments only).
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix step count mismatch when sequence packing is enabled
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Emit a single step-0 progress event and guard applyStatus totalSteps
Merge the two consecutive _update_progress calls before train() so the
step-0 gate in _on_progress fires once instead of twice, avoiding a
duplicate startup event and a null-metric step-0 row in training_metrics.
Apply the same positive-number guard to applyStatus that applyProgress
uses, so a stale or startup status poll can no longer overwrite the
packed step count with 0 or replace it with a stale total.
* Log debug message when train_dataset length is unavailable
The TypeError fallback for length-less datasets (e.g. streaming
IterableDataset) was silent, leaving no trace that the step estimate
came from the raw dataset rather than the packed one.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
IOReport energy counters can reset (sleep/wake, power gating), making a poll
delta negative. Return None for a negative total so the monitor shows -- for
that poll instead of a bogus negative wattage; it self-corrects next poll.
* Studio: Add inline confirmation (Allow/Always allow/Deny) for tool calls
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix race in tool-call confirmation gate
* Studio: gate built-in tool calls and harden the confirmation handshake
The Allow / Always allow / Deny controls only lived in the fallback tool
card, but the built-in tools (web search, python, terminal, code
execution, image generation) render with their own components and so
never showed the buttons. Those calls paused after tool_start with no way
to approve them, hanging until the 1 hour timeout. Only MCP tools, which
use the fallback renderer, actually worked.
Render the controls for every tool card by wrapping each registered tool
component (and the fallback) in thread.tsx with a shared
ToolConfirmationControls, so the gate applies uniformly.
Also make the handshake robust:
- The gate keys on a per-call approval_id minted by the backend and
echoed in tool_start, instead of session_id alone, so a stale or
concurrent confirmation can no longer resolve the wrong call.
- The approval slot is registered before tool_start is yielded, closing
the race where a fast click or an auto "Always allow" could reach the
backend before the waiter existed.
- The frontend resolves with the same session id the request was sent
with (plus the approval_id), fixing the new-thread mismatch where the
confirmation targeted a different session than the blocked stream.
- The confirm endpoint returns {resolved}; the UI keeps the buttons and
shows a retry hint until the backend confirms a match, instead of
hiding them on a failed or mistargeted post.
- The gate runs after the disabled-tool and duplicate-call checks, so a
call that will not execute is not put up for approval. A denied call is
still excluded from duplicate detection, so re-issuing and approving it
works.
- "Always allow" is scoped per session to match the backend gate.
Add backend tests for the approval registry, the SSE no-deadlock
handshake, and the loop integration (allow, deny, disabled, duplicate,
re-issue after deny).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Move "Confirm tool calls" to the Tools section
* Studio: Keep tool group open while a tool call awaits confirmation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix tool confirmation session scope for PR #5869
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix confirmation follow-ups for PR #5869
* Apply pre-commit formatting for PR #5869
* Fix confirmation cleanup for PR #5869
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden confirmation lookups for PR #5869
* Studio: make the tool-call confirmation decision immutable
resolve_tool_decision accepted a second confirmation for the same approval_id
and overwrote slot["decision"] in the window before the waiter reads it and
pops the slot, so a duplicate or out-of-order POST could flip an Allow to Deny
(and returned a misleading resolved:true). Reject once the slot's event is
already set so the first decision wins. Adds a regression test.
* Fix/adjust tool confirmations for PR #5869
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: show Apple GPU temperature and power in the GPU monitor (macOS)
The GPU monitor on Apple Silicon always showed -- for Temperature and
Power: the MLX branch of get_gpu_utilization() hardcoded None because
ioreg's AGXAccelerator PerformanceStatistics carries neither metric.
Add utils/hardware/apple.py, mirroring macmon's no-sudo approach:
- Temperature: average of the AppleSMC "Tg*" float keys via the
AppleSMCKeysEndpoint user client (ctypes/IOKit, macOS 14+).
- Power: IOReport "Energy Model" group, "GPU Energy" channels; each
poll diffs the energy counter against the previous poll's sample, so
the value is the average wattage over the polling window. The first
poll only sets the baseline and returns None.
Both readers latch to None on first failure and never raise, so
non-Mac platforms and locked-down hosts keep the previous behavior.
* Sample IOReport with the subscribed channels descriptor for PR #6187
IOReportCreateSubscription writes the channel descriptor that later samples
must use; sampling with the original requested group can return no Energy
Model entries on hosts that normalize the channel set, leaving power_draw_w
null after the baseline. Use the subscribed descriptor (matching macmon) and
fall back to the requested channels if the OS leaves it unset.
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: danielhanchen <23090290+danielhanchen@users.noreply.github.com>
Daniel Han