* fix(studio): forward OpenAI tools/tool_choice to llama-server (#4999)
Studio's /v1/chat/completions silently stripped standard OpenAI `tools`
and `tool_choice` fields, so clients using standard function calling
(opencode, Claude Code, Cursor, Continue, ...) never got structured
tool_calls back. Adds a client-side pass-through path mirroring the
existing Anthropic /v1/messages flow: when `tools` is present without
Studio's `enable_tools` shorthand, the request is forwarded to
llama-server verbatim so the client sees native id, finish_reason
("tool_calls"), delta.tool_calls, and accurate usage tokens.
Also wires Anthropic tool_choice forwarding: /v1/messages previously
accepted tool_choice on the request model but silently dropped it with
a warning. Translate the four Anthropic shapes to OpenAI format and
forward them so agentic clients can actually enforce tool use.
- ChatCompletionRequest: add tools, tool_choice, stop; extra="allow"
- ChatMessage: accept role="tool", optional tool_call_id / tool_calls /
name; content is now optional (assistant with only tool_calls)
- routes/inference.py: _openai_passthrough_stream /
_openai_passthrough_non_streaming helpers, routing branch in
openai_chat_completions, vision+tools via content-parts injection
- _build_passthrough_payload: tool_choice parameter (default "auto")
- anthropic_compat: anthropic_tool_choice_to_openai() translator
- tests/test_openai_tool_passthrough.py: Pydantic + translator unit tests
- tests/test_studio_api.py: 5 new E2E tests (non-stream, stream,
multi-turn, OpenAI SDK, Anthropic tool_choice=any regression)
* fix(studio): surface httpx transport errors from OpenAI passthrough
When the managed llama-server subprocess crashes mid-request, the
async pass-through helpers in routes/inference.py used to return a
bare 500 (non-streaming) or an "An internal error occurred" SSE chunk
(streaming) because _friendly_error only recognized the sync path's
"Lost connection to llama-server" substring -- httpx transport
failures (ConnectError / ReadError / RemoteProtocolError /
ReadTimeout) stringify differently and fell through to the generic
case.
- _friendly_error: map any httpx.RequestError subclass to the same
"Lost connection to the model server" message the sync chat path
emits. Placed before the substring heuristics so the streaming path
automatically picks it up via its existing except Exception catch.
- _openai_passthrough_non_streaming: wrap the httpx.AsyncClient.post
in a try/except httpx.RequestError and re-raise as HTTPException
502 with the friendly detail.
- tests/test_openai_tool_passthrough.py: new TestFriendlyErrorHttpx
class pinning the mapping for ConnectError, ReadError,
RemoteProtocolError, ReadTimeout, and confirming non-httpx paths
(context-size heuristic, generic fallback) are unchanged.
* fix(studio): close aiter_bytes/aiter_lines explicitly in passthroughs
The httpcore asyncgen cleanup fix in 5cedd9a5 is incomplete on Python
3.13 + httpcore 1.0.x: it switched to manual client/response lifecycle
but still used anonymous `async for raw_line in resp.aiter_lines():`
patterns in all three streaming paths. Python's async for does NOT
auto-close the iterator on break/return, so the aiter_lines /
aiter_bytes async generator remains alive, reachable only from the
surrounding coroutine frame. Once `_stream()` returns the frame is
GC'd and the orphaned asyncgen is finalized on a LATER GC pass in a
DIFFERENT asyncio task, where httpcore's
HTTP11ConnectionByteStream.aclose() enters anyio.CancelScope.__exit__
with a mismatched task and prints "Exception ignored in: <async
generator>" / "async generator ignored GeneratorExit" / "Attempted
to exit cancel scope in a different task" to the server log.
User observed this on /v1/messages after successful (status 200)
requests, with the traceback pointing at HTTP11ConnectionByteStream
.__aiter__ / .aclose inside httpcore.
Fix: save resp.aiter_lines() / resp.aiter_bytes() as a variable and
explicitly `await iter.aclose()` in the finally block BEFORE
resp.aclose() / client.aclose(). This closes the asyncgen inside the
current task's event loop, so the internal httpcore byte stream is
cleaned up before Python's asyncgen GC hook has anything orphaned to
finalize. Each aclose is wrapped in try/except Exception so nested
anyio cleanup noise can't bubble out.
Applied to all three streaming passthrough paths:
- _anthropic_passthrough_stream (/v1/messages client-side tool path)
- _openai_passthrough_stream (/v1/chat/completions client-side tool
path, new in this PR)
- openai_completions (/v1/completions bytes proxy from PR #4956)
* fix(studio): default ChatCompletionRequest.stream to false per OpenAI spec
OpenAI's /v1/chat/completions spec defaults `stream` to false, so
clients that omit the field (naive curl, minimal integrations) expect
a single JSON response back. Studio was defaulting to true, silently
switching those clients into SSE and breaking any parser that didn't
also handle streaming. ResponsesRequest and AnthropicMessagesRequest
already default to false correctly; only ChatCompletionRequest was
wrong.
Studio's own frontend always sets `stream` explicitly on every
chat-adapter / chat-api / runtime-provider call site, so the flip has
no UI impact. SDK users (OpenAI Python/JS SDK, opencode, Claude Code,
Cursor, Continue) also always pass `stream` explicitly, so they're
unaffected. The only clients feeling the change are raw-curl users
who were relying on the wrong default -- those get the correct OpenAI
behavior now.
Added a regression test pinning the default so it can't silently
flip back.
* fix(studio): reject images in OpenAI tool passthrough for text-only GGUFs
The new tool passthrough branch runs before _extract_content_parts,
skipping the existing not is_vision guard. Requests combining tools
with an image on a text-only tool-capable GGUF were forwarded to
llama-server, producing opaque upstream errors instead of the
pre-existing clear 400. Restore the guard inline at the dispatch
point, checking both legacy image_base64 and inline image_url parts.
* fix(studio): require tool_call_id on role=tool chat messages
Enforce the OpenAI spec rule that role="tool" messages must carry a
tool_call_id. Without it, upstream backends cannot associate a tool
result with the assistant's prior tool_calls entry and the request
fails in non-obvious ways through the passthrough path. Reject at the
request boundary with a 422 instead.
* fix(studio): harden OpenAI tool passthrough validation and error surfacing
Three related fixes called out by the PR review:
1. Preserve upstream status codes in the streaming passthrough. The
httpx request is now dispatched before the StreamingResponse is
constructed. Non-200 upstream responses and httpx RequestError
transport failures raise HTTPException with the real status
instead of being buried inside a 200 SSE error frame, so OpenAI
SDK clients see APIError/BadRequestError/... as expected.
2. Require non-empty content on user/system/tool messages. Per the
OpenAI spec, content may only be omitted on assistant messages
that carry tool_calls; enforce that at the request boundary so
malformed messages never reach the passthrough path.
3. Role-constrain tool-call metadata. tool_calls is only valid on
role=assistant, tool_call_id and name only on role=tool. Without
this, a user/system message with tool_calls would flip the
passthrough branch on and be forwarded to llama-server, surfacing
as an opaque upstream error.
* fix(studio): normalize image mode and passthrough JSON verbatim
Two Gemini-code-assist review findings on PR #5099:
1. Unconditionally convert decoded images to RGB before PNG encoding.
The prior code only handled RGBA, letting CMYK/I/F images crash
at img.save(format="PNG") and surface as opaque 400s. Applied to
both the passthrough helper and the non-passthrough GGUF path
that originally carried this pattern, keeping the two sites in
sync.
2. Return the upstream JSON body as raw bytes via Response rather
than parse-then-re-serialize with JSONResponse. Matches the
passthrough helper's "verbatim" contract and drops a redundant
round-trip.
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: Ollama support, recommended folders, Custom Folders UX polish
Backend:
- Add _scan_ollama_dir that reads manifests/registry.ollama.ai/library/*
and creates .gguf symlinks under <ollama_dir>/.studio_links/ pointing
at the content-addressable blobs, so detect_gguf_model and llama-server
-m work unchanged for Ollama models
- Filter entries under .studio_links from the generic models/hf/lmstudio
scanners to avoid duplicate rows and leaked internal paths in the UI
- New GET /api/models/recommended-folders endpoint returning LM Studio
and Ollama model directories that currently exist on the machine
(OLLAMA_MODELS env var + standard paths, ~/.lmstudio/models, legacy
LM Studio cache), used by the Custom Folders quick-add chips
- detect_gguf_model now uses os.path.abspath instead of Path.resolve so
the readable symlink name is preserved as display_name (e.g.
qwen2.5-0.5b-Q4_K_M.gguf instead of sha256-abc...)
- llama-server failure with a path under .studio_links or .cache/ollama
surfaces a friendlier message ("Some Ollama models do not work with
llama.cpp. Try a different model, or use this model directly through
Ollama instead.") instead of the generic validation error
Frontend:
- ListLabel supports an optional leading icon and collapse toggle; used
for Downloaded (download icon), Custom Folders (folder icon), and
Recommended (star icon)
- Custom Folders header gets folder icon on the left, and +, search,
and chevron buttons on the right; chevron uses ml-auto so it aligns
with the Downloaded and Recommended chevrons
- New recommended folder chips render below the registered scan folders
when there are unregistered well-known paths; one click adds them as
a scan folder
- Custom folder rows that are direct .gguf files (Ollama symlinks) load
immediately via onSelect instead of opening the GGUF variant expander
(which is for repos containing multiple quants, not single files)
- When loading a direct .gguf file path, send max_seq_length = 0 so the
backend uses the model's native context instead of the 4096 chat
default (qwen2.5:0.5b now loads at 32768 instead of 4096)
- New listRecommendedFolders() helper on the chat API
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: log silent exceptions and support read-only Ollama dirs
Replace silent except blocks in _scan_ollama_dir and the
recommended-folders endpoint with narrower exception types plus debug
or warning logs, so failures are diagnosable without hiding signal.
Add _ollama_links_dir helper that falls back to a per-ollama-dir hashed
namespace under Studio's own cache (~/.unsloth/studio/cache/ollama_links)
when the Ollama models directory is read-only. Common for system installs
at /usr/share/ollama/.ollama/models and /var/lib/ollama/.ollama/models
where the Studio process has read but not write access. Previously the
scanner returned an empty list in that case and Ollama models would
silently not appear.
The fallback preserves the .gguf suffix on symlink names so
detect_gguf_model keeps recognising them. The prior "raw sha256 blob
path" fallback would have missed the suffix check and failed to load.
* Address review: detect mmproj next to symlink target for vision GGUFs
Codex P1 on model_config.py:1012: when detect_gguf_model returns the
symlink path (to preserve readable display names), detect_mmproj_file
searched the symlink's parent directory instead of the target's. For
vision GGUFs surfaced via Ollama's .studio_links/ -- where the weight
file is symlinked but any mmproj sidecar lives next to the real blob
-- mmproj was no longer detected, so the model was misclassified as
text-only and llama-server would start without --mmproj.
detect_mmproj_file now adds the resolved target's parent to the scan
order when path is a symlink. Direct (non-symlink) .gguf paths are
unchanged, so LM Studio and HF cache layouts keep working exactly as
before. Verified with a fake layout reproducing the bug plus a
regression check on a non-symlink LM Studio model.
* Address review: support all Ollama namespaces and vision projector layers
- Iterate over all directories under registry.ollama.ai/ instead of
hardcoding the "library" namespace. Custom namespaces like
"mradermacher/llama3" now get scanned and include the namespace
prefix in display names, model IDs, and symlink names to avoid
collisions.
- Create companion -mmproj.gguf symlinks for Ollama vision models
that have an "application/vnd.ollama.image.projector" layer, so
detect_mmproj_file can find the projector alongside the model.
- Extract symlink creation into _make_symlink helper to reduce
duplication between model and projector paths.
* Address review: move imports to top level and add scan limit
- Move hashlib and json imports to the top of the file (PEP 8).
- Remove inline `import json as _json` and `import hashlib` from
function bodies, use the top-level imports directly.
- Add `limit` parameter to `_scan_ollama_dir()` with early exit
when the threshold is reached.
- Pass `_MAX_MODELS_PER_FOLDER` into the scanner so it stops
traversing once enough models are found.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: Windows fallback, all registry hosts, collision safety
_make_link (formerly _make_symlink):
- Falls back to os.link() hardlink when symlink_to() fails (Windows
without Developer Mode), then to shutil.copy2 as last resort
- Uses atomic os.replace via tmp file to avoid race window where the
.gguf path is missing during rescan
Scanner now handles all Ollama registry layouts:
- Uses rglob over manifests/ instead of hardcoding registry.ollama.ai
- Discovers hf.co/org/repo:tag and any other host, not just library/
- Filenames include a stable sha1 hash of the manifest path to prevent
collisions between models that normalize to the same stem
Per-model subdirectories under .studio_links/:
- Each model's links live in their own hash-keyed subdirectory
- detect_mmproj_file only sees the projector for that specific model,
not siblings from other Ollama models
Friendly Ollama error detection:
- Now also matches ollama_links/ (the read-only fallback cache path)
and model_identifier starting with "ollama/"
Recommended folders:
- Added os.access(R_OK | X_OK) check so unreadable system directories
like /var/lib/ollama/.ollama/models are not advertised as chips
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: filter ollama_links from generic scanners
The generic scanners (models_dir, hf_cache, lmstudio) already filter
out .studio_links to avoid duplicate Ollama entries, but missed the
ollama_links fallback cache directory used for read-only Ollama
installs. Add it to the filter.
* Address review: idempotent link creation and path-component filter
_make_link:
- Skip recreation when a valid link/copy already exists (samefile or
matching size check). Prevents blocking the model-list API with
multi-GB copies on repeated scans.
- Use uuid4 instead of os.getpid() for tmp file names to avoid race
conditions from concurrent scans.
- Log cleanup errors instead of silently swallowing them.
Path filter:
- Use os.sep-bounded checks instead of bare substring match to avoid
false positives on paths like "my.studio_links.backup/model.gguf".
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: drop copy fallback, targeted glob, robust path filter
_make_link:
- Drop shutil.copy2 fallback -- copying multi-GB GGUFs inside a sync
API request would block the backend. Log a warning and skip the
model when both symlink and hardlink fail.
Scanner:
- Replace rglob("*") with targeted glob patterns (*/*/* and */*/*/*)
to avoid traversing unrelated subdirectories in large custom folders.
Path filter:
- Use Path.parts membership check instead of os.sep substring matching
for robustness across platforms.
Scan limit:
- Skip _scan_ollama_dir when _generic already fills the per-folder cap.
* Address review: sha256, top-level uuid import, Path.absolute()
- Switch hashlib.sha1 to hashlib.sha256 for path hashing consistency.
- Move uuid import to the top of the file instead of inside _make_link.
- Replace os.path.abspath with Path.absolute() in detect_gguf_model
to match the pathlib style used throughout the codebase.
* Address review: fix stale comments (sha1, rglob, copy fallback)
Update three docstrings/comments that still referenced the old
implementation after recent changes:
- sha1 comment now says "not a security boundary" (no hash name)
- "rglob" -> "targeted glob patterns"
- "file copies as a last resort" -> removed (copy fallback was dropped)
* Address review: fix stale links, support all manifest depths, scope error
_make_link:
- Drop size-based idempotency shortcut that kept stale links after
ollama pull updates a tag to a same-sized blob. Only samefile()
is used now -- if the link doesn't point at the exact same inode,
it gets replaced.
Scanner:
- Revert targeted glob back to rglob so deeper OCI-style repo names
(5+ path segments) are not silently skipped.
Ollama error:
- Only show "Some Ollama models do not work with llama.cpp" when the
server output contains GGUF compatibility hints (key not found,
unknown architecture, failed to load). Unrelated failures like
OOM or missing binaries now show the generic error instead of
being misdiagnosed.
---------
Co-authored-by: Daniel Han <info@unsloth.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Studio: add folder browser modal for Custom Folders
The Custom Folders row in the model picker currently only accepts a
typed path. On a remote-served Studio (Colab, shared workstation) that
means the user has to guess or paste the exact server-side absolute
path. A native browser folder picker can't solve this: HTML
`<input type="file" webkitdirectory>` hides the absolute path for
security, and the File System Access API (Chrome/Edge only) returns
handles rather than strings, neither of which the server can act on.
This PR adds a small in-app directory browser that lists paths on the
server and hands the chosen string back to the existing
`POST /api/models/scan-folders` flow.
## Backend
* New endpoint `GET /api/models/browse-folders`:
* `path` query param (expands `~`, accepts relative or absolute; empty
defaults to the user's home directory).
* `show_hidden` boolean to include dotfiles/dotdirs.
* Returns `{current, parent, entries[], suggestions[]}`. `parent` is
null at the filesystem root.
* Immediate subdirectories only (no recursion); files are never
returned.
* `entries[].has_models` is a cheap hint: the directory looks like it
holds models if it is named `models--*` (HF hub cache layout) or
one of the first 64 children is a .gguf/.safetensors/config.json/
adapter_config.json or another `models--*` subfolder.
* Sort order: model-bearing dirs, then plain, then hidden; case-
insensitive alphabetical within each bucket.
* Suggestions auto-populate from HOME, the HF cache root, and any
already-registered scan folders, deduplicated.
* Error surface: 404 for missing path, 400 for non-directory, 403 on
permission errors. Auth-required like the other models routes.
* New Pydantic schemas `BrowseEntry` and `BrowseFoldersResponse` in
`studio/backend/models/models.py`.
## Frontend
* New `FolderBrowser` component
(`studio/frontend/src/components/assistant-ui/model-selector/folder-browser.tsx`)
using the existing `Dialog` primitive. Features:
* Clickable breadcrumb with a `..` row for parent navigation.
* Quick-pick chips for the server-provided suggestions.
* `Show hidden` checkbox.
* In-flight fetch cancellation via AbortController so rapid
navigation doesn't flash stale results.
* Badges model-bearing directories inline.
* `chat-api.ts` gains `browseFolders(path?, showHidden?)` and matching
types.
* `pickers.tsx` adds a folder-magnifier icon next to the existing `Add`
button. Opening the browser seeds it with whatever the user has
already typed; confirming fills the text input, leaving the existing
validation and save flow unchanged.
## What it does NOT change
* The existing text-input flow still works; the browser is additive.
* No new permissions or escalation; the endpoint reads only directories
the server process is already allowed to read.
* No model scanning or filesystem mutation happens from the browser
itself -- it just returns basenames for render.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: cap folder-browser entries and expose truncated flag
Pointing the folder browser at a huge directory (``/usr/lib``,
``/proc``, or a synthetic tree with thousands of subfolders) previously
walked the whole listing and stat-probed every child via
``_looks_like_model_dir``. That is both a DoS shape for the server
process and a large-payload surprise for the client.
Introduce a hard cap of 2000 subdirectory entries and a
``truncated: bool`` field on the response. The frontend renders a small
hint below the list when it fires, prompting the user to narrow the
path. Below-cap directories are unchanged.
Verified end-to-end against the live backend with a synthetic tree of
2050 directories: response lands at 2000 entries, ``truncated=true``,
listing finishes in sub-second time (versus tens of seconds if we were
stat-storming).
* Studio: suggest LM Studio / Ollama dirs + 2-level model probe
Three improvements to the folder-browser, driven by actually dropping
an LM Studio-style install (publisher/model/weights.gguf) into the
sandbox and walking the UX:
## 1. Quick-pick chips for other local-LLM tools
`well_known_model_dirs()` (new) returns paths commonly used by
adjacent tools. Only paths that exist are returned so the UI never
shows dead chips.
* LM Studio current + legacy roots + user-configured
`downloadsFolder` from its `settings.json` (reuses the existing
`lmstudio_model_dirs()` helper).
* Ollama: `$OLLAMA_MODELS` env override, then `~/.ollama/models`,
`/usr/share/ollama/.ollama/models`, and `/var/lib/ollama/.ollama/models`
(the systemd-service install path surfaced in the upstream "where is
everything?" issue).
* Generic user-choice locations: `~/models`, `~/Models`.
Dedup is stable across all sources.
## 2. Two-level model-bearing probe
LM Studio and Ollama both use `root/publisher/model/weights.gguf`.
The previous `has_models` heuristic only probed one level, so the
publisher dir (whose immediate children are model dirs, not weight
files) was always marked as non-model-bearing. Pulled the direct-
signal logic into `_has_direct_model_signal` and added a grandchild
probe so the classic layout is now recognised.
Still O(PROBE^2) worst-case, still returns immediately for
`models--*` names (HF cache layout) and for any direct weight file.
## 3. model_files_here hint on response body
A leaf model dir (just GGUFs, no subdirs) previously rendered as
`(empty directory)` in the modal, confusing users into thinking the
folder wasn't scannable. Added a `model_files_here` count on the
response (capped at 200) and a small hint row in the modal: `N model
files in this folder. Click "Use this folder" to scan it.`
## Verification
Simulated an LM Studio install by downloading the real 84 MB
`unsloth/SmolLM2-135M-Instruct-Q2_K.gguf` into
`~/.lmstudio/models/unsloth/SmolLM2-135M-Instruct-GGUF/`. Confirmed
end-to-end:
* Home listing suggests `~/.lmstudio/models` as a chip.
* Browsing `~/.lmstudio/models` flags `unsloth` (publisher) as
`has_models=true` via the 2-level probe.
* Browsing the publisher flags `SmolLM2-135M-Instruct-GGUF` (model
dir) as `has_models=true`.
* Browsing the model dir returns empty entries but
`model_files_here=1`, and the frontend renders a hint telling the
user it is a valid target.
* Studio: one-click scan-folder add + prominent remove + plain search icon
Three small Custom Folders UX fixes after real-use walkthrough:
* **One-click add from the folder browser**. Confirming `Use this
folder` now submits the path directly to
`POST /api/models/scan-folders` instead of just populating the text
input. `handleAddFolder` takes an optional explicit path so the
submit lands in the same tick as `setFolderInput`, avoiding a
state-flush race. The typed-path + `Add` button flow is unchanged.
* **Prominent remove X on scan folders**. The per-folder delete
button was `text-muted-foreground/40` and hidden entirely on
desktop until hovered (`md:opacity-0 md:group-hover:opacity-100`).
Dropped the hover-only cloak, bumped color to `text-foreground/70`,
added a red hover/focus background, and sized the icon up from
`size-2.5` to `size-3`. Always visible on every viewport.
* **Plain search icon for the Browse button**. `FolderSearchIcon`
replaced with `Search01Icon` so it reads as a simple "find a
folder" action alongside the existing `Add01Icon`.
* Studio: align Custom Folders + and X buttons on the same right edge
The Custom Folders header used `px-2.5` with a `p-0.5` icon button,
while each folder row used `px-3` with a `p-1` button. That put the
X icon 4px further from the right edge than the +. Normalised both
rows to `px-2.5` with `p-1` so the two icons share a column.
* Studio: empty-state button opens the folder browser directly
The first-run empty state for Custom Folders was a text link reading
"+ Add a folder to scan for local models" whose click toggled the
text input. That's the wrong default: a user hitting the empty state
usually doesn't know what absolute path to type, which is exactly
what the folder browser is for.
* Reword to "Browse for a models folder" with a search-icon
affordance so the label matches what the click does.
* Click opens the folder browser modal directly. The typed-path +
Add button flow is still available via the + icon in the
section header, so users who know their path keep that option.
* Slightly bump the muted foreground opacity (70 -> hover:foreground)
so the button reads as a primary empty-state action rather than a
throwaway hint.
* Studio: Custom Folders header gets a dedicated search + add button pair
The Custom Folders section header had a single toggle button that
flipped between + and X. That put the folder-browser entry point
behind the separate empty-state link. Cleaner layout: two buttons in
the header, search first, then add.
* Search icon (left) opens the folder browser modal directly.
* Plus icon (right) toggles the text-path input (unchanged).
* The first-run empty-state link is removed -- the two header icons
cover both flows on every state.
Both buttons share the same padding / icon size so they line up with
each other and with the per-folder remove X.
* Studio: sandbox folder browser + bound caps + UX recoveries
PR review fixes for the Custom Folders folder browser. Closes the
high-severity CodeQL path-traversal alert and addresses the codex /
gemini P2 findings.
Backend (studio/backend/routes/models.py):
* New _build_browse_allowlist + _is_path_inside_allowlist sandbox.
browse_folders now refuses any target that doesn't resolve under
HOME, HF cache, Studio dirs, registered scan folders, or the
well-known third-party model dirs. realpath() is used so symlink
traversal cannot escape the sandbox. Also gates the parent crumb
so the up-row hides instead of 403'ing.
* _BROWSE_ENTRY_CAP now bounds *visited* iterdir entries, not
*appended* entries. Dirs full of files (or hidden subdirs when
show_hidden is False) used to defeat the cap.
* _count_model_files gets the same visited-count fix.
* PermissionError no longer swallowed silently inside the
enumeration / counter loops -- now logged at debug.
Frontend (folder-browser.tsx, pickers.tsx, chat-api.ts):
* splitBreadcrumb stops mangling literal backslashes inside POSIX
filenames; only Windows-style absolute paths trigger separator
normalization. The Windows drive crumb value is now C:/ (drive
root) instead of C: (drive-relative CWD-on-C).
* browseFolders accepts and forwards an AbortSignal so cancelled
navigations actually cancel the in-flight backend enumeration.
* On initial-path fetch error, FolderBrowser now falls back to HOME
instead of leaving the modal as an empty dead end.
* When the auto-add path (one-click "Use this folder") fails, the
failure now surfaces via toast in addition to the inline
paragraph (which is hidden when the typed-input panel is closed).
* Studio: rebuild browse target from trusted root for CodeQL clean dataflow
CodeQL's py/path-injection rule kept flagging the post-validation
filesystem operations because the sandbox check lived inside a
helper function (_is_path_inside_allowlist) and CodeQL only does
intra-procedural taint tracking by default. The user-derived
``target`` was still flowing into ``target.exists`` /
``target.is_dir`` / ``target.iterdir``.
The fix: after resolving the user-supplied ``candidate_path``,
locate the matching trusted root from the allowlist and rebuild
``target`` by appending each individually-validated segment to
that trusted root. Each segment is rejected if it isn't a single
safe path component (no separators, no ``..``, no empty/dot).
The downstream filesystem ops now operate on a Path constructed
entirely from ``allowed_roots`` (trusted) plus those validated
segments, so CodeQL's dataflow no longer sees a tainted source.
Behavior is unchanged for all valid inputs -- only the
construction of ``target`` is restructured. Live + unit tests
all pass (58 selected, 7 deselected for Playwright env).
* Studio: walk browse paths from trusted roots for CodeQL
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ubuntu <ubuntu@h100-8-cheapest.us-east5-a.c.unsloth.internal>
* fix: support GGUF variant selection for non-suffixed repos
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden GGUF detection across cached models and picker flows
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* chore: use shared GGUF picker helper for search rows
* fix: avoid mixed cache duplication and preserve GGUF fallback detection
* fix: unify GGUF cache matching and merge picker hints
* fix: normalize local GGUF matching across picker and model config
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: robust cached-gguf classification + hint-aware click routing
- _repo_gguf_size_bytes: treat size_on_disk=None as 0 and dedupe fallback
by commit_hash so partial/interrupted downloads don't TypeError out of
sum() and wipe the entire cached list.
- list_cached_gguf / list_cached_models: narrow per-repo try/except so
one malformed repo no longer poisons the whole response.
- handleModelClick: route through isKnownGgufRepo instead of the
suffix-only isGgufRepo, so non-suffixed GGUF repos still open the
variant expander from every call site.
- Replace the modelIsGgufById/resultIsGgufById Maps with Sets of known
GGUF ids to stop conflating "no hint" with "known not-GGUF".
- Make HfModelResult.isGguf required (it is always set in makeMapModel).
- Add regression tests for the None size case, mixed-repo inclusion in
cached-gguf, and per-repo error isolation.
* fix: exclude mmproj from GGUF classification and case-normalize hint lookups
- _repo_gguf_size_bytes now filters mmproj vision-adapter files so
safetensors+mmproj.gguf repos stay on the cached-models path and
non-GGUF rows no longer show zero pickable variants. A vision-capable
GGUF repo (main weight + mmproj adapter) still classifies as GGUF and
reports the main weight size.
- modelGgufIds / resultGgufIds now key on lowercased ids and
isKnownGgufRepo lowercases its lookup, so store and HF-search ids
that differ only by casing still match the same GGUF hint.
- New regression tests: mmproj-only repo excluded from cached-gguf,
same repo included in cached-models, vision-capable repo still
classified as GGUF with correct size.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* Studio: live model-load progress + rate/ETA on download and load
Two UX fixes for the opaque multi-minute wait between clicking Load
and being able to chat, visible most clearly on large MoE GGUFs like
MiniMax-M2.7 (131 GB of weights on a 97 GB GPU):
1. **Model-load phase is now observable.** The existing chat flow
transitions the toast to "Starting model..." as soon as the
download hits 100%, then shows a spinner with no other feedback
until llama-server reports healthy. For a 130 GB model that spinner
freezes for five-plus minutes while the kernel pages shards into
the page cache. A new `GET /api/inference/load-progress` endpoint
samples `/proc/<pid>/status VmRSS` on the llama-server subprocess
against the sum of shard file sizes on disk, so the UI can render
a real bar plus rate / ETA during that window.
2. **Rate and ETA on downloads and loads.** Both the chat toast and
the training-start overlay used to show a static pair of numbers
(for example "15.4 of 140.8 GB"). A rolling 15-second window over
the existing byte-series now surfaces "85.3 MB/s, 24m 23s left"
beside that pair. The estimator is shared between the download
and load phases so the numbers don't reset when the phase flips.
Also fixes a pre-existing assignment bug uncovered while wiring this
up: `load_model` was storing the caller's `gguf_path` kwarg into
`self._gguf_path`, which is `None` on the HF-download code path. The
resolved on-disk path (`model_path`) is what llama-server actually
mmaps; downstream consumers need that. No existing reader used
`_gguf_path`, so this is a correctness fix for the new endpoint.
- Backend: `LlamaCppBackend.load_progress()`, `GET /api/inference/load-progress`, `LoadProgressResponse` Pydantic model.
- Frontend: `useTransferStats` hook, `formatRate` / `formatEta` helpers, `getLoadProgress` client, rewired chat toast and `DownloadRow` in the training overlay.
- Tests: `studio/backend/tests/test_llama_cpp_load_progress.py` covers empty states, mmap phase, ready phase, sharded total aggregation, missing gguf_path, and unreadable /proc (7 cases). `tsc -b` and `vite build` on the frontend both clean.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: stream export worker output into the export dialog
The Export Model dialog only showed a spinner on the "Exporting..."
button while the worker subprocess was doing the actual heavy lifting.
For Merged to 16bit and GGUF / Llama.cpp exports this meant several
minutes (or more, for large models) of opaque silence, with no way to
tell whether save_pretrained_merged, convert_hf_to_gguf.py, or
llama-quantize was making progress.
This adds a live terminal-style output panel inside the export dialog,
rendered just above the Cancel / Start Export buttons and scrollable
with auto-follow-tail. It shows stdout and stderr from both the worker
process itself and any child process it spawns (GGUF converter,
llama-quantize), coloured by stream.
Backend
- core/export/worker.py: new _setup_log_capture(resp_queue) installed
before LogConfig.setup_logging. It saves the original stdout/stderr
fds, creates pipes, os.dup2's the write ends onto fds 1 and 2 (so
every child process inherits the redirected fds), and spins up two
daemon reader threads. Each thread reads bytes from a pipe, echoes
them back to the original fd (so the server console keeps working),
splits on \n and \r, and forwards each line to the resp queue as
{"type":"log","stream":"stdout|stderr","line":...,"ts":...}.
PYTHONUNBUFFERED=1 is set so nested Python converters flush
immediately.
- core/export/orchestrator.py:
- Thread-safe ring buffer (collections.deque, maxlen 4000) with a
monotonically increasing seq counter. clear_logs(),
get_logs_since(cursor), get_current_log_seq(), is_export_active().
- _wait_response handles rtype == "log" by appending to the buffer
and continuing the wait loop. Status messages are also surfaced as
a "status" stream so users see high level progress alongside raw
subprocess output.
- load_checkpoint, _run_export, and cleanup_memory now wrap their
bodies with the existing self._lock (previously unused), clear the
log buffer at the start of each op, and flip _export_active in a
try/finally so the SSE endpoint can detect idle.
- routes/export.py:
- Wrapped every sync orchestrator call (load_checkpoint,
cleanup_memory, export_merged_model, export_base_model,
export_gguf, export_lora_adapter) in asyncio.to_thread so the
FastAPI event loop stays free during long exports. Without this
the new SSE endpoint could not be served concurrently with the
blocking export POST.
- New GET /api/export/logs/stream SSE endpoint. Honors
Last-Event-ID and a since query param for reconnect, emits log /
heartbeat / complete / error events, uses the id field to carry
the log seq so clients can resume cleanly. On first connect
without an explicit cursor it starts from the current seq so old
lines from a previous run are not replayed.
Frontend
- features/export/api/export-api.ts: streamExportLogs() helper that
authFetches the SSE endpoint and parses id / event / data fields
manually (same pattern as streamTrainingProgress in train-api.ts).
- features/export/components/export-dialog.tsx:
- Local useExportLogs(exporting) hook that opens the SSE stream on
exporting transitions to true, accumulates up to 4000 lines in
component state, and aborts on cleanup.
- New scrollable output panel rendered above DialogFooter, only
shown for Merged to 16bit and GGUF / Llama.cpp (LoRA adapter is
a fast disk write with nothing to show). Dark terminal styling
(bg-black/85, emerald text, rose for stderr, sky for status),
max-height 14rem, auto-scrolls to the bottom on new output but
stops following if the user scrolls up. A small streaming / idle
indicator is shown next to the panel title.
- DialogContent widens from sm:max-w-lg to sm:max-w-2xl when the
output panel is visible so the logs have room to breathe.
Verified
- Python smoke test (tests/smoke_export_log_capture.py): spawns a
real mp.get_context("spawn") process, installs _setup_log_capture,
confirms that parent stdout prints, parent stderr prints, AND a
child subprocess invoked via subprocess.run (both its stdout and
stderr) are all captured in the resp queue. Passes.
- Orchestrator log helpers tested in isolation: _append_log,
get_logs_since (with and without a cursor), clear_logs not
resetting seq so reconnecting clients still progress. Passes.
- routes.export imports cleanly in the studio venv and /logs/stream
shows up in router.routes.
- bun run build: tsc -b plus vite build, no TypeScript errors.
No existing export behavior is changed. If the subprocess, the SSE
endpoint, or the frontend hook fails, the export itself still runs to
completion the same way it did before, with or without logs visible.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* export dialog: trim bootstrap noise, scope logs per screen, show realpath
Several follow-ups to the live export log work:
1. Worker bootstrap noise (transformers venv activation, Unsloth banner,
"Top GGUF/hub models" lists, vision detection, 2k-step weight load
bar) is dropped from the export-dialog stream. A threading.Event
gate in worker.py defaults closed and only opens once _handle_export
actually starts; until then the reader thread still echoes lines to
the saved console fd for debugging but does not push them onto the
resp_queue. The orchestrator already spawns a fresh subprocess for
every checkpoint load, so the gate is naturally reset between runs.
2. tqdm in non-tty mode defaults to a 10s mininterval, which makes
multi-step bars look frozen in the panel. Set TQDM_MININTERVAL=0.5
in the worker env so any tqdm-driven progress emits more often.
3. The dialog's useExportLogs hook now also clears its line buffer
when exportMethod or open changes, so re-opening the dialog into a
different action's screen no longer shows the previous action's
saved output. A useElapsedSeconds tick + "Working Xs" badge in the
log header gives users a visible sign that long single-step phases
(cache copies, GGUF conversion) are still running when no new lines
are arriving.
4. ExportBackend.export_{merged,base,gguf,lora} now return
(success, message, output_path); the worker forwards output_path on
each export_*_done response, the orchestrator's _run_export passes
it to routes/export.py, which surfaces it via
ExportOperationResponse.details.output_path. The dialog's Export
Complete screen renders the resolved on-disk realpath under "Saved
to" so users can find their exported model directly.
* fix(cli): unpack 3-tuple return from export backend
ExportOrchestrator.export_{merged,base,gguf,lora} now return
(success, message, output_path) so the studio dialog can show
the on-disk realpath. The CLI still unpacked 2 values, so every
`unsloth export --format ...` crashed with ValueError before
reporting completion. Update the four call sites and surface
output_path via a "Saved to:" echo.
* fix(studio): anchor export log SSE cursor at run start
The export dialog SSE defaulted its cursor to get_current_log_seq()
at connect time, so any line emitted between the POST that kicks
off the export and the client opening the stream was buffered with
seqs 1..k and then skipped (seq <= cursor). Long-running exports
looked silent during their first seconds.
Snapshot _log_seq into _run_start_seq inside clear_logs() and
expose it via get_run_start_seq(). The SSE default cursor now uses
that snapshot, so every line emitted since the current run began
is reachable regardless of when the client connects. Old runs
still can't leak in because their seqs are <= the snapshot.
* fix(studio): reconnect export log SSE on stream drop
useExportLogs launched streamExportLogs once per exporting
transition and recorded any drop in .catch(). Long GGUF exports
behind a proxy with an idle kill-timeout would silently lose the
stream for the rest of the run even though the backend already
supports Last-Event-ID resume. The "retry: 3000" directive emitted
by the backend is only meaningful to native EventSource; this
hook uses a manual fetch + ReadableStream parse so it had no
effect.
Wrap streamExportLogs in a retry loop that tracks lastSeq from
ExportLogEvent.id and passes it as since on reconnect. Backoff is
exponential with jitter, capped at 5s, reset on successful open.
The loop stops on explicit backend `complete` event or on effect
cleanup.
* fix(studio): register a second command so Typer keeps `export` as a subcommand
The CLI export unpacking tests wrap `unsloth_cli.commands.export.export`
in a fresh Typer app with a single registered command. Typer flattens a
single-command app into that command, so the test's
`runner.invoke(cli_app, ["export", ckpt, out, ...])` treats the leading
`"export"` token as an unexpected extra positional argument -- every
parametrized case failed with:
Got unexpected extra argument (.../out)
Register a harmless `noop` second command so Typer preserves subcommand
routing and the tests actually exercise the 3-tuple unpack path they
were written to guard.
Before: 4 failed
After: 4 passed
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: studio-install <studio@local.install>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
* studio: show HF model download progress in training start overlay
During the training setup phase, the overlay only displayed a static
"Loading model..." line while model weights were being downloaded from
Hugging Face. On slow connections this looked like the app had frozen.
This adds a small self-contained progress block inside the existing
TrainingStartOverlay that polls the existing
GET /api/models/download-progress endpoint and renders a Progress bar
with bytes downloaded, total bytes, and percent complete.
Notes:
- Frontend only change. No backend, worker, SSE, or runtime store edits.
- Reuses the existing getDownloadProgress client wrapper and the
existing /api/models/download-progress endpoint that already scans
the HF blob cache for completed and .incomplete files.
- selectedModel is read directly from useTrainingConfigStore inside the
overlay, so no prop drilling and live-training-view.tsx is unchanged.
- Polling runs at 1500 ms and is gated on the HF repo regex
(^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$), the same regex the backend uses,
so local paths and empty form state never hit the endpoint.
- Polling stops once progress reaches 1.0 so the bar can stay at 100
until the overlay hides on the first training step.
- Network errors are silently swallowed, matching the chat side flow
(the bar simply freezes at the last value).
- When downloadedBytes is 0 the block is hidden entirely, so cached
models do not flash a progress bar.
- When the HF API cannot determine the total size, the block falls
back to "X downloaded" with no percent and no bar.
Verified with bun run build (tsc -b plus vite build, no TypeScript
errors).
* training overlay: track dataset download + show on-disk realpath
Adds a dedicated "Downloading dataset..." section to the training-start
overlay alongside the existing model-weights one, so an HF dataset that
is downloading mid-startup is no longer mislabeled as model weights or
hidden entirely. The new GET /api/datasets/download-progress endpoint
mirrors /api/models/download-progress against the datasets-- prefix in
HF_HUB_CACHE.
Both endpoints now also return cache_path, the resolved on-disk
realpath of the snapshot directory (or the cache repo root if no
snapshot is materialized yet). The overlay surfaces this under each
download row so users can immediately see where the model and dataset
landed without digging through server logs.
The frontend's existing useModelDownloadProgress hook is generalized
to a single useHfDownloadProgress(repoId, fetcher) hook that the
model and dataset variants both delegate to, keeping polling, gating,
and completion semantics in one place.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: Polish training start overlay download progress UI (#4957)
* studio: polish training start overlay download progress visuals
* Fix formatCachePath cross-platform support and redundant sizeLabel
- Extend formatCachePath regex to also shorten macOS /Users/<user> paths to ~
- Suppress sizeLabel when no byte info is available (cachePath-only state),
since the "Preparing" badge already conveys the status
* Fix misleading status badge when download total is unknown
- Hide badge when totalBytes is 0 but downloadedBytes > 0, since we cannot
determine if the download is still in progress or already complete (happens
when HF size metadata lookup fails for gated/private repos)
- Keep "Preparing" badge for the zero-bytes cachePath-only state
- Add Windows native path shortening to formatCachePath (C:\Users\<name>)
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
---------
Co-authored-by: studio-install <studio@local.install>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Show non exported models in chat UI
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Distinguish b/w LoRa and full fine tune saves. Cleanup
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* Studio: add API key authentication for programmatic access
External users want to hit the Studio API (chat completions with tool
calling, training, export, etc.) without going through the browser
login flow. This adds sk-unsloth- prefixed API keys that work as a
drop-in replacement for JWTs in the Authorization: Bearer header.
Backend:
- New api_keys table in SQLite (storage.py)
- create/list/revoke/validate functions with SHA-256 hashed storage
- API key detection in _get_current_subject before the JWT path
- POST/GET/DELETE /api/auth/api-keys endpoints on the auth router
Frontend:
- /api-keys page with create form, one-time key reveal, keys table
- API Keys link in desktop and mobile navbar
- Route registered with requireAuth guard
Zero changes to any existing route handler -- every endpoint that uses
Depends(get_current_subject) automatically works with API keys.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use actual origin in API key usage examples
The examples on /api-keys were hardcoded to localhost:8888 which is
wrong for remote users. Use window.location.origin so the examples
show the correct URL regardless of where the user is connecting from.
* Add `unsloth studio run` CLI command for one-liner model serving
Adds a `run` subcommand that starts Studio, loads a model, creates an
API key, and prints a ready-to-use curl command -- similar to
`ollama run` or `vllm serve`.
Usage: unsloth studio run -m unsloth/Qwen3-1.7B-GGUF --gguf-variant UD-Q4_K_XL
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add end-to-end tests for `unsloth studio run` and API key usage
Tests the 4 usage examples from the API Keys page:
1. curl basic (non-streaming) chat completions
2. curl streaming (SSE) chat completions
3. OpenAI Python SDK streaming completions
4. curl with tools (web_search + python)
Also tests --help output, invalid key rejection, and no-key rejection.
All 7 tests pass against Qwen3-1.7B-GGUF.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add /v1/completions, /v1/embeddings, /v1/responses endpoints and --parallel support
- llama_cpp.py: accept n_parallel param, pass to llama-server --parallel
- run.py: plumb llama_parallel_slots through to app.state
- inference.py: add /completions and /embeddings as transparent proxies to
llama-server, add /responses as application-level endpoint that converts
to ChatCompletionRequest; thread n_parallel through load_model
- studio.py: set llama_parallel_slots=4 for `unsloth studio run` path
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make /v1/responses endpoint match OpenAI Responses API format
The existing /v1/responses shim returned Chat Completions format, which
broke OpenAI SDK clients using openai.responses.create(). This commit
replaces the endpoint with a proper implementation that:
- Returns `output` array with `output_text` content parts instead of
`choices` with `message`
- Uses `input_tokens`/`output_tokens` instead of `prompt_tokens`/
`completion_tokens` in usage
- Sets `object: "response"` and `id: "resp_..."`
- Emits named SSE events for streaming (response.created,
response.output_text.delta, response.completed, etc.)
- Accepts all OpenAI Responses API fields (tools, store, metadata,
previous_response_id) without erroring -- silently ignored
- Maps `developer` role to `system` and `input_text`/`input_image`
content parts to the internal Chat format
Adds Pydantic schemas for request/response models and 23 unit tests
covering schema validation, input normalisation, and response format.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add Anthropic-compatible /v1/messages endpoint (#4981)
* Add Anthropic-compatible /v1/messages endpoint with tool support
Translate Anthropic Messages API format to/from internal OpenAI format
and reuse the existing server-side agentic tool loop. Supports streaming
SSE (message_start, content_block_delta, etc.) and non-streaming JSON.
Includes offline unit tests and e2e tests in test_studio_run.py.
* Add enable_tools, enabled_tools, session_id to /v1/messages endpoint
Support the same shorthand as /v1/chat/completions: enable_tools=true
with an optional enabled_tools list uses built-in server tools without
requiring full Anthropic tool definitions. session_id is passed through
for sandbox isolation. max_tokens is now optional.
* Strip leaked tool-call XML from Anthropic endpoint content
Apply _TOOL_XML_RE to content events in both streaming and
non-streaming tool paths, matching the OpenAI endpoint behavior.
* Emit custom tool_result SSE event in Anthropic stream
Adds a non-standard tool_result event between the tool_use block close
and the next text block, so clients can see server-side tool execution
results. Anthropic SDKs ignore unknown event types.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Split /v1/messages into server-side and client-side tool paths
enable_tools=true runs the existing server-side agentic loop with
built-in tools (web_search/python/terminal). A bare tools=[...] field
now triggers a client-side pass-through: client-provided tools are
forwarded to llama-server and any tool_use output is returned to the
caller with stop_reason=tool_use for client execution.
This fixes Claude Code (and any Anthropic SDK client) which sends
tools=[...] expecting client-side execution but was previously routed
through execute_tool() and failing with 'Unknown tool'.
Adds AnthropicPassthroughEmitter to convert llama-server OpenAI SSE
chunks into Anthropic SSE events, plus unit tests covering text
blocks, tool_use blocks, mixed, stop reasons, and usage.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix httpcore GeneratorExit in /v1/messages passthrough stream
Explicitly aclose aiter_lines() before the surrounding async with
blocks unwind, mirroring the prior fix in external_provider.py
(a41160d3) and cc757b78's RuntimeError suppression.
* Wire stop_sequences through /v1/messages; warn on tool_choice
Plumb payload.stop_sequences to all three code paths (server-side
tool loop, no-tool plain, client-side passthrough) so Anthropic SDK
clients setting stop_sequences get the behavior they expect. The
llama_cpp backend already accepted `stop` on both generate_chat_
completion and generate_chat_completion_with_tools; the Anthropic
handler simply wasn't passing it.
tool_choice remains declared on the request model for Anthropic SDK
compatibility (the SDK often sets it by default) but is not yet
honored. Log a structured warning on each request carrying a non-
null tool_choice so the silent drop is visible to operators.
* Wire min_p / repetition_penalty / presence_penalty through /v1/messages
Align the Anthropic endpoint's sampling surface with /v1/chat/completions.
Adds the three fields as x-unsloth extensions on AnthropicMessagesRequest
and threads them through all three code paths: server-side tool loop,
no-tool plain, and client-side passthrough.
The passthrough builder emits "repeat_penalty" (not "repetition_penalty")
because that is llama-server's field name; the backend methods already
apply the same rename internally.
* Fix block ordering and prev_text reset in non-streaming tool path
_anthropic_tool_non_streaming was building the response by appending
all tool_use blocks first, then a single concatenated text block at
the end — losing generation order and merging pre-tool and post-tool
text into one block. It also never reset prev_text between synthesis
turns, so the first N characters of each post-tool turn were dropped
(where N = length of the prior turn's final cumulative text).
Rewrite to build content_blocks incrementally in generation order,
matching the streaming emitter's behavior: deltas within a turn are
merged into the trailing text block, tool_use blocks interrupt the
text sequence, and prev_text is reset on tool_end so turn N+1 diffs
against an empty baseline.
Caught by gemini-code-assist[bot] review on #4981.
* Make test_studio_run.py e2e tests pytest-compatible
Add a hybrid session-scoped studio_server fixture in conftest.py that
feeds base_url / api_key into the existing e2e test functions. Three
invocation modes are now supported:
1. Script mode (unchanged) — python tests/test_studio_run.py
2. Pytest + external server — point at a running instance via
UNSLOTH_E2E_BASE_URL / UNSLOTH_E2E_API_KEY env vars, no per-run
GGUF load cost
3. Pytest + fixture-managed server — pytest drives _start_server /
_kill_server itself via --unsloth-model / --unsloth-gguf-variant,
CI-friendly
The existing _start_server / _kill_server helpers and main() stay
untouched so the script entry point keeps working exactly as before.
Test function signatures are unchanged — the (base_url, api_key)
parameters now resolve via the new fixtures when running under
pytest.
* Rename test_studio_run.py -> test_studio_api.py
The file is entirely about HTTP API endpoint testing (OpenAI-compatible
/v1/chat/completions, Anthropic-compatible /v1/messages, API key auth,
plus a CLI --help sanity check on the command that runs the API). None
of its tests cover training, export, chat-UI, or internal-Python-API
concerns.
The old name misleadingly suggested "tests for the unsloth studio run
CLI subcommand" — the new name reflects the actual scope.
Updates:
- git mv the file (rename tracked, history preserved)
- Rewrite opening docstring to state the API surface focus and call
out what is explicitly out of scope
- Update all 4 Usage-block path references to the new filename
- LOG_FILE renamed to test_studio_api.log
- conftest.py fixture import rewritten from test_studio_run to
test_studio_api, plus 7 docstring/comment references updated
No functional changes to test logic, signatures, or main().
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix httpcore asyncgen cleanup in /v1/messages and /v1/completions
The earlier fix in 985e92a9 was incomplete: it closed aiter_lines()
explicitly but still used `async with httpx.AsyncClient()` /
`async with client.stream()` inside the generator. When the generator
is orphaned (e.g. client disconnects mid-stream and Starlette drops
the StreamingResponse iterator without explicitly calling aclose()),
Python's asyncgen finalizer runs the cleanup in a DIFFERENT task than
the one that originally entered the httpx context managers. The
`async with` exits then trigger httpcore's HTTP11ConnectionByteStream
.aclose(), which enters anyio.CancelScope.__exit__ with a mismatched
task and raises RuntimeError("Attempted to exit cancel scope in a
different task"). That error escapes any user-owned try/except
because it happens during GC finalization.
Replace `async with` with manual client/response lifecycle in both
/v1/messages passthrough and /v1/completions proxy. Close the
response and client in a finally block wrapped in
`try: ... except Exception: pass`. This suppresses RuntimeError (and
other Exception subclasses) from the anyio cleanup noise while
letting GeneratorExit (a BaseException, not Exception) propagate
cleanly so the generator terminates as Python expects.
Traceback observed in user report:
File ".../httpcore/_async/connection_pool.py", line 404, in __aiter__
yield part
RuntimeError: async generator ignored GeneratorExit
...
File ".../anyio/_backends/_asyncio.py", line 455, in __exit__
raise RuntimeError(
RuntimeError: Attempted to exit cancel scope in a different task
* Expand unsloth studio run banner with SDK base URL and more curl examples
Add an explicit "OpenAI / Anthropic SDK base URL" line inside the info
box so SDK users don't accidentally copy the bare server URL (without
/v1) into their OpenAI/Anthropic SDK constructors and hit 404s.
Replace the single /v1/chat/completions curl example with three
labeled blocks: chat/completions, Anthropic /messages, and OpenAI
Responses. The Anthropic example includes max_tokens (Anthropic SDKs
require it even though Studio accepts None).
All examples derived from a computed sdk_base_url so the /v1 prefix
stays in sync if the public path ever changes.
* Hash API keys with HMAC-SHA256 + persistent server secret
Stores the HMAC secret in a new app_secrets singleton table. Fixes
CodeQL py/weak-sensitive-data-hashing alert on storage.py:74-76,
394-395. Refresh tokens stay on plain SHA-256 (unchanged _hash_token)
so existing user sessions survive upgrade — API keys are new on this
branch so there is no migration.
* Use PBKDF2 for API key hashing per CodeQL recommendation
HMAC-SHA256 was still flagged by py/weak-sensitive-data-hashing.
Switch to hashlib.pbkdf2_hmac, which is in CodeQL's recommended
allowlist (Argon2/scrypt/bcrypt/PBKDF2). Persistent server-side
salt stays in app_secrets for defense-in-depth. 100k iterations to
match auth/hashing.py's password hasher.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
* feat: inject local model provider into recipe jobs via JWT
* feat: auto-generate JWT for local model providers in recipes
* feat: add is_local flag to model provider config types and utils
* fix(studio): skip endpoint validation for local providers
* feat(studio): add local/external model source toggle to provider dialog
* feat(studio): thread localProviderNames through model config dialog chain
* feat(studio): show 'Local model (Chat)' label for local model_provider configs
* fix: hardcode loopback for local endpoint, clear stale creds on toggle
* fix: document TOCTOU/JWT rotation, add deferred import comments, fix is_local serialization
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): clear stale local model state on provider toggle and validation
* fix(studio): override empty local endpoint in validation and skip model gate for unused providers
* fix(studio): resolve loopback port from app.state, clear stale local provider fields, sync model id on toggle
Address review feedback on the local-model-provider flow:
- Backend (jobs.py): _resolve_local_v1_endpoint now reads the actual bound
port from app.state.server_port (set in run.py after binding) instead of
parsing it out of request.base_url, which is wrong behind any reverse
proxy or non-default port. The two duplicated urlparse blocks are gone.
- Backend (jobs.py): defensively pop api_key_env, extra_headers, extra_body
from local providers so a previously external provider that flipped to
local cannot leak invalid JSON or rogue auth headers into the local /v1
call. Also dedupe the post-loop assignment and tighten the local-name
intersection so empty names cannot match.
- Backend (jobs.py): hoist datetime and urllib.parse imports to the top
import block for consistency with the rest of the file.
- Backend (run.py): expose the bound port on app.state.server_port after
the uvicorn server is constructed.
- Frontend (model-provider-dialog.tsx): clear extra_headers and extra_body
when toggling to local mode. Hidden inputs would otherwise keep stale
JSON blocking validate/run.
- Frontend (model-config-dialog.tsx): factor the local-aware provider
selection logic into applyProviderChange and call it from both
onValueChange and onBlur, so manually typing a provider name and tabbing
away keeps the model field consistent.
- Frontend (recipe-studio.ts store): handle both directions of the
is_local toggle in the cascade. external -> local now backfills
model: "local" on already-linked model_configs so they pass validation
immediately, mirroring the existing local -> external clear path.
- Frontend (validate.ts + build-payload.ts): thread localProviderNames
into validateModelConfigProviders and skip the "model is required"
check for local-linked configs. Local providers do not need a real
model id since the inference endpoint uses the loaded Chat model.
* fix(studio): narrow store cascade types, sync model placeholder on graph relink and node removal, harden ephemeral port path
Loop 2 review fixes:
- recipe-studio.ts: type-narrow next.is_local by also checking
next.kind === "model_provider". TS otherwise raised TS2339 because
next was typed as the union NodeConfig after the spread. The behavior
is unchanged but the code now compiles cleanly.
- model-config-dialog.tsx: convert the lastProviderRef / providerInputRef
ref-during-render pattern (pre-existing react-hooks/refs lint error)
to a useEffect that syncs providerInputRef from config.provider. The
combobox blur path still uses applyProviderChange and remains stable.
- recipe-graph-connection.ts: when a graph drag links a model_provider
to a model_config, mirror the dialog applyProviderChange behavior:
fill model: "local" if the new provider is local and the model field
is blank, clear model when relinking from a local placeholder to an
external provider, otherwise leave the model alone.
- reference-sync.ts: when a referenced provider node is removed, clear
the synthetic model: "local" placeholder along with the provider
field, so a future relink to an external provider does not pass
validation with a stale value that fails at runtime.
- run.py: only publish app.state.server_port when the bound port is a
real positive integer; for ephemeral binds (port==0) leave it unset
and let request handlers fall back to request.base_url.
- jobs.py: _resolve_local_v1_endpoint also falls back when
app.state.server_port is non-positive, and uses `is None` instead of
the truthy fallback so a literal 0 is handled correctly.
* fix(studio): strict is_local check, narrow loaded-model gate to LLM-reachable configs, add scope-server port fallback
Loop 3 review fixes:
- jobs.py, validate.py: require `is_local is True` instead of truthy
check. Malformed payloads such as is_local: "false" or is_local: 1
would otherwise be treated as local and silently rewritten to the
loopback endpoint.
- jobs.py: _resolve_local_v1_endpoint now tries request.scope["server"]
(the actual uvicorn-assigned (host, port) tuple) as a second
resolution step before falling back to parsing request.base_url.
This covers direct-uvicorn startup paths and ephemeral binds that
never publish app.state.server_port.
- jobs.py: new _used_llm_model_aliases helper collects the set of
model_aliases that an LLM column actually references, and the
"Chat model loaded" gate is now only triggered when a local
provider is reachable from that set. Orphan model_config nodes on
the canvas no longer block unrelated recipe runs.
* fix(studio): force skip_health_check on local-linked configs, skip JSON parsing for local providers, local-aware inline editor
Loop 4 review fixes:
- jobs.py: after rewriting local providers, also force
skip_health_check: true on any model_config linked to a local
provider. The /v1/models endpoint only advertises the real loaded
model id, so data_designer's default model-availability health check
would otherwise fail against the placeholder "local" id before the
first chat completion call. The inference route already ignores the
model id in chat completions, so skipping the check is safe.
- builders-model.ts: buildModelProvider now short-circuits for local
providers and emits only { name, endpoint: "", provider_type, is_local }
without running parseJsonObject on the hidden extra_headers/extra_body
inputs. Imported or hydrated recipes with stale invalid JSON in those
fields no longer block client-side validate/run.
- inline-model.tsx: the model_config branch now accepts an optional
localProviderNames prop and mirrors the dialog applyProviderChange
behavior. Changing provider to/from a local one auto-fills or clears
the "local" placeholder consistently with the other edit paths.
- recipe-graph-node.tsx: derive localProviderNames from the store via
useMemo (stable identity) and pass it through renderNodeBody to
<InlineModel>. Hooks order is preserved by declaring them above the
early return for markdown_note nodes.
- run.py: minor comment tweak - loop 3 already added the scope-server
fallback path, note that in the comment.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <info@unsloth.ai>
Fix custom folder scanning when pointing directly at a model directory.
When a user adds a custom scan folder that points directly at a model
directory (e.g. /path/to/gemma-4-e2b-it-gguf/ containing config.json
and gemma-4-E2B-it-BF16.gguf), the model list previously showed
individual .gguf files as separate entries instead of recognizing the
directory as a single model. Clicking any entry showed "No GGUF
variants found" because list_local_gguf_variants received a file path
and immediately returned empty.
Changes:
- Add _is_model_directory() helper that detects directories with both
config metadata and actual model weight files (excludes mmproj GGUFs
and non-weight .bin files like tokenizer.bin)
- _scan_models_dir: detect self-model and return single directory entry
- _scan_lmstudio_dir: surface model directories directly instead of
descending into them as publisher folders; handle both root and child
model directories
- Add _resolve_gguf_dir() helper for GGUF path resolution that only
falls back to parent directory when parent has model metadata
- list_local_gguf_variants / _find_local_gguf_by_variant: use resolver
so .gguf file paths inside model directories work correctly
* Add vision detection cache to is_vision_model() to avoid redundant subprocess spawns
is_vision_model() is called 4-5 times per training run for the same model
with zero caching. For transformers 5.x models, each call spawns a full
subprocess (~6s each). This adds a module-level _vision_detection_cache dict
following the same pattern as the existing _audio_detection_cache used by
detect_audio_type(). The function is refactored into a thin cache wrapper
around _is_vision_model_uncached(), saving ~12s per training run.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Include hf_token in vision cache key for gated model correctness
Cache key is now (model_name, hf_token) instead of just model_name.
This prevents stale False results when an unauthenticated probe for a
gated model is followed by an authenticated call.
* Remove test file from main PR - will be submitted separately
* Fix vision cache: normalize model names and skip caching transient failures
- Normalize model names in cache key using resolve_cached_repo_id_case()
to avoid duplicate entries for different casings of the same HF repo
(aligns with case normalization from #4822)
- Return None instead of False on transient failures (network errors,
subprocess timeouts, HF API issues) so the cache layer can distinguish
"definitely not a vision model" from "failed to check"
- Only cache definitive True/False results; transient failures are retried
on the next call instead of being permanently locked in as False
* Refine failure handling: cache deterministic failures, guard normalization
- Subprocess non-zero exit, JSON errors, and general exceptions return
False (deterministic, cached) instead of None (retryable). Only
subprocess.TimeoutExpired returns None since timeouts are transient.
- Wrap cache key normalization in try/except so resolve_cached_repo_id_case
or normalize_path failures fall back to raw model_name instead of
crashing callers.
* Harden vision detection cache: fix transient failure handling, thread safety, token security
- All subprocess failure paths now return None (transient) instead of False,
preventing permanent misclassification of VLMs after temporary HF/auth/network errors
- Use SHA256 fingerprint for hf_token in cache key instead of raw bearer token
- Add threading.Lock with double-checked locking to prevent thundering herd
of concurrent subprocess spawns for the same uncached model
- Distinguish permanent failures (RepositoryNotFoundError, GatedRepoError,
ValueError) from transient ones in _is_vision_model_uncached
- Pass resolved/normalized model name to detection (not just cache key)
- Log normalization fallback at debug level instead of silent swallow
- Thread hf_token through callers in routes/models.py and trainer.py
that previously omitted it
* Refine lock strategy and token fingerprint
- Move detection computation outside the lock to avoid serializing
long-running subprocess spawns (60s timeout) and HF API calls across
all concurrent model checks. Lock is now only held for cache writes.
- Use full SHA256 digest for token fingerprint instead of truncated
16-char prefix to eliminate collision risk.
* Fix huggingface_hub import fallback and use atomic cache read
- Add fallback import path for RepositoryNotFoundError/GatedRepoError
from huggingface_hub.utils (older hub versions) when .errors is
not available
- Use sentinel-based dict.get() for single atomic cache read instead
of two-step in/[] pattern (future-proof for no-GIL runtimes)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* studio: add speculative decoding support (ngram-mod, on by default)
Enable n-gram speculative decoding for GGUF models in Unsloth Studio.
Uses llama.cpp's ngram-mod mode which gives 10-40% faster generation
with zero VRAM cost via a 4MB fixed hash table that auto-resets on
low acceptance rates.
Backend:
- Add speculative_type field to LoadRequest, LoadResponse, and
InferenceStatusResponse pydantic models
- Add speculative_type parameter to LlamaCppBackend.load_model()
with allowlist validation (ngram-simple, ngram-mod)
- Pass --spec-type, --spec-ngram-size-n 16, --draft-max 24 flags
to llama-server when ngram-mod is active
- Default to ngram-mod for non-vision GGUF models server-side
- Silently skip speculative decoding for vision models (unsupported
in llama.cpp server-context.cpp)
Frontend:
- Add speculative_type to TS API types
- Add speculativeType/loadedSpeculativeType to chat runtime store
with default value of "ngram-mod"
- Add On/Off toggle in Model settings section (GGUF only, hidden
for vision models), included in dirty check for Apply/Reset
- Wire speculative_type through model load request and response
- Restore speculative type state on page refresh/reconnect
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: remove server-side speculative decoding override
The backend was overriding speculative_type=None to "ngram-mod" for
non-vision GGUF models, which prevented users from disabling spec
decoding via the UI toggle. The frontend store already defaults to
"ngram-mod", so the backend fallback was redundant and blocked the
explicit "Off" setting.
* fix: use recommended ngram-mod params from llama.cpp docs
Update speculative decoding params to match the recommended values
from llama.cpp docs (docs/speculative.md):
--spec-ngram-size-n 24 (was 16, docs say small n not recommended)
--draft-min 48 (was 0)
--draft-max 64 (was 24, docs note MoEs need long drafts)
Also fix comment: ngram-mod uses ~16 MB (4M entries * 4 bytes),
not 4 MB.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add benchmark table and references to speculative decoding comment
Include speedup numbers from llama.cpp PRs #18471 and #19164 as an
inline comment so future readers understand the expected gains.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio): reuse HF cached repo casing to prevent duplicate downloads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Move cache case resolution tests to separate PR
Tests for resolve_cached_repo_id_case and get_model_config case resolution
belong in their own PR to keep this change focused on the runtime fix.
* fix(studio): debug-log HF_HUB_CACHE fallback in path_utils
* Fix stale memoization in resolve_cached_repo_id_case
- Check exact-case path before memo to ensure a newly-appeared exact
match always wins over a previously memoized variant
- Validate memoized entries still exist on disk before returning them
to prevent stale results when cache dirs are deleted/recreated
* Minor cleanups for cache case resolution
- Use .is_dir() instead of .exists() for exact-case cache check
(cache entries are always directories)
- Remove redundant fallback in _detect_audio_from_tokenizer since
get_cache_path already handles case resolution and returns None
when the model is not cached
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Small GGUF models (<9B) frequently generate full code or lengthy
explanations instead of calling tools, bypassing the existing
plan-without-action re-prompt mechanism. Three issues:
1. _REPROMPT_MAX_CHARS=500 was too low -- models that output full
HTML/code responses (often 1000+ chars) never triggered the
re-prompt at all, since it only fires on short responses.
2. _MAX_REPROMPTS=1 gave the model only one chance to comply.
Small models often need 2-3 nudges before switching from
text generation to tool calling.
3. The re-prompt text ("Please use the available tools...") was
too polite for small models to follow reliably.
4. Tool-calling detection missed chat templates using Jinja
whitespace-trimming syntax ({%- if tools -%}) since only
({%- if tools %}) and ({% if tools %}) were checked.
Changes:
- Raise _REPROMPT_MAX_CHARS from 500 to 2000 so longer responses
(code blocks, multi-paragraph plans) still trigger re-prompts
- Raise _MAX_REPROMPTS from 1 to 3 for more retry budget
- Use direct, imperative re-prompt language that small models
follow more reliably ("STOP. You MUST call a tool NOW.")
- Strengthen the system prompt tool nudge to explicitly forbid
outputting code blocks (redirect to the python tool instead)
- Add Jinja whitespace-trimmed variants to the tool_markers
list so all template styles are detected correctly
* feat(studio): display images from Python tool execution in chat UI
When the model calls the Python tool to create a matplotlib plot or
other image file, the image now displays inline in the chat output
instead of being invisible to the user.
Backend:
- Detect new image files (png/jpg/gif/webp/bmp) after Python subprocess
completes by diffing os.listdir before/after execution
- Append __IMAGES__ sentinel to tool result for frontend consumption
- Strip sentinel before injecting result into LLM context (role: tool)
so the model never sees file paths
- Add GET /sandbox/{session_id}/{filename} endpoint with JWT auth
(header or query param), path traversal protection, extension
allowlist, realpath containment check, and nosniff header
Frontend:
- Parse __IMAGES__ sentinel in tool_end SSE events, create structured
result with text/images/sessionId
- Render <img> tags in Python tool UI pointing at the sandbox endpoint
Also fixes a bug where SyntaxError in user code was misreported as
"unsafe code detected" instead of showing the actual Python traceback.
The _check_code_safety function now lets SyntaxError pass through to
the subprocess for a proper error message.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): improve SVG detection and strip XML preamble
Handle <?xml ...?> declarations before <svg> tags in code fences,
strip XML declaration from SVGs before data URI rendering, and
update the sloth suggestion prompt to request showing code.
* fix(studio): persist parentId so retries survive reload
The append() handler was destructuring only { message } from
ExportedMessageRepositoryItem and discarding parentId. When loading
a saved thread, load() used ExportedMessageRepository.fromArray()
which chains all messages sequentially, flattening retry branches
into a linear list.
Now append() writes parentId to the MessageRecord, and load()
reconstructs the tree when parentIds are present. Old threads
without parentId fall back to the existing fromArray() behavior.
* fix(studio): address review findings for image display and retry persistence
Image detection:
- Use mtime comparison instead of filename-only diff so overwritten
files (e.g. plt.savefig("chart.png") called twice) are detected
Sentinel parsing:
- Use rsplit/lastIndexOf instead of split/indexOf so user code that
prints __IMAGES__: does not collide with the backend sentinel
Mixed legacy/new threads:
- For old messages without a stored parentId, infer sequential parent
from the previous message instead of null, preventing multiple roots
Sandbox endpoint:
- Change Cache-Control from "public, max-age=3600" to "private,
no-store" since these are authenticated responses
---------
Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio): prevent small models from stalling on tool-calling tasks
Small GGUF models (< 9B params) in "Think, Search, Code" mode would
often describe what they planned to do ("Let me create this dashboard")
and then stop generating without ever calling a tool.
Three changes:
1. Simplify web_tips for small models: remove the "fetch its full content
by calling web_search with the url parameter" guidance for models < 9B.
This multi-step instruction causes small models to plan elaborate
search-then-fetch-then-code sequences they cannot reliably execute.
2. Add "always call tools directly" imperative to the system prompt nudge
so models act immediately instead of narrating their intentions.
3. Add plan-without-action re-prompt in the agentic loop: when the model
emits planning text (matching patterns like "let me", "I'll", etc.)
without calling any tool, inject a nudge asking it to call the tool
and continue the loop. Capped at 2 re-prompts per request.
Benchmarked with Qwen3.5-4B-GGUF (N=5 trials per variant):
- Baseline: 40% of requests had any tool call
- Combined fix: 100% of requests had at least one tool call
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio): allow context length slider to reach model's native limit
The context length slider was hard-capped to the VRAM-estimated maximum,
preventing users from requesting higher context even though the backend
already handles it safely (multi-GPU selection, --fit fallback). Expose
the model's native context length from GGUF metadata as a separate API
field and use it as the slider ceiling instead. Add an amber warning
when the selected context exceeds the estimated VRAM capacity.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Raise VRAM budget to 90% and add native_context_length tests
Increase the GPU memory utilization threshold from 70% to 90% across
_select_gpus and _fit_context_to_vram, allowing longer context lengths
before VRAM capping kicks in.
Add 33 tests for the native_context_length feature covering the backend
property, context value separation invariants, Pydantic models, route
completeness, edge cases, and cross-platform binary I/O.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: add scan_folders table and CRUD functions to studio_db
* feat: add scan folders API endpoints and integrate into model scan
* feat: add scan folders API client and update source types
* feat: add custom source to model filters and selector
* feat: add Model Folders section to chat settings sidebar
* style: fix biome formatting in ModelFoldersSection
* fix: address review findings for custom scan folders
empty string bypass, concurrent delete crash guard,
Windows case normalization, response_model on endpoints,
logging, deduplicated filter/map, module level cache for
custom folder models, consistent source labels, handleRemove
error surfacing, per folder scan cap
* fix: show custom folders section regardless of chatOnly mode
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* refactor: extract shared refreshLocalModelsList in pickers
* Harden custom scan folder validation and scanning
- Validate path exists, is a directory, and is readable before persisting
- Apply per-folder model cap during traversal instead of after (avoids
scanning millions of inodes in large directories)
- Wrap per-folder scan in try/except so one unreadable folder does not
break the entire /api/models/local endpoint for all callers
- Normalize case on Windows before storing so C:\Models and c:\models
dedup correctly
- Extend macOS denylist to cover /private/etc and /private/tmp (realpath
resolves /etc -> /private/etc, bypassing the original denylist)
- Add /boot and /run to Linux denylist
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Improve scan robustness and preserve Windows path casing
- Preserve original Windows path casing in DB instead of lowercasing
(normcase used only for dedup comparison, not storage)
- Catch PermissionError per child directory so one unreadable subdirectory
does not skip the entire custom folder scan
- Wrap list_scan_folders() DB call in try/except so a DB issue does not
break the entire /api/models/local endpoint
* fix: scan custom folders for both flat and HF cache layouts
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Windows case-insensitive path dedup with COLLATE NOCASE
Use COLLATE NOCASE on the scan_folders.path column so that the UNIQUE
constraint correctly deduplicates C:\Models and c:\models on Windows
without lowercasing the stored path. Also use COLLATE NOCASE in the
pre-insert lookup query on Windows to catch existing rows with
different casing.
* Restore early-exit limit in _scan_models_dir for custom folders
Keep the limit parameter so _scan_models_dir stops iterating once
enough models are found, avoiding unbounded traversal of large
directories. The post-traversal slice is still applied after combining
with _scan_hf_cache results.
* feat: scan custom folders with LM Studio layout too
* Fix custom folder models being hidden by dedup
Custom folder entries were appended after HF cache and models_dir
entries. The dedup loop kept the first occurrence of each model id,
so custom models with the same id as an existing HF cache entry were
silently dropped -- they never appeared in the "Custom Folders" UI
section.
Use a separate dedup key for custom-source entries so they always
survive deduplication. This way a model can appear under both
"Downloaded" (from HF cache) and "Custom Folders" (from the
user-registered directory) at the same time.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden LM Studio scan and fix COLLATE NOCASE on Linux
- Add per-child and per-publisher OSError handling in _scan_lmstudio_dir
so one unreadable subdirectory does not discard the entire custom
folder's results
- Only apply COLLATE NOCASE on the scan_folders schema on Windows where
paths are case-insensitive; keep default BINARY collation on Linux
and macOS where /Models and /models are distinct directories
* Use COLLATE NOCASE in post-IntegrityError fallback SELECT on Windows
The fallback SELECT after an IntegrityError race now uses the same
case-insensitive collation as the pre-insert check, so a concurrent
writer that stored the path with different casing does not cause a
false "Folder was concurrently removed" error.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* studio: improve GGUF tool calling accuracy and reliability
- Add URL fetching to web_search tool so models can read full page
content instead of only getting search snippets. Uses html2text for
clean markdown conversion with regex fallback.
- Inject current date and behavioral guidance (URL fetch workflow,
no repeated queries, use code for data processing) into the
tool-use system prompt.
- Append error recovery nudge to tool results that indicate failure,
helping small models avoid looping on the same broken call.
- Strip leaked <tool_call> XML from assistant messages in conversation
history and from the outgoing SSE stream.
- Raise default max tool iterations from 10 to 25 across backend,
model schema, and frontend defaults.
- Increase _MAX_PAGE_CHARS from 4k to 16k so fetched pages contain
enough content for the model to extract useful information.
- Add "IMPORTANT: These are only short snippets" hint to search
results so models know to fetch full pages when needed.
Tested with Qwen3.5-4B-GGUF (UD-Q4_K_XL), 10 runs before/after:
- XML leaks in responses: 10/10 -> 0/10
- URL fetch usage: 0 -> 4/10 runs
- Runs producing actual correct answers: 0/10 -> 2/10
- Average tool calls per query: 5.5 -> 3.8 (more efficient)
- Average response time: 12.3s -> 9.8s
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add tool calling benchmark results across model sizes and quants
Tested 16 configurations (4 models x 2 quants x 2 KV cache types)
with 10 runs each on NVIDIA B200.
Best config: 27B UD-Q4_K_XL + bf16 KV -- 6/10 runs found all 4
correct songs, 0 XML leaks, 131s average response time.
* Add duplicate tool-call detection and final-answer synthesis
When the model repeats the exact same tool call (same name + arguments)
twice in a row, skip execution and return a redirect message telling it
to try a different approach. This prevents the 8x-repeated-query loops
observed on 27B and 35B models.
When the tool iteration cap (25) is reached, inject a "provide your
final answer now" message before the final streaming pass. This lets
the model synthesize a useful answer from everything it gathered
instead of being silently cut off.
Tested on Qwen3.5-27B UD-Q4_K_XL (10 runs):
- Repeated query runs: 4/10 -> 2/10
- Cap hits: 1/10 -> 0/10
- All 4/4 accuracy: 5/10 -> 7/10
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix CodeQL alert: handle whitespace in script/style closing tags
The regex fallback for HTML stripping did not match closing tags
with whitespace before the angle bracket (e.g. </script >).
Use \s* before > in both script and style patterns.
* Address reviewer findings: SSRF, timeout crash, XML regex, dedup
- SSRF: resolve hostname via getaddrinfo and reject private, loopback,
link-local, multicast, and reserved addresses before fetching
- Timeout: handle timeout=None (unlimited mode) in URL fetch path
by defaulting to 60s instead of crashing on min(None, 60)
- Download cap: read at most max_chars*4+1 bytes instead of the
full response body before truncating
- XML regex: match both <tool_call> and <function=...> markup in
the history/stream cleanup (inference.py)
- CodeQL: use [^>]* in closing script/style tags to handle any
whitespace or attributes before >
- Dedup: track whether each tool call failed so retries after
transient errors are allowed; only block consecutive identical
calls that both succeeded
- Final-answer synthesis: guard on max_tool_iterations > 0 so
callers who disable tools do not get a false "used all calls" turn
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix redirect SSRF, SSE streaming regression, dedup off-by-one
- SSRF redirect bypass: disable auto-redirect in urllib, manually
follow up to 5 hops with host validation at each step. Prevents
public URLs from redirecting to loopback/private targets.
- SSE streaming: track prev_text on the raw cumulative and strip
XML from the delta only, so completed tool_call tags do not cause
the cumulative to shrink and drop trailing real text.
- Dedup off-by-one: check the immediately previous call (window=1)
instead of requiring 2 matching history entries, so the second
identical successful call is blocked rather than the third.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix redirect HTTPError handling and tighten error prefixes
- Redirect fix: urllib raises HTTPError (not a normal response) when
the redirect handler returns None. Catch HTTPError for 3xx codes
and extract the Location header from the exception object.
- Error prefixes: remove overly broad "No " prefix that matched
"No results found." (a valid empty-search outcome, not an error).
Replace with specific prefixes like "Blocked:", "No query provided",
"Failed to resolve". This ensures empty search results are correctly
classified as non-errors for duplicate-call tracking.
* Fix SSE cross-chunk XML leaks, cleanup review findings
- SSE streaming: sanitize the full cumulative text before diffing
against the previous sanitized snapshot, so XML tags that span
chunk boundaries are stripped correctly. The previous delta-based
approach leaked split tags.
- DRAINING fallback: use _strip_tool_markup() helper instead of a
manual regex that only handled <tool_call> but not <function=...>.
- Move hashlib import, _TOOL_XML_RE compile, and datetime import to
module level per style guide.
- Remove unused _hit_tool_cap variable.
* Fix DNS rebinding, charset detection, HTTPError handling, dedup double-record
- DNS rebinding: resolve hostname once via getaddrinfo, pin the
returned IP, rewrite the URL to connect to the pinned IP with
a Host header. Each redirect hop re-resolves and re-validates.
Closes the TOCTOU window between validation and connection.
- Charset: use resp.headers.get_content_charset() instead of
hardcoding utf-8, so pages with other encodings decode correctly.
- HTTPError: return descriptive "HTTP {code} {reason}" instead of
re-raising into a generic "Search failed" message.
- Dedup: remove redundant _record_tool_call in the duplicate branch;
the single call at the end of the loop handles all cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [WIP] balanced device map for studio
* gpus as a request parameter
* API for multi GPU stuff
* return multi gpu util in new API
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use balanced_low0 instead of balanced
* Use balanced_low0 instead of balanced
* Fix device_map typo, UUID parsing crash, set() filter bug, and broken tests
- balanced_low0 -> balanced_low_0 (transformers/accelerate rejects the old string)
- get_parent_visible_gpu_ids() now handles UUID/MIG CUDA_VISIBLE_DEVICES
gracefully instead of crashing on int() parse
- _get_backend_visible_gpu_info() set() or None bug: empty set is falsy so
CUDA_VISIBLE_DEVICES=-1 would disable filtering and report all GPUs
- test_gpu_selection.py: add missing get_visible_gpu_utilization import and
add required job_id arg to start_training() calls
* Smart GPU determinism using estimates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* disallow gpu selection for gguf for now
* cleanup
* Slightly larger baseline
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Treat empty list as auto
* Verbose logging/debug
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cleanup and revert unnecessary deletions
* Cleanup excessive logs and guard against disk/cpu offload
* auth for visibility API. cleanup redundant imports. Adjust QLoRA estimate
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* support for non cuda gpus
* Fix multi-GPU auto-selection memory accounting
The multi_gpu_factor was applied uniformly to all GPUs including the
first one, which unfairly penalizes single-GPU capacity when
transitioning to multi-GPU. This created a discontinuity where a model
that barely fits 1 GPU would suddenly require 2 GPUs because the first
GPU's free memory was discounted by 20%.
Now the first GPU keeps its full free memory, and only additional GPUs
have an overhead factor (0.85) applied to account for inter-GPU
communication and sharding overhead. This gives more accurate
auto-selection and avoids unnecessary multi-GPU for models that
comfortably fit on one device.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add sandbox tests for multi-GPU selection logic
24 tests covering model size estimation, memory requirements, automatic
GPU selection, device map generation, GPU ID validation, and multi-GPU
overhead accounting. All tests use mocks so they run without GPUs on
Linux, macOS, and Windows.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix reviewer findings: 4bit inference estimate, fallback, GGUF gpu_ids, retry
1. 4-bit inference now uses reduced memory estimate (model_size/3 + buffer)
instead of the FP16 1.3x multiplier. This prevents over-sharding
quantized models across unnecessary GPUs.
2. When model size estimation fails, auto_select_gpu_ids now falls back to
all visible GPUs instead of returning None (which could default to
single-GPU loading for an unknown-size model).
3. GGUF inference route now treats gpu_ids=[] as auto-selection (same as
None) instead of rejecting it as an unsupported explicit request.
4. Training retry path for "could not get source code" now preserves the
gpu_ids parameter so the retry lands on the same GPUs.
5. Updated sandbox tests to cover the new 4-bit inference estimate branch.
* Remove accidentally added unsloth-zoo submodule
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix UUID/MIG visibility and update test expectations
1. nvidia.py: When CUDA_VISIBLE_DEVICES uses UUID/MIG tokens, the
visibility APIs now return "unresolved" with empty device lists instead
of exposing all physical GPUs. This prevents the UI from showing GPUs
that the backend process cannot actually use.
2. test_gpu_selection.py: Updated test expectations to match the new
multi-GPU overhead accounting (first GPU at full capacity, 0.85x for
additional GPUs) and 4-bit inference memory estimation formula.
All 60 tests now pass.
* Add CPU/disk offload guard to audio inference path
The audio model loading branch returned before the common
get_offloaded_device_map_entries() check, so audio models loaded with a
multi-GPU device_map that spilled layers to CPU/disk would be accepted
instead of rejected. Now audio loads also verify no modules are offloaded.
* Improve VRAM requirement estimates
* Replace balanced_low_0 with balanced
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* refine calculations for slightly easier nums
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* adjust estimates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use nums instead of obj to avoid seralisation error
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden nvidia-smi parsing and fix fallback GPU list
1. nvidia.py: Wrap int() casts for GPU index and memory in try/except
so MIG slices, N/A values, or unexpected nvidia-smi output skip the
unparseable row instead of aborting the entire GPU list.
2. nvidia.py: Handle GPU names containing commas by using the last
field as memory instead of a fixed positional index.
3. hardware.py: fallback_all now uses gpu_candidates (GPUs with verified
VRAM data) instead of raw devices list, which could include GPUs
with null VRAM that were excluded from the ranking.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* consolidate raise_if_offload
* Improve MoE support. Guard against nvidia-smi failures
* Improve MoE support. Guard against nvidia-smi failures
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix shared-expert LoRA undercount, torch VRAM fallback, and apply_gpu_ids edge case
1. vram_estimation.py: compute_lora_params now includes shared experts
(n_shared_experts) alongside routed experts when computing MoE LoRA
adapter parameters. Previously only n_experts were counted, causing
the estimator to undercount adapter, optimizer, and gradient memory
for DeepSeek/GLM-style models with shared experts.
2. hardware.py: _torch_get_per_device_info now uses mem_get_info (which
reports system-wide VRAM usage) instead of memory_allocated (which
only reports this process's PyTorch allocations). This prevents
auto-selection from treating a GPU as mostly free when another
process is consuming VRAM. Falls back to memory_allocated when
mem_get_info is unavailable.
3. hardware.py: apply_gpu_ids([]) now returns early instead of setting
CUDA_VISIBLE_DEVICES="" which would disable CUDA entirely. Empty
list inherits the parent visibility, same as None.
4. hardware.py: Upgraded fallback_all GPU selection log from debug to
warning so operators are notified when the model likely will not fit
in available VRAM.
* Guard nvidia-smi subprocess calls against OSError and TimeoutExpired
get_visible_gpu_utilization and get_backend_visible_gpu_info now catch
OSError (nvidia-smi not found) and TimeoutExpired internally instead
of relying on callers to wrap every invocation. Returns the standard
available=False sentinel on failure so the torch-based fallback in
hardware.py can take over.
* Guard get_primary_gpu_utilization and reset GPU caches between tests
1. nvidia.py: get_primary_gpu_utilization now catches OSError and
TimeoutExpired internally, matching the pattern already used in
get_visible_gpu_utilization and get_backend_visible_gpu_info. All
three nvidia-smi callers are now self-contained.
2. test_gpu_selection.py: Added _GpuCacheResetMixin that resets the
module-level _physical_gpu_count and _visible_gpu_count caches in
tearDown. Applied to all test classes that exercise GPU selection,
device map, or visibility functions. This prevents stale cache
values from leaking between tests and causing flaky results on
machines with real GPUs.
* Fix nvidia-smi fallback regression and physical GPU count validation
1. hardware.py: get_gpu_utilization, get_visible_gpu_utilization, and
get_backend_visible_gpu_info now check result.get("available") before
returning the nvidia-smi result. When nvidia-smi is unavailable or
returns no data (e.g., containers without nvidia-smi, UUID/MIG masks),
the functions fall through to the torch-based fallback instead of
returning an empty result. This fixes a regression where the internal
exception handling in nvidia.py prevented the caller's except block
from triggering the fallback.
2. hardware.py: resolve_requested_gpu_ids now separates negative-ID
validation from physical upper-bound validation. The physical count
check is only enforced when it is plausibly a true physical count
(i.e., higher than the largest parent-visible ID), since
torch.cuda.device_count() under CUDA_VISIBLE_DEVICES returns the
visible count, not the physical total. The parent-visible-set check
remains authoritative in all cases. This prevents valid physical IDs
like [2, 3] from being rejected as "out of range" when nvidia-smi is
unavailable and CUDA_VISIBLE_DEVICES="2,3" makes torch report only
2 devices.
* Fix UUID/MIG torch fallback to enumerate devices by ordinal
When CUDA_VISIBLE_DEVICES uses UUID or MIG identifiers,
get_parent_visible_gpu_ids() returns [] because the tokens are
non-numeric. The torch fallback in get_visible_gpu_utilization() and
get_backend_visible_gpu_info() previously passed that empty list to
_torch_get_per_device_info(), getting nothing back.
Now both functions detect the empty-list case and fall back to
enumerating torch-visible ordinals (0..device_count-1) with
index_kind="relative". This means the UI and auto-selection still
see real device data in Kubernetes, MIG, and Slurm-style UUID
environments where nvidia-smi output cannot be mapped to physical
indices.
Updated test_uuid_parent_visibility to verify the new torch fallback
path returns available=True with relative ordinals.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add type hint for gpu_ids parameter in InferenceOrchestrator.load_model
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Fixes#4670
Separates the GGUF context slider ceiling from the currently active context length so lowering context via Chat Settings no longer locks the slider max to the reduced value.
- Backend: adds `max_context_length` to GGUF load/status responses, computed from the largest VRAM/KV-fit cap across all usable GPU subsets
- Frontend: stores `ggufMaxContextLength` and uses it for Context Length slider/input bounds; hydrates from both `/api/inference/load` and `/api/inference/status`
- Defaults UI ceiling to native context for CPU-only and fallback paths
- Seeds `effective_ctx` and `max_available_ctx` before GPU probing to prevent `UnboundLocalError` on probe failure
- Property fallback uses native `_context_length`, not effective `context_length`
* fix: default HF cache to standard platform path instead of legacy Unsloth cache
* feat: show LM Studio and local models in chat Fine-tuned tab
* feat: show LM Studio models in Hub models tab
* fix: fetch local models after auth refresh completes
* Revert "fix: fetch local models after auth refresh completes"
This reverts commit cfd61f0ac7.
* fix: increase llama-server health check timeout to 600s for large models
* feat: expandable GGUF variant picker for LM Studio local models
* fix: show GGUF variant label for locally loaded LM Studio models
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: show publisher name in LM Studio model labels
* fix: set model_id for loose GGUF files in LM Studio publisher dirs
* fix: show publisher prefix in Fine-tuned tab LM Studio models
* fix: only use model_id for lmstudio source models
* fix: only show LM Studio models in Hub tab on Mac/chat-only mode
* fix: respect XDG_CACHE_HOME, handle Windows paths in isLocalPath, refresh LM Studio on remount
- _setup_cache_env now reads XDG_CACHE_HOME (falls back to ~/.cache)
instead of hard-coding ~/.cache/huggingface. This follows the standard
HF cache resolution chain and respects distro/container overrides.
- isLocalPath in GgufVariantExpander uses a regex that covers Windows
drive letters (C:\, D:/), UNC paths (\\server\share), relative paths
(./, ../), and tilde (~/) -- not just startsWith("/").
- HubModelPicker.useEffect now calls listLocalModels() before the
alreadyCached early-return gate so LM Studio models are always
refreshed on remount. Also seeds useState from _lmStudioCache for
instant display on re-open.
* fix: add comment explaining isLocalPath regex for Windows/cross-platform paths
* fix: prioritize unsloth publisher in LM Studio model list
* fix: scope unsloth-first sort to LM Studio models on all platforms
* fix: add missing _lmStudioCache module-level declaration
* fix: prioritize unsloth publisher before timestamp sort in LM Studio group
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Detect always-on reasoning models and show Think button as locked-on
Models with hardcoded <think>/<think> tags or reasoning_content in
their chat template (e.g. distilled reasoning models) always produce
thinking output regardless of any toggle. Previously these models
were not detected as reasoning-capable at all, so the Think button
was grayed out even though the model was actively reasoning.
Backend:
- Detect <think>/<think> and reasoning_content in GGUF chat templates
as a fallback when enable_thinking is not present
- Add reasoning_always_on flag to LoadResponse and InferenceStatusResponse
- Pass the flag through all GGUF load and status response paths
Frontend:
- Add reasoningAlwaysOn to the chat runtime store and API types
- When reasoning_always_on is true, show the Think button as lit
(active) but not clickable, with a tooltip explaining the model
always uses thinking
- Force reasoningEnabled=true when the model always reasons
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use pointer-events-none instead of disabled for always-on Think button
The HTML disabled attribute was not fully blocking clicks on the Think
button for always-on reasoning models. Switch to pointer-events-none
CSS class which prevents all mouse interaction at the CSS level.
* Use a static span instead of disabled button for always-on Think
Replace the button element with a plain span when reasoning is
always on. This makes it physically impossible to toggle since
there is no clickable element at all, avoiding any CSS or
disabled-attribute edge cases.
* Simplify always-on Think button to stay lit and remain toggleable
Keep the Think button as a normal toggleable button but ensure it
shows as lit when reasoning_always_on is true. The model always
reasons regardless of the toggle state so there is no need to
block interaction.
---------
Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix ~1.2s TTFT penalty when tools are enabled in Studio
When users enable web search, Python execution, or terminal tools,
every message gets a ~1.2s delay before any text appears -- even when
the model does not call any tool. This happens because
generate_chat_completion_with_tools() does a non-streaming detection
pass (stream: False) first, waits for the complete response, then
checks for tool calls. For the ~90% of messages that don't trigger a
tool call, this blocking wait is entirely wasted.
Root cause: the detection pass payload uses stream: False, forcing
llama-server to generate the entire response before returning any
tokens.
Fix: replace the non-streaming detection pass with a streaming pass
(stream: True) and a speculative buffer state machine that detects
tool signals in the first 1-2 SSE chunks:
- BUFFERING: accumulate content tokens, check first chars for tool
signal prefixes (<tool_call>, <function=)
- STREAMING: no tool detected, yield tokens to caller immediately
- DRAINING: tool signal found, silently accumulate rest of stream
Three detection paths:
1. Structured delta.tool_calls -- detected instantly, transition to
DRAINING, accumulate fragments, assemble at stream end.
2. XML tool markup in content -- buffer holds up to 32 chars checking
for <tool_call> or <function= prefix, then transitions to DRAINING.
3. No tool signal -- first non-whitespace, non-XML char triggers
immediate transition to STREAMING (fast path, ~90% of requests).
Safety net: after any stream ends in STREAMING state, check accumulated
content for XML tool signals. Handles rare "content before tool call"
edge case.
Additional supporting changes:
- Add headers parameter to _stream_with_retry for auth forwarding
- Share _strip_tool_markup and regex patterns between the detection
pass and the final streaming pass (removes duplication)
- Remove the iteration==0 non-streaming content shortcut (no longer
needed since all iterations stream directly)
- Keep the final streaming pass as fallback for max_tool_iterations
exhaustion
Benchmarked on Qwen3.5-4B Q4_K_XL:
- No tools: TTFT ~112ms (unchanged)
- Tools enabled, no call: TTFT ~112ms (was ~1207ms)
- Decode TPS: 226 (unchanged in all cases)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add unit tests for streaming tool detection state machine
16 tests covering every tool call parsing path:
- Plain text (no tool call) streaming
- Structured delta.tool_calls detection and fragment assembly
- XML <tool_call>JSON</tool_call> detection via buffer
- XML <function=name> tag detection via buffer
- Whitespace before tool XML
- Safety net (content then tool XML)
- Parallel multi-tool calls
- Reasoning token bypass (thinking models)
- Reasoning then tool call
- Empty response handling
- Buffer prefix timeout (HTML not mistaken for tool)
- Non-XML first char instant streaming
- False positive rejection (<tool_tip> vs <tool_call>)
- Arguments split across multiple chunks
- auto_heal_tool_calls=False respects the flag
- Metrics accumulation across tool iterations
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix reasoning-only BUFFERING, pre-tool content emission, and code duplication
Addresses review feedback on the streaming tool detection:
1. Reasoning tokens are no longer yielded during BUFFERING/DRAINING
states. The consumer in routes/inference.py tracks prev_text across
tool iterations without resetting it, so yielding reasoning during
a detection pass that resolves to a tool call would corrupt the
delta computation for subsequent iterations. Reasoning is now
silently accumulated during detection (matching the old non-streaming
behavior) and flushed together with content when the buffer resolves
to STREAMING.
2. Handle reasoning-only responses in the BUFFERING resolver. When a
thinking model emits only reasoning_content with no content tokens,
the stream ends while still in BUFFERING state. The resolver now
detects this case and yields reasoning as plain text (without
<think> wrapper), matching the final streaming pass behavior for
models like Qwen3 in always-think mode.
3. Replace duplicated re.sub calls for stripping tool markup with
the existing _strip_tool_markup(content_text, final=True) helper,
removing ~40 lines of redundant regex code.
4. Update tests: adjust reasoning test expectations to match the new
behavior (reasoning batched with content, not streamed individually
during BUFFERING). Add test_reasoning_only_no_content for the
reasoning-only edge case. 17/17 tests pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address remaining reviewer findings: late tool_call IDs and XML speculation
1. Late-arriving tool_calls.id: when a provider sends the real ID on a
later delta chunk (after the initial one with index and function
name), the accumulator now updates the ID instead of keeping the
synthetic "call_{idx}" placeholder. (P2, 2/10 reviewers)
2. XML speculation respects auto_heal_tool_calls: when auto_heal is
explicitly disabled, _TOOL_XML_SIGNALS is empty so the BUFFERING
state never speculatively holds content for XML prefix detection.
Content starting with literal "<tool_call>" or "<function=" text
flows straight through without delay. (P2, 1/10 reviewers)
Skipped: finish_reason="tool_calls" without delta.tool_calls fallback
(P1, 1/10 reviewers). llama-server always sends delta.tool_calls
fragments in streaming mode. A non-streaming fallback for this edge
case would add complexity for a scenario that does not occur in
practice with the supported backend.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Check request.is_disconnected() every 20 tokens instead of every token
The disconnect check is an async round-trip that adds overhead on every
loop iteration. Since the cancel watcher in llama_cpp.py already
handles connection teardown (closes the streaming response on cancel),
this route-layer check is a secondary safety net that does not need to
run on every single token.
Check every 20 tokens across all 4 streaming paths:
- gguf_tool_stream (tool-enabled GGUF)
- gguf_stream_chunks (standard GGUF)
- audio_input_generate (audio/whisper input)
- generic backend stream (non-GGUF fallback)
* Fix safety net, DRAINING metadata, and test import path
1. Safety net no longer retroactively executes tools after visible
content was already emitted to the user. Once _last_emitted is
non-empty, the stream is committed to normal content mode.
Retroactive tool execution after visible output would violate the
streaming contract and corrupt the route-layer cumulative delta
tracker (prev_text). The tool XML is still stripped by
_strip_tool_markup so the user sees clean content.
2. DRAINING false-positive path now merges accumulated metrics from
prior tool iterations instead of dropping them. Uses the same
merge formula as the STREAMING path.
3. Test import path fixed to use repo root instead of hardcoded
sibling directory. Works in clean checkouts and CI.
4. Renamed test_content_then_tool_xml_safety_net to
test_content_then_tool_xml_no_retroactive_execution to reflect
the corrected behavior.
17/17 tests pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Redact --api-key value from llama-server startup log
When UNSLOTH_DIRECT_STREAM=1, the generated bearer token was logged
verbatim in the startup command. Replace the secret with <redacted>
before logging.
* Remove test file temporarily
* Revert disconnect throttle, reset prev_text on tool_start, restore XML safety net
Addresses all P1 findings from reviewer round 3 (10 reviewers):
1. Revert disconnect check to every iteration (was every 20th).
All 10 reviewers flagged this as a correctness regression for
short streams and sparse tool event loops. The cancel watcher in
llama_cpp.py is the primary mechanism but the route-layer check
must remain per-iteration for completeness. [10/10]
2. Reset prev_text on tool_start in gguf_tool_stream. When a tool
cycle begins after visible content was already streamed, the
route-layer cumulative delta tracker (prev_text) must be reset
so the post-tool synthesis response is not truncated or dropped.
[9/10]
3. Remove the _last_emitted gate from the XML safety net. The gate
was added to prevent retroactive tool execution after visible
content, but with prev_text now reset on tool_start (#2), the
root cause is fixed and the safety net can correctly handle
content-then-tool-XML responses (matching pre-PR behavior).
[8/10]
* Use None instead of {} for empty auth headers in TTS methods
* Include accumulated metrics in STREAMING metadata check
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat(studio): editable context length with Apply/Reset for GGUF model settings
Previously the Context Length field was read-only and the backend
hardcoded `-c 0`, ignoring custom values entirely. KV Cache Dtype also
triggered an immediate model reload with no way to cancel.
Backend:
- llama_cpp.py: pass the actual n_ctx value to `-c` instead of always 0
- models/inference.py: relax max_seq_length to 0..1048576 (0 = model
default) so GGUF models with large context windows are supported
Frontend:
- chat-runtime-store: add customContextLength and loadedKvCacheDtype
state fields for dirty tracking
- chat-settings-sheet: make Context Length an editable number input,
stop KV Cache Dtype from auto-reloading, show Apply/Reset buttons
when either setting has been changed
- use-chat-model-runtime: send customContextLength as max_seq_length
in the load request, reset after successful load
* fix: preserve maxSeqLength for non-GGUF models in load request
customContextLength ?? 0 sent max_seq_length=0 for non-GGUF models,
breaking the finetuning/inference path that needs the slider value.
Now uses a three-way branch:
- customContextLength set: use it (user edited GGUF context)
- GGUF without custom: 0 (model's native context)
- Non-GGUF: maxSeqLength from the sampling slider
* fix: keep max_seq_length default at 4096 for non-GGUF callers
Only relax the bounds (ge=0 for GGUF's "model default" mode,
le=1048576 for large context windows). The default stays at 4096
so API callers that omit max_seq_length still get a sane value
for non-GGUF models.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): rename trust remote code toggle and hide when no model selected
- Rename "Trust remote code" to "Enable custom code"
- Shorten subtitle to "Only enable if sure"
- Hide the toggle when no model is loaded (already hidden for GGUFs)
* fix: restore ge=128 for max_seq_length validation
Keep the minimum at 128 so the API rejects nonsensical values.
GGUF path now sends the model's native context length (from
ggufContextLength) instead of 0 when the user has not customized it.
The upper bound stays at 1048576 for large-context GGUF models.
* feat(studio): replace Context Length input with slider
Use a ParamSlider (512 to model's native context, step 512) instead
of a small number input. Shows "Max" when at the model's native
context length. Consistent with the other slider controls in the
settings panel.
* feat(studio): add editable number input alongside Context Length slider
The slider and number input stay synced -- dragging the slider updates
the number, typing a number moves the slider. The input also accepts
values beyond the slider range for power users who need custom context
lengths larger than the model default.
* fix(studio): widen context length input and use 1024 step for slider
Make the number input wider (100px) so large values like 262144 are
fully visible. Change slider step from 512 to 1024 and min from 512
to 1024.
* fix(studio): context length number input increments by 1024
* fix(studio): cap context length input at model's native max
Adds max attribute and clamps typed/incremented values so the context
length cannot exceed the GGUF model's reported context window.
* fix(studio): point "What's new" link to changelog page
Changed from /blog to /docs/new/changelog.
* fix(studio): preserve custom context length after Apply, remove stale subtitle
- After a reload with a custom context length, keep the user's value
in the UI instead of snapping back to the model's native max.
ggufContextLength always reports the model's native metadata value
regardless of what -c was passed, so we need to preserve
customContextLength when it differs from native.
- Remove "Reload to apply." from KV Cache Dtype subtitle since the
Apply/Reset buttons now handle this.
* feat(studio): auto-enable Search and Code tools when model supports them
Previously toolsEnabled and codeToolsEnabled stayed false after loading
a model even if it reported supports_tools=true. Now both toggles are
automatically enabled when the loaded model supports tool calling,
matching the existing behavior for reasoning.
* fix(studio): auto-enable tools in autoLoadSmallestModel path
The suggestion cards trigger autoLoadSmallestModel which bypasses
selectModel entirely. It was hardcoding toolsEnabled: false and
codeToolsEnabled: false even when the model supports tool calling.
Now both are set from the load response, matching the selectModel
behavior. Also sets kvCacheDtype/loadedKvCacheDtype for dirty
tracking consistency.
* fix(studio): re-read tool flags after auto-loading model
The runtime state was captured once at the start of the chat adapter's
run(), before autoLoadSmallestModel() executes. After auto-load enables
tools in the store, the request was still built with the stale snapshot
that had toolsEnabled=false. Now re-reads the store after auto-load so
the first message includes tools.
* fix(studio): re-read entire runtime state after auto-load, not just tools
The runtime snapshot (including params.checkpoint, model id, and all
tool/reasoning flags) was captured once before auto-load. After
autoLoadSmallestModel sets the checkpoint and enables tools, the
request was still built with stale params (empty checkpoint, tools
disabled). Now re-reads the full store state after auto-load so the
first message has the correct model, tools, and reasoning flags.
* feat(studio): add Hugging Face token field in Preferences
Adds a password input under Configuration > Preferences for users to
enter their HF token. The token is persisted in localStorage and
passed to all model validate/load/download calls, replacing the
previously hardcoded null. This enables downloading gated and private
models.
* fix(studio): use model native context for GGUF auto-load, show friendly errors
The auto-load paths and selectModel for GGUF were sending
max_seq_length=4096 which now actually limits the context window
(since we fixed the backend to respect n_ctx). Changed to send 0
for GGUF, which means "use model's native context size".
Also replaced generic "An internal error occurred" messages with
user-friendly descriptions for known errors like context size
exceeded and lost connections.
LoadRequest validation changed to ge=0 to allow the GGUF "model
default" signal. The frontend slider still enforces min=128 for
non-GGUF models.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): filter out FP8 models from model search results
Hide models matching *-FP8-* or *FP8-Dynamic* from both the
recommended list and HF search results. These models are not
yet supported in the inference UI.
---------
Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: multi-source model discovery (HF default, legacy cache, LM Studio)
* Fix multi-source model discovery bugs
- Fix lmstudio_model_dirs: add ~/.lmstudio/models as default path,
remove dead sys.platform branch, add dedup via seen set
- Fix _setup_cache_env: preserve legacy HF cache env vars when the
legacy hub directory exists and is non-empty
- Fix _scan_lmstudio_dir: use absolute path for id field so
is_local_path() returns True
- Remove LM Studio dirs from allowed_roots (scanned unconditionally)
- Replace bare except passes with logger.warning in legacy cache blocks
- Fix delete_cached_model to search both default and legacy HF caches
- Make lmstudio_dirs non-optional in TS interface (matches Python schema)
- Exclude lmstudio source from trainable model filter
- Remove unused import sys
* Scan HF default cache alongside legacy and active caches
When _setup_cache_env overrides HF_HUB_CACHE to the legacy Unsloth
path, the standard HF default cache (~/.cache/huggingface/hub) was
never scanned, hiding models downloaded before Unsloth Studio was
installed.
Add hf_default_cache_dir() and _all_hf_cache_scans() helper that
deduplicates and scans all three HF cache locations (active, legacy,
default). Used in list_local_models, list_cached_gguf,
list_cached_models, and delete_cached_model.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat(db): add SQLite storage layer for training history
* feat(api): add training history endpoints and response models
* feat(training): integrate DB persistence into training event loop
* feat(ui): add training history views and card grid
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): address review issues in training history persistence
- Strip hf_token/wandb_token from config before SQLite storage
- Add UUID suffix to job_id for collision resistance
- Use isfinite() for 0.0 metric handling throughout
- Respect _should_stop in error event finalization
- Run schema DDL once per process, not per connection
- Close connection on schema init failure
- Guard cleanup_orphaned_runs at startup
- Cap _metric_buffer at 500 entries
- Make FLUSH_THRESHOLD a class constant
- Map 'running' to 'training' phase in historical view
- Derive LR/GradNorm from history arrays in historical view
- Fix nested button with div[role=button] in history cards
- Guard String(value) against null/undefined in config popover
- Clear selectedHistoryRunId on auto tab switch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): address round-2 review findings across training backend and frontend
Backend (training.py):
- Move state mutation after proc.start() so a failed spawn does not wedge
the backend with is_training=True
- Create DB run row eagerly after proc.start() so runs appear in history
during model loading, not after first metric event
- Rewrite _flush_metrics_to_db() with snapshot-before-insert pattern to
preserve metrics arriving during the write and retain buffer on failure
- Guard eval_loss with float() coercion and math.isfinite(), matching the
existing grad_norm guard
- Increase pump thread join timeout from 3s to 8s to cover SQLite's
default 5s lock timeout
Frontend (studio-page.tsx):
- Fix history navigation: check isTrainingRunning instead of
showTrainingView in onSelectRun so completed runs are not misrouted
- Replace activeTab state + auto-switch useEffect with derived tab to
eliminate react-hooks/set-state-in-effect lint violation
Frontend (historical-training-view.tsx):
- Add explicit "running" branch to message ternary so running runs no
longer fall through to "Training errored"
- Derive loading from detail/error state and move cleanup to effect
return to eliminate react-hooks/set-state-in-effect lint violation
Frontend (progress-section.tsx):
- Derive stopRequested from isTrainingRunning && stopRequestedLocal to
eliminate react-hooks/set-state-in-effect lint violation and remove
unused useEffect import
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): resolve 3 remaining bugs from round-2 review
1. Stuck on Current Run tab [12/20]: Only force "current-run" tab when
isTrainingRunning is true, not when stale completed-run data exists.
After training ends, users can freely navigate to Configure.
2. Incomplete metric sanitization [7/20]: Apply float() coercion and
isfinite() guards to loss and learning_rate, matching the existing
pattern used by grad_norm and eval_loss. Prevents TypeError from
string values and NaN leaks into history arrays.
3. Stop button state leak across runs [10/20]: Add key={runtime.jobId}
to ProgressSection so React remounts it when a new run starts,
resetting stopRequestedLocal state.
* fix(studio): deduplicate loss/lr sanitization in training event handler
Reuse _safe_loss/_safe_lr from the progress update block instead of
re-sanitizing the same raw event values for metric history.
* fix(studio): restore loss > 0 guard to prevent eval steps injecting 0.0 into metric histories
Round-2/3 fixes relaxed the history append guard from `loss > 0` to
`loss is not None`, which let eval-only log events (where loss defaults
to 0.0) append fake zeros into loss_history and lr_history. Restore the
`loss > 0` check to match the worker's own has_train_loss gate. The
float() coercion and isfinite() sanitization from round-3 remain intact.
* fix(studio): resolve training history bugs — nullable loss/lr, tab nav, sparkline
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Fix Studio silently exiting on Windows without error output
On Windows, `unsloth studio` launches a child process via
subprocess.Popen to run the server in the studio venv. If the child
crashes (e.g. due to a missing package), the parent just calls
typer.Exit(rc) with no message -- the user sees "Launching Unsloth
Studio... Please wait..." and then the prompt returns with zero
feedback.
Root cause: `data_designer_unstructured_seed` is imported at the top
level in seed.py. If this package is not installed in the studio venv,
the entire import chain (seed.py -> routes/__init__.py -> main.py ->
run_server()) crashes with ModuleNotFoundError. Since run.py has no
try/except around run_server() and studio.py does not report nonzero
exit codes, the failure is completely silent.
Changes:
- run.py: wrap run_server() in try/except, print clear error with
traceback to stderr. Also reconfigure stderr encoding on Windows so
tracebacks with non-ASCII paths do not cause secondary failures.
- studio.py: print an error message when the child process exits with
a nonzero code on Windows, so the user knows something went wrong.
- seed.py: make data_designer_unstructured_seed import optional with
a try/except fallback. The server starts normally and only returns
HTTP 500 if the unstructured seed endpoints are actually called.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Skip Anaconda/Miniconda Python when creating Studio venv on Windows
Conda-bundled CPython ships modified DLL search paths that prevent
torch from loading c10.dll on Windows. The Studio server fails
silently at startup because the venv was created with conda's Python.
Standalone CPython (python.org, winget, uv) does not have this issue.
Both install.ps1 and setup.ps1 now skip any Python binary whose path
contains conda, miniconda, anaconda, miniforge, or mambaforge when
selecting the interpreter for the studio venv. If only conda Python
is available, the scripts print an error with instructions to install
standalone CPython.
* Fix multi-file preview crash and improve setup.ps1 Python discovery
Addresses review findings [10/10] and [8/10]:
1. seed.py: _read_preview_rows_from_multi_files() had a hard import
of build_multi_file_preview_rows inside the function body, bypassing
the optional-plugin guard. Moved it into the top-level try/except
block and added a None guard matching the other functions.
2. setup.ps1: Python discovery now probes py.exe (Python Launcher)
first, uses Get-Command -All to look past conda entries that shadow
standalone CPython further down PATH, skips WindowsApps stubs, and
resolves the actual executable path so venv creation does not
re-resolve back to a conda interpreter.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Check sys.base_prefix to catch venvs created from conda Python
A venv created from conda Python (e.g. C:\Users\danie\.venv) has a
path that does not contain "conda", but sys.base_prefix still points
to the conda install (e.g. C:\Users\danie\miniconda3). The previous
path-only check missed this case entirely.
Both install.ps1 and setup.ps1 now use a Test-IsConda helper that
checks both the executable path AND sys.base_prefix against the
conda/miniconda/anaconda/miniforge/mambaforge pattern. This catches:
- Direct conda Python executables
- Venvs created from conda Python (base_prefix reveals the origin)
* Fix install.ps1 passing version string to uv venv instead of resolved path
Find-CompatiblePython returned a bare version string (e.g. "3.13")
which was passed to `uv venv --python 3.13`. uv performs its own
interpreter discovery and can resolve that version string back to a
conda Python, defeating the entire conda-skip logic.
Now Find-CompatiblePython returns a hashtable with both .Version (for
display) and .Path (the resolved absolute executable path). The venv
is created with `uv venv --python <absolute-path>`, ensuring uv uses
the exact interpreter we validated.
* Quote resolved Python path in uv venv call for paths with spaces
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: handle Windows subprocess crash during dataset.map()
Windows uses spawn (not fork) for multiprocessing. Spawned workers
cannot resolve Unsloth's dynamically compiled cache modules from
unsloth_compiled_cache/, causing ModuleNotFoundError and RuntimeError
during dataset.map() tokenization.
Add two platform-guarded patches for sys.platform == "win32":
1. Force HF_DATASETS_MULTITHREADING_MAX_WORKERS=1 and set spawn method
2. Monkey-patch Dataset.map() to force num_proc=None
Fixes#4490
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address review: extend spawn fix to macOS, add multiprocess fallback
- Change platform checks from sys.platform == "win32" to
sys.platform != "linux" so macOS (also spawn-based) is covered
- Wrap multiprocess import in try/except falling back to stdlib
multiprocessing when the multiprocess package isn't installed
- Rename _win32_safe_map to _spawn_safe_map to reflect broader scope
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: replace global Dataset.map monkey-patch with targeted num_proc routing
The previous approach had issues: Patch 1 set HF_DATASETS_MULTITHREADING_MAX_WORKERS
and forced set_start_method (dead code on platforms already using spawn), and Patch 2
globally monkey-patched Dataset.map() (too broad, missed Dataset.filter()).
Replace with a two-layer fix:
1. Studio layer: Add dataset_map_num_proc() that returns None on spawn platforms
(Windows, macOS). Unlike num_proc=1 which still creates Pool(1) and spawns a
worker, num_proc=None runs Dataset.map()/filter() truly in-process.
Update all dataset.map() callsites to use it. ThreadPoolExecutor callers
(format_conversion.py) keep using safe_num_proc() since threads are unaffected.
2. Root-cause layer: Propagate UNSLOTH_COMPILE_LOCATION via PYTHONPATH on spawn
platforms so spawned workers can import compiled modules. Mirrors the .venv_t5
pattern in worker.py. Does not import unsloth_zoo.compiler (heavy torch/triton
imports). Completely skipped on Linux.
Also extend safe_num_proc() to return 1 on macOS (was only guarding Windows),
and narrow the transformers 5.x dataloader guard from != "linux" to explicit
("win32", "darwin").
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: add safe_thread_num_proc() for ThreadPoolExecutor callsites
safe_num_proc() correctly caps to 1 on macOS/Windows for process-based
multiprocessing, but format_conversion.py reuses it for ThreadPoolExecutor
workers. Threads share address space and are unaffected by spawn, so
capping to 1 makes image URL downloads sequential -- a real regression.
Add safe_thread_num_proc() that skips the platform guard but keeps the
cpu_count heuristic, and switch both ThreadPoolExecutor callsites in
format_conversion.py to use it.
* fix: remove double-wrap in dataset_num_proc + fix num_proc=1 in datasets route
- trainer.py:3009: Replace safe_num_proc(max(1, os.cpu_count() // 4))
with max(1, (os.cpu_count() or 1) // 4) to avoid double-wrapping
inside dataset_map_num_proc which already calls safe_num_proc
- trainer.py:15-20: Clarify comment on PYTHONPATH propagation
- datasets.py:445: Change num_proc=1 to num_proc=None for 10-row
preview slice (avoids unnecessary multiprocessing overhead)
* fix: guard os.cpu_count() against None in worker-count helpers
os.cpu_count() can return None on some platforms. Use (os.cpu_count() or 1)
to prevent TypeError in safe_num_proc() and safe_thread_num_proc().
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* feat(chat): add server-side timings and context display for GGUF
Extract timings/usage metadata from llama-server SSE stream and forward
through the full stack. Replace client-side estimates with accurate
server-reported metrics (prompt eval, tok/s, token counts, cache hits).
Add context window usage bar to chat top nav.
* feat(chat): source badges with hover cards and 2-row collapse
- Add hover cards to source badges showing favicon, title, URL and
snippet description on hover
- Limit source badges to 2 rows with +X more expand/collapse
- Parse snippet from web search results for hover card descriptions
- Replace individual Source rendering with grouped SourcesGroup component
* fix(chat): add null guards for server timings edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(chat): reset contextUsage on thread switch, remove unused context-display
* fix(chat): stop double-counting completion tokens in tool-calling path
* fix(chat): skip metadata events in llm_assist consumers
* fix(chat): hide context usage bar in compare mode
* fix(chat): harden timings pipeline and context usage persistence
Accumulate prompt_ms, predicted_ms, and predicted_n from intermediate
tool-detection passes so the final metadata reflects total server work.
Persist contextUsage in message metadata (Dexie) and restore on thread
load. Add type guard in gguf_stream_chunks for unexpected dict events.
Clear contextUsage when entering compare mode.
* feat(chat): make GGUF stream metadata OpenAI-compatible
* fix(chat): address PR review feedback
* feat(chat): address PR review feedback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(recipe-studio): prevent fitView from zooming to wrong location on recipe load
* feat: add pymupdf/python-docx deps and unstructured uploads storage root
* feat: add POST /seed/upload-unstructured-file endpoint
* feat: add multi-file chunking with source_file column
* feat: update frontend types and API layer for multi-file upload
* feat: round-robin preview rows across source files
Ensures every uploaded file is represented in the preview table
by cycling through sources instead of just taking the first N rows.
* fix: disable OCR, fix auto-load timing, fix persistence on reload
- Disable pymupdf4llm OCR with write_images=False, show_progress=False
- Replace onAllUploaded callback with useEffect that detects uploading→done
transition (avoids stale closure reading empty file IDs)
- Fix importer to preserve file IDs from saved recipes instead of clearing
(clearing only happens at share time via sanitizeSeedForShare)
* fix: harden unstructured upload with input validation and state fixes
Validate block_id/file_id with alphanumeric regex to prevent path
traversal, use exact stem match for file deletion, add error handling
for metadata writes and empty files, fix React stale closures and
object mutations in upload loop, and correct validation logic for
unstructured seed resolved_paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: address PR review - legacy path import, share sanitizer, sync effect
Promote legacy source.path into resolved_paths for old unstructured
recipes, clear source.paths in share sanitizer to prevent leaking local
filesystem paths, and gate file sync effect to dialog open transition
so users can actually delete all uploaded files.
* fix: CSV column fix (BOM + whitespace + unnamed index re-save) for #4470
* fix: harden unstructured upload flow and polish dialog UX
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: support full model GGUF export, disable incompatible methods in UI
* fix: resolve base model from config.json for venv_t5 export switching
* feat: detect BNB-quantized models and disable all export methods for quantized non-PEFT checkpoints
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: relocate Ollama Modelfile alongside GGUFs during non-PEFT export cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add elapsed timer to tool status pill in Studio
Show a count-up seconds timer (0s, 1s, 2s, ...) next to the tool status
text in the composer area. Helps users gauge how long a tool call (web
search, code execution) has been running. Timer resets when a new tool
starts and disappears when all tools finish.
* Fix tool call parsing, add tool outputs panel and reasoning copy button
Backend:
- Rewrite tool call XML parser to use balanced-brace JSON extraction
instead of greedy regex, fixing truncation on nested braces in
code/JSON arguments
- Handle optional closing tags (</tool_call>, </function>, </parameter>)
that models frequently omit
- Support bare <function=...> tags without <tool_call> wrapper
- Strip tool call markup from streamed content so raw XML never leaks
into the chat UI
- Use a persistent ~/studio_sandbox/ working directory for tool
execution so files persist across calls within a session
- Emit tool_start/tool_end SSE events so the frontend can display
tool inputs and outputs
Frontend:
- Add collapsible "Tool Outputs" panel below assistant messages showing
each tool call's input and output with copy buttons
- Add copy button to reasoning blocks
- Add elapsed timer to tool status pill
- Update project URLs in pyproject.toml (http -> https, add docs link)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add interactive HTML preview with fullscreen toggle for code blocks
HTML code fences now render an interactive sandboxed iframe preview
below the syntax-highlighted code, similar to how SVG fences show
an image preview. The iframe uses sandbox="allow-scripts" to allow
JavaScript execution while blocking access to the parent page.
Includes a fullscreen toggle (enlarge/minimize button) that expands
the preview into a viewport overlay, dismissible via button, Escape
key, or backdrop click. A streaming placeholder prevents partial
HTML from rendering mid-stream.
* Add tool call settings: auto-heal toggle, max iterations, timeout
Add three user-configurable tool call settings to the Studio Settings panel:
- Auto Heal Tool Calls: toggle to control fallback XML parsing of malformed
tool calls from model output (default: on)
- Max Tool Calls Per Message: slider 0-40 + Max to cap tool call iterations
per message (default: 10)
- Max Tool Call Duration: slider 1-30 minutes + Max to set per-tool-call
execution timeout (default: 5 minutes)
All settings persist to localStorage and flow through the full stack:
frontend store -> API request -> Pydantic model -> route -> llama_cpp -> tools.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix tool call timeout: respect no-limit and apply to web search
- Use a sentinel to distinguish timeout=None (no limit) from the default
(300s). Previously None was silently replaced with _EXEC_TIMEOUT.
- Pass the configured timeout to DDGS() for web searches so the setting
applies uniformly to all tool types.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add input validation bounds and per-thread sandbox isolation
- Add ge=0 constraint to max_tool_calls_per_message (rejects negative values)
- Add ge=1 constraint to tool_call_timeout (minimum 1 second)
- Thread session_id from frontend through backend to tool execution
- Scope sandbox directories per conversation: ~/studio_sandbox/{thread_id}/
- Backwards compatible: API callers without session_id use ~/studio_sandbox/
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix non-monotonic streaming and Python temp script path
- Split tool markup stripping into closed-only (mid-stream) and full
(final flush) to prevent cumulative text from shrinking mid-stream
- Enforce monotonicity: only emit when cleaned text grows, so the
proxy's delta logic (cumulative[len(prev_text):]) never breaks
- Place Python temp scripts in the sandbox workdir instead of /tmp so
sys.path[0] points to the sandbox and cross-call imports work
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Sanitize session_id to prevent path traversal in sandbox
Strip path separators and parent-dir references from session_id before
using it as a directory name. Verify the resolved path stays under
~/studio_sandbox/ as a second guard.
* feat(chat): proper assistant-ui tool call UIs with sources
Replace custom metadata-based ToolOutputsGroup with native assistant-ui
tool-call content parts. Backend SSE tool_start/tool_end events now emit
proper { type: "tool-call" } parts from the adapter, enabling per-tool
UIs registered via tools.by_name in MessagePrimitive.Parts.
- Web search: Globe icon, Source badges with favicons, auto-collapse
when LLM starts responding
- Python: Code icon, syntax-highlighted code via Streamdown/shiki,
output block with copy
- Terminal: Terminal icon, command in trigger, output with copy
- ToolGroup wraps consecutive tool calls (skips for single calls)
- Sources component renders URL badges at end of message
- Flattened code block CSS (single border, no nested boxes)
* fix(inference): respect empty enabled_tools allowlist
`if payload.enabled_tools:` is falsy for [], falling through to
ALL_TOOLS. Use `is not None` so an explicit empty list disables
all tools as intended.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Shine1i <wasimysdev@gmail.com>
* studio: improve onboarding UX, tooltips, and training defaults
- Change splash text to "Train and run LLMs locally"
- Add "Chat Only" card with BubbleChatIcon to skip directly to chat
- Add Skip/Skip to Chat buttons in sidebar and footer
- Back button on step 1 returns to splash screen instead of being disabled
- Change "Watch video guide" to "Get started with our guide" with new URL
- Update intro text to mention all model types + chat
- Make all tooltips clickable (in addition to hover) via React context
- Strip surrounding quotes from pasted HF tokens
- Rename "Eval Split" to "Evaluation Split"
- Add SparklesIcon to "Auto Detect" format option
- Change step 4 heading to "Choose your training parameters"
- Default max_steps to 60
- Learning rate displayed in scientific notation with +/- stepper
- Context length options capped by model's max_position_embeddings (via AutoConfig)
- Fix "QLORA"/"LORA" to "QLoRA"/"LoRA" in summary step
- Backend: add max_position_embeddings to model config endpoint
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* compare for 2 diff models
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* resolving gemini comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: disable thinking for Qwen3.5 <9B and always for AI Assist
- Change Qwen3.5 thinking threshold from <=2B to <9B (0.8B, 2B, 4B
all disable thinking by default; 9B+ enables it)
- Always pass enable_thinking=False in AI Assist helper calls
(_run_with_helper and _generate_with_backend) regardless of chat
thinking settings
* studio: address PR review comments
- Extract _get_max_position_embeddings helper to DRY config extraction
- Fix "Skip to Chat" to navigate to /chat on step 1 (was /studio)
* fix: comment out debug print statements
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: skip Shiki highlighting for incomplete SVG code fences
While streaming SVG content, the syntax highlighter (Shiki) re-parses
the entire growing SVG on every token, blocking the main thread and
freezing the code area until the fence closes. Show a plain-text
preview for incomplete SVG fences instead, similar to how Mermaid
diagrams show a placeholder while streaming.
* studio: fix default top_k from 50/40 to 20 for chat inference
Per Qwen3.5 docs (unsloth.ai/docs/models/qwen3.5), top_k should be 20
for both thinking and non-thinking modes. The model-specific config in
inference_defaults.json already had top_k=20 for Qwen3.5, but the
generic fallback defaults were wrong:
- Frontend DEFAULT_INFERENCE_PARAMS.topK: 50 -> 20
- Backend generate_chat_completion top_k: 40 -> 20
- Backend generate_chat_completion_with_tools top_k: 40 -> 20
- Frontend title generation top_k: 40 -> 20
* studio: set universal inference defaults for unknown models
Default params for any model without specific config:
temperature=0.6, top_p=0.95, top_k=20, min_p=0.01,
presence_penalty=0.0, repetition_penalty=1.0
Models with entries in inference_defaults.json (Qwen3.5, Gemma-3,
Llama, etc.) override these with their recommended values.
Updated in: frontend DEFAULT_INFERENCE_PARAMS, backend Pydantic
request models, and backend generate_chat_completion defaults.
* studio: only trust_remote_code for unsloth/ models in AutoConfig
Only set trust_remote_code=True when the model name starts with
"unsloth/". All other models default to False for safety.
* studio: move Generating spinner above the composer
The "Generating" spinner was below the send message bar, causing
the bar to jump up and down. Move it above the composer in both
the regular thread view and the welcome/empty view.
* studio: adjust toast close button position away from edge
Move the X close button on toasts (like "Starting model...") from
top-1.5 to top-3 and add right-3, giving more breathing room from
the top-right corner.
* studio: make Think button smaller with tighter icon-text gap
Reduce gap from 1.5 to 0.5, padding from px-2.5/py-1 to px-2/py-0.5,
and icon from size-3.5 to size-3.
* studio: multiple onboarding and chat UX improvements
- Move Generating spinner above composer (fixes jumping send bar)
- Make Think button smaller with tighter icon-text gap
- Chat card now inside grid (same size as Audio/Embeddings cards)
- Rename "Chat Only" to "Chat"
- Chat card requires Continue to proceed (no auto-advance)
- Continue on Chat selection skips onboarding and goes to /chat
- Tooltip (i) click on Chat card doesn't trigger navigation
- Step 1 footer Back button goes back to splash (label is "Back")
- Splash "Skip Onboarding" renamed to "Skip to Chat", navigates to /chat
- Toast close button moved away from edge
* studio: align Skip to Chat button, add Skip to footer
- Sidebar "Skip to Chat" now uses primary (green) Button style with
arrow icon, full width, aligned like step items. Shows on all steps.
- Footer: added "Skip" outline button next to Continue that goes
directly to /studio with progress saved (markOnboardingDone)
* studio: change default max steps from 30 to 60 in toggle hook
The DEFAULT_MAX_STEPS in use-max-steps-epochs-toggle.ts was still 30,
used as fallback when toggling from epochs back to max steps.
* studio: extend context length options to 262K
CONTEXT_LENGTHS now includes 65536, 131072, 262144 in addition to
the existing 512-32768 range. The onboarding step filters these by
the model's max_position_embeddings (e.g. Nemotron-3-Nano-4B has
262144), showing powers of 2 up to the model's maximum.
* studio: auto-select LoRA vs QLoRA based on model size and GPU memory
After selecting a model in onboarding, detect the total model weight
file size from HF Hub (safetensors/bin files). Then estimate memory
needed: model_size_gb * 1.5 * context_scale, where context_scale is:
- <=8192 tokens: 1.0x
- >8192 tokens: 1.7x
- >=16384 tokens: 2.0x
- >=32768 tokens: 4.0x
If the estimate fits in free GPU VRAM, default to LoRA (16-bit).
Otherwise default to QLoRA (4-bit).
Backend changes:
- Add model_size_bytes to ModelDetails (models.py)
- Add _get_model_size_bytes() using HfApi.repo_info (routes/models.py)
- Add vram_free_gb to get_gpu_summary (hardware.py)
Frontend changes:
- Add autoSelectTrainingMethod() in training-config-store.ts
- Called after model defaults are loaded
- Add model_size_bytes to ModelConfigResponse type
- Add vramFreeGb to HardwareInfo hook
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: rename "Importing ML libraries..." to "Importing Unsloth..."
* studio: show model/dataset in training status, fix LoRA/QLoRA casing
- Training status now shows 'Training "model_name"' and 'Dataset = ...'
instead of generic "Starting training..."
- Fix Studio progress section to show QLoRA/LoRA instead of QLORA/LORA
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: rename 'Skip to Chat' to 'Skip Onboarding' on splash screen
* studio: add presence_penalty support for chat inference
Add presence_penalty as a parameter across the full stack:
- Backend: llama_cpp.py generate_chat_completion/with_tools, Pydantic
models (inference.py), routes/inference.py pass-through
- Frontend: InferenceParams type, DEFAULT_INFERENCE_PARAMS (0.0),
chat-adapter.ts payload, chat-settings-sheet.tsx slider (0-2),
model defaults loading from inference_defaults.json
- Set Qwen3.5 default presence_penalty to 1.5 per official docs
- Default for unknown models is 0.0 (off)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix Chat card deselecting Text and aligning with other cards
* studio: fix presence_penalty not loading from inference defaults
The inference_config.py load_inference_config() was not including
presence_penalty in the returned config dict, so the Qwen3.5
default of 1.5 from inference_defaults.json never reached the
frontend. Added it to the config builder.
* studio: add delete button for cached models in model selector
Add trash icon on each downloaded model row (GGUF and safetensors) with
confirmation dialog. Backend DELETE /api/models/delete-cached endpoint
uses huggingface_hub scan_cache_dir + delete_revisions to cleanly remove
cached repos, refusing if the model is currently loaded.
* studio: restore inference defaults, reasoning, and tools on page refresh
On page refresh with a model already loaded, the frontend was not
re-applying model-specific inference defaults (presence_penalty,
temperature, etc.) or restoring reasoning/tools support flags.
Backend: Add inference config, supports_reasoning, supports_tools,
and context_length to InferenceStatusResponse.
Frontend: In the refresh callback, when an active model is detected,
apply mergeRecommendedInference and restore reasoning/tools flags
with proper Qwen3.5 size-based defaults.
* studio: fix delete dialog closing before async completes
Prevent AlertDialogAction's default close behavior with
e.preventDefault() so the dialog stays open during deletion.
Also block onOpenChange dismiss while deleting is in progress.
* fix: add Dict and Any imports to inference models
* studio: fix Qwen3.5 reasoning threshold in frontend load path
The frontend loadModel handler had the old threshold (<=2) for
disabling reasoning on small Qwen3.5 models. Changed to <9 to
match the backend. This was causing 4B to not properly disable
thinking by default when auto-loaded.
* studio: move GGUF delete to per-variant level
For GGUF repos, the trash icon now appears on each downloaded variant
row inside the quantization expander instead of on the repo-level row.
Backend accepts optional variant param to delete specific GGUF files
(blob + symlink) rather than the entire repo cache.
* studio: restore ggufContextLength on page refresh
The Max Tokens slider was capped at 32768 on page refresh because
ggufContextLength was not restored from the status response.
Now set it from statusRes.context_length on reconnect.
* fix: remove <think> from Qwen3.5 response template marker
The train-on-responses-only feature uses template markers to find
where the assistant response starts. The Qwen3.5 response marker
included '<think>\n' which is only present when thinking mode is
enabled. With thinking disabled (default for <9B), the marker
never matched, causing 100% of samples to be dropped.
Changed response marker from '<|im_start|>assistant\n<think>\n'
to '<|im_start|>assistant\n' which works regardless of thinking mode.
* studio: fix sloth ASCII art alignment in training overlay
* fix: correct sloth ASCII art alignment to match Unsloth banner
* studio: add Python and terminal tool calling to chat
Register python and terminal tools alongside web search. Python
executor validates imports (stdlib only) via unsloth_zoo
rl_environments, runs code in a subprocess sandbox with 5-min
timeout and cancel support. Terminal executor blocks dangerous
commands (rm, sudo, etc.) and runs in a temp directory.
Update llama_cpp tool loop to show tool-specific status messages
and pass cancel_event through to executors. Rename composer
toggle from "Search" to "Tools" and show TerminalIcon for
execution status pills.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix Nemotron/transformers 5.x support, onboarding navigation, port binding
Backend:
- Dynamic transformers 5.x detection via tokenizer_config.json fetch
(checks for TokenizersBackend class, cached per-model)
- Bump transformers 5.x version from 5.2.0 to 5.3.0 across all workers,
setup scripts (setup.sh, setup.ps1)
- Auto-enable trust_remote_code for unsloth/* models needing transformers 5.x
(workaround for NemotronH config parsing bug in transformers)
- Auto-install mamba-ssm/causal-conv1d for SSM models (NemotronH, Falcon-H1)
with --no-build-isolation --no-deps to avoid torch version conflicts
- Add SO_REUSEADDR to port check in run.py (fixes Colab proxy stale connection
falsely reporting port as in-use)
Frontend:
- Fix "Skip to Chat" navigation: use window.location.href instead of React
Router navigate() to bypass useEffect redirect race
- Fix "Skip Onboarding" on splash: navigates to /studio (not /chat)
- Fix onboarding guard: only check isOnboardingDone() on initial mount
- Fix Chat card on step 1: add sr-only spacer for consistent alignment
- Fix Chat+Text both selected: clear RadioGroup value when Chat is selected
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: split tools toggle into Search and Code buttons
Replace the single "Tools" toggle with two independent toggles:
- "Search" (globe icon) enables web search only
- "Code" (terminal icon) enables Python and terminal execution
Add enabled_tools list field to the inference payload so the
backend only registers the tools the user has toggled on. Both
toggles appear in the main composer and the compare composer.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix tool calling import validation and error logging
Replace unsloth_zoo-dependent import checker with a standalone
ast-based validator using sys.stdlib_module_names. This properly
blocks non-stdlib imports (numpy, requests, etc.) and returns a
clear error message to the model so it can rewrite using only
stdlib.
Add full traceback to tool streaming error logs for debugging.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: parse gpt-oss harmony channels for clean safetensors chat output
gpt-oss models emit multi-channel output via harmony protocol tokens
(<|channel|>analysis<|message|>... and <|channel|>final<|message|>...).
TextIteratorStreamer with skip_special_tokens=True strips the special
tokens but leaves channel names concatenated with content, producing
garbled output like "analysisWe need to...assistantfinalHello!".
Add HarmonyTextStreamer that decodes with skip_special_tokens=False,
parses harmony markup via regex, and emits <think>analysis</think>
for the analysis channel and plain text for the final channel --
reusing the existing frontend reasoning UI.
Also expose supports_reasoning=True for non-GGUF gpt-oss models in
the /status endpoint so the frontend enables the Think toggle.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: use unsloth_zoo for Python sandbox validation
Set UNSLOTH_IS_PRESENT=1 and import check_python_modules and
check_signal_escape_patterns directly from unsloth_zoo instead
of a standalone fallback. This gives us the full Unsloth
validation including stdlib-only import checks and signal/timeout
escape pattern detection.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: allow all imports in Python tool sandbox
Remove stdlib-only import restriction. Keep signal escape
pattern detection via unsloth_zoo for safety.
* studio: fix ReadTimeout on tool streaming final pass
The 0.5s read timeout used for cancel-checking during streaming
also fires when waiting for the first response from llama-server
(e.g. reasoning model thinking for 15+ seconds). Add
_stream_with_retry() context manager that retries on ReadTimeout
while checking cancel_event, so the model has unlimited time to
think before producing the first token. Applied to both the
regular streaming path and the tool-calling final pass.
* fix: rewrite HarmonyTextStreamer with stateful incremental parsing
The delta-on-transformed approach had two critical bugs:
1. Before the full <|channel|>X<|message|> pattern was complete, the
strip-tokens fallback emitted "analysis" as plain text. Then when
the regex matched, _transform returned a completely different format
(<think>...</think>) and the delta was computed against the wrong
base string, producing fragments like "think>", "nk>", ">".
2. Even with full matches, the closing </think> tag shifted position
as content grew, so text[prev_len:] produced garbled deltas.
Replace with stateful incremental parsing that:
- Buffers until a complete channel+message pair is seen
- Emits <think> once when analysis channel first appears
- Streams analysis content deltas (computed on channel content directly)
- Emits </think> once when final channel first appears
- Streams final content deltas
- Closes open think tags in end()
Also skip the generic all_special_tokens stripping in
_clean_generated_text for gpt-oss since HarmonyTextStreamer already
produces clean output and the generic stripping was mangling <think>
tags.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: strip all <|...|> tokens in gpt-oss cleanup, not just harmony subset
The gpt-oss tokenizer has added tokens like <|return|> (id=200002) that
are not part of the harmony channel protocol but can leak into output.
The previous regex only stripped channel|message|start|end tokens.
Broaden the _clean_generated_text regex for gpt-oss to <\|[a-z_]+\|>
which catches all pipe-delimited tokens (return, constrain, reserved,
etc.) without matching <think>/<\/think> tags.
Verified: gpt-oss all_special_tokens are only <|return|>,
<|reserved_200017|>, <|startoftext|> -- none overlap with <think>.
The harmony tokens (channel, message, start, end) are added_tokens
but not in all_special_tokens.
* fix: hide config-only model repos from cached models list
Repos that only have metadata/config files cached (no .safetensors or
.bin weight files) were showing up in the Downloaded list with tiny
sizes like "1.8 KB" or "24 KB". These are just leftover config
snapshots from architecture checks, not usable models.
Filter the cached-models endpoint to only include repos that contain
actual model weight files (.safetensors or .bin).
* studio: fix toast description text contrast in dark mode
Add explicit !text-muted-foreground to toast description classNames
so secondary text (e.g. "Releases VRAM and resets inference state.")
is readable in dark mode.
* studio: fix Chat card icon alignment with size-4 spacer
Replace sr-only span (takes no space) with a size-4 shrink-0 div
matching the RadioGroupItem dimensions in other cards, so the Chat
icon aligns vertically with Text/Audio/Vision/Embeddings icons.
---------
Co-authored-by: workspace <user@workspace.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Manan17 <shahmanan170602@gmail.com>
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
* studio: switch helper model to Qwen3.5-4B-GGUF
Replace Qwen3-4B-Instruct-2507-GGUF with Qwen3.5-4B-GGUF as the
default helper model for LLM-assisted dataset detection. Same
UD-Q4_K_XL variant.
* studio: fix stale GGUF metadata when switching models (#4347)
Reset _supports_reasoning, _supports_tools, _context_length, and
_chat_template at the start of _read_gguf_metadata() to prevent
stale settings from a previous model leaking into the next load.
Co-authored-by: Daniel Han <daniel@unsloth.ai>
* studio: change login error to "Incorrect password", add reset-password CLI
- Login error now says "Incorrect password" instead of the generic
"Incorrect username or password" since Studio only has one account.
- Add `unsloth studio reset-password` command that deletes the auth
database so a fresh admin account with a new random password is
created on the next server start.
* studio: include reset command in login error message
* studio: change password setup subtitle wording
## Summary
- Add web search tool calling for GGUF models (Search toggle, DuckDuckGo via ddgs)
- Add KV cache dtype dropdown (f16/bf16/q8_0/q5_1/q4_1) in Chat Settings
- Fix Qwen3/3.5 inference defaults per official docs (thinking on/off params)
- Enable reasoning by default for Qwen3.5 4B and 9B
- Replace "Generating" toast with inline spinner
- Fix stop button via asyncio.to_thread (event loop no longer blocked)
- Fix CUDA 12 compat lib paths for llama-server on CUDA 13 systems
- Fix auto-load model name not appearing in selector
- Training progress messages + dataset_num_proc fix
Integrated PRs:
- #4327 (imagineer99): BETA badge alignment (already in tree)
- #4340 (Manan Shah): prioritize training models in model selection
- #4344 (Roland Tannous): setup.sh macOS python version compatibility
- #4345 (Manan Shah): revamp model+dataset checking logic
* studio: extract param count from model name as fallback
When HuggingFace API doesn't return totalParams for a model,
extract the param count from the model name (e.g. "Qwen3-0.6B"
-> "0.6B", "Llama-3.2-1B-Instruct" -> "1B"). Applied to both
the recommended list and HF search results.
* studio: read GGUF context_length via fast header parser, set max tokens
- Fast GGUF metadata reader (~30-55ms) parses only KV header, skips
tensor data and large arrays (tokenizer vocab etc)
- Extracts context_length and chat_template from GGUF metadata
- Returns context_length in LoadResponse for frontend to use
- Frontend sets maxTokens to actual context_length for GGUFs (e.g.
262144 for Qwen3.5-9B, 131072 for Qwen2.5-7B)
- Max Tokens slider shows "Max" and is locked for GGUFs
- Auto-load path also uses actual context_length from load response
- Toast auto-dismiss (5s) and close button for auto-load toast
* studio: GGUF TTS audio support (from PR #4318)
Add GGUF TTS audio generation via llama-server. When a GGUF model
loads, the backend probes its vocabulary to detect audio codecs
(SNAC/BiCodec/DAC/CSM/Whisper). If detected, the codec is pre-loaded
and the model is reported as audio to the frontend.
During chat, TTS models route to the audio generation path which sends
a per-codec prompt to llama-server's /completion endpoint, extracts
generated tokens/text, and decodes to WAV using AudioCodecManager.
Also strips base64 audio data from prior assistant messages to prevent
context overflow.
Co-authored-by: Manan Shah <mananshah511@gmail.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove package-lock.json from tracking
* studio: per-model inference defaults, GGUF max tokens fix, reasoning toggle
- Add inference_defaults.json with per-model-family sampling parameters
for ~50 families (Qwen3.5, Qwen3, Gemma-3, Llama-3, DeepSeek, etc.).
Values sourced from unslothai/docs and Ollama params blobs.
- Family-based lookup in inference_config.py: extracts model family from
identifier, matches against patterns (longest match first), merges with
priority: model-specific YAML > family JSON > default.yaml.
- Fix GGUF Max Tokens slider locked at "Max": store ggufContextLength
separately from maxTokens so the slider is adjustable (step=64).
- Fix Ministral YAML: top_p was literal string "default", now 0.95.
- Add reasoning toggle for thinking models (Qwen3.5, Qwen3, DeepSeek-R1,
DeepSeek-V3.1, etc.): detect enable_thinking support from GGUF chat
template metadata, pass --jinja to llama-server, send
chat_template_kwargs per-request. Frontend shows "Reasoning is ON/OFF"
pill button next to attachment button in composer.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: remove default system prompt injection
Backend was injecting "You are a helpful AI assistant." when no system
prompt was provided. Neither unslothai/docs nor Ollama specify a default
system prompt for most models. Now defaults to empty string, letting the
model's own chat template handle system behavior.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: use lightbulb icons and "Think" label for reasoning toggle
Lightbulb on when thinking enabled, lightbulb-off when disabled.
Label is just "Think" in both states; grayed out styling when off.
* studio: fix HTML file upload breaking chat
Replace SimpleTextAttachmentAdapter with custom TextAttachmentAdapter
(excludes text/html) and HtmlAttachmentAdapter that strips tags via
DOMParser, removing scripts/styles and extracting readable text content
instead of dumping raw HTML markup into the conversation.
* studio: show chat template in Configuration panel
Display the model's Jinja2 chat template in a new "Chat Template"
section under Settings (now open by default). For GGUFs, reads from
GGUF metadata; for safetensors, reads from tokenizer.chat_template.
Template is editable with a "Restore default chat template" button
that appears when modified. Section only shows when a model with a
chat template is loaded.
* studio: editable chat template with Apply & Reload
Chat template section now functional:
- Editing the template shows "Apply & Reload" (reloads model with
custom template) and "Revert changes" buttons
- For GGUFs: writes template to temp .jinja file, passes
--chat-template-file to llama-server on reload
- For non-GGUF: passes chat_template_override in load request
- Settings section now open by default
- selectModel supports forceReload to reload same model
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix DeepSeek reasoning detection and auto-load metadata
- Set _model_identifier before _read_gguf_metadata so DeepSeek
"thinking" template detection works (was always None before)
- Populate ggufContextLength, supportsReasoning, reasoningEnabled,
defaultChatTemplate in autoLoadSmallestModel GGUF path
* studio: add spacing before BETA badge in navbar
Add gap-1.5 on the logo Link container to space the BETA label
from the wordmark.
Co-authored-by: Imagineer99 <Imagineer99@users.noreply.github.com>
* studio: vertically center BETA badge with logo
---------
Co-authored-by: Manan Shah <mananshah511@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Imagineer99 <Imagineer99@users.noreply.github.com>
- GGUF: use -c 0 for model's native context size (no 4096 cap)
- GGUF: hide Max Seq Length slider (irrelevant), set Max Tokens to Max
- Non-GGUF: default Max Tokens to 4096
- Max Tokens slider shows "Max" label when at ceiling for GGUFs
- Run non-GGUF load_model in asyncio.to_thread for progress polling
- Auto-load smallest downloaded model when chatting without selection
- Wait for in-progress model load before inference (modelLoading store flag)
- Recommended list: 4 GGUFs + 4 hub models after case-insensitive dedup
- Model selector waits for cached data before rendering
- Toast close button repositioned, Sampling section open by default
- Add logging to _get_repo_size_cached exception handler
- Use -c 0 for llama-server (model's native context size, no 4096 cap)
- Run non-GGUF backend.load_model in asyncio.to_thread for progress polling
- Auto-load smallest downloaded model when user chats without selecting one
- Wait for in-progress model load before inference (no "No model loaded" error)
- Add modelLoading flag to zustand store for cross-component coordination
- Dynamic top models: send 8 GGUFs + 8 hub models, frontend caps 4+4 after dedup
- Case-insensitive dedup: downloaded models correctly hide from recommended list
- Prevent duplicate toasts: guard against double selectModel calls
- Model selector waits for cached data before rendering (no empty flash)
- Toast close button positioned at top-right with proper spacing
- Sampling section expanded by default in chat settings
- Global toast close button styling fix
llama-server uses stb_image internally which does not support WebP,
TIFF, AVIF, and other formats that browsers accept for upload.
Uploading a WebP image to a vision GGUF model caused a 400 error:
"Failed to load image or audio file" / "failed to decode image bytes".
Convert all uploaded images to PNG via PIL before base64-encoding and
forwarding to llama-server. This handles WebP, TIFF, BMP, GIF, AVIF,
and any other format PIL supports. RGBA images are converted to RGB
first since PNG with alpha can cause issues in some vision pipelines.
1. Backend: When a model fails with "No config file found" or similar
unsupported-model errors, wrap the message with "This model is not
supported yet. Try a different model." instead of showing the raw
Unsloth exception.
2. Frontend: Compute estimated download size from the HF search API's
safetensors.parameters dtype breakdown (BF16=2B/param, I32=4B/param,
F32=4B/param, etc.) and show it in the model picker instead of just
the param count. For example, Kimi-K2.5 now shows "~554 GB" instead
of "171B" (which was misleading since 171B params != 171GB download).
Three fixes on top of the download progress feature:
1. Backend: Replace broken "no .incomplete = done" completion check
with a 95% byte threshold. HF downloads files sequentially, so
between files there are briefly no .incomplete files even though
the download is far from done (e.g. Kimi-K2.5 reported "done"
after downloading 22KB of config files out of 595GB).
2. Frontend: Track hasShownProgress flag. Only show "Download
complete. Loading into memory..." if we actually displayed
download progress before. For already-cached models where the
first poll returns progress=1.0, this avoids the misleading
"Download complete" message.
3. Frontend: Deduplicate recommended vs downloaded -- filter out
models already in the "Downloaded" section. Cache the fetched
lists at module level so re-mounting the popover does not flash
an empty "Downloaded" section.
Roland Tannous