* UX: Refine chat preset and group built-in presets
* fix: reuse built-in preset names and unify GGUF state reads
* fix: built-in chat preset save and refresh behavior
* Add chat preset invariant tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: decouple chat presets from model-specific settings
Limit chat preset compare/apply/save behavior to temperature, topP, topK, minP, repetitionPenalty, presencePenalty, maxTokens, and systemPrompt.
Preserve legacy stored preset data on load for backwards compatibility, but stop treating model-specific settings such as checkpoint, trustRemoteCode, and maxSeqLength as part of preset identity.
Also align legacy prompt migration dedupe with the new preset semantics and add invariant coverage for preset-owned config comparisons.
* fix: detect built-in preset edits from param changes
* fix: correct built-in preset dirty state and speculative select values
* fix: preserve default preset sync and keep qwen think pristine
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* UX: single chat header error placement and selector alignment
* fix: centre model dropdown chevron
* fix: revert view mode single
* fix: allow model selector label to truncate in narrow headers
* fix(studio): use py.exe to detect supported Python on Windows
Description:
The previous detection looked at `python --version` on PATH and
hard-failed if the resolved Python wasn't 3.11-3.13. On systems
where Python 3.14 sits ahead of 3.13 in PATH order, this aborted
the installer even though a supported interpreter was installed.
Prefer the py.exe launcher and probe `py -3.13`, `py -3.12`,
`py -3.11` in turn. Fall back to `python --version` only when py.exe
is absent, and surface a clearer error when no supported version
can be found via either path.
* Studio: consolidate Windows studio overlay into single Tauri-gated block
Replace the in-file sentinel hotfix and the unconditional file-copy
overlay with a single block gated on $TauriMode. Hash-compare makes
re-runs no-ops, removing the sentinel-clobbering bug that occurred
when the second copy path overwrote the marker without re-adding it.
Non-Tauri --local installs no longer need a copy overlay: the
editable install above (uv pip install -e $RepoRoot --no-deps) makes
_PACKAGE_ROOT in unsloth_cli/commands/studio.py resolve to the repo
source tree via PEP 660 __file__-relative resolution, so
`unsloth studio setup` finds the local setup.ps1 and
install_python_stack.py without any file copying.
Plain PyPI installs invoked from a checked-out repo directory are
also no longer silently overlaid from cwd.
* fix(studio): work around uv space-in-path truncation on Windows
uv 0.11.x truncates `-c <path>` and `-r <path>` arguments at the
first space, breaking installs on Windows when the venv or repo
sits under a path containing spaces (e.g. C:\Users\First Last\...).
Pass paths through GetShortPathNameW to convert to 8.3 short form
before handing them to uv. Plain pip is unaffected and keeps the
original long path. No-op on Linux/Mac (gated on IS_WINDOWS and
on the path actually containing a space).
* Refactor Python stack overlay logic in install.ps1
Refactor overlay logic for Python stack installation and improve handling of missing target directories.
* Update Python installation logic in setup.ps1
* Studio: add github_repo seed reader and GitHub Support Bot recipe
Adds a first-party Data Designer seed reader that scrapes GitHub issues,
pull requests, and commits from one or more repositories via the GraphQL
API, and a learning recipe (GitHub Support Bot) that turns those rows into
synthetic support Q&A pairs for fine-tuning.
Backend (new plugin studio/backend/plugins/data-designer-github-repo-seed):
* GitHubRepoSeedSource config: repos, token (falls back to GH_TOKEN /
GITHUB_TOKEN env var), item_types (issues / pulls / commits),
per-resource limit (0 means all), max_comments_per_item.
* Rate-limit-aware GraphQL client (GitHubClient + RepoScraper) shared
across repos; flattens each item into a uniform row with columns
item_type, repo, number, title, body, state, author, created_at,
closed_at, url, labels, comments.
* Registered via the data_designer.plugins entry point.
Frontend:
* New seed_github block variant so the seed node card shows
"GitHub repositories" instead of the generic "Document file"
placeholder, with its own icon and inline summary (repo count +
item-type list).
* Rewritten seed dialog github_repo form: repos textarea pre-filled with
unslothai/unsloth + unslothai/unsloth-zoo, password input for the GH
token, items-per-repo number with an "All" toggle, and the noisier
options (item types, max comments, include comments) tucked under an
Advanced collapsible.
* Local model auto-load on Run: if a recipe uses an is_local provider
and the inference server is not already serving that model, the
executions hook calls /api/inference/load first. Removes the "open
/chat to load a model" prerequisite that users kept tripping on.
* Honor the recipe's run.rows value in the Run dialog (previously the
store reset to 5 regardless of what the template shipped).
Recipe (studio/frontend/src/features/data-recipes/learning-recipes/
github-support-bot.json):
* Defaults to the Local Model provider + unsloth/gemma-4-E2B-it-GGUF.
* Scrapes unslothai/unsloth and unslothai/unsloth-zoo, issues and pulls,
up to 100 items per resource.
* Two LLM blocks: normalized_question (llm-text) rewrites each thread
into a clean support question, support_answer (llm-structured)
produces JSON with answer / diagnosis_questions / cites / confidence.
* Run defaults to 10 rows for a quick smoke test.
Verified end-to-end on a running Studio: card renders, source-data
dialog is pre-populated, All toggle disables the limit input, the
recipe executes and produces rows against a loaded local GGUF.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: improve GitHub recipe support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: speed up GitHub scraper and harden the support-bot recipe
Addresses a perf issue found while demoing the github_repo seed reader:
Scraper is too slow at scale. The PRs GraphQL query pulls deeply nested
fields (reviewThreads, reviews, commits, timelineItems, etc.) so the
page size was pinned at 3 to stay under GitHub's node-count ceiling. 100
PRs meant 34 serial round trips. Added lighter query variants
(PRS_PAGE_QUERY_LIGHT, ISSUES_PAGE_QUERY_LIGHT) that drop the fields the
Studio flatten layer does not use (it only reads title, body, state,
author, labels, comments). With the light query PR pages can safely go
to 25 per page and issues to 50. The plugin scraper now passes
light=True to RepoScraper so Studio always uses the fast path; the heavy
query remains available for other callers.
Recipe defaults are now demo-ready with production knobs called out:
- max_parallel_requests: 1 and max_tokens: 800 so small local models
stay stable when running the support_answer structured column.
- support_answer prompt trimmed to 80-200 words so gemma-4-E2B GGUF can
actually comply with the schema. The canonical 150-300 word codex
prompt is still documented in the node3 markdown note for
production upgrades.
* Studio: rename GitHub recipe to 'GitHub Scraper' and add Easy mode
Changes the recipe framing from a single-purpose 'Support Bot' pipeline
to a general-purpose scraper that produces {user_request,
grounded_response} training pairs. Aligns with the canonical
github_data_gatherer dataset (11 enrichment tasks mirrored in pr_requests_20
/ issue_requests_20 on the input side and explain_pr / issue_fix_plan /
issue_solution on the output side).
Recipe JSON changes:
- columns[0] renamed normalized_question -> user_request, prompt now
inverts a GitHub thread into a realistic user ask instead of
normalising it.
- columns[1] renamed support_answer -> coauthor_response, emits
{response, followups, cites, task, confidence} and branches on
issue vs PR thread type.
- Notes rewritten to document the 11-task catalog and the canonical
production prompt to paste in for a full dataset backfill.
Frontend: Easy mode for github_repo recipes. The drag-and-drop canvas is
hidden behind an 'Advanced' tab; Easy mode is the default for any recipe
whose seed_source_type is github_repo. The Easy form reuses the existing
GithubRepoSeedForm (promoted to exported), adds a rows input bound to
previewRows, a model field bound to the model_config, and a single Run
button that calls runPreview() directly (no modal). Non-github recipes
see the same Editor / Runs tabs as before.
View mode persists per-recipe-id in localStorage under
recipe-studio:view-mode:<recipeId>.
* Studio: auto-detect server GH_TOKEN and widen Easy-mode detection
The GitHub seed form now fetches /api/data-recipe/seed/github/env-token
on mount and, when the server exposes a GH_TOKEN / GITHUB_TOKEN env var
and the token field is blank, shows a small 'Using server env var' badge
and swaps the placeholder text. The token value itself is never returned
to the UI.
Widens Easy-mode detection in recipe-studio-page.tsx so that recipes
saved before ui.seed_source_type was persisted also get the Easy tab:
falls back to recipe.seed_config.source.seed_type, which is always
present for github_repo seeds.
* fix: polish GitHub recipe UI
* Studio: default llama-server --threads to -1 (auto)
Previously we passed --threads only when the caller set an explicit
value, which meant llama-server fell back to its internal default.
That default has varied across llama.cpp builds (some versions use
hardware concurrency including hyperthreads, which hurts throughput on
CPU-heavy inference). Always passing --threads -1 pins the behaviour
to llama.cpp's auto-detect (physical cores).
Caller-supplied n_threads still wins when non-None.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch Easy mode to Runs pane on run start
Easy mode had no progress island or canvas overlay, so after clicking Run
the only visible state was the button label flipping to "Running..." while
the screen otherwise stayed identical. This reads as stuck even though the
job is progressing.
Wire an onExecutionStart callback from recipe-studio-page.tsx through to
useRecipeExecutions so that when a run is kicked off from easy mode, the
page flips to the executions view where the Runs sidebar, progress bar,
rate/ETA panel, and live log are rendered. Advanced/editor mode keeps its
existing behavior and stays on the canvas (it already has the floating
ExecutionProgressIsland).
* fix: clean up GitHub scraper layout
* Studio: forward llm-structured output_format as llama-server response_format
Local GGUF runs of llm-structured columns used to generate the full
max_tokens budget before the prompt-level "return JSON in a ```json
fence" instruction got parsed. Small models (e.g. gemma-4-E2B-it)
routinely broke format, so each row took ~65s and frequently failed
with "No parsable JSON structure within ```json markdown fence".
For any local-provider model_config referenced by an llm-structured
column, clone the model_config and inject response_format into the
clone's inference_parameters. Uses llama.cpp server's flat shape
(tools/server/README.md):
{"type": "json_schema", "schema": <output_format>}
Not the OpenAI-nested form; data_designer's OpenAI adapter forwards
response_format verbatim via facade._COMPLETION_REQUEST_FIELDS, and
llama-server's documented schema path expects the flat variant.
The clone is per (model_alias, column) so:
- llm-text / llm-judge columns that share the same alias keep
free-form sampling.
- Each structured column gets its own schema, so columns with
different output_formats don't collide.
Effect on gemma-4-E2B-it demos: every row parses cleanly, and the
model terminates immediately after the closing brace instead of
running to max_tokens. Net wall-clock is usually faster even though
grammar-constrained sampling is slightly slower per token.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: flip Easy to Runs pane before validation scrape, not after
Previously onExecutionStart fired inside runExecution, which runs AFTER
validateRecipe() -- and validation re-invokes the seed reader. For the
github_repo reader that is a full GraphQL scrape, so the user sat on a
"Running..." button with an otherwise unchanged Easy form for 10-15s
before anything moved.
Call onExecutionStart at the top of runWithValidation, right after we
have a payload to send. The view flips immediately; ensureLocalModelLoaded
+ validateRecipe now run against the Runs pane instead of a frozen Easy
form. runExecution still calls onExecutionStart downstream, but the
callback is idempotent (the page's easy -> executions guard skips the
second call), so no behaviour change for runs that pass validation.
If validation fails the toast + runErrors path still fires; the Easy
form's error banner still reads runErrors when the user switches back.
* Studio: unify data-recipe workflow auth on sk-unsloth-* keys
The previous commit (a61b4cc9) assumed storage.create_api_key(..., internal=True)
and storage.revoke_internal_api_key(key_id) existed, but those helpers were
only in the working tree, never committed. Recipe runs in local-model mode
were therefore crashing with 500 when _inject_local_providers tried to mint
a workflow key. This commit ships the missing pieces.
auth/storage.py:
- api_keys schema gains is_internal INTEGER DEFAULT 0 (with a guarded
ALTER TABLE migration so existing auth.db files upgrade in place).
- create_api_key takes an internal=False kwarg; internal keys are flagged
so they can be hidden from user-facing listings.
- list_api_keys takes include_internal=False so UIs never see workflow keys.
- New revoke_internal_api_key(key_id): id-only revoke for keys minted by
non-user subjects (the JobManager does not know a username).
core/data_recipe/jobs/manager.py:
- JobManager.start accepts internal_api_key_id and stores it on Job so
lifecycle handlers can revoke eagerly.
- _handle_event revokes on EVENT_JOB_COMPLETED / _ERROR / _CANCELLED.
- _pump_loop subprocess-died fallback also retires the key so a crashed
worker cannot leak a live sk-unsloth-* beyond its TTL.
- Revocation is best-effort (swallow exceptions) -- the 24h TTL is the
safety net if storage hiccups.
core/data_recipe/jobs/types.py:
- Job dataclass gains internal_api_key_id: int | None = None.
Replaces the bespoke 24h JWT path that jobs.py used to mint for local
providers. One mint/revoke/verify surface for every API key the server
issues, and revocation is now eager (seconds, not 24h) instead of TTL-only.
* Studio: plug workflow-key leak on unexpected create_job errors
Review follow-up on the sk-unsloth-* workflow-key lifecycle in
create_job. Previously the revoke handlers wrapped mgr.start(...) but
only caught RuntimeError and ValueError, and get_job_manager() sat
outside the try block entirely. Any other exception type (TypeError
from a mismatched kwarg, OSError from the queue write, etc.) would
bubble up to FastAPI and leave the minted key live until its 24h TTL.
Fix: one try block covers both get_job_manager() and mgr.start(), with
a trailing except Exception that revokes and re-raises. The
RuntimeError -> 409 and ValueError -> 400 paths are unchanged so
specific client-facing status codes still surface. Revocation is still
best-effort (_revoke_internal_api_key_safe swallows errors) because we
never want revoke failures to mask the original crash.
Severity is low -- the key can't bootstrap longer access and the 24h
TTL bounds the window -- but the reviewer's point stands: eager revoke
on every failure path is the right invariant.
* Studio: nest response_format under extra_body so pydantic accepts it
The previous commit dropped response_format at the top level of a cloned
model_config's inference_parameters, which BuilderConfig rejected with:
ValidationError: Extra inputs are not permitted [type=extra_forbidden]
data_designer.model_configs.1.inference_parameters.response_format
data_designer's BaseInferenceParams is a pydantic model with extra=forbid
and only a fixed set of fields (temperature, top_p, max_tokens,
max_parallel_requests, timeout, extra_body). The pass-through path for
anything the schema doesn't know about is `extra_body`, which the
OpenAI SDK spreads into the chat-completions request body at the top
level -- which is exactly where llama-server reads response_format from.
Inject under extra_body (merging with any existing extra_body contents)
so the clone validates. llama-server still receives
{"type": "json_schema", "schema": <output_format>} at the top level of
the request body, which is the flat shape llama.cpp's server expects.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: forward response_format to llama-server and fence-wrap the reply
Two-part fix for the llm-structured data-recipe path:
(1) The /v1/chat/completions proxy was dropping response_format. The
route's passthrough branch only triggered on tools / tool messages, so
requests carrying a JSON schema fell into the non-passthrough GGUF path
which calls generate_chat_completion (no response_format kwarg). The
schema never reached llama-server, so guided decoding was a no-op and
the model emitted free-form text that happened to parse a fraction of
the time. Widen the passthrough trigger and teach _build_passthrough_payload
to forward response_format so llama-server's GBNF grammar actually runs.
Guided decoding does not require supports_tools, so split the condition:
a request is now passthrough-routed if it carries tools/tool messages
(existing behavior) OR carries response_format (new). The vision guard,
streaming fork, and tools-choice defaulting are unchanged.
(2) data_designer's llm-structured parser looks for a ```json ... ```
markdown fence and discards anything else. Guided decoding emits only
the JSON object (the GBNF grammar has no fence tokens), so a
100%-valid schema-constrained run still ended up 0 ok / N failed with
"No parsable JSON structure within ```json markdown fence". In
_openai_passthrough_non_streaming, wrap each choice's content in the
expected fence when the caller asked for guided decoding. Already-fenced
content is left alone so other clients that prefer raw JSON are not
affected; the wrap is scoped to requests that carried response_format.
Net effect on the GitHub Support Bot recipe on a local GGUF: schema
actually binds during sampling, content arrives wrapped in the fence
data_designer expects, and generation terminates immediately after the
closing brace instead of running out to max_tokens.
* Studio: Easy mode runs a full run, capped at the user's row count
Easy mode used to call runPreview, which produces a test run: no
artifact persisted, reduced progress tracking, and framed in the Runs
pane as "Test run". The whole point of the form is to let a user kick
off a real dataset build with one click, so wire it to runFull instead
and bind the Rows input to fullRows (not previewRows).
runFull requires a non-empty fullRunName. The Easy form has no run-name
input, so seed a default on mount whenever Easy is active and
fullRunName is still empty. Uses `<recipe name> <iso-timestamp>` so
each Easy run gets a stable-ish default that still sorts chronologically
in the Runs pane. User can override it from the Advanced run dialog
before clicking Run.
Rename GithubScraperEasyView's rows props from previewRows/setPreviewRows
to rows/setRows so the view stays agnostic to which hook state the page
chooses to bind. Loading indicator now follows fullLoading.
* Studio: clamp GitHub scrape page size and memoize the materialization
Two wins for the "before Generating fires" gap on small previews:
(1) scrape_{issues,prs,commits} hardcoded per_page (50 / 25 / 100) and
only checked the trial limit AFTER the page was written, so a 1-row
Easy run still asked GitHub for a full 50-issue + 25-PR page, wrote
them all to JSONL, and then stopped because total_new already exceeded
the trial cap. Cap per_page at min(page_cap, trial_limit) so
github_limit=1 actually asks for first:1.
(2) GitHubRepoSeedReader.get_dataset_uri used to scrape fresh on every
invocation. data_designer calls the seed reader multiple times per
recipe job (validation, preview, per-column sampling), so a 2-repo
Easy preview ran the full GraphQL scrape three times back-to-back,
burning ~15s of dead air before any LLM generation began.
Added a module-level in-process cache keyed on
(repos, item_types, limit, include_comments, max_comments_per_item,
sha256(token)[:16]) that stores the JSONL path of the first
materialization. Subsequent calls with the same signature return the
cached path, guarded by a staleness check that drops the entry if the
file was tmp-cleaned. Raw token values never land in the key.
Net effect on a 1-row Easy run, 2 repos, limit=1: 2 GraphQL round
trips instead of ~12, and the first-to-Generating gap collapses from
~15s to roughly 2-3s.
* Studio: make Easy mode Rows input editable instead of snapping to 1
The Rows to generate input used type="number" with value bound directly
to the rows state and an onChange that coerced any non-positive parse
result back to 1. The moment the user pressed backspace to clear the
field, the parent re-rendered with value=1 and the caret jumped, making
it impossible to change the value without arrowing the browser's +/-
spinner.
Switch to a text input with inputMode="numeric" and pattern="[0-9]*"
(so mobile still shows a numeric keyboard, and the browser drops the
spinner buttons the user did not want). Add a local rowsText buffer so
the field can hold transient empty / partial digit strings while
editing without fighting the parent state; the canonical rows value
only advances when the buffer parses to a valid integer in [1, 10000],
and onBlur clamps back to 1 or 10000 if the user left it out of range.
No behavior change for valid numeric edits - the downstream runFull()
still sees a clean positive integer.
* Studio: expand dataset cells horizontally by column on click
Click a long cell to expand that whole column. Click again to collapse.
Replaces the prior row-level vertical expansion which made it hard to
compare cells across columns. State is scoped per execution and per
column; the row itself is no longer a click target.
* Studio: force expanded dataset column to grow wide enough to read
* Studio: disable thinking for local recipe inference and plumb the kwarg
Reasoning-capable models (gemma-3n, qwen3.5, etc.) emit a
<think>...</think> preamble ahead of the answer by default, which
roughly doubles the generated token count per row on a local GGUF
and pushes the actual answer past data_designer's json-fence regex
on llm-structured columns. Recipes want the terse answer, not the
scratchpad.
Two halves of the fix:
(1) routes/data_recipe/jobs.py: when _inject_local_providers walks
the recipe's model_configs to point them at the local endpoint, also
stash chat_template_kwargs={"enable_thinking": false} under each
config's inference_parameters.extra_body. OpenAI SDK spreads
extra_body into the top-level request body, so llama-server and the
Studio /v1/chat/completions route both see it.
(2) routes/inference.py: the chat-completions route previously
dropped chat_template_kwargs on the floor because the whitelist
body builder only forwarded known fields.
- At the top of openai_chat_completions, lift
chat_template_kwargs.enable_thinking from payload.model_extra
onto the typed payload.enable_thinking field when the caller
did not set the latter, so the non-passthrough GGUF path's
generate_chat_completion(...) call honors the override.
- Teach _build_passthrough_payload to forward a
chat_template_kwargs dict, and have _build_openai_passthrough_body
derive that dict from payload.enable_thinking so
response_format requests (structured columns) also land at
llama-server with the reasoning preamble suppressed.
Net effect on a 10-row support-bot run with gemma-4-E2B-it-GGUF:
responses arrive without <think> tags, wall-clock per call drops
roughly in half, and structured columns stop leaking reasoning
tokens through the GBNF-constrained output.
* Studio: update GitHub Support Bot learning recipe with maintainer layout
Replace the template with the hand-laid-out export from the maintainer
so note nodes ship with real x/y positions (scattered around the
graph instead of all stacked at x=480) and the edges / canvas pan look
correct on first load. Also picks up the maintainer's prompt tweaks and
output schema names (coauthor_response / user_request / followups / task /
cites / confidence).
Diff is mostly ui.nodes positions and prompt bodies; runtime shape is
unchanged (seed_config / columns still target model_1 against the Local
Model provider).
* Studio: auto-size dataset sample columns; wide text gets a wide column
Drop the per-column click-to-expand toggle and the 180-char truncation.
Every column now renders its full value. Columns with long text get a
min-w of 48rem so the text is readable without wrapping into a tall
block; narrow-content columns get a 12rem min-w. The table wrapper
already has overflow-x-auto, so wide-column totals cause a horizontal
scrollbar instead of cramming everything into the viewport.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix GitHub scrape progress
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add resetApiBase export for test setup
* Studio: rename github-support-bot output columns to User / Assistant
Previously emitted user_request and coauthor_response, which did not
match the canonical User / Assistant chat-pair shape that downstream
SFT consumers expect. Renamed the columns in the recipe JSON (columns,
UI node ids, edges, notes, prompt Jinja refs) and the matching copy in
the learning-recipes index, data-recipes-page, and easy view.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: make stop button actually stop generation
The UI stop button routes through assistant-ui's cancelRun, which aborts
the frontend fetch. Four issues combined to let llama-server keep decoding
long after the user clicked stop:
1. request.is_disconnected() does not fire reliably behind proxies
(e.g. Colab) that don't propagate fetch aborts.
2. llama-server defaults n_predict to n_ctx when max_tokens is not sent,
so a cancelled request keeps producing tokens up to 262144.
3. The httpx.Client pool keeps TCP keep-alive, so even a cleanly closed
stream reuses the same connection and llama-server's liveness poll
never sees a disconnect.
4. No explicit backend route to cancel - every cancel path relied on
is_disconnected.
Changes:
- Add POST /api/inference/cancel keyed by session_id/completion_id, with
a registry populated for the lifetime of each streaming response.
- Have the frontend (chat-adapter.ts) POST /inference/cancel on
AbortController abort, alongside the existing fetch teardown.
- Send max_tokens=4096 + t_max_predict_ms=120000 as defaults on every
outbound chat completion to llama-server; honoured by user overrides.
- Disable httpx keep-alive on the streaming client so connection close
reaches llama-server and its 1s liveness check fires.
No behaviour changes for non-streaming paths or for existing callers
that already pass max_tokens/session_id.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: harden stop-button cancel path and scope cancel route
- Require at least one identifier for /api/inference/cancel so a missing
thread id cannot silently cancel every in-flight generation.
- Scope /cancel to a dedicated studio_router so it is not exposed under
the /v1 OpenAI-compat prefix as a surprise endpoint.
- Store a set of cancel events per key in _CANCEL_REGISTRY so concurrent
requests on the same session_id do not overwrite each other, and
deduplicate in _cancel_by_keys so the cancelled count reflects unique
requests.
- Always send session_id with chat completions (not only when tools are
enabled) so non-tool GGUF streams register under it and are reachable
from /cancel.
- Register the non-GGUF stream_chunks path in the cancel registry too,
so transformers-based stop-button works behind proxies that swallow
fetch aborts.
- Only apply the 2-minute t_max_predict_ms wall-clock cap when the
caller did not pass max_tokens, so legitimate long generations on
slow CPU/macOS/Windows supported installs are not silently truncated.
- Remove the abort listener on normal stream completion so reused
AbortSignals cannot fire a spurious cancel POST after the fact.
* studio: close cancel-race and stale-cancel gaps in stop path
- Register the cancel tracker before returning StreamingResponse so a
stop POST that arrives during prefill / warmup / proxy buffering
finds an entry in _CANCEL_REGISTRY. Cleanup now runs via a Starlette
BackgroundTask instead of a finally inside the async generator body.
- Add a per-run cancel_id on the frontend (crypto.randomUUID) and in
ChatCompletionRequest so /api/inference/cancel matches one specific
generation. Removes the stale-cancel bug where pressing stop then
starting a new run in the same thread would cancel the retry.
- Apply t_max_predict_ms unconditionally in all three llama-server
payload builders (previously gated on max_tokens=None, which made it
dead code for UI callers that always send params.maxTokens). Raise
the default to 10 minutes so slow CPU / macOS / Windows installs are
not cut off mid-generation.
- Make _cancel_by_keys refuse empty input (return 0) so a future
internal caller can not accidentally mass-cancel every in-flight
request.
- Accept cancel_id (primary), session_id, and completion_id on the
/api/inference/cancel route. Unify the three streaming sites on the
same _cancel_keys / _tracker variable names.
- Annotate _CANCEL_REGISTRY as dict[str, set[threading.Event]].
* Add review tests for PR #5069
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: harden stop-button cancel semantics and wall-clock cap
- Make /inference/cancel match cancel_id EXCLUSIVELY when supplied.
Previously the handler iterated ('cancel_id','session_id','completion_id')
and unioned matches, so a stale cancel POST carrying {cancel_id:old,
session_id:thr} would still cancel a later run on the same thread via
the shared session_id. cancel_id is now a per-run exclusive key;
session_id / completion_id are only used as fallbacks when cancel_id
is absent.
- Close the early-cancel race. If /inference/cancel lands before the
streaming handler reaches _TrackedCancel.__enter__() (stop clicked
during prefill / warmup / proxy buffering), the cancel was silently
dropped. Stash unmatched cancel_ids in _PENDING_CANCELS with a 30 s
TTL; _TrackedCancel.__enter__() now replays any matching pending
cancel by set()-ing the event immediately after registration.
- Make t_max_predict_ms = _DEFAULT_T_MAX_PREDICT_MS conditional on
max_tokens is None at all three llama-server payload sites. The cap
is a safety net for callers who leave max_tokens unset (otherwise
llama-server defaults n_predict to n_ctx, up to 262144). Callers who
set an explicit max_tokens are already self-limiting and must not be
silently truncated at 10 minutes on slow CPU / macOS / Windows
legitimate long generations.
- Guard each StreamingResponse return with try/except BaseException so
_tracker.__exit__ runs even if StreamingResponse construction or any
preceding statement raises between _tracker.__enter__() and the
BackgroundTask attachment. Prevents a registry leak on that narrow
window.
* studio: close TOCTOU race and restore wall-clock backstop on UI path
- Close TOCTOU race in the pending-cancel mechanism. The previous fix
split cancel_inference's (cancel_by_keys + remember_pending_cancel)
and _TrackedCancel.__enter__'s (register + consume_pending) into
four separate lock acquisitions. Under contention a cancel POST
could acquire-then-release the lock, find the registry empty, and
stash ONLY AFTER __enter__ had already registered and consumed an
empty pending map -- silently dropping the cancel. Both call sites
now do their work inside a single _CANCEL_LOCK critical section, via
the new atomic helper _cancel_by_cancel_id_or_stash() and an
inlined consume-pending step in __enter__. Reproduced the race under
forced interleaving pre-fix; 0/2000 drops post-fix under parallel
stress.
- Apply t_max_predict_ms UNCONDITIONALLY at all three llama-server
payload sites. The previous iteration gated the cap on
`max_tokens is None`, which turned out to be dead code on the
primary Studio UI path: chat-adapter.ts sets
maxTokens=loadResp.context_length after every model load, so every
chat request carries an explicit max_tokens and the wall-clock
safety net never fired. The cap's original purpose is to bound
stuck decodes regardless of the token budget; it must always apply.
- Raise _DEFAULT_T_MAX_PREDICT_MS from 10 minutes to 1 hour. 10
minutes was too aggressive for legitimate slow-CPU chat responses
(a 4096-token reply at 2 tok/s takes ~34 min); 1 hour accommodates
that and still catches genuine zombie decodes.
- Prune _PENDING_CANCELS inside _cancel_by_keys as well, so stashed
entries expire proportionally to overall cancel traffic rather than
only to cancel_id-specific POSTs.
* studio: trim verbose comments and docstrings in cancel path
* studio/llama_cpp: drop upstream PR hashes from benchmark comment
* Add review tests for Studio stop button
* Consolidate review tests for Studio stop button
* Align cancel-route test with exclusive cancel_id semantics
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: move cancel cleanup to generator finally; drop dead helper
- Move _tracker.__exit__ from Starlette BackgroundTask into each
streaming generator's finally block. Starlette skips the background
callback when stream_response raises (OSError / ClientDisconnect),
which leaked _CANCEL_REGISTRY entries on abrupt disconnect.
- Check cancel_event.is_set() at the top of each GGUF while loop so a
pending-replay cancel falls through to final_chunk + [DONE] instead
of propagating GeneratorExit out of _stream_with_retry.
- Remove unused _remember_pending_cancel; _cancel_by_cancel_id_or_stash
superseded it.
* Add review tests for Studio stop-button
* studio: wire audio-input stream into cancel registry
- Register cancel_event with _TrackedCancel on the audio-input streaming
path so POST /api/inference/cancel can stop whisper / audio-input GGUF
runs. Previously the registry stayed empty on this branch, so the stop
button returned {"cancelled":0} and the decode ran to completion.
- Apply the same finally-based cleanup and pre-iteration cancel-event
check used on the other three streaming paths.
- Update the _CANCEL_REGISTRY block comment to list cancel_id as the
primary key (was stale "session_id preferred").
* Consolidate review tests for Studio stop-button cancel flow
- Merge the 6 behavioral tests from test_stream_cleanup_on_disconnect.py
(finally cleanup on normal/exception/aclose, pre-set cancel_event
pattern, and its regressions) into test_stream_cancel_registration_timing.py,
which is the PR's existing file covering the same area.
- Extend structural invariants to include audio_input_stream alongside the
three GGUF / Unsloth streaming generators: no _tracker.__enter__ inside
the async gen body, cleanup via try/finally, no background= on
StreamingResponse.
- Delete test_stream_cleanup_on_disconnect.py (now empty).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: make cancel-via-POST interrupt Unsloth and audio-input streams
Close two remaining gaps in the stop-button cancellation wiring:
- stream_chunks (Unsloth path): add a top-of-loop cancel_event check and
call backend.reset_generation_state() so cancel POSTs flush GPU state
and close the SSE cleanly instead of relying on request.is_disconnected
(which does not fire through proxies like Colab's).
- audio_input_stream: run the synchronous audio_input_generate() via
asyncio.to_thread so blocking whisper chunks do not freeze the event
loop, matching the pattern already used by the GGUF streaming paths.
* Add review tests for Studio stop-button cancel flow
* Consolidate review tests for Studio stop-button cancel flow
- Delete standalone test_cancel_registry.py at repo root: tests duplicated
test_cancel_atomicity.py / test_cancel_id_wiring.py and re-implemented
registry primitives inline (scaffolding).
- Extend tests/studio/test_stream_cancel_registration_timing.py with
regression guards for the iter-1 cancel-loop fixes:
structural: each streaming generator checks cancel_event in its loop;
audio_input_stream offloads next() via asyncio.to_thread;
stream_chunks cancel branch calls reset_generation_state().
runtime: Unsloth loop breaks on external cancel and resets state;
audio loop stays responsive under blocking next();
both loops emit zero tokens on pre-set cancel (replay path).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: extend stop-path to passthrough streams; tighten wall-clock cap
- Lower _DEFAULT_T_MAX_PREDICT_MS from 1 hour to 10 minutes so the
wall-clock backstop actually bounds runaway decodes when cancel
signaling fails.
- Wire _TrackedCancel and cancel_event.is_set() into
_openai_passthrough_stream and _anthropic_passthrough_stream and
disable httpx keepalive so stop requests from /v1 and /v1/messages
tool-calling clients reach llama-server.
- Apply t_max_predict_ms to the tool-passthrough request body so the
backstop covers passthrough paths as well.
- Symmetric pre-registration stash for session_id/completion_id
cancels (_cancel_by_keys_or_stash) so early cancels by those keys
replay on later registration like cancel_id.
- Drop dead except BaseException guards around StreamingResponse()
at four streaming sites; cleanup lives in the generator's finally.
* studio: harden cancel registry against ghost-cancel and leak paths
- Revert the session_id/completion_id stash in the fallback cancel
helper. session_id is thread-scoped and reused across runs, so
stashing it on an unmatched POST would fire cancel_event for the
user's next unrelated request via _TrackedCancel.__enter__.
cancel_id remains the only per-run unique key that gets stashed.
- Default max_tokens to _DEFAULT_MAX_TOKENS in the tool-passthrough
body. Mirror the direct GGUF path so OpenAI/Anthropic passthrough
callers who omit max_tokens get the same zombie-decode cap instead
of relying on the wall-clock backstop alone.
- Wrap _openai_passthrough_stream setup with an outer try/except
BaseException. The inner except httpx.RequestError does not catch
asyncio.CancelledError at await client.send, which would otherwise
leave _tracker registered in _CANCEL_REGISTRY indefinitely.
- Frontend stop POST uses plain fetch + manual Authorization header
instead of authFetch. A 401 on the cancel POST no longer refreshes
tokens or redirects the user to the login page mid-stop.
* Add review tests for Studio stop-button cancel flow
* studio: trim comments on stop-button review changes
Collapse multi-paragraph rationale blocks on the cancel registry,
_openai_passthrough_stream, and the frontend onAbortCancel handler
into one-line explanations of why the non-obvious behaviour exists.
Drop authFetch import that became unused when the cancel POST
switched to plain fetch.
* Consolidate review tests for Studio stop-button cancel flow
Move review-added tests out of test_cancel_dispatch_edges.py into the
existing PR test files that already cover the same areas:
- backend registry fan-out / exclusivity / idempotency / falsy-keys
edge cases moved into tests/studio/test_cancel_atomicity.py
- frontend plain-fetch (not authFetch) + manual Authorization header
moved into tests/studio/test_cancel_id_wiring.py
Delete the now-empty test_cancel_dispatch_edges.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop default-capping responses at 4096 tokens (follow-up to #5069) (#5174)
* Studio: stop default-capping responses at 4096 tokens
Follow-up to #5069. The 4096 default introduced for runaway-decode
defense silently truncates any caller that omits max_tokens. The
Studio chat UI sets params.maxTokens = loadResp.context_length after
a GGUF load, so it's fine, but every other consumer is not:
- OpenAI-API direct callers (/v1/chat/completions, /v1/responses,
/v1/messages, /v1/completions) where the OpenAI default is
effectively unlimited per response. langchain, llama-index, raw
curl, and the openai SDK all rely on that.
- Reasoning models. Qwen3 / gpt-oss reasoning traces routinely exceed
4096 tokens before the model emits a single visible content token.
The user sees the trace cut off mid-thought.
- Long-form generation ("write a chapter", "produce a full SVG").
Reproduced on this branch: gemma-4-E2B-it-GGUF Q8_0, prompt asking
for a 10000-word story, no max_tokens in the request:
finish_reason: stop (misleading -- should be 'length')
content_chars: 19772
content_tail: ...'a comforting, yet immense, pressure.\n\n*"'
Body ended mid-sentence on a stray opening quote, right at the 4096
token mark.
After this patch the same request returns 38357 chars ending with
'...held in a perfect, dynamic equilibrium.' -- a natural stop, not
a truncation.
Implementation: rename the constant to _DEFAULT_MAX_TOKENS_FLOOR and
set it to 32768. Each call site now uses the model's effective
context length when known, falling back to the floor:
default_cap = self._effective_context_length or _DEFAULT_MAX_TOKENS_FLOOR
The 10-minute t_max_predict_ms wall-clock backstop from #5069 is
preserved as the second line of defense.
Plumbed _build_passthrough_payload + _build_openai_passthrough_body
through the routes layer so the Anthropic and OpenAI passthrough
paths also respect the model's context length.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: cancel passthrough streams during llama-server prefill + route through apiUrl for Tauri
Three reviewer-flagged correctness gaps in the stop-button mechanism.
1) `_openai_passthrough_stream` could not honor cancel during prefill.
The cancel check ran inside the `async for raw_line in lines_iter`
body, so a cancel POST that arrived before llama-server emitted the
first SSE line was unobservable until prefill completed. With a long
prompt under proxy/Colab conditions -- the exact target scenario for
this PR -- that left the model decoding for a long time after the
user clicked Stop. Add an asyncio watcher task that closes `resp` as
soon as `cancel_event` is set, raising in `aiter_lines` so the
generator can exit. The watcher polls a threading.Event because the
cancel registry is keyed by threading.Event for the synchronous
/cancel handler.
2) `_anthropic_passthrough_stream` had the same blocking-prefill pattern.
Same fix.
3) The frontend's stop-button cancel POST used a bare relative
`fetch("/api/inference/cancel", ...)`, which targets the webview
origin in Tauri production builds (where the backend is at
`http://127.0.0.1:8888`). Route through the existing `apiUrl()`
helper from `lib/api-base.ts` to match every other Studio call.
Browser/dev builds get the empty base, so behavior is unchanged
there.
Verified via temp/pr_simulation/sim_5069_prefill_cancel.py: cancel
during prefill terminates within ~250ms on both passthrough paths
(was 145s+ on the Anthropic path before this change), and the standard
non-passthrough chat path still cancels with no regression.
* Studio: log cancel-body parse errors instead of silently swallowing
Reviewer-flagged defensive logging gap. The bare `except Exception: pass`
in `cancel_inference` would mask malformed payloads that hint at a buggy
client or a transport issue. Log at debug so future investigation isn't
left guessing whether `body={}` came from a missing body or a parse
failure. Behavior is unchanged: an unparseable body still falls through
to the empty-dict path and the cancel call returns `{"cancelled": 0}`.
* Studio: Anthropic passthrough cancel parity with OpenAI passthrough
Two reviewer-flagged consistency gaps in the cancel surface for
/v1/messages.
1) Anthropic passthrough did not register cancel_id, so a per-run cancel
POST (the cleanest Studio-style cancel path) silently missed when
the route hit `_anthropic_passthrough_stream`. The OpenAI passthrough
has registered (cancel_id, session_id, completion_id) since this PR
was first opened; mirror that here. Also add `cancel_id` to
`AnthropicMessagesRequest` so the route handler can plumb it through.
2) The cancel handler's fallback key list checked only completion_id
and session_id, never message_id. Anthropic clients that send their
native `id` (returned in the SSE message_start event) for cancel had
no way to hit the registry. Add message_id to the fallback list.
Verified via temp/pr_simulation/sim_5069_prefill_cancel.py: P2 now
cancels by cancel_id in 137ms (was hanging pre-fix), and the new P2b
case cancels by message_id in 77ms. P1 (OpenAI) and P3 (standard chat)
still pass with no regression.
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: probe AMD GPUs in llama-server VRAM detection
_get_gpu_free_memory in studio/backend/core/inference/llama_cpp.py
only queried nvidia-smi. On AMD ROCm hosts that returns nothing, so
the GPU list is empty, the auto-fit logic falls into the no-gpus
branch, and llama-server gets --fit on with no -ngl to anchor it.
The model loads on CPU even though the GPU is detected elsewhere in
Studio. Addresses #5106.
Add a torch-based fallback that runs after nvidia-smi fails or returns
empty:
import torch
if torch.cuda.is_available() and hasattr(torch.cuda, "mem_get_info"):
for ordinal in range(torch.cuda.device_count()):
free, _total = torch.cuda.mem_get_info(ordinal)
gpus.append((ordinal, free // (1024 * 1024)))
Works on AMD because the ROCm torch wheels Studio installs reuse the
entire torch.cuda.* namespace via HIP. Also rescues NVIDIA hosts
where nvidia-smi is missing from PATH (a secondary cause of the bug
on Windows). Matches the convention
studio/backend/utils/hardware/hardware.py:412 already uses for the
same fallback purpose.
Verified locally: nvidia-smi path returns the expected GPU and free
MiB; torch fallback returns valid VRAM when nvidia-smi is forced to
fail. Note: PR #4874 is a draft taking a different approach
(parsing vulkaninfo); the two are complementary.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review feedback on PR #5172
torch.cuda.device_count() enumerates GPUs RELATIVE to the current
CUDA_VISIBLE_DEVICES (or HIP_VISIBLE_DEVICES on ROCm). Returning
those visible ordinals directly lets _select_gpus rewrite
CUDA_VISIBLE_DEVICES with the wrong physical IDs: a process started
with CUDA_VISIBLE_DEVICES=2,3 would get its child llama-server
relaunched with CUDA_VISIBLE_DEVICES=0,1, targeting the wrong GPUs
and violating any scheduler pinning.
Translate visible ordinals back through the active CVD/HIP/ROCR
mask before returning. Falls through to bare ordinal when no mask
is set. Also drop the redundant int() cast on // -- bytes // 2**20
already returns int.
Verified: with CUDA_VISIBLE_DEVICES=6 and nvidia-smi forced to fail,
the torch fallback now returns (6, free_mib) instead of (0, free_mib).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix ROCm visibility precedence + narrow ROCm child env
Two reviewer-flagged correctness bugs in the AMD GPU probe path.
1) ROCm visibility precedence was reversed. torch.cuda enumerates GPUs
relative to HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES on ROCm builds,
but the probe's env-var lookup checked CUDA_VISIBLE_DEVICES first. With
CUDA_VISIBLE_DEVICES=0,1 and HIP_VISIBLE_DEVICES=6,7 the probe returned
[(0, ...), (1, ...)] when torch's view was actually [(6, ...), (7, ...)].
The wrong physical IDs flowed downstream into CUDA_VISIBLE_DEVICES for
the llama-server subprocess, pinning it to GPUs 0,1 instead of 6,7.
Fix: branch on torch.version.hip. On ROCm, prefer HIP > ROCR > CUDA
(matches torch's own ordering). On NVIDIA, use CUDA only -- ignoring
any HIP/ROCR vars the parent happens to have set.
2) Child env narrowing only set CUDA_VISIBLE_DEVICES. On ROCm, llama-server
honors HIP/ROCR; if the parent shell exported HIP_VISIBLE_DEVICES=4,5
and the selector picked just GPU 4, the child still saw both because
we never narrowed HIP/ROCR. Now we set all three on ROCm so the AMD
subprocess actually sees the planned subset.
Both branches verified via temp/pr_simulation/sim_5172_rocm_precedence.py
(7/7 cases pass), including the reviewer's verbatim R5 case
(CVD=0,1 + HIP/ROCR=6,7).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: sort GPU probe result + honor explicitly empty ROCm masks
Two reviewer-flagged correctness nits on top of eff55fb8.
1) Gemini medium: the torch fallback returned an unsorted list when the
visibility mask was non-sequential (e.g. CUDA_VISIBLE_DEVICES=5,2,9),
diverging from the docstring guarantee and the nvidia-smi path. Now
sorted by physical id.
2) Codex P2: an explicitly empty HIP_VISIBLE_DEVICES="" should mean
"no GPUs" per the codebase convention in
utils/hardware/hardware.py::_get_parent_visible_gpu_spec. The previous
`or` chain treated empty string as falsy and silently fell through to
ROCR / CUDA, producing wrong physical IDs. Switch to `is not None`
checks to match.
Verified via sim_5172_rocm_precedence.py (9/9 cases pass) including the
two new R8 (sort) and R9 (empty-HIP honored) cases.
* Studio: align nvidia-smi probe with torch fallback (sort + robust CVD)
Two follow-up Gemini-medium nits on PR #5172.
1) Fragile CVD parsing on the nvidia-smi path: `cvd.split(",")` would
raise ValueError on a trailing comma like "0,1," because the empty
trailing token is not skipped. The torch fallback already filters
empty tokens via `if x.strip()`; mirror that here.
2) Missing sort guarantee on the nvidia-smi path: the docstring promises
sort-by-id, the torch fallback now sorts, but the nvidia-smi path
relied on driver enumeration order. Add an explicit sort.
Both changes match what shipped in 6b1cccd6 for the torch fallback, so
the two probe paths now have identical CVD parsing + ordering semantics.
* Studio: drop cvd.strip() truthiness so empty CVD filters all GPUs
Reviewer-flagged correctness bug. The previous `if cvd is not None and
cvd.strip():` guard treated `CUDA_VISIBLE_DEVICES=""` as if the variable
were unset, leaving `allowed=None` (and `physical_ids=None` on the torch
path). On the nvidia-smi path that mattered: nvidia-smi ignores CVD
entirely, so the probe's `allowed` filter is the only thing that
respects the parent's "no GPUs" intent. Pre-fix the probe returned every
physical GPU when the parent had explicitly hidden them.
Drop the `.strip()` truthiness check on both paths. The downstream
`if x.strip()` token filter still keeps trailing-comma masks like
"0,1," safe, and an empty mask now produces an empty allowed/physical
set as expected (matching utils/hardware/hardware.py convention).
Verified via sim_5172_rocm_precedence.py R10 + R11 (now 11/11 cases
pass): nvidia-smi path with `CUDA_VISIBLE_DEVICES=""` now returns []
instead of leaking the hidden GPUs.
* Studio: log ROCm env-var failures instead of silently swallowing
Reviewer-flagged defensive logging gap. The bare `except Exception: pass`
around the HIP/ROCR env-var assignment would mask anything from a
missing torch import to an unexpected version object shape. Log at
debug so a failed AMD child-env narrowing is at least traceable.
Behavior is unchanged: torch missing or version probe failing still
leaves the child with only CUDA_VISIBLE_DEVICES set.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: stop currency escape from breaking inline LaTeX
The currency-escape preprocessor in studio/frontend/src/lib/latex.ts
matched the opening dollar of any $<digits>...$ span and inserted a
backslash. The result was that text like "$30^\circ$" or
"**$90 - x$**" rendered as raw characters with stray dollar signs.
Fixes#5164.
Add two helpers in front of the escape:
- hasInlineMathCloser looks for an unescaped, non-doubled closing
dollar within the same line. Bold-wrapped spans (**$X$**) are always
treated as math since LLMs use that form for bold math.
- looksLikeMathBody filters multi-token bodies that look like prose
between two currency tokens ($5 to $10, $5, $10).
Verified against 111 inputs: the issue body, common LaTeX patterns
(Greek vars, fractions, integrals, vectors, exponents), prose currency
in lists and sentences, code blocks, and headings. All pass.
* Address review feedback on PR #5170
- Drop ^ and _ from MATH_OP_RE since LATEX_CHAR_RE already short-
circuits on those before MATH_OP_RE is consulted (Gemini comment).
- Treat compact currency ranges like $5-$10 and $5/$10 as currency
rather than math. The body between the first two dollars in those
forms is "5-" or "5/", a single non-whitespace token that previously
hit the math shortcut. Extend TRAIL_PUNCT_RE to strip - and / so the
trimmed body comes back as pure currency. (Codex comment.)
- Honour __underscore-bold__ around math the same way as **-bold**.
Markdown allows both delimiters and LLMs do reach for the underscore
form. (Gemini comment.)
Verified against the existing 18 cases plus 5 new ones for the range,
slash, and underscore-bold scenarios. All pass.
* Studio: fix numeric inline math + currency-as-closer in LaTeX preprocess
Two reviewer-flagged real-world misses in the inline-math heuristic.
1) Numeric-only operator forms like $2 + 2$, $100 < 200$, $1,000 - 500$
were getting their leading $ escaped, so the renderer never saw them
as math. The body has a math op but no lone-letter variable, so the
old looksLikeMathBody required the lone-letter clause and rejected
purely numeric expressions. Add SIMPLE_MATH_RE to recognise number-
or-letter operands joined by math operators.
2) Prose like "Starts at $5 + a $10 add-on" was being treated as one
math span "5 + a " with the second currency token mistaken for the
closer. The body satisfied the math-op + lone-letter check, so the
span got accepted and the renderer ate "10 add-on". In hasInlineMathCloser,
reject any candidate $ whose next character is a digit -- that's almost
always another currency token starting, not the closer of a real math
span (math doesn't follow $ with a bare digit).
Verified via temp/pr_simulation/sim_5170_latex.mjs: 25/25 cases pass,
including the 6 reviewer numeric-math cases, 3 currency-as-closer cases,
and 16 regression checks against the originally shipped behavior.
* Studio: kill in-flight llama-server before spawning a new one
Two rapid Apply clicks in the chat settings panel can race two
load_model calls. Both pass the Phase 1 _kill_process (because neither
has stored its Popen handle yet), both download / read metadata, and
both reach Phase 3 and spawn a server. Only the last reference is
tracked in self._process. The first server becomes an orphan that
holds the model in RAM until the kernel OOM kicks in. Addresses #5161.
Two complementary changes in studio/backend/core/inference/llama_cpp.py:
1. At load_model entry, set the existing _cancel_event so any in-flight
load aborts at its next checkpoint, then bind a fresh Event for
the new load. Subsequent _kill_process and download phases pick up
the new event.
2. Inside the Phase 3 lock, immediately before subprocess.Popen, run a
defensive _kill_process that removes any orphan handle a racing
load might have stored after the first kill ran.
Speculative decoding cleanup (also touched while in this code path):
The chat UI used to send "ngram-mod" as the wire value when the
speculative dropdown was On, and the backend mapped that to the
4-flag combo --spec-type ngram-mod --spec-ngram-size-n 24 --draft-min
48 --draft-max 64. Switch the wire value to "default" and have the
backend pass the single llama-server flag --spec-default. That flag
expands to the exact same params (see common/arg.cpp:3905-3914 in
llama.cpp). Default-on for non-vision models is preserved.
Verified:
- Unit test: planted Popen orphan handle is terminated before
self._process is overwritten.
- All ten spec-cmd mappings produce the expected llama-server args
("default" -> --spec-default, "off" / null -> no flag, vision ->
always disabled, manual "ngram-mod" / "ngram-simple" still work).
* Address review feedback on PR #5171
The previous attempt at cancelling in-flight loads via
``self._cancel_event.set()`` followed by
``self._cancel_event = threading.Event()`` was broken in two ways
(flagged independently by Gemini and Codex):
1. Sub-methods like _download_gguf consult ``self._cancel_event``
on every check. After the rebind, the in-flight thread reads the
FRESH unset Event, not the one we just set, so cancellation never
propagates.
2. Worse, if unload_model() lands between ``set()`` and the rebind,
unload's signal hits the OLD event and is then immediately
discarded when load_model swaps in a fresh Event. The user's
stop-request silently no-ops.
Revert to the original ``self._cancel_event.clear()``. The Phase 3
defensive ``_kill_process()`` introduced in this branch still closes
the orphan-process race that #5161 reports: even if two concurrent
loads both pass Phase 1 with self._process == None, the loser's
Phase 3 kill terminates the winner's Popen handle before overwriting
it, so we end up with exactly one llama-server process.
* Studio: coerce legacy speculative-type values for the simplified dropdown
The Speculative Decoding control was simplified to On (default) / Off,
but the backend still accepts and reports the older manual modes
(ngram-mod, ngram-simple). When a load response or status refresh comes
back with one of those values -- whether from an external API caller, a
model loaded before this PR landed, or a not-yet-upgraded backend -- the
controlled Select renders with an empty trigger because the value is not
in the SelectItem list.
Add a tiny normaliser at both entry points (status refresh + post-load)
so legacy manual modes coerce to "default". The user sees "On" instead
of a blank dropdown, and reapplying lets llama.cpp pick its own preferred
strategy via --spec-default.
Reviewer-flagged finding on PR #5171.
* fix: use (gguf) context length before max seq length
For GGUF models context length is used instead of `maxSeqLength`
Fixes#4893
* Studio: split rollback max_seq_length from target load/validate
Restore params.maxSeqLength as the value passed to the target model's
validateModel and effectiveMaxSeqLength fallback, and introduce a
separate rollbackMaxSeqLength used only in the rollback loadModel call.
Without the split, switching from a GGUF model with a large native
context to a non-GGUF target polluted the new load via the
effectiveMaxSeqLength else-branch.
Also:
- Detect a previous GGUF via isGguf, activeGgufVariant, or a .gguf
suffix on the checkpoint, so local/LM Studio paths not present in
the models catalog still take the GGUF rollback path.
- Use 0 as the GGUF rollback fallback to match the sentinel used at
the normal GGUF load site (ggufContextLength fallback to 0); 4096
would reintroduce the original truncation bug when both
customContextLength and ggufContextLength are null.
- Reuse the captured stateBeforeUnload for modelRequiresTrustRemoteCode
and drop the now-unused DEFAULT_INFERENCE_PARAMS value import.
* Studio: drop pending customContextLength from GGUF rollback
rollbackMaxSeqLength previously preferred customContextLength over
ggufContextLength, but customContextLength is a pending, not-yet-applied
slider edit: chat-settings-sheet.tsx treats it as the "dirty" marker
(ctxDirty = customContextLength !== null) and use-chat-model-runtime.ts
clears it to null on every successful load. Rollback exists to restore
the previously-loaded model at its confirmed running context, so leaking
a pending slider value can load the rollback target at a context the
user never validated against VRAM, potentially causing the rollback
itself to fail.
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: detect reasoning_effort and preserve_thinking in chat templates
Previously Studio's chat template sniffer only recognized Qwen's
enable_thinking and DeepSeek's thinking markers. For gpt-oss (Harmony
templates) and newer Qwen3.6 templates, the Think toggle was hidden or
could only be flipped on/off.
This change adds two new detections and corresponding UI controls:
1. reasoning_effort style (gpt-oss). When the chat template contains
reasoning_effort, the Think button becomes a Low / Medium / High
dropdown and the backend forwards {"reasoning_effort": <level>} in
chat_template_kwargs. Load-time --chat-template-kwargs flag is also
switched to the new style.
2. preserve_thinking kwarg (Qwen3.6). Independent of the reasoning
toggle. When the template mentions preserve_thinking, a new
Preserve Thinking on/off pill is shown next to Think. Off by
default, persisted via localStorage. When on, the backend adds
{"preserve_thinking": true} to chat_template_kwargs so past-turn
<think> blocks are kept in the prompt instead of being stripped.
Backend helper _request_reasoning_kwargs now merges all applicable
kwargs into a single chat_template_kwargs dict based on the model's
detected style and template capabilities. Inputs are validated with
Literal types in the Pydantic request model.
Tested end to end against cached GGUFs for unsloth/gpt-oss-20b-GGUF and
unsloth/Qwen3.6-35B-A3B-GGUF. Confirmed the llama-server startup
--chat-template-kwargs flag and per-request JSON body carry the
expected keys for all combinations.
* Studio: review pass and CI format fixes for reasoning-styles PR
Addresses review feedback and pre-commit CI:
- Preserve Thinking pill in shared-composer now gates on modelLoaded
only, matching the thread.tsx toggle. Previously the inline version
disabled whenever supports_reasoning was false.
- The non-GGUF already_loaded LoadResponse now emits reasoning_style
(and supports_preserve_thinking=False) so a reconnecting frontend
sees the correct style for an already-running gpt-oss safetensors
model.
- use-chat-model-runtime reconnect path now always clears
reasoningEnabled for models without reasoning support instead of
inheriting the previous model's state.
- _reasoning_default is now reset alongside the other reasoning flags
in both backend reset blocks.
- supports_reasoning description updated to mention reasoning_effort
alongside enable_thinking.
- Ran scripts/run_ruff_format.py on the touched Python files to
satisfy pre-commit.ci.
* Studio: detect reasoning flags on safetensors load + share Qwen param helper
Addresses bot review feedback:
- Extract the chat-template substring sniffer out of _read_gguf_metadata
into a module-level detect_reasoning_flags(template, model_id) helper.
Also runs on the safetensors / transformers load paths:
- POST /api/inference/load non-GGUF LoadResponse
- already_loaded non-GGUF early return
- GET /api/inference/status non-GGUF branch
The gpt-oss fallback via backend._is_gpt_oss_model() is preserved so
safetensors gpt-oss still surfaces reasoning controls even when no
chat_template is stored on the model record.
- Deduplicate the Qwen3 / Qwen3.5 / Qwen3.6 Think-toggle parameter
adjustment into a single features/chat/utils/qwen-params.ts. Both
the assistant-ui Think toggle (thread.tsx) and the shared composer
(shared-composer.tsx) now import the same helper. The superset that
applies presence_penalty=1.5 for Qwen3.5 and Qwen3.6 is now used by
both sites (thread.tsx previously did not apply it).
* Studio: fill missing reasoning flags on safetensors status + add always_on reset
Round 4 review fixes:
- routes/inference.py safetensors status response now populates
reasoning_always_on and supports_tools from detect_reasoning_flags.
Previously Pydantic defaulted both to False, so safetensors models
with always-on <think> templates or tool-calling templates were
silently losing those flags on /api/inference/status reconnect.
- routes/inference.py already_loaded safetensors branch now falls back
to backend._is_gpt_oss_model() when the chat template is missing,
matching the status-endpoint behaviour.
- Safetensors status endpoint log_source set to "Safetensors status"
so the emitted template-detection log lines are attributable.
- chat-runtime-store clearCheckpoint now also resets reasoningAlwaysOn
so switching from an always-on reasoning model to a non-always-on
one does not leave the Think button permanently locked on.
* Studio: skip reasoning kwargs when always-on; narrow non-GGUF advertisement
Round 5 addresses reviewer feedback:
- _request_reasoning_kwargs and the load-time --chat-template-kwargs
emission now skip when _reasoning_always_on is true. Templates with
hardcoded <think> tags do not consume enable_thinking / reasoning_effort
so sending them was noise.
- Non-GGUF (Unsloth / transformers) LoadResponse and InferenceStatusResponse
paths no longer advertise template-derived supports_reasoning /
reasoning_style / supports_preserve_thinking / supports_tools. The
transformers generation path does not yet forward chat_template_kwargs
to tokenizer.apply_chat_template, so exposing the UI controls on those
models was misleading. Only the gpt-oss Harmony case is kept
(reasoning_style = reasoning_effort) because it is handled via the
HarmonyTextStreamer at the tokenizer level. A follow-up PR can thread
chat_template_kwargs through the transformers path and re-enable the
broader detection.
- GGUF / llama-server paths keep the full detect_reasoning_flags output.
* add unsloth studio desktop app
* Fix review findings
- studio/src-tauri/tauri.conf.json: retarget updater to staging repo
(danielhanchen/unsloth-staging-2); switch to unslothai/unsloth on upstream merge.
- studio/src-tauri/linux/postremove.sh: drop the interactive read loop and the
/home/* iteration. Package maintainer scripts must stay non-interactive and
must not touch other users' data.
- studio/frontend/src/app/auth-guards.ts: honor tauriAutoAuth() boolean. Failed
auto-auth now redirects to /login; requireGuest/requirePasswordChangeFlow
only redirect to /chat when auth succeeds. The new early-return on failed
auth is intentional so the login / change-password flows remain reachable
when desktop auth is not yet established.
- studio/frontend/src/config/env.ts: keep fetched=false on health failure so
later calls retry instead of caching the client-side platform guess.
- studio/src-tauri/src/install.rs: pick the available system package manager
(apt-get, dnf, zypper, pacman); AppImage bundles run on non-Debian distros.
- studio/frontend/src/lib/open-link.ts + markdown-text/sources callers: return
boolean from openLink so callers only preventDefault on handled URLs; relative
hrefs now navigate natively.
- studio/frontend/src/features/settings/tabs/about-tab.tsx: fetch(apiUrl(...))
so the version request targets the backend port in desktop mode. The bare
/api/health predates the Tauri webview (blame: the earlier onboarding commit,
which ran with same-origin frontend/backend); in desktop mode the webview
origin is tauri://localhost so the bare path fails.
- install.ps1: gate the install_python_stack.py hotfix on a sentinel comment
instead of a content regex; append the sentinel after applying so reruns
are unambiguous.
- unsloth_cli/commands/studio.py _write_auth_secret: use the atomic mkstemp +
os.replace path on Windows too; chmod calls are wrapped in try/except OSError.
- studio/src-tauri/src/preflight.rs probe_existing_backends: fan out the health
probes concurrently; desktop-auth status still runs sequentially per candidate.
reqwest::Client is internally Arc-wrapped so the in-loop .clone() is a
refcount bump, not a deep clone; annotated inline.
- studio/src-tauri/src/preflight.rs run_cli_probe: wait() after kill() to reap
the child, matching probe_cli_capability.
- studio/src-tauri/src/process.rs + main.rs: add stop_backend_detached and use
it from the tray quit handler so the 5s graceful-wait does not block the
Tauri main loop. RunEvent::Exit keeps the synchronous safety-net call.
- studio/backend/main.py: drop the permissive localhost CORS regex in
api-only mode; the explicit allow_origins list is sufficient.
- .github/workflows/release-desktop.yml: drop max-parallel: 1 so platform
builds run in parallel, and lift releaseBody to an env var so the three
tauri-action invocations share one source of truth.
* Fix review findings (loop 2)
- studio/backend/auth/storage.py update_password: clear_desktop_secret()
alongside clear_bootstrap_password() so rotating the admin password
also revokes any previously provisioned .desktop_secret. Without this,
an old local desktop credential keeps minting fresh admin tokens via
/api/auth/desktop-login after a password rotation.
- studio/src-tauri/src/desktop_auth.rs provision_desktop_auth: wrap
cmd.output().await in tokio::time::timeout(30s). DESKTOP_AUTH_LOCK is
held across the whole desktop_auth flow, and previously a hanging
`unsloth studio provision-desktop-auth` subprocess would pin the lock
indefinitely and freeze every subsequent desktop_auth call.
* Add review tests
* Consolidate review tests
Merge review-added tests into the existing studio/backend/tests/test_desktop_auth.py
(the PR's authoritative desktop-auth test file). Drops three scaffolding files under
tests/python/ in favor of five focused tests next to the tests they extend:
- test_update_password_clears_desktop_secret (runtime)
- test_update_password_on_unknown_user_leaves_desktop_secret_intact (runtime)
- test_cli_provisioning_delegates_to_storage_create_desktop_secret (source-level)
- test_cli_connect_auth_db_reads_storage_db_path (source-level)
- test_desktop_auth_provision_has_bounded_timeout (Rust source-level)
* Revert auth-guards.ts Tauri branches to unconditional form
The review loop on PR 5144 introduced a regression: the isTauri branch of
requireAuth redirected to /login when tauriAutoAuth() returned false, and
requireGuest / requirePasswordChangeFlow silently fell through on the same
condition. The Tauri desktop app authenticates via a local auto-generated
secret; it must never surface /login or /change-password to the user. A
failed auto-auth should let the startup layer retry, not expose a password
form.
Restore the three Tauri branches to the author's original unconditional
form (requireAuth: return; requireGuest / requirePasswordChangeFlow: throw
redirect({to: '/chat'})). Keep the rest of the review fixes -- the
apiUrl() fetch wrapping, authRedirect helper, and fetchAuthStatus refactor
are all legitimate improvements and are preserved.
* Revert release-desktop.yml to author's version
The review loop's workflow-file tweaks (drop max-parallel: 1, lift releaseBody
to an env var) are cosmetic. OAuth tokens cannot push workflow-file changes,
and fine-grained PATs cannot honor maintainerCanModify on a third-party fork.
Reverting the workflow file to wasimysaid's version lets the push go through
without needing a classic PAT with both repo and workflow scopes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Daniel Han <unslothai@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
`TestVisionCacheOnException::test_exception_result_cached` currently
patches `load_model_config` with `side_effect=OSError("network down")`
and asserts `assert_called_once()`. That assertion is impossible by
design: `_is_vision_model_uncached` in
`studio/backend/utils/models/model_config.py` intentionally returns
`None` for `OSError` so `is_vision_model` does not cache the fallback
and retries on the next call. The module docstring on
`_vision_detection_cache` itself spells this out:
Only definitive results (True/False from successful detection) are
cached; transient failures (network errors, timeouts) are NOT
cached so they can be retried.
The test has been failing identically on every downstream review run
against `unslothai/unsloth` main (e.g. `unsloth#5115`, `unsloth#5080`),
but the failure is not introduced by any of those PRs and does not
gate correctness.
Fix the collision by splitting the class into the two contracts the
code actually implements:
1. `test_permanent_exception_result_cached` keeps the original
intent ("exception falls back to False and that False is cached")
but uses `ValueError`, which is one of the exception types
`_is_vision_model_uncached` treats as permanent and caches. No
`huggingface_hub` import needed.
2. `test_transient_exception_not_cached` pins the opposite contract
with the original `OSError("network down")`: the call returns
False but the second invocation re-runs detection
(`call_count == 2`). This guards against a future regression
where somebody caches transient failures and then users with a
flaky network permanently see wrong detection for a model.
Both tests use `assert ... is False` on the public API and mock-count
assertions on `load_model_config`; no private helpers are touched.
* Replace assistant-ui's shared-store autoscroll with independent per-pane state
- assistant-ui's useThreadViewportAutoScroll uses shared state that causes random scrolling issues on the compare page
- Replaced with independent per-pane autoscroll implementation
* Fix scroll-click scrolling for slow movements; polish sidebar animations and UI
- Fix middle-click scroll not responding to very slow movements
- Add animations to sidebar navigation icons; font adjustments
- Align app menu styling for consistency
- Fix rare lag on sidebar expand/collapse
- Smooth right sidebar (parameter tuning) animations and expand/collapse lag
* Address PR feedback: clean up unused code and polish sidebar
- Remove unused _SuggestionItem component and its dependencies (SuggestionPrimitive import, SUGGESTION_TOOLS, toolIconMap)
- Log delete-message errors to console before showing toast for better observability
- Further sidebar menu design adjustments
* Address PR feedback: auto-follow guard and NavItem cleanup
- Prevent auto-follow from breaking on short threads: gate wheel/touch detach on scrollTop > 0 so no-op upward gestures on non-scrollable viewports don't flip userDetachedRef
- Simplify NavItem: collapse identical-branch ternary and drop unused variant prop
* Address PR feedback: model selector consistency and nested scroll fix
- Adjust model selector to be consistent with the sidebar
- Fix auto-follow breaking when user scrolls inside nested scrollable regions (reasoning/tool panels): gate wheel/touch detach on innerScrollWillConsumeUpward(target) so bubbled events from inner scrollers don't flip viewport intent (Codex review)
* Filter layout-induced upward scroll deltas from auto-follow detach
Accumulator now counts distanceFromBottom growth instead of raw -scrollTop delta, so browser scroll-anchoring on auto-collapsing panels (reasoning, tool outputs) no longer falsely detaches auto-follow mid-stream.
---------
Co-authored-by: sneakr <hauzin@hotmail.com>
* Studio: support images on /v1/messages (Anthropic-compat)
Translate Anthropic `image` content blocks (base64 and url sources) to
OpenAI `image_url` multimodal parts so the Anthropic endpoint reaches
llama-server's native vision path. Mirrors the `/v1/chat/completions`
vision behavior: 400 when the active GGUF isn't a vision model, and
embedded images are re-encoded to PNG (stb_image format coverage).
Server-side agentic loop is disabled when images are present, matching
the existing `not image_b64` gate on /v1/chat/completions.
Adds translator + normalizer unit tests.
* Studio: address gemini-code-assist review on /v1/messages image support
- Preserve interleaving of Anthropic text + image content blocks in the
translator (previously flattened all text first, then all images).
- Let _normalize_anthropic_openai_images return has_image so the route
skips the second scan it was doing.
- Use module-level base64/io in the helper instead of re-importing.
* Studio: forward standard OpenAI tools / tool_choice on /v1/responses
Mirrors the /v1/chat/completions client-side tool pass-through from #5099
so clients (OpenAI Codex CLI, OpenAI Python SDK, ...) that target the
Responses API receive structured function_call output items instead of
plain text with tool-call tokens leaking into content.
- ResponsesRequest: type tools/tool_choice properly, add parallel_tool_calls;
accept function_call and function_call_output input items for multi-turn
- Translate flat Responses tool / tool_choice shape to the nested Chat
Completions shape before forwarding to llama-server
- _normalise_responses_input: map function_call_output -> role="tool",
function_call -> assistant tool_calls (preserving call_id)
- Non-streaming: map returned tool_calls -> top-level function_call
output items keyed by call_id
- Streaming: emit response.output_item.added (function_call),
response.function_call_arguments.delta/.done, and response.output_item.done
per tool call while keeping the text message at output_index 0
- Pytest coverage: tools/tool_choice translation, multi-turn input mapping,
non-streaming tool_calls mapping, response round-trip
* Studio: merge system messages and close inner stream on /v1/responses
Fixes two issues surfacing when OpenAI Codex CLI drives /v1/responses
against a GGUF with a strict chat template (gpt-oss harmony, Qwen3, ...).
1. "System message must be at the beginning" upstream errors
Codex sends `instructions` AND a `role:"developer"` message in `input`,
producing two separate system-role messages. Strict templates raise
when a second system message exists or when one appears after a user
turn. _normalise_responses_input now hoists all instructions / system /
developer content into a single merged system message at the top of
the Chat Completions message list.
2. "async generator ignored GeneratorExit" / "Attempted to exit cancel
scope in a different task"
_responses_stream consumed the inner chat-completions body_iterator
without an explicit aclose() in a finally block. On client disconnect
(Codex frequently cancels mid-stream), Python 3.13 finalized the inner
async generator on a different task, tripping anyio's cancel-scope
check. Mirrored the same try/finally + aclose pattern used by the
/v1/messages, /v1/chat/completions, and /v1/completions passthroughs.
Tests: hoisting of instructions + developer, developer mid-conversation,
multiple system messages in input, no-system passthrough.
* Studio: accept Codex multi-turn shapes and fix cross-task stream close on /v1/responses
Two issues observed driving /v1/responses from OpenAI Codex CLI against a
GGUF backend.
1. 422 on every turn after the first
Codex replays prior assistant turns with
`content:[{"type":"output_text","text":...,"annotations":[],"logprobs":[]}]`
and carries forward `reasoning` items (o-series / gpt-5) between turns.
Our `ResponsesContentPart` union only accepted input_text / input_image,
and `ResponsesInputItem` only message / function_call / function_call_output,
so Pydantic failed the whole list and FastAPI returned
`"Input should be a valid string"` against the `str` branch of the
outer union.
- Add `ResponsesOutputTextPart` for assistant-replay content.
- Add `ResponsesUnknownContentPart` and `ResponsesUnknownInputItem`
as permissive catch-alls (drop during normalisation).
- Wire an explicit `Discriminator` so dispatch is deterministic and
the fallthrough reaches the catch-all instead of misreporting via
the outer `Union[str, list[...]]`.
- `_normalise_responses_input` now accepts output_text parts, flattens
single-part assistant text to a plain string (keeps legacy chat
templates happy), and silently drops reasoning / unknown items.
2. "async generator ignored GeneratorExit" / cross-task cancel scope
`_responses_stream` awaited `openai_chat_completions` in the parent
route-handler task, which opens the httpx client for the inner
passthrough on *that* task. The outer `StreamingResponse` then iterates
in a child task, so the asyncgen GC finalises the inner httpcore byte
stream on the child task, tripping anyio's "Attempted to exit cancel
scope in a different task". Move the `await` inside `event_generator`
so the httpx lifecycle stays within the single streaming child task,
and surface any HTTPException as a `response.failed` SSE frame.
Tests: assistant output_text replay, reasoning-item tolerance, unknown
content-part tolerance, end-to-end Codex-shape payload (developer + user +
reasoning + function_call + function_call_output + assistant output_text +
user), and single-part assistant flattening to plain string.
* Studio: call llama-server directly from streaming /v1/responses
The previous fix (running the inner await inside event_generator) was not
enough. Wrapping the existing `openai_chat_completions` pass-through still
stacks two async generators: when the outer generator is closed, the
innermost `HTTP11ConnectionByteStream.__aiter__` in httpcore doesn't
receive GeneratorExit before Python's asyncgen GC finalises it in a
sibling task, tripping "Attempted to exit cancel scope in a different
task" and "async generator ignored GeneratorExit" — the same Python 3.13
+ httpcore 1.0.x interaction already seen in PRs #4956, #4981, #5099.
Cure both pass-throughs had: a single same-task httpx lifecycle with
explicit `aiter_lines().aclose()` BEFORE `resp.aclose()` / `client.aclose()`
in the generator's finally block.
Apply it at the Responses layer by dropping the wrapper entirely for GGUF:
open httpx, consume `resp.aiter_lines()`, parse `chat.completion.chunk`,
emit Responses SSE events, close everything in finally — all in the
single StreamingResponse child task. Non-GGUF streaming is rejected with
a 400 (wrapping the transformers backend would re-introduce the
double-layer pattern and isn't a Codex-compatible path today anyway).
Also surfaces upstream httpx.RequestError / non-200 as a
`response.failed` SSE frame rather than a dropped stream now that the
request is dispatched after SSE headers have gone out.
* Studio: silence benign httpcore asyncgen GC warnings on Python 3.13
The streaming pass-throughs (/v1/chat/completions, /v1/messages,
/v1/responses, /v1/completions) all use the proven #4981 / #5099 pattern
— single-task httpx lifecycle with explicit aiter_lines().aclose() ahead
of resp.aclose() / client.aclose() in the generator's finally block.
That handles our own iterators correctly.
The residual noise ("async generator ignored GeneratorExit" /
"Attempted to exit cancel scope in a different task") comes from an
innermost HTTP11ConnectionByteStream.__aiter__ that httpcore creates
internally inside its pool. We hold no reference to it, so we cannot
aclose it ourselves. Python 3.13's asyncgen GC hook finalises it on the
finaliser task, its aclose path enters an anyio CancelScope shield, and
Python flags the cross-task exit. The response has already been
delivered with a 200 by then — it is purely log noise, not a functional
failure. Same interaction seen in modelcontextprotocol/python-sdk #831,
agno #3556, chainlit #2361, langchain-mcp-adapters #254.
Install a targeted sys.unraisablehook that swallows this specific tuple
— RuntimeError mentioning "cancel scope" or "GeneratorExit" plus an
object repr referencing HTTP11ConnectionByteStream — and defers to the
default hook for every other unraisable. Idempotent; guarded by a
sentinel attribute so repeated imports don't stack filters.
* Chatbox, scroll, and menu fixes
- Fixed chatbox auto-expand height for multi-line text on the compare page
- Fixed chatbox UI to be consistent across compare and new chat
- Fixed scrolling being enabled on pages with no content, which also triggered the scroll-to-bottom button
- Fixed scroll-to-bottom button to only appear after scrolling up a reasonable amount instead of instantly
- Added shutdown studio button to the menu for easier access
- Fixed pop-up menu width to match the user button width
(cherry picked from commit cd4e390dfa84fe311fae79a781b96cc0ef5970a9)
* fix: correct compare scroll viewport and clean up chat composer UI polish
* Dark theme refactor and sidebar/chat UI refinements
- Complete refactoring of dark theme
- Replaced square rounded-corner user profile image with a circular bordered one
- Replaced user profile icon with 'U' initial and renamed label from 'Studio' to 'User'
- Chat bubbles now have a pointy top-right edge
- Sidebar menu tab line color selection is now consistent across all menus
- Tab-selection color animation now also applies to recent chats
- Removed 'Compare' menu autoselect when a compare chat conversation is selected
- Fixed UI consistency in Compare to match New Chat
- Removed sidebar animation and tab line, replaced with rounded selection for consistency
- Further adjustments to sidebar UI
- Further adjustments to compare chat UI
* Fixed sidebar collapse/expand for recent chats and recent runs not being clickable
* Chatbox, scroll, and menu fixes
- Fixed chatbox auto-expand height for multi-line text on the compare page
- Fixed chatbox UI to be consistent across compare and new chat
- Fixed scrolling being enabled on pages with no content, which also triggered the scroll-to-bottom button
- Fixed scroll-to-bottom button to only appear after scrolling up a reasonable amount instead of instantly
- Added shutdown studio button to the menu for easier access
- Fixed pop-up menu width to match the user button width
* Sidebar, fonts, and chat UI refinements
- Replaced logo PNG with real font text for 'unsloth' and 'BETA' label
- Added Hellix font and applied it across menus and UI elements
- Lighter scrollbar in the sidebar compared to other areas of the app
- Adjusted chat font and chat bubble styling
- Adjusted app menu design to stay consistent with the sidebar
- Adjusted text style for 'New Chat' and repositioned content/chatbox
- Adjusted model selector and top area UI
- Fixed footer text from 'LLM's' to 'LLMs'
- Fixed active selection border color incorrectly appearing on page refresh and during general navigation
- Logo now defaults to 'New Chat' when clicked
* Sidebar, model selector, and mobile UI fixes
- Further adjustments to sidebar UI and logo
- Changed right bar icon
- Model selector adjustments
- Collapsed sidebar now matches the content area background
- Adjusted Hellix font spacing across pages
- Fixed sidebar icon overlap on mobile screens
* Adjust sidebar icons
* Adjust sidebar icons
* Fixed compare chat UI and scrolling issues
* Fixed inference settings icon behavior and context info positioning
- Fixed top right inference settings icon to move into sidepanel during expand/collapse, matching left sidebar behavior
- Adjusted context information element positioning
* Fix: textarea overflow in system prompt editor
* Code block redesign, font, and chat bubble adjustments
- Redesigned code block colors and theme
- Changed code block font to Fira Code
- Fixed scrollbar disappearing when expanding/collapsing tool calls in chats
- Adjusted chat bubble background color
* Fix chat bubble background color in dark theme
* fix: restore textarea auto-sizing and scope prompt editor sizing
* fix: add explicit textarea field sizing for prompt editor overflow
* fix: generate chat nonce on click instead of render
* fix: respect training lock on logo navigation
* Refactor compare page dual chat scrolling behavior
* Revert "Refactor compare page dual chat scrolling behavior"
This reverts commit d056ec09f2.
---------
Co-authored-by: sneakr <hauzin@hotmail.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* export: update GGUF quant list and ordering
* gguf: add Q2_K_L quantize flags for output and embeddings
* export: add live console logs for LoRA export flow
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: stream q2_k_l quantize logs and include subprocess error details
* fix: route Q2_K_L preset to q2_k ftype with q8_0 output+embeddings
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Trashing a thread mid-stream used to delete the Dexie rows while the
model kept generating, because the sidebar has no access to the
@assistant-ui aui context. Expose per-thread cancelRun() through the
chat runtime store and call it from deleteChatItem so trash behaves
like Stop → Trash. Covers compare pairs by cancelling each paired
thread.
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* fix(studio): forward OpenAI tools/tool_choice to llama-server (#4999)
Studio's /v1/chat/completions silently stripped standard OpenAI `tools`
and `tool_choice` fields, so clients using standard function calling
(opencode, Claude Code, Cursor, Continue, ...) never got structured
tool_calls back. Adds a client-side pass-through path mirroring the
existing Anthropic /v1/messages flow: when `tools` is present without
Studio's `enable_tools` shorthand, the request is forwarded to
llama-server verbatim so the client sees native id, finish_reason
("tool_calls"), delta.tool_calls, and accurate usage tokens.
Also wires Anthropic tool_choice forwarding: /v1/messages previously
accepted tool_choice on the request model but silently dropped it with
a warning. Translate the four Anthropic shapes to OpenAI format and
forward them so agentic clients can actually enforce tool use.
- ChatCompletionRequest: add tools, tool_choice, stop; extra="allow"
- ChatMessage: accept role="tool", optional tool_call_id / tool_calls /
name; content is now optional (assistant with only tool_calls)
- routes/inference.py: _openai_passthrough_stream /
_openai_passthrough_non_streaming helpers, routing branch in
openai_chat_completions, vision+tools via content-parts injection
- _build_passthrough_payload: tool_choice parameter (default "auto")
- anthropic_compat: anthropic_tool_choice_to_openai() translator
- tests/test_openai_tool_passthrough.py: Pydantic + translator unit tests
- tests/test_studio_api.py: 5 new E2E tests (non-stream, stream,
multi-turn, OpenAI SDK, Anthropic tool_choice=any regression)
* fix(studio): surface httpx transport errors from OpenAI passthrough
When the managed llama-server subprocess crashes mid-request, the
async pass-through helpers in routes/inference.py used to return a
bare 500 (non-streaming) or an "An internal error occurred" SSE chunk
(streaming) because _friendly_error only recognized the sync path's
"Lost connection to llama-server" substring -- httpx transport
failures (ConnectError / ReadError / RemoteProtocolError /
ReadTimeout) stringify differently and fell through to the generic
case.
- _friendly_error: map any httpx.RequestError subclass to the same
"Lost connection to the model server" message the sync chat path
emits. Placed before the substring heuristics so the streaming path
automatically picks it up via its existing except Exception catch.
- _openai_passthrough_non_streaming: wrap the httpx.AsyncClient.post
in a try/except httpx.RequestError and re-raise as HTTPException
502 with the friendly detail.
- tests/test_openai_tool_passthrough.py: new TestFriendlyErrorHttpx
class pinning the mapping for ConnectError, ReadError,
RemoteProtocolError, ReadTimeout, and confirming non-httpx paths
(context-size heuristic, generic fallback) are unchanged.
* fix(studio): close aiter_bytes/aiter_lines explicitly in passthroughs
The httpcore asyncgen cleanup fix in 5cedd9a5 is incomplete on Python
3.13 + httpcore 1.0.x: it switched to manual client/response lifecycle
but still used anonymous `async for raw_line in resp.aiter_lines():`
patterns in all three streaming paths. Python's async for does NOT
auto-close the iterator on break/return, so the aiter_lines /
aiter_bytes async generator remains alive, reachable only from the
surrounding coroutine frame. Once `_stream()` returns the frame is
GC'd and the orphaned asyncgen is finalized on a LATER GC pass in a
DIFFERENT asyncio task, where httpcore's
HTTP11ConnectionByteStream.aclose() enters anyio.CancelScope.__exit__
with a mismatched task and prints "Exception ignored in: <async
generator>" / "async generator ignored GeneratorExit" / "Attempted
to exit cancel scope in a different task" to the server log.
User observed this on /v1/messages after successful (status 200)
requests, with the traceback pointing at HTTP11ConnectionByteStream
.__aiter__ / .aclose inside httpcore.
Fix: save resp.aiter_lines() / resp.aiter_bytes() as a variable and
explicitly `await iter.aclose()` in the finally block BEFORE
resp.aclose() / client.aclose(). This closes the asyncgen inside the
current task's event loop, so the internal httpcore byte stream is
cleaned up before Python's asyncgen GC hook has anything orphaned to
finalize. Each aclose is wrapped in try/except Exception so nested
anyio cleanup noise can't bubble out.
Applied to all three streaming passthrough paths:
- _anthropic_passthrough_stream (/v1/messages client-side tool path)
- _openai_passthrough_stream (/v1/chat/completions client-side tool
path, new in this PR)
- openai_completions (/v1/completions bytes proxy from PR #4956)
* fix(studio): default ChatCompletionRequest.stream to false per OpenAI spec
OpenAI's /v1/chat/completions spec defaults `stream` to false, so
clients that omit the field (naive curl, minimal integrations) expect
a single JSON response back. Studio was defaulting to true, silently
switching those clients into SSE and breaking any parser that didn't
also handle streaming. ResponsesRequest and AnthropicMessagesRequest
already default to false correctly; only ChatCompletionRequest was
wrong.
Studio's own frontend always sets `stream` explicitly on every
chat-adapter / chat-api / runtime-provider call site, so the flip has
no UI impact. SDK users (OpenAI Python/JS SDK, opencode, Claude Code,
Cursor, Continue) also always pass `stream` explicitly, so they're
unaffected. The only clients feeling the change are raw-curl users
who were relying on the wrong default -- those get the correct OpenAI
behavior now.
Added a regression test pinning the default so it can't silently
flip back.
* fix(studio): reject images in OpenAI tool passthrough for text-only GGUFs
The new tool passthrough branch runs before _extract_content_parts,
skipping the existing not is_vision guard. Requests combining tools
with an image on a text-only tool-capable GGUF were forwarded to
llama-server, producing opaque upstream errors instead of the
pre-existing clear 400. Restore the guard inline at the dispatch
point, checking both legacy image_base64 and inline image_url parts.
* fix(studio): require tool_call_id on role=tool chat messages
Enforce the OpenAI spec rule that role="tool" messages must carry a
tool_call_id. Without it, upstream backends cannot associate a tool
result with the assistant's prior tool_calls entry and the request
fails in non-obvious ways through the passthrough path. Reject at the
request boundary with a 422 instead.
* fix(studio): harden OpenAI tool passthrough validation and error surfacing
Three related fixes called out by the PR review:
1. Preserve upstream status codes in the streaming passthrough. The
httpx request is now dispatched before the StreamingResponse is
constructed. Non-200 upstream responses and httpx RequestError
transport failures raise HTTPException with the real status
instead of being buried inside a 200 SSE error frame, so OpenAI
SDK clients see APIError/BadRequestError/... as expected.
2. Require non-empty content on user/system/tool messages. Per the
OpenAI spec, content may only be omitted on assistant messages
that carry tool_calls; enforce that at the request boundary so
malformed messages never reach the passthrough path.
3. Role-constrain tool-call metadata. tool_calls is only valid on
role=assistant, tool_call_id and name only on role=tool. Without
this, a user/system message with tool_calls would flip the
passthrough branch on and be forwarded to llama-server, surfacing
as an opaque upstream error.
* fix(studio): normalize image mode and passthrough JSON verbatim
Two Gemini-code-assist review findings on PR #5099:
1. Unconditionally convert decoded images to RGB before PNG encoding.
The prior code only handled RGBA, letting CMYK/I/F images crash
at img.save(format="PNG") and surface as opaque 400s. Applied to
both the passthrough helper and the non-passthrough GGUF path
that originally carried this pattern, keeping the two sites in
sync.
2. Return the upstream JSON body as raw bytes via Response rather
than parse-then-re-serialize with JSONResponse. Matches the
passthrough helper's "verbatim" contract and drops a redundant
round-trip.
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Add Qwen3.6 inference defaults for Studio
Add qwen3.6 family entry to inference_defaults.json with the
recommended sampling parameters from Qwen's documentation:
temperature=0.7, top_p=0.8, top_k=20, min_p=0.0,
presence_penalty=1.5, repetition_penalty=1.0.
Without this, Qwen3.6 models fall through to the generic qwen3
pattern which uses different defaults (temperature=0.6,
top_p=0.95, no presence_penalty).
* Add Qwen3.6-35B-A3B-GGUF to default model lists
* Add Qwen3.5/3.6 presence_penalty to thinking toggle and small-model disable logic
- Thinking toggle (on-load + button click) now sets presencePenalty: 1.5 for
Qwen3.5 and Qwen3.6 models (both thinking-ON and thinking-OFF states)
- Small-model thinking-disable check (<9B defaults to no-thinking) extended
from Qwen3.5-only to also cover Qwen3.6, in all 3 locations:
frontend on-load, frontend refresh, backend llama_cpp.py
* auth: default to chat
* settings: relaunch onboarding
* onboarding: return to launch page
* studio: stop auto guided tour
* ui: soften global radius
* cleanup: rename onboarding exit prop
* fix onboarding redirect safety
* Show real Unsloth version in settings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat(studio): replace navbar navigation with collapsible sidebar
Add an app-wide sidebar with hover-expand and pin-to-dock behavior.
Navigation items (Studio, Recipes, Export, Chat) move from the center
pill navbar to the sidebar. Chat threads and recipes render as
collapsible sub-lists. Navbar simplified to logo + update + close.
- Extend SidebarProvider with pinned/hovered state model
- New AppSidebar with animated active indicator, sloth profile menu,
theme toggle, guided tour, back/forward navigation
- Chat page refactored to URL-driven view state via search params
- Extract reusable hooks for chat thread and recipe sidebar data
- Guard startViewTransition for browser compatibility
- Wrap chat deletions in Dexie transaction for data integrity
* feat(studio): move logo to sidebar and make navbar overlay
- Sidebar is now full-height with logo in SidebarHeader
- Collapsed sidebar shows sticker.png, expanded shows full logo
- Navbar is absolute-positioned overlay (no layout space)
- Main content extends to top, aligning with navbar controls
* feat(studio): full-height sidebar with recents, edge-to-edge nav buttons
- Sidebar outside max-w-7xl, pinned to left edge
- Remove sidebar rounding, menu buttons rounded-md
- Nav buttons flush to sidebar edges with no left rounding
- Replace collapsible recipes/chat with flat nav items
- Add Recents section with chat history (1 item when not on chat, full on chat)
- New Chat as first nav item with PencilEdit02Icon
- Cursor pointer on all sidebar buttons
- Navbar temporarily hidden for screenshots
* fix(studio): fix chat scroll, action bar hover, collapsible recents
- Fix sticky composer by removing `relative` override on viewport footer
- Action bar buttons only show on hover (autohide=always)
- Remove floating border/shadow from action bar
- Add scroll space above composer for last message actions
- Back/forward buttons use router history (stay in-app)
- Recents section collapsible with chevron on chat route
- Set html/body/#root height for proper h-full chain
* fix(studio): address review feedback, clean up unused code
- Unhide navbar (was left hidden from screenshot)
- Remove unused imports: SidebarMenuSub*, BubbleChatIcon, ColumnInsertIcon
- Remove unused vars: recipeItems, activeRecipeId, canCompare, recipesOpen
- Include compare query id in active sidebar selection
- Use store type for contextUsage instead of inline type
- Simplify noop in sidebar.tsx
- Remove empty className prop
* feat(studio): add mobile sidebar, recent runs section, and misc UX fixes
* feat(studio): scaffold settings feature module with dialog store
* feat(studio): add tri-state theme store for settings
* feat(chat): add clear-all-chats and export-chat-history utils
* feat(studio): add settings dialog shell with tab rail
* feat(studio): add appearance tab with theme and sidebar pin
* feat(studio): add settings general tab with hf token, auto-title, reset prefs
* feat(studio): add settings chat tab with export and clear
* feat(studio): add api keys tab with list and revoke flow
* feat(studio): add create-key form and reveal dialog
* feat(studio): add usage examples panel to api keys tab
* feat(studio): add settings about tab with update and shutdown
* feat(studio): add settings dropdown item and cmd-comma shortcut
* feat(studio): remove legacy api-keys route and chat-sheet preference rows
* fix(studio): settings dialog a11y + polish pass
* feat(studio): inline api key reveal card replacing nested dialog
* fix(studio): hide revoked keys from settings list
* refactor(studio): strip navbar and hoist training unload guard
* feat(studio): explicit sidebar toggle, remove hover-open and pin icons
* fix(studio): use SidebarRight01Icon for collapsed sidebar open toggle
* fix(studio): address code review findings for settings dialog
* feat(studio): collapsible navigate group with standalone new-chat and compare
* fix(studio): chat-only standalone actions, use ColumnInsertIcon for compare
* fix(studio): sidebar new-chat/compare state reset and icon-mode collapsible
* feat(studio): add compact logo assets for sidebar header
* Fixed sidebar design
* fix(studio): sidebar delete icon hover contrast and sizing
* feat(studio): route-gate sidebar recents (chats off /studio, runs on /studio)
* feat(studio): add chat search store
* feat(studio): add chat search index hook with snapshot-on-open
* feat(studio): add chat search command dialog with global shortcut
* feat(studio): wire chat search into sidebar
* fix(studio): trim hf token on save, add show/hide toggle, commit on close
* revert(studio): restore original sidebar/border colors, brighten sidebar
* feat(studio): forward overlayClassName through CommandDialog
* fix(studio): wrap search dialog in Command context, redesign as flat 635px card
* fix(studio): reserve right padding on recent items so delete icon stops overlapping title
* fix(studio): skip hf token unmount-commit during reset-prefs reload
* chore(studio): drop unused icon import and unreachable runs navigate branch
* fix(studio): chat search index filters archived before limit, batches message query, picks up reasoning text
* fix(studio): keep CommandEmpty in tree so empty state renders correctly
* fix(studio): cap system prompt and chat template textareas so they scroll instead of growing
* fix(studio): attach chat-compare tour anchor to sidebar compare button
* fix(studio): persist system theme explicitly so next-themes does not clobber on reload
* fix(studio): auto-switch to history tab when selecting a recent run from sidebar
* UI overhaul: chatbox, scrollbar, sidebar, and compare view
UI Changes:
- Redesigned the Compare UI with general cleanup
- Redesigned the Chatbox UI
- Reduced the width of the user chat bubble for improved readability
- Narrowed the user chat box across the content page
- Adjusted thinking-box text color to be slightly darker
- Removed faded text effect from chat messages
- Removed faded text effect from the thinking box
- Added a small LLM chat safety note at the bottom of the chatbox
- Restyled the scrollbar
Layout & Behavior:
- Reworked the scrollbar to span the full height of the page (no top/bottom padding) and remain persistently visible when content is scrollable, rather than only on hover
- Reworked the Configuration sidebar to span full height — removed rounded corners and borders, with the scrollbar adjusted to match the full top-to-bottom layout
- Adjusted the top menu and bottom chatbox content areas to work correctly with the new full-page scroll behavior
- Made chat content match the chatbox width, with content sliding slightly behind the chatbox when scrolling
- Aligned chat text width with the chatbox for visual consistency, including how far the text extends behind the chatbox
Fixes:
- Fixed the chatbox not auto-expanding when typing multi-line input while bottom-positioned during an active chat (previously only worked before a chat had started)
- Fixed positioning and design of the user chat hover menu buttons to match the assistant chat box — now displayed below the chat bubble instead of on the left side
* Fix user message layout in thread component
* swap code icon
* fix compare layout
* fix compare pane flex
* Sidebar improvements and fixes
- Added scrolling support to the sidebar so menus and recent chats no longer get hidden
- Recent chats are now always visible in the sidebar, not hidden when in Studio, Recipes, or Export
- Recent chat is now deselected when selecting other navigations
- Fixed sidebar glitch where browser resize could make the sidebar and expand button disappear completely
- Fixed glitch where the open-sidebar hover tooltip appeared above the logo when clicking expand sidebar
- Reduced sidebar width on mobile to around 2/3 of the screen (was too wide)
- Made the close-sidebar hover tooltip consistent with the rest of the design
- Removed sidebar collapse/expand animation
- Small adjustment to chat width
* Fix route scrolling, polling, and theme sync issues
* Fix Studio page scrolling
---------
Co-authored-by: sneakr <hauzin@hotmail.com>
* Studio: Ollama support, recommended folders, Custom Folders UX polish
Backend:
- Add _scan_ollama_dir that reads manifests/registry.ollama.ai/library/*
and creates .gguf symlinks under <ollama_dir>/.studio_links/ pointing
at the content-addressable blobs, so detect_gguf_model and llama-server
-m work unchanged for Ollama models
- Filter entries under .studio_links from the generic models/hf/lmstudio
scanners to avoid duplicate rows and leaked internal paths in the UI
- New GET /api/models/recommended-folders endpoint returning LM Studio
and Ollama model directories that currently exist on the machine
(OLLAMA_MODELS env var + standard paths, ~/.lmstudio/models, legacy
LM Studio cache), used by the Custom Folders quick-add chips
- detect_gguf_model now uses os.path.abspath instead of Path.resolve so
the readable symlink name is preserved as display_name (e.g.
qwen2.5-0.5b-Q4_K_M.gguf instead of sha256-abc...)
- llama-server failure with a path under .studio_links or .cache/ollama
surfaces a friendlier message ("Some Ollama models do not work with
llama.cpp. Try a different model, or use this model directly through
Ollama instead.") instead of the generic validation error
Frontend:
- ListLabel supports an optional leading icon and collapse toggle; used
for Downloaded (download icon), Custom Folders (folder icon), and
Recommended (star icon)
- Custom Folders header gets folder icon on the left, and +, search,
and chevron buttons on the right; chevron uses ml-auto so it aligns
with the Downloaded and Recommended chevrons
- New recommended folder chips render below the registered scan folders
when there are unregistered well-known paths; one click adds them as
a scan folder
- Custom folder rows that are direct .gguf files (Ollama symlinks) load
immediately via onSelect instead of opening the GGUF variant expander
(which is for repos containing multiple quants, not single files)
- When loading a direct .gguf file path, send max_seq_length = 0 so the
backend uses the model's native context instead of the 4096 chat
default (qwen2.5:0.5b now loads at 32768 instead of 4096)
- New listRecommendedFolders() helper on the chat API
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: log silent exceptions and support read-only Ollama dirs
Replace silent except blocks in _scan_ollama_dir and the
recommended-folders endpoint with narrower exception types plus debug
or warning logs, so failures are diagnosable without hiding signal.
Add _ollama_links_dir helper that falls back to a per-ollama-dir hashed
namespace under Studio's own cache (~/.unsloth/studio/cache/ollama_links)
when the Ollama models directory is read-only. Common for system installs
at /usr/share/ollama/.ollama/models and /var/lib/ollama/.ollama/models
where the Studio process has read but not write access. Previously the
scanner returned an empty list in that case and Ollama models would
silently not appear.
The fallback preserves the .gguf suffix on symlink names so
detect_gguf_model keeps recognising them. The prior "raw sha256 blob
path" fallback would have missed the suffix check and failed to load.
* Address review: detect mmproj next to symlink target for vision GGUFs
Codex P1 on model_config.py:1012: when detect_gguf_model returns the
symlink path (to preserve readable display names), detect_mmproj_file
searched the symlink's parent directory instead of the target's. For
vision GGUFs surfaced via Ollama's .studio_links/ -- where the weight
file is symlinked but any mmproj sidecar lives next to the real blob
-- mmproj was no longer detected, so the model was misclassified as
text-only and llama-server would start without --mmproj.
detect_mmproj_file now adds the resolved target's parent to the scan
order when path is a symlink. Direct (non-symlink) .gguf paths are
unchanged, so LM Studio and HF cache layouts keep working exactly as
before. Verified with a fake layout reproducing the bug plus a
regression check on a non-symlink LM Studio model.
* Address review: support all Ollama namespaces and vision projector layers
- Iterate over all directories under registry.ollama.ai/ instead of
hardcoding the "library" namespace. Custom namespaces like
"mradermacher/llama3" now get scanned and include the namespace
prefix in display names, model IDs, and symlink names to avoid
collisions.
- Create companion -mmproj.gguf symlinks for Ollama vision models
that have an "application/vnd.ollama.image.projector" layer, so
detect_mmproj_file can find the projector alongside the model.
- Extract symlink creation into _make_symlink helper to reduce
duplication between model and projector paths.
* Address review: move imports to top level and add scan limit
- Move hashlib and json imports to the top of the file (PEP 8).
- Remove inline `import json as _json` and `import hashlib` from
function bodies, use the top-level imports directly.
- Add `limit` parameter to `_scan_ollama_dir()` with early exit
when the threshold is reached.
- Pass `_MAX_MODELS_PER_FOLDER` into the scanner so it stops
traversing once enough models are found.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: Windows fallback, all registry hosts, collision safety
_make_link (formerly _make_symlink):
- Falls back to os.link() hardlink when symlink_to() fails (Windows
without Developer Mode), then to shutil.copy2 as last resort
- Uses atomic os.replace via tmp file to avoid race window where the
.gguf path is missing during rescan
Scanner now handles all Ollama registry layouts:
- Uses rglob over manifests/ instead of hardcoding registry.ollama.ai
- Discovers hf.co/org/repo:tag and any other host, not just library/
- Filenames include a stable sha1 hash of the manifest path to prevent
collisions between models that normalize to the same stem
Per-model subdirectories under .studio_links/:
- Each model's links live in their own hash-keyed subdirectory
- detect_mmproj_file only sees the projector for that specific model,
not siblings from other Ollama models
Friendly Ollama error detection:
- Now also matches ollama_links/ (the read-only fallback cache path)
and model_identifier starting with "ollama/"
Recommended folders:
- Added os.access(R_OK | X_OK) check so unreadable system directories
like /var/lib/ollama/.ollama/models are not advertised as chips
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: filter ollama_links from generic scanners
The generic scanners (models_dir, hf_cache, lmstudio) already filter
out .studio_links to avoid duplicate Ollama entries, but missed the
ollama_links fallback cache directory used for read-only Ollama
installs. Add it to the filter.
* Address review: idempotent link creation and path-component filter
_make_link:
- Skip recreation when a valid link/copy already exists (samefile or
matching size check). Prevents blocking the model-list API with
multi-GB copies on repeated scans.
- Use uuid4 instead of os.getpid() for tmp file names to avoid race
conditions from concurrent scans.
- Log cleanup errors instead of silently swallowing them.
Path filter:
- Use os.sep-bounded checks instead of bare substring match to avoid
false positives on paths like "my.studio_links.backup/model.gguf".
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: drop copy fallback, targeted glob, robust path filter
_make_link:
- Drop shutil.copy2 fallback -- copying multi-GB GGUFs inside a sync
API request would block the backend. Log a warning and skip the
model when both symlink and hardlink fail.
Scanner:
- Replace rglob("*") with targeted glob patterns (*/*/* and */*/*/*)
to avoid traversing unrelated subdirectories in large custom folders.
Path filter:
- Use Path.parts membership check instead of os.sep substring matching
for robustness across platforms.
Scan limit:
- Skip _scan_ollama_dir when _generic already fills the per-folder cap.
* Address review: sha256, top-level uuid import, Path.absolute()
- Switch hashlib.sha1 to hashlib.sha256 for path hashing consistency.
- Move uuid import to the top of the file instead of inside _make_link.
- Replace os.path.abspath with Path.absolute() in detect_gguf_model
to match the pathlib style used throughout the codebase.
* Address review: fix stale comments (sha1, rglob, copy fallback)
Update three docstrings/comments that still referenced the old
implementation after recent changes:
- sha1 comment now says "not a security boundary" (no hash name)
- "rglob" -> "targeted glob patterns"
- "file copies as a last resort" -> removed (copy fallback was dropped)
* Address review: fix stale links, support all manifest depths, scope error
_make_link:
- Drop size-based idempotency shortcut that kept stale links after
ollama pull updates a tag to a same-sized blob. Only samefile()
is used now -- if the link doesn't point at the exact same inode,
it gets replaced.
Scanner:
- Revert targeted glob back to rglob so deeper OCI-style repo names
(5+ path segments) are not silently skipped.
Ollama error:
- Only show "Some Ollama models do not work with llama.cpp" when the
server output contains GGUF compatibility hints (key not found,
unknown architecture, failed to load). Unrelated failures like
OOM or missing binaries now show the generic error instead of
being misdiagnosed.
---------
Co-authored-by: Daniel Han <info@unsloth.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Reduce inline comments from ~160 lines to ~25 across both files.
Keep one-line summaries of the "why"; drop multi-paragraph rationale
blocks that repeated information already captured in commit messages
and PR discussion.
* fix: replacing SetEnvironmentVariable with direct registry API
* apply reviews
* Use CreateSubKey for HKCU\Environment
* Store PATH backup under HKCU\Software\Unsloth
* Fix $backupKey registry handle leak in PATH backup block
Wrap $backupKey operations in try/finally so the handle is closed even
if GetValue or SetValue throws. The Add-ToUserPath helper already uses
this pattern for its registry key -- the backup block was the only
place missing it.
* Isolate WM_SETTINGCHANGE broadcast from PATH write error handling
Wrap the broadcast dummy-variable calls in their own try/catch so a
broadcast failure does not mask a successful registry PATH write.
Previously, if SetEnvironmentVariable threw after SetValue already
committed the new PATH, Add-ToUserPath would return $false and the
caller would skip Refresh-SessionPath.
* PATH helper polish: venv precedence, quoted entries, raw/expanded dedup
Three small follow-ups surfaced by a 10-reviewer pass against the rebased
PR head. None fix a regression vs main; each strictly improves the new
helpers.
Refresh-SessionPath / Refresh-Environment:
- Move $env:Path to the front of the merge so an activated venv keeps
precedence over machine/user PATH after a refresh. Pre-PR dropped
process-only entries entirely; post-PR kept them but at the back.
- Dedup on both raw and expanded forms so %USERPROFILE%\foo and the
already-expanded C:\Users\me\foo do not both survive.
Add-ToUserPath:
- Trim whitespace and surrounding double-quotes from each compared entry
so quoted PATH entries like "C:\Program Files\CMake\bin" deduplicate
against an unquoted directory of the same path.
* Back up User PATH inside Add-ToUserPath, before first mutation
Previously only studio/setup.ps1 took a one-time PATH backup, at script
top (line ~547). install.ps1 (the irm | iex entry point) had no backup,
so users who installed via that path had no recovery surface if anything
clobbered their PATH. The PR description's "one-time backup before any
modifications" promise only held for the studio installer flow.
Move the backup into Add-ToUserPath itself: just before the first actual
SetValue mutation, write the pristine raw PATH to
HKCU\Software\Unsloth\PathBackup if no backup already exists. This:
- Covers both entry points (install.ps1 and studio/setup.ps1).
- Captures the TRUE pristine PATH even when install.ps1 runs first and
studio/setup.ps1 runs afterwards (the script-top backup in setup.ps1
would otherwise see an already-modified PATH).
- Is idempotent: once a backup exists, subsequent calls preserve it.
- Skips when nothing would mutate (dedup match) or PATH is empty.
The script-top backup in studio/setup.ps1 is kept for defense in depth.
* Refresh PATH: venv-aware merge order
Reconcile two competing concerns about Refresh-SessionPath /
Refresh-Environment surfaced by separate review rounds:
- venv at the back -> activated venv loses precedence to system Python
- process at the front -> stale shims (old node, old python, etc.)
still on $env:Path can beat a freshly installed tool
New merge order:
1. Activated venv Scripts dir, only if $env:VIRTUAL_ENV is set
2. Machine PATH freshly read from registry
3. User PATH freshly read from registry
4. Current $env:Path as fallback
This way an explicitly-activated venv keeps priority while a tool the
script just installed wins over any stale entry that was already on
the inherited shell PATH. When no venv is active, fresh registry
entries take precedence as expected.
* Append to User PATH by default, close $envKey in finally
Add-ToUserPath gains a -Position Append|Prepend parameter defaulting to
Append so installing unsloth no longer prepends the bundled venv Scripts
directory ahead of the user's existing python / pip on new shells. The
four current call sites (install.ps1 launcher, studio/setup.ps1 CMake,
nvcc, Python user Scripts) all take the Append default because each one
that needs in-session precedence already does an inline $env:Path prepend
independently. This matches rustup / cargo / nvm / pyenv / uv behavior.
Also wrap the script-top $envKey.GetValue in a try/finally so the
registry handle is released even if the read throws. Matches the pattern
already used for $backupKey five lines below.
* Prepend cmake, nvcc, Python Scripts; keep venv Scripts appended
The previous commit switched Add-ToUserPath to append by default so that
installing unsloth would not silently hijack the user's system python /
pip. That was correct for the venv Scripts dir (which contains python.exe
and pip.exe alongside unsloth.exe), but wrong for the three studio/setup
call sites. Those persist cmake, the driver-compatible nvcc, and the
Python user Scripts dir for future shells, and in all three cases an
older tool already earlier in the user PATH would keep winning after the
install finished. The nvcc case is especially load-bearing: setup selects
a driver-compatible CUDA toolkit, then llama.cpp builds against whatever
wins PATH resolution, so a stale older nvcc produces broken builds.
Pass -Position 'Prepend' explicitly at the three setup.ps1 call sites
(cmake at line 754, nvcc bin at line 1025, Python user Scripts at line
1191). None of those directories holds python.exe, so prepending them
does not re-introduce the original hijack problem. Leave the install.ps1
venv Scripts call on the default Append with a comment explaining why.
* Symmetric dedup, Prepend reorders duplicates, unsloth shim dir
Address three separate findings surfaced by review:
1. Dedup asymmetry (Gemini high-priority): the existing dedup expanded
registry entries via ExpandEnvironmentVariables but did NOT expand the
new directory. Passing "%USERPROFILE%\foo" when "C:\Users\me\foo" was
already in PATH produced a duplicate. Expand both sides so the check
is symmetric.
2. -Position Prepend no-op on existing duplicates: the dedup loop
returned $false as soon as it saw a match, regardless of position.
That left a late-position duplicate in place instead of moving it to
the front, so "prepend the newly selected cmake/nvcc" did not always
beat an older copy earlier in PATH. Partition entries into kept and
dropped lists, then reinsert a single copy at the requested position.
Append still returns $false on any match so user-curated orderings
are not reshuffled. Prepend also returns $false when the only copy
is already at position 0 so we preserve the user's casing.
3. Stop adding the venv Scripts dir to User PATH entirely. That dir
holds python.exe and pip.exe alongside unsloth.exe, so neither
Prepend nor Append worked: prepend hijacked the user's system python
and pip, append made the freshly-installed unsloth.exe lose to any
older unsloth.exe earlier on PATH. Replace the Scripts-dir PATH add
with a dedicated shim directory that contains only unsloth.cmd, and
prepend that dir. The shim calls the venv's unsloth.exe by absolute
path so future pip upgrades inside the venv propagate automatically.
* Shim via hardlink, Append user Scripts, drop venv sysconfig fallback
Three follow-ups to the c0ab1ab shim commit, targeting concerns raised in
the second 20-reviewer pass:
1. Shim uses unsloth.exe (hardlink, copy fallback) instead of unsloth.cmd.
The batch-file approach had three distinct regressions:
- cmd.exe expanded %...% sequences inside user arguments, so prompts
like "What does 50% mean?" got mangled before reaching the CLI
- Git Bash / MSYS2 / POSIX-style shells on Windows do not resolve
bare-name lookups to .cmd files, so `unsloth` stopped working there
- Set-Content -Encoding ASCII replaced non-ASCII profile characters
with '?', so installs under C:\Users\Jörg\... wrote a broken shim
A hardlink (fallback: copy) of unsloth.exe is a native Windows
executable with no shell indirection. PATHEXT picks .exe before .cmd
in cmd.exe and PowerShell, Git Bash honors .exe natively, subprocess
callers hit it directly, and a hardlink stays in sync with the venv
on pip upgrades because both names point at the same inode.
2. studio/setup.ps1 Python user Scripts dir is added with default Append
instead of -Position Prepend. That directory holds every pip-installed
user console script (pip, pytest, huggingface-cli, and so on), not
just unsloth, so reordering it silently changed resolution order for
unrelated tools. The new install.ps1 shim at PATH position 0 already
guarantees `unsloth` resolves to the freshly installed copy, so the
Python user Scripts entry only needs to be present, not at the front.
3. The sysconfig lookup in studio/setup.ps1 no longer falls back to
sysconfig.get_path('scripts') when the nt_user scheme dir does not
exist. When setup.ps1 is invoked from an activated venv (a flow the
linked issue actually hits) that fallback returns the venv's Scripts
directory, which would then be added to the persisted User PATH and
re-introduce the python / pip hijack the shim dir is meant to avoid.
Stick strictly to the nt_user scheme; skip the block if it does not
exist on disk.
* Do not crash installer when unsloth.exe shim is locked
The shim update sequence at install.ps1:1095 did a bare Remove-Item /
New-Item HardLink / Copy-Item. Under the script's $ErrorActionPreference
a locked target (most commonly 'unsloth studio' still running while the
user re-invokes the installer) turns the Remove-Item failure into a
terminating error that aborts the install with no actionable message.
The existing shim is perfectly usable in that state, so there is no
reason to abort. Wrap the whole remove/link/copy sequence in a try/catch
that logs the probable cause (Studio still running), points at the fix
(close Studio and re-run), and lets the installer finish with the old
launcher still serving the command.
Also only emit the "added unsloth launcher to PATH" step line when the
launcher was actually (re)created AND the PATH entry was newly added --
previously the message fired even when the shim refresh silently failed,
which was confusing.
* Guard shim PATH entry on existence, use NullString for broadcast delete
Two follow-ups surfaced by the latest review pass:
1. Do not add the shim directory to User PATH when the launcher was not
actually created. Antivirus blocking unsloth.exe, a disk-full volume,
or restrictive filesystem permissions can make both the hardlink and
the copy fallback fail on a fresh install. In that case the existing
sequence would report "added unsloth launcher to PATH" warnings but
still prepend the empty $ShimDir to User PATH -- the user sees an
install that claims success but then cannot resolve `unsloth` in a
new shell. Gate Add-ToUserPath on Test-Path $ShimExe so the PATH
entry is only persisted when the launcher is really there.
2. Pass [NullString]::Value instead of $null to the broadcast-delete
call in Add-ToUserPath. On PowerShell 7.5 and later (running on .NET
9), a bare $null going into [Environment]::SetEnvironmentVariable
can be coerced to an empty string rather than a true .NET null,
which sets the dummy UnslothPathRefresh_XXXXXXXX variable to "" in
HKCU\Environment instead of deleting it. The leaked variable is
visible in System Properties and accumulates one entry per install
run. [NullString]::Value is a PowerShell-specific sentinel that
crosses the interop boundary as a real null and works on both PS 5.1
and PS 7.x. See PowerShell/PowerShell#24637 for the underlying issue.
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: add folder browser modal for Custom Folders
The Custom Folders row in the model picker currently only accepts a
typed path. On a remote-served Studio (Colab, shared workstation) that
means the user has to guess or paste the exact server-side absolute
path. A native browser folder picker can't solve this: HTML
`<input type="file" webkitdirectory>` hides the absolute path for
security, and the File System Access API (Chrome/Edge only) returns
handles rather than strings, neither of which the server can act on.
This PR adds a small in-app directory browser that lists paths on the
server and hands the chosen string back to the existing
`POST /api/models/scan-folders` flow.
## Backend
* New endpoint `GET /api/models/browse-folders`:
* `path` query param (expands `~`, accepts relative or absolute; empty
defaults to the user's home directory).
* `show_hidden` boolean to include dotfiles/dotdirs.
* Returns `{current, parent, entries[], suggestions[]}`. `parent` is
null at the filesystem root.
* Immediate subdirectories only (no recursion); files are never
returned.
* `entries[].has_models` is a cheap hint: the directory looks like it
holds models if it is named `models--*` (HF hub cache layout) or
one of the first 64 children is a .gguf/.safetensors/config.json/
adapter_config.json or another `models--*` subfolder.
* Sort order: model-bearing dirs, then plain, then hidden; case-
insensitive alphabetical within each bucket.
* Suggestions auto-populate from HOME, the HF cache root, and any
already-registered scan folders, deduplicated.
* Error surface: 404 for missing path, 400 for non-directory, 403 on
permission errors. Auth-required like the other models routes.
* New Pydantic schemas `BrowseEntry` and `BrowseFoldersResponse` in
`studio/backend/models/models.py`.
## Frontend
* New `FolderBrowser` component
(`studio/frontend/src/components/assistant-ui/model-selector/folder-browser.tsx`)
using the existing `Dialog` primitive. Features:
* Clickable breadcrumb with a `..` row for parent navigation.
* Quick-pick chips for the server-provided suggestions.
* `Show hidden` checkbox.
* In-flight fetch cancellation via AbortController so rapid
navigation doesn't flash stale results.
* Badges model-bearing directories inline.
* `chat-api.ts` gains `browseFolders(path?, showHidden?)` and matching
types.
* `pickers.tsx` adds a folder-magnifier icon next to the existing `Add`
button. Opening the browser seeds it with whatever the user has
already typed; confirming fills the text input, leaving the existing
validation and save flow unchanged.
## What it does NOT change
* The existing text-input flow still works; the browser is additive.
* No new permissions or escalation; the endpoint reads only directories
the server process is already allowed to read.
* No model scanning or filesystem mutation happens from the browser
itself -- it just returns basenames for render.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: cap folder-browser entries and expose truncated flag
Pointing the folder browser at a huge directory (``/usr/lib``,
``/proc``, or a synthetic tree with thousands of subfolders) previously
walked the whole listing and stat-probed every child via
``_looks_like_model_dir``. That is both a DoS shape for the server
process and a large-payload surprise for the client.
Introduce a hard cap of 2000 subdirectory entries and a
``truncated: bool`` field on the response. The frontend renders a small
hint below the list when it fires, prompting the user to narrow the
path. Below-cap directories are unchanged.
Verified end-to-end against the live backend with a synthetic tree of
2050 directories: response lands at 2000 entries, ``truncated=true``,
listing finishes in sub-second time (versus tens of seconds if we were
stat-storming).
* Studio: suggest LM Studio / Ollama dirs + 2-level model probe
Three improvements to the folder-browser, driven by actually dropping
an LM Studio-style install (publisher/model/weights.gguf) into the
sandbox and walking the UX:
## 1. Quick-pick chips for other local-LLM tools
`well_known_model_dirs()` (new) returns paths commonly used by
adjacent tools. Only paths that exist are returned so the UI never
shows dead chips.
* LM Studio current + legacy roots + user-configured
`downloadsFolder` from its `settings.json` (reuses the existing
`lmstudio_model_dirs()` helper).
* Ollama: `$OLLAMA_MODELS` env override, then `~/.ollama/models`,
`/usr/share/ollama/.ollama/models`, and `/var/lib/ollama/.ollama/models`
(the systemd-service install path surfaced in the upstream "where is
everything?" issue).
* Generic user-choice locations: `~/models`, `~/Models`.
Dedup is stable across all sources.
## 2. Two-level model-bearing probe
LM Studio and Ollama both use `root/publisher/model/weights.gguf`.
The previous `has_models` heuristic only probed one level, so the
publisher dir (whose immediate children are model dirs, not weight
files) was always marked as non-model-bearing. Pulled the direct-
signal logic into `_has_direct_model_signal` and added a grandchild
probe so the classic layout is now recognised.
Still O(PROBE^2) worst-case, still returns immediately for
`models--*` names (HF cache layout) and for any direct weight file.
## 3. model_files_here hint on response body
A leaf model dir (just GGUFs, no subdirs) previously rendered as
`(empty directory)` in the modal, confusing users into thinking the
folder wasn't scannable. Added a `model_files_here` count on the
response (capped at 200) and a small hint row in the modal: `N model
files in this folder. Click "Use this folder" to scan it.`
## Verification
Simulated an LM Studio install by downloading the real 84 MB
`unsloth/SmolLM2-135M-Instruct-Q2_K.gguf` into
`~/.lmstudio/models/unsloth/SmolLM2-135M-Instruct-GGUF/`. Confirmed
end-to-end:
* Home listing suggests `~/.lmstudio/models` as a chip.
* Browsing `~/.lmstudio/models` flags `unsloth` (publisher) as
`has_models=true` via the 2-level probe.
* Browsing the publisher flags `SmolLM2-135M-Instruct-GGUF` (model
dir) as `has_models=true`.
* Browsing the model dir returns empty entries but
`model_files_here=1`, and the frontend renders a hint telling the
user it is a valid target.
* Studio: one-click scan-folder add + prominent remove + plain search icon
Three small Custom Folders UX fixes after real-use walkthrough:
* **One-click add from the folder browser**. Confirming `Use this
folder` now submits the path directly to
`POST /api/models/scan-folders` instead of just populating the text
input. `handleAddFolder` takes an optional explicit path so the
submit lands in the same tick as `setFolderInput`, avoiding a
state-flush race. The typed-path + `Add` button flow is unchanged.
* **Prominent remove X on scan folders**. The per-folder delete
button was `text-muted-foreground/40` and hidden entirely on
desktop until hovered (`md:opacity-0 md:group-hover:opacity-100`).
Dropped the hover-only cloak, bumped color to `text-foreground/70`,
added a red hover/focus background, and sized the icon up from
`size-2.5` to `size-3`. Always visible on every viewport.
* **Plain search icon for the Browse button**. `FolderSearchIcon`
replaced with `Search01Icon` so it reads as a simple "find a
folder" action alongside the existing `Add01Icon`.
* Studio: align Custom Folders + and X buttons on the same right edge
The Custom Folders header used `px-2.5` with a `p-0.5` icon button,
while each folder row used `px-3` with a `p-1` button. That put the
X icon 4px further from the right edge than the +. Normalised both
rows to `px-2.5` with `p-1` so the two icons share a column.
* Studio: empty-state button opens the folder browser directly
The first-run empty state for Custom Folders was a text link reading
"+ Add a folder to scan for local models" whose click toggled the
text input. That's the wrong default: a user hitting the empty state
usually doesn't know what absolute path to type, which is exactly
what the folder browser is for.
* Reword to "Browse for a models folder" with a search-icon
affordance so the label matches what the click does.
* Click opens the folder browser modal directly. The typed-path +
Add button flow is still available via the + icon in the
section header, so users who know their path keep that option.
* Slightly bump the muted foreground opacity (70 -> hover:foreground)
so the button reads as a primary empty-state action rather than a
throwaway hint.
* Studio: Custom Folders header gets a dedicated search + add button pair
The Custom Folders section header had a single toggle button that
flipped between + and X. That put the folder-browser entry point
behind the separate empty-state link. Cleaner layout: two buttons in
the header, search first, then add.
* Search icon (left) opens the folder browser modal directly.
* Plus icon (right) toggles the text-path input (unchanged).
* The first-run empty-state link is removed -- the two header icons
cover both flows on every state.
Both buttons share the same padding / icon size so they line up with
each other and with the per-folder remove X.
* Studio: sandbox folder browser + bound caps + UX recoveries
PR review fixes for the Custom Folders folder browser. Closes the
high-severity CodeQL path-traversal alert and addresses the codex /
gemini P2 findings.
Backend (studio/backend/routes/models.py):
* New _build_browse_allowlist + _is_path_inside_allowlist sandbox.
browse_folders now refuses any target that doesn't resolve under
HOME, HF cache, Studio dirs, registered scan folders, or the
well-known third-party model dirs. realpath() is used so symlink
traversal cannot escape the sandbox. Also gates the parent crumb
so the up-row hides instead of 403'ing.
* _BROWSE_ENTRY_CAP now bounds *visited* iterdir entries, not
*appended* entries. Dirs full of files (or hidden subdirs when
show_hidden is False) used to defeat the cap.
* _count_model_files gets the same visited-count fix.
* PermissionError no longer swallowed silently inside the
enumeration / counter loops -- now logged at debug.
Frontend (folder-browser.tsx, pickers.tsx, chat-api.ts):
* splitBreadcrumb stops mangling literal backslashes inside POSIX
filenames; only Windows-style absolute paths trigger separator
normalization. The Windows drive crumb value is now C:/ (drive
root) instead of C: (drive-relative CWD-on-C).
* browseFolders accepts and forwards an AbortSignal so cancelled
navigations actually cancel the in-flight backend enumeration.
* On initial-path fetch error, FolderBrowser now falls back to HOME
instead of leaving the modal as an empty dead end.
* When the auto-add path (one-click "Use this folder") fails, the
failure now surfaces via toast in addition to the inline
paragraph (which is hidden when the typed-input panel is closed).
* Studio: rebuild browse target from trusted root for CodeQL clean dataflow
CodeQL's py/path-injection rule kept flagging the post-validation
filesystem operations because the sandbox check lived inside a
helper function (_is_path_inside_allowlist) and CodeQL only does
intra-procedural taint tracking by default. The user-derived
``target`` was still flowing into ``target.exists`` /
``target.is_dir`` / ``target.iterdir``.
The fix: after resolving the user-supplied ``candidate_path``,
locate the matching trusted root from the allowlist and rebuild
``target`` by appending each individually-validated segment to
that trusted root. Each segment is rejected if it isn't a single
safe path component (no separators, no ``..``, no empty/dot).
The downstream filesystem ops now operate on a Path constructed
entirely from ``allowed_roots`` (trusted) plus those validated
segments, so CodeQL's dataflow no longer sees a tainted source.
Behavior is unchanged for all valid inputs -- only the
construction of ``target`` is restructured. Live + unit tests
all pass (58 selected, 7 deselected for Playwright env).
* Studio: walk browse paths from trusted roots for CodeQL
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ubuntu <ubuntu@h100-8-cheapest.us-east5-a.c.unsloth.internal>
* Reapply "updated models template mappers. added lfm2.5vl450m to transformers 5…" (#4945)
This reverts commit 33503ea248.
* Add missing gemma-4-31B-it bnb-4bit mapper entry and LFM2.5 upstream namespace for PR #4950
- Add unsloth/gemma-4-31B-it-unsloth-bnb-4bit to __INT_TO_FLOAT_MAPPER so
the int-to-float resolution works for this model (already listed in
TEMPLATE_TO_MODEL_MAPPER but had no mapper entry).
- Add LiquidAI/LFM2.5-1.2B-Instruct to lfm-2.5 TEMPLATE_TO_MODEL_MAPPER
entry so the canonical upstream namespace is mapped consistently with lfm-2.
* Add missing gemma-4-31B-it bnb-4bit Ollama mapping and lfm-2.5 chat template alias
- Add unsloth/gemma-4-31B-it-unsloth-bnb-4bit to OLLAMA_TEMPLATE_TO_MODEL_MAPPER
so Ollama export works for this model (E2B-it and E4B-it bnb-4bit variants were
already present, 31B-it was inconsistently omitted)
- Register CHAT_TEMPLATES["lfm-2.5"] as alias of the lfm-2 template to prevent
KeyError when Studio resolves LFM2.5 models through MODEL_TO_TEMPLATE_MAPPER
* Add missing LFM2 bnb-4bit INT_TO_FLOAT_MAPPER entry
unsloth/LFM2-1.2B-unsloth-bnb-4bit is referenced in model_mappings.py
but had no mapper.py entry, so model resolution would fail when users
load that variant with load_in_4bit=False or when the float name is
used with load_in_4bit=True.
* Fix review findings for PR #16
1. ollama_template_mappers.py: Restore dropped Gemma-4 base model IDs
(E2B, E4B, 31B, 26B-A4B) and add missing google/ upstream IDs to
the gemma4 Ollama mapper for consistency with other gemma entries.
2. mapper.py: Remove self-mapping non-bnb-4bit entries from
__INT_TO_FLOAT_MAPPER that were polluting FLOAT_TO_INT_MAPPER with
lowercase 16-bit names, causing load_in_4bit=True to return bad
model names. Add direct MAP_TO_UNSLOTH_16bit entries to preserve
the google->unsloth 16-bit redirects.
3. mapper.py: Add LFM2.5 MAP_TO_UNSLOTH_16bit redirect so
LiquidAI/LFM2.5-1.2B-Instruct resolves to its unsloth mirror.
* Add review tests for PR #4950
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove top-level test files
These test_*.py files were added at the repo root rather than under tests/.
Removing them from this PR; the production mapper changes remain.
* Add gemma-4-26B-A4B-it mapping
Adds unsloth/gemma-4-26B-A4B-it to __INT_TO_FLOAT_MAPPER as a 2-tuple so
google/gemma-4-26B-A4B-it routes to unsloth/gemma-4-26B-A4B-it across
INT_TO_FLOAT_MAPPER, FLOAT_TO_INT_MAPPER, and MAP_TO_UNSLOTH_16bit.
The 26B-A4B (MoE) model has no bnb-4bit variant, so the key uses the
plain unsloth name rather than the -unsloth-bnb-4bit suffix.
Removes the now-redundant standalone _add_with_lower call for the -it
variant; the 16bit mapping is registered via the dict loop.
* Add unsloth-bnb-4bit mappings for gemma-4 base (non-it) models
Adds E2B, E4B, 31B base unsloth-bnb-4bit entries to __INT_TO_FLOAT_MAPPER.
The 26B-A4B (MoE) base has no bnb-4bit variant on HF, so it stays on the
standalone _add_with_lower line for the 16bit-only routing.
Removes the redundant _add_with_lower lines for E2B, E4B, 31B base since
the dict loop now registers the same google->unsloth route through the
2-tuple entries, plus full FLOAT_TO_INT and INT_TO_FLOAT coverage.
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: refresh Downloaded GGUF list and recurse into variant subdirs
Two fixes for the model picker's "Downloaded" section.
Frontend (`pickers.tsx`):
* `HubModelPicker`'s mount effect short-circuited the cached-gguf and
cached-models refetch whenever the module-level cache already had
entries (`if (alreadyCached) return;`). After downloading a new repo
in the same session, reopening the picker rendered the stale cache
and the new repo never appeared in "Downloaded" until a full page
reload. The early return is removed so the lists are always refreshed
on mount; the module cache still drives the initial render so there
is no spinner flash when we already had data.
Backend (`utils/models/model_config.py`):
* `list_local_gguf_variants` and `_find_local_gguf_by_variant` used a
non-recursive `Path.glob("*.gguf")`. Some HF GGUF repos (e.g.
`unsloth/gemma-4-26B-A4B-it-GGUF`) place the largest quants under a
variant-named subdirectory such as `BF16/...gguf`, which the
top-level glob missed. Both helpers now use `rglob` and the variant
filename is stored as a path relative to the scan root so the
locator can still find the file.
The flat-layout case (variants directly in the snapshot root) is
unchanged: verified against `unsloth/gemma-4-E2B-it-GGUF` which still
returns its UD-Q4_K_XL variant correctly.
* Studio: emit posix-style relative filenames for local GGUF subdirs
`list_local_gguf_variants` was doing `str(f.relative_to(p))`, which on
Windows produces backslash-separated paths like `BF16\foo.gguf`. The
remote `list_gguf_variants` (HF API path) always returns forward-slash
filenames such as `BF16/foo.gguf`, so the two would diverge on Windows.
Switch to `.as_posix()` so the local and remote variant filenames stay
identical across Linux, macOS, and Windows. Verified by simulating with
`PureWindowsPath` in the test suite.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: detect mmproj at snapshot root for nested-variant layouts
When _find_local_gguf_by_variant returns a weight file inside a
quant-named subdir (e.g. snapshot/BF16/foo.gguf), detect_mmproj_file
was scanning only the immediate parent and missing the mmproj file
sitting at the snapshot root. The model was then loaded without
--mmproj, silently breaking vision support for repos that ship
nested variants.
detect_mmproj_file now takes an optional search_root and walks up
from the weight file to that root, in order, so the mmproj at the
snapshot root is picked up. Sibling quant subdirs are not scanned,
so an unrelated variant's mmproj does not leak in.
Also apply the suggested micro-optimization on relative_to in
list_local_gguf_variants -- only build the posix path when storing
the first file for a quant.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(rocm): tighten gfx regex to ignore generic ISA lines
ROCm 6.1+ rocminfo emits generic ISA names such as
"amdgcn-amd-amdhsa--gfx11-generic" and "amdgcn-amd-amdhsa--gfx9-4-generic"
alongside the real GPU name. The previous `gfx[1-9]` regex used in
`_has_rocm_gpu` matched both, so a host with only a generic ISA entry
would be reported as having a usable AMD GPU.
Tighten the pattern to `gfx[1-9][0-9a-z]{2,3}` so only real gfx ids
match. This covers every documented target from GFX6 (gfx600) through
GFX12 (gfx1201), including letter-suffixed ids like gfx90a (MI250 /
MI250X) and gfx90c. Documented generic ISA names always have 1 or 2
digits before the dash and no longer match.
Applied to both `studio/install_python_stack.py` and
`studio/install_llama_prebuilt.py` so the two detection paths agree.
Co-authored-by: Martin Hoyer <mhoyer@redhat.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Martin Hoyer <mhoyer@redhat.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: support GGUF variant selection for non-suffixed repos
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden GGUF detection across cached models and picker flows
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* chore: use shared GGUF picker helper for search rows
* fix: avoid mixed cache duplication and preserve GGUF fallback detection
* fix: unify GGUF cache matching and merge picker hints
* fix: normalize local GGUF matching across picker and model config
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: robust cached-gguf classification + hint-aware click routing
- _repo_gguf_size_bytes: treat size_on_disk=None as 0 and dedupe fallback
by commit_hash so partial/interrupted downloads don't TypeError out of
sum() and wipe the entire cached list.
- list_cached_gguf / list_cached_models: narrow per-repo try/except so
one malformed repo no longer poisons the whole response.
- handleModelClick: route through isKnownGgufRepo instead of the
suffix-only isGgufRepo, so non-suffixed GGUF repos still open the
variant expander from every call site.
- Replace the modelIsGgufById/resultIsGgufById Maps with Sets of known
GGUF ids to stop conflating "no hint" with "known not-GGUF".
- Make HfModelResult.isGguf required (it is always set in makeMapModel).
- Add regression tests for the None size case, mixed-repo inclusion in
cached-gguf, and per-repo error isolation.
* fix: exclude mmproj from GGUF classification and case-normalize hint lookups
- _repo_gguf_size_bytes now filters mmproj vision-adapter files so
safetensors+mmproj.gguf repos stay on the cached-models path and
non-GGUF rows no longer show zero pickable variants. A vision-capable
GGUF repo (main weight + mmproj adapter) still classifies as GGUF and
reports the main weight size.
- modelGgufIds / resultGgufIds now key on lowercased ids and
isKnownGgufRepo lowercases its lookup, so store and HF-search ids
that differ only by casing still match the same GGUF hint.
- New regression tests: mmproj-only repo excluded from cached-gguf,
same repo included in cached-models, vision-capable repo still
classified as GGUF with correct size.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* Add configurable PyTorch mirror via UNSLOTH_PYTORCH_MIRROR env var
When set, UNSLOTH_PYTORCH_MIRROR overrides the default
https://download.pytorch.org/whl base URL in all four install scripts
(install.sh, install.ps1, studio/setup.ps1, studio/install_python_stack.py).
When unset or empty, the official URL is used. This lets users behind
corporate proxies or in regions with poor connectivity to pytorch.org
point at a local mirror without patching scripts.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add pytest for UNSLOTH_PYTORCH_MIRROR in install_python_stack.py
Tests that _PYTORCH_WHL_BASE picks up the env var when set, falls back
to the official URL when unset or empty, and preserves the value as-is
(including trailing slashes).
* Remove stale test assertions for missing install.sh messages
* Fix GPU mocking in test_get_torch_index_url.sh
Extract _has_usable_nvidia_gpu and _has_amd_rocm_gpu alongside
get_torch_index_url so the GPU-presence checks work in tests.
Add -L flag handling to mock nvidia-smi so it passes the GPU listing
check. All 26 tests now pass on CPU-only machines.
* Strip trailing slash from UNSLOTH_PYTORCH_MIRROR to avoid double-slash URLs
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: hard-stop at n_ctx with a dedicated 'Context limit reached' toast
llama-server's default behavior when the KV cache fills is to silently
drop the oldest non-``n_keep`` tokens and keep generating. The UI has
no way to tell the user that earlier turns were evicted -- they just
see degraded continuity and a confusing ``5,361 / 4,096`` on the
context usage bar.
Launch llama-server with ``--no-context-shift`` so it returns a clean
error once the request would exceed ``n_ctx``. In the chat adapter,
catch the error, identify it as a context-limit error via
``isContextLimitError()``, and surface a dedicated toast that names
the exact control to adjust: the ``Context Length`` field in the chat
Settings panel.
Also add a lightweight tooltip hint on ``ContextUsageBar`` when usage
crosses 85%, so users see the "raise Context Length in Settings"
suggestion before they hit the hard stop.
Tests:
* ``test_llama_cpp_no_context_shift.py`` pins the ``--no-context-shift``
flag in the static launch-command template, and pins it inside the
unconditional ``cmd = [ ... ]`` block so a future refactor can't
hide it behind a branch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shorten --no-context-shift comment to 1 line
* Match backend _friendly_error rewrite in isContextLimitError
Codex review on PR caught that ``backend/routes/inference.py::_friendly_error``
rewrites the raw llama-server text
"request (X tokens) exceeds the available context size (Y tokens)"
into
"Message too long: X tokens exceeds the Y-token context window. ..."
on the main streaming GGUF path. The heuristic only looked for
"context size" / "exceeds the available context" / "context shift",
none of which survive the rewrite, so the new "Context limit reached"
toast would never fire for the most common case. Add matches for
"message too long" and "context window" so both wordings hit.
Also addresses Gemini feedback on the launch-flag test:
* Use ``inspect.getsource(LlamaCppBackend.load_model)`` instead of
reading ``__file__`` directly; scopes the assertions to the
function that actually launches llama-server.
* Replace the hardcoded ``" ]"`` indent search with a
line-at-a-time scan for a line that is just ``]``, so the test
survives reformatting.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: split model-load progress label across two rows
The chat flow and training overlay both compose a progress label like
"112.6 of 122.3 GB • 331.0 MB/s • 30s left" and render it next to the
percent badge in a single flex row. Once the rate + ETA part shows up,
the label outgrows the row width and wraps mid-phrase, orphaning the
percent ("19 left %") onto a second ragged line.
Fix in model-load-status.tsx: split the label on the first " • " into
a primary (size) chunk that stays on row 1 with the percent, and a
secondary (rate/ETA) chunk that renders on its own muted row below.
Labels without a bullet (e.g. "22.8 GB downloaded") collapse cleanly
to one row. The inline-status variant keeps only the primary and
surfaces the full label via the tooltip.
Also extracts the rate/ETA math out of useTransferStats into a pure
``transfer-stats.ts`` module (appendSample + computeTransferStats) so
it can be reasoned about and tested without React. The hook is now a
thin wrapper that feeds sample history through the pure functions.
Backend: adds two companion test files for load_progress():
* test_llama_cpp_load_progress_matrix.py (21 tests) -- platform
matrix (Linux /proc, macOS/Windows absence), VmRSS parsing
variants (tab/space/missing/malformed), filesystem edges (HF-cache
symlinks, broken symlinks, nonexistent paths, relative paths),
shard aggregation (partial multi-shard, two series in same dir,
mmproj-* exclusion, single-file), lifecycle races, concurrent
sampling (10 threads x 50 iters against real /proc), fraction
bounds.
* test_llama_cpp_load_progress_live.py (5 tests) -- no-mock live
integration: real subprocess allocating 100 MB to match VmRSS,
real ready phase, real dead-pid degradation, real shard
aggregation, repeated polling. Skipped on non-Linux.
Both complement the existing test_llama_cpp_load_progress.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Hoist splitProgressLabel out of JSX IIFE (review feedback)
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: live model-load progress + rate/ETA on download and load
Two UX fixes for the opaque multi-minute wait between clicking Load
and being able to chat, visible most clearly on large MoE GGUFs like
MiniMax-M2.7 (131 GB of weights on a 97 GB GPU):
1. **Model-load phase is now observable.** The existing chat flow
transitions the toast to "Starting model..." as soon as the
download hits 100%, then shows a spinner with no other feedback
until llama-server reports healthy. For a 130 GB model that spinner
freezes for five-plus minutes while the kernel pages shards into
the page cache. A new `GET /api/inference/load-progress` endpoint
samples `/proc/<pid>/status VmRSS` on the llama-server subprocess
against the sum of shard file sizes on disk, so the UI can render
a real bar plus rate / ETA during that window.
2. **Rate and ETA on downloads and loads.** Both the chat toast and
the training-start overlay used to show a static pair of numbers
(for example "15.4 of 140.8 GB"). A rolling 15-second window over
the existing byte-series now surfaces "85.3 MB/s, 24m 23s left"
beside that pair. The estimator is shared between the download
and load phases so the numbers don't reset when the phase flips.
Also fixes a pre-existing assignment bug uncovered while wiring this
up: `load_model` was storing the caller's `gguf_path` kwarg into
`self._gguf_path`, which is `None` on the HF-download code path. The
resolved on-disk path (`model_path`) is what llama-server actually
mmaps; downstream consumers need that. No existing reader used
`_gguf_path`, so this is a correctness fix for the new endpoint.
- Backend: `LlamaCppBackend.load_progress()`, `GET /api/inference/load-progress`, `LoadProgressResponse` Pydantic model.
- Frontend: `useTransferStats` hook, `formatRate` / `formatEta` helpers, `getLoadProgress` client, rewired chat toast and `DownloadRow` in the training overlay.
- Tests: `studio/backend/tests/test_llama_cpp_load_progress.py` covers empty states, mmap phase, ready phase, sharded total aggregation, missing gguf_path, and unreadable /proc (7 cases). `tsc -b` and `vite build` on the frontend both clean.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: pin peft to 0.18.1 to fix export subprocess issues
peft 0.19.0 causes export subprocess shutdown failures in Studio.
Reverting to 0.18.1 resolves the issue.
* studio: move peft pin to extras-no-deps to prevent torch upgrade
Installing peft via overrides.txt would resolve its deps and pull in
torch>=0.11.0, breaking other pinned packages. Moving the pin to
extras-no-deps.txt ensures --no-deps is used during install.
* studio: stream export worker output into the export dialog
The Export Model dialog only showed a spinner on the "Exporting..."
button while the worker subprocess was doing the actual heavy lifting.
For Merged to 16bit and GGUF / Llama.cpp exports this meant several
minutes (or more, for large models) of opaque silence, with no way to
tell whether save_pretrained_merged, convert_hf_to_gguf.py, or
llama-quantize was making progress.
This adds a live terminal-style output panel inside the export dialog,
rendered just above the Cancel / Start Export buttons and scrollable
with auto-follow-tail. It shows stdout and stderr from both the worker
process itself and any child process it spawns (GGUF converter,
llama-quantize), coloured by stream.
Backend
- core/export/worker.py: new _setup_log_capture(resp_queue) installed
before LogConfig.setup_logging. It saves the original stdout/stderr
fds, creates pipes, os.dup2's the write ends onto fds 1 and 2 (so
every child process inherits the redirected fds), and spins up two
daemon reader threads. Each thread reads bytes from a pipe, echoes
them back to the original fd (so the server console keeps working),
splits on \n and \r, and forwards each line to the resp queue as
{"type":"log","stream":"stdout|stderr","line":...,"ts":...}.
PYTHONUNBUFFERED=1 is set so nested Python converters flush
immediately.
- core/export/orchestrator.py:
- Thread-safe ring buffer (collections.deque, maxlen 4000) with a
monotonically increasing seq counter. clear_logs(),
get_logs_since(cursor), get_current_log_seq(), is_export_active().
- _wait_response handles rtype == "log" by appending to the buffer
and continuing the wait loop. Status messages are also surfaced as
a "status" stream so users see high level progress alongside raw
subprocess output.
- load_checkpoint, _run_export, and cleanup_memory now wrap their
bodies with the existing self._lock (previously unused), clear the
log buffer at the start of each op, and flip _export_active in a
try/finally so the SSE endpoint can detect idle.
- routes/export.py:
- Wrapped every sync orchestrator call (load_checkpoint,
cleanup_memory, export_merged_model, export_base_model,
export_gguf, export_lora_adapter) in asyncio.to_thread so the
FastAPI event loop stays free during long exports. Without this
the new SSE endpoint could not be served concurrently with the
blocking export POST.
- New GET /api/export/logs/stream SSE endpoint. Honors
Last-Event-ID and a since query param for reconnect, emits log /
heartbeat / complete / error events, uses the id field to carry
the log seq so clients can resume cleanly. On first connect
without an explicit cursor it starts from the current seq so old
lines from a previous run are not replayed.
Frontend
- features/export/api/export-api.ts: streamExportLogs() helper that
authFetches the SSE endpoint and parses id / event / data fields
manually (same pattern as streamTrainingProgress in train-api.ts).
- features/export/components/export-dialog.tsx:
- Local useExportLogs(exporting) hook that opens the SSE stream on
exporting transitions to true, accumulates up to 4000 lines in
component state, and aborts on cleanup.
- New scrollable output panel rendered above DialogFooter, only
shown for Merged to 16bit and GGUF / Llama.cpp (LoRA adapter is
a fast disk write with nothing to show). Dark terminal styling
(bg-black/85, emerald text, rose for stderr, sky for status),
max-height 14rem, auto-scrolls to the bottom on new output but
stops following if the user scrolls up. A small streaming / idle
indicator is shown next to the panel title.
- DialogContent widens from sm:max-w-lg to sm:max-w-2xl when the
output panel is visible so the logs have room to breathe.
Verified
- Python smoke test (tests/smoke_export_log_capture.py): spawns a
real mp.get_context("spawn") process, installs _setup_log_capture,
confirms that parent stdout prints, parent stderr prints, AND a
child subprocess invoked via subprocess.run (both its stdout and
stderr) are all captured in the resp queue. Passes.
- Orchestrator log helpers tested in isolation: _append_log,
get_logs_since (with and without a cursor), clear_logs not
resetting seq so reconnecting clients still progress. Passes.
- routes.export imports cleanly in the studio venv and /logs/stream
shows up in router.routes.
- bun run build: tsc -b plus vite build, no TypeScript errors.
No existing export behavior is changed. If the subprocess, the SSE
endpoint, or the frontend hook fails, the export itself still runs to
completion the same way it did before, with or without logs visible.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* export dialog: trim bootstrap noise, scope logs per screen, show realpath
Several follow-ups to the live export log work:
1. Worker bootstrap noise (transformers venv activation, Unsloth banner,
"Top GGUF/hub models" lists, vision detection, 2k-step weight load
bar) is dropped from the export-dialog stream. A threading.Event
gate in worker.py defaults closed and only opens once _handle_export
actually starts; until then the reader thread still echoes lines to
the saved console fd for debugging but does not push them onto the
resp_queue. The orchestrator already spawns a fresh subprocess for
every checkpoint load, so the gate is naturally reset between runs.
2. tqdm in non-tty mode defaults to a 10s mininterval, which makes
multi-step bars look frozen in the panel. Set TQDM_MININTERVAL=0.5
in the worker env so any tqdm-driven progress emits more often.
3. The dialog's useExportLogs hook now also clears its line buffer
when exportMethod or open changes, so re-opening the dialog into a
different action's screen no longer shows the previous action's
saved output. A useElapsedSeconds tick + "Working Xs" badge in the
log header gives users a visible sign that long single-step phases
(cache copies, GGUF conversion) are still running when no new lines
are arriving.
4. ExportBackend.export_{merged,base,gguf,lora} now return
(success, message, output_path); the worker forwards output_path on
each export_*_done response, the orchestrator's _run_export passes
it to routes/export.py, which surfaces it via
ExportOperationResponse.details.output_path. The dialog's Export
Complete screen renders the resolved on-disk realpath under "Saved
to" so users can find their exported model directly.
* fix(cli): unpack 3-tuple return from export backend
ExportOrchestrator.export_{merged,base,gguf,lora} now return
(success, message, output_path) so the studio dialog can show
the on-disk realpath. The CLI still unpacked 2 values, so every
`unsloth export --format ...` crashed with ValueError before
reporting completion. Update the four call sites and surface
output_path via a "Saved to:" echo.
* fix(studio): anchor export log SSE cursor at run start
The export dialog SSE defaulted its cursor to get_current_log_seq()
at connect time, so any line emitted between the POST that kicks
off the export and the client opening the stream was buffered with
seqs 1..k and then skipped (seq <= cursor). Long-running exports
looked silent during their first seconds.
Snapshot _log_seq into _run_start_seq inside clear_logs() and
expose it via get_run_start_seq(). The SSE default cursor now uses
that snapshot, so every line emitted since the current run began
is reachable regardless of when the client connects. Old runs
still can't leak in because their seqs are <= the snapshot.
* fix(studio): reconnect export log SSE on stream drop
useExportLogs launched streamExportLogs once per exporting
transition and recorded any drop in .catch(). Long GGUF exports
behind a proxy with an idle kill-timeout would silently lose the
stream for the rest of the run even though the backend already
supports Last-Event-ID resume. The "retry: 3000" directive emitted
by the backend is only meaningful to native EventSource; this
hook uses a manual fetch + ReadableStream parse so it had no
effect.
Wrap streamExportLogs in a retry loop that tracks lastSeq from
ExportLogEvent.id and passes it as since on reconnect. Backoff is
exponential with jitter, capped at 5s, reset on successful open.
The loop stops on explicit backend `complete` event or on effect
cleanup.
* fix(studio): register a second command so Typer keeps `export` as a subcommand
The CLI export unpacking tests wrap `unsloth_cli.commands.export.export`
in a fresh Typer app with a single registered command. Typer flattens a
single-command app into that command, so the test's
`runner.invoke(cli_app, ["export", ckpt, out, ...])` treats the leading
`"export"` token as an unexpected extra positional argument -- every
parametrized case failed with:
Got unexpected extra argument (.../out)
Register a harmless `noop` second command so Typer preserves subcommand
routing and the tests actually exercise the 3-tuple unpack path they
were written to guard.
Before: 4 failed
After: 4 passed
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: studio-install <studio@local.install>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
* studio: show HF model download progress in training start overlay
During the training setup phase, the overlay only displayed a static
"Loading model..." line while model weights were being downloaded from
Hugging Face. On slow connections this looked like the app had frozen.
This adds a small self-contained progress block inside the existing
TrainingStartOverlay that polls the existing
GET /api/models/download-progress endpoint and renders a Progress bar
with bytes downloaded, total bytes, and percent complete.
Notes:
- Frontend only change. No backend, worker, SSE, or runtime store edits.
- Reuses the existing getDownloadProgress client wrapper and the
existing /api/models/download-progress endpoint that already scans
the HF blob cache for completed and .incomplete files.
- selectedModel is read directly from useTrainingConfigStore inside the
overlay, so no prop drilling and live-training-view.tsx is unchanged.
- Polling runs at 1500 ms and is gated on the HF repo regex
(^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$), the same regex the backend uses,
so local paths and empty form state never hit the endpoint.
- Polling stops once progress reaches 1.0 so the bar can stay at 100
until the overlay hides on the first training step.
- Network errors are silently swallowed, matching the chat side flow
(the bar simply freezes at the last value).
- When downloadedBytes is 0 the block is hidden entirely, so cached
models do not flash a progress bar.
- When the HF API cannot determine the total size, the block falls
back to "X downloaded" with no percent and no bar.
Verified with bun run build (tsc -b plus vite build, no TypeScript
errors).
* training overlay: track dataset download + show on-disk realpath
Adds a dedicated "Downloading dataset..." section to the training-start
overlay alongside the existing model-weights one, so an HF dataset that
is downloading mid-startup is no longer mislabeled as model weights or
hidden entirely. The new GET /api/datasets/download-progress endpoint
mirrors /api/models/download-progress against the datasets-- prefix in
HF_HUB_CACHE.
Both endpoints now also return cache_path, the resolved on-disk
realpath of the snapshot directory (or the cache repo root if no
snapshot is materialized yet). The overlay surfaces this under each
download row so users can immediately see where the model and dataset
landed without digging through server logs.
The frontend's existing useModelDownloadProgress hook is generalized
to a single useHfDownloadProgress(repoId, fetcher) hook that the
model and dataset variants both delegate to, keeping polling, gating,
and completion semantics in one place.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: Polish training start overlay download progress UI (#4957)
* studio: polish training start overlay download progress visuals
* Fix formatCachePath cross-platform support and redundant sizeLabel
- Extend formatCachePath regex to also shorten macOS /Users/<user> paths to ~
- Suppress sizeLabel when no byte info is available (cachePath-only state),
since the "Preparing" badge already conveys the status
* Fix misleading status badge when download total is unknown
- Hide badge when totalBytes is 0 but downloadedBytes > 0, since we cannot
determine if the download is still in progress or already complete (happens
when HF size metadata lookup fails for gated/private repos)
- Keep "Preparing" badge for the zero-bytes cachePath-only state
- Add Windows native path shortening to formatCachePath (C:\Users\<name>)
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
---------
Co-authored-by: studio-install <studio@local.install>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: anchor ctx-slider warning threshold at 4096 when weights exceed VRAM
The chat settings sheet's ctx slider reads `max_context_length` from
`/api/inference/status` and renders
Exceeds estimated VRAM capacity (N tokens). The model may use
system RAM.
when the user drags the slider above that value. For models whose
weights fit on some GPU subset, `_max_context_length` was already set
to the binary-search cap and the warning fired correctly.
For models whose weights exceed 90% of every GPU subset's free memory
(e.g. MiniMax-M2.7-GGUF at 131 GB on a 97 GB GPU), the ceiling-probe
loop never matched a subset, so `max_available_ctx` stayed at the
native context (e.g. 196608). The slider ran all the way to native
with no indication that any value above the 4096 spec default would
trigger `--fit on` and degrade performance.
Anchor `max_available_ctx` at `min(4096, native_context_length)` when
no subset fits, so the warning fires at the right threshold and the
user sees the correct safe-zone / warning-zone split:
Before (MiniMax-M2.7 on 97 GB GPU):
slider 0 .. 196608, warning threshold = 196608 (never fires)
After:
slider 0 .. 196608, warning threshold = 4096 (fires correctly)
No frontend changes required: `chat-settings-sheet.tsx` already
consumes `ggufMaxContextLength` (= status.max_context_length) as the
warning threshold and `ggufNativeContextLength` as the slider max.
Adds tests/test_llama_cpp_max_context_threshold.py covering
weights-exceed-VRAM (single / multi-GPU), a native-ctx below the 4096
fallback case (don't lie about supported ctx), fittable-model
regressions (small / multi-GPU / tiny on huge GPU), and the
`max_context_length` property's fallback semantics.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
imagineer99