Follow-up to the merged #5940. Trims verbose comments across the
Windows/WSL installer and Studio backend to their load-bearing content
(constraints, env-var names, issue refs, magic-value rationale),
removing restated-code narration and multi-sentence justifications.
Comment-only and machine-verified: every .py file is AST-dump-identical
(docstrings normalized), every .ps1 is non-comment-token identical, and
every .sh has zero non-comment-line changes with bash -n clean, all
checked against main. Install suites (340 passed) and backend suites
(282 passed) unchanged. Net -159 comment lines across 15 files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Studio: keep llama-server discovery from crashing on an access-denied candidate
_find_llama_server_binary probed candidates with Path.is_file(), which raises
PermissionError (WinError 5) when a path exists but is momentarily inaccessible
(antivirus lock, an install replace in flight, an elevated-install ACL),
aborting model validation. Treat a denied-but-present path as the real binary
so discovery returns it; absent paths still skip.
* Retry a transiently locked binary instead of returning a denied path
Returning a still-denied path only moved the PermissionError to the next
is_file() (probe_server_capabilities). Retry briefly so a transient lock
clears and discovery returns an accessible path; on a persistent lock return
nothing rather than a path downstream cannot stat.
* Studio: do not fall back to another llama-server when a pinned one is locked
A denied LLAMA_SERVER_PATH made discovery skip the explicit pin and run a
lower-priority managed or PATH binary, so a load could silently use a stale or
incompatible server. Split the probe into a file/absent/denied status: when the
pinned path exists but stays access-denied, warn and stop rather than falling
back to a different executable.
* Studio: never downgrade past a denied pinned or managed llama-server
Extend the no-fallback rule beyond LLAMA_SERVER_PATH: a present-but-denied
UNSLOTH_LLAMA_CPP_PATH or managed ($STUDIO_HOME/llama.cpp, ~/.unsloth/llama.cpp)
binary now reports temporarily-unavailable instead of silently launching a
lower-priority legacy or PATH server. Shared _scan_pinned/_unavailable helpers;
legacy in-tree and PATH stay genuine fallbacks (a denied candidate there just
continues).
* Studio: let diffusion asset lookup use a locked llama-server path for its dir
DiffusionGemma does not run llama-server; _find_diffusion_assets only needs the
install dir to find the adjacent llama-diffusion-gemma-visual-server. The
no-fallback rule returning None on a transiently locked llama-server therefore
hid an available visual-server and raised 'runner not found'. Add an
include_denied option so diffusion lookup gets the locked path (its dir is all
it needs), while inference keeps the no-denied-path, no-downgrade behavior.
* Studio: report a locked llama-server as temporarily unavailable, not missing
When the pinned/managed binary stays access-denied through the retries, discovery
returns None and load_model raised 'binary not found', a terminal error that
points users at reinstalling rather than retrying a transient AV/install lock.
Reuse include_denied to detect the locked path and raise a distinct
temporarily-unavailable, retry message instead.
* Studio: GGUF preflight treats a locked llama-server as present
The pre-download preflight (and so /api/inference/validate) used the default
discovery, which returns None for a transiently access-denied binary, so it
raised 'binary not found' for a binary that merely needs the lock to clear. Use
include_denied so the existence check counts a locked binary as present; the
load itself still reports a still-locked binary as temporarily unavailable.
* fix: extend llama.cpp first-token timeout
* fix: timeout label pluralization
* studio: distinguish llama stream timeout phases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust timeout handling for PR #5841
* Fix lint failure for PR #5841
* Fix/adjust stream timeout handling for PR #5841
* Fix/adjust first token timeout for PR #5841
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust passthrough timeouts for PR #5841
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust preheader stream cancellation for PR #5841
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust timeout PR diff for PR #5841
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust Python 3.9 stream iteration for PR #5841
* Fix first body timeout for PR #5841
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix first token timeout deadlines for PR #5841
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
The studio backend lazily imports unsloth_zoo submodules (export's
llama_cpp, hardware's vllm_utils, mlx, chat_templates). Each imports the
top-level unsloth_zoo, which raises 'Please install Unsloth via pip
install unsloth' unless UNSLOTH_IS_PRESENT is set (normally by import
unsloth). Only hardware.py set it per-site; the rest crash on a clean
install, and it surfaced on Windows. Set it once at backend startup (as
unsloth does on import) so every site, and the DiffusionGemma runner,
works across platforms. Follow-up to #6259 (which covered only the shim
subprocess env).
list_gguf_variants() keys files by _extract_quant_label(), which only captured the base quant token. Repos that ship the same base quant at multiple bits-per-weight (e.g. byteshape/Qwen3.6-35B-A3B-MTP-GGUF with three IQ4_XS files at 3.53/3.97/4.19 bpw) collapsed into a single row and Studio summed their sizes (~48 GB).
Extend the regex to capture an optional trailing -<N>(.<N>)?bpw modifier so each flavor produces a unique label. Round-trips through _find_local_gguf_by_variant and _download_gguf since both sides use the same extractor.
Fixes#5728.
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
A clean install could not run DiffusionGemma: the runner spawns
python -m unsloth_zoo.diffusion_studio.shim, and unsloth_zoo refuses to
import unless UNSLOTH_IS_PRESENT is set (normally by import unsloth). The
shim never imports unsloth, so the subprocess died with
'Please install Unsloth via pip install unsloth!' and the model load
failed with a 500. Set the flag in the runner child env, as unsloth does
on import.
* fix(studio): keep local GGUF vision on llama-server
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): lower local GGUF vision log level
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): find GGUF companions from variant dirs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Studio: account for mmproj VRAM in GGUF fit budget (#5825)
Vision GGUFs load the mmproj projector onto the GPU via --mmproj
alongside the weights, but the context auto-sizing / GPU-selection
budget sized off _get_gguf_size_bytes(model_path), which counts only
the weight file(s). The projector was never added, so the budget was
too optimistic: context got mis-estimated and tight vision loads
spilled to system RAM / OOM'd.
Resolve the launch projector once before GPU selection and fold its
size into the fit budget. The same resolved path feeds both the budget
and the --mmproj launch flag, so the two cannot disagree. The summary
log now reports the projector size separately, keeping "GGUF size"
accurate.
Adds _mmproj_vram_bytes() + unit tests (no GPU / network / subprocess).
* Studio: simplify mmproj summary-log concatenation (#5825)
Address review: the summary log mixed explicit `+` with implicit
f-string concatenation. Extract the optional projector fragment into
`mmproj_note` so the logger.info uses uniform implicit concatenation.
No behavioral change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: trim mmproj VRAM comments
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
DiffusionGemma serves via the visual runner, which streams per-step
canvas frames so the answer resolves live in the bubble. The agentic
tool loop (generate_chat_completion_with_tools) does not forward those
frames, so whenever a tool pill (Search/Code) was on the live canvas
silently vanished while text still streamed. DiffusionGemma is not a
tool-calling target anyway, so report supports_tools=False for it: the
chat always takes the frame-forwarding path, and the Search/Code pills
disable themselves (a local model has no builtin web search either).
Also turn the artifacts canvas on by default for DiffusionGemma so a
full-HTML answer (e.g. a playable game) renders as an interactive
sandboxed card without the user flipping the global artifacts toggle.
* Studio: make project sources work with RAG and polish project UI
Projects had a disabled Sources tab with an Add sources placeholder.
This wires it up end to end on top of the RAG engine:
- Add a project scope to the RAG store, ingestion and retrieval
- New endpoints: POST/GET /api/rag/projects/{id}/documents
- search_knowledge_base resolves kb, project and thread scopes; an
explicit KB stays exclusive, project and thread scopes combine
- Multi-scope search: FTS uses scope IN (...), vec0 KNN runs per
scope and merges by cosine score
- Lazy ALTER TABLE adds documents.project_id on existing databases
- Deleting a project also removes its indexed sources
- Sources tab now uploads with progress chips and drag and drop
- Chats inside a project auto-enable retrieval over project sources
when the project has indexed documents (cached probe, no Docs pill
needed); external providers still never receive rag_scope
UI polish:
- Rounder project cards with folder icon chip and softer shadow
- Project header icon in a rounded chip
- Chats/Sources pills and Add sources button without borders
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: match Add sources button shadow to the chat composer in light mode
* Studio: round project switcher hover pill and pad the folder icon
* Studio: remove border from project sources box
* Studio: grey hover on project cards and menu, move search into header, widen page spacing
* Studio: shorten sources copy, white header pills with composer shadow, fixed-width search, hub-size page headings
* Studio: align project landing blocks to the composer width
* Studio: restore muted background and flat look on projects header controls
* Studio: darker grey hover on project cards in light mode
* Studio: soften project card hover grey
* Studio: keep project card menu button visible while its menu is open
* Studio: drop focus outlines and rings on buttons and clickable icons, keep input focus styles
* Studio: address review feedback on project sources
- Remove uploaded files from disk when a project is deleted, confined
to the uploads root
- 404 project uploads when the project does not exist, matching the KB
endpoint
- Guard lexical search against an empty scope list
- Re-invalidate the project sources probe after uploads and removals
settle so a chat sent mid-upload cannot cache a stale negative
- Keep keyboard focus rings: only mouse focus drops the Tailwind ring,
the browser default outline stays removed
* Studio: add a green New badge to the project Sources tab
* Studio: unify New pills, fully round with soft emerald fill and no border
* Studio: a touch more vertical padding on New pills
* Fix project RAG source edge cases for PR #6205
* Fix duplicate RAG upload cleanup for PR #6205
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: cache MCP tool discovery instead of re-probing every chat send
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Stop re-probing offline/down MCP servers every time
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add tests for mid-probe delete and OAuth cool-off paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Don't cool-off a server edited or deleted mid-probe
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard MCP refresh cache writes
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* feat(studio): add S3 dataset configuration foundation (#4539)
Add foundational types and configuration for S3 bucket dataset loading:
- Add S3Config type to frontend training types
- Add S3Config Pydantic model to backend training models
- Add "s3" as a DatasetSource option
- Add s3Config state and setS3Config action to training config store
- Add i18n translations for S3 configuration (English and Chinese)
This provides the type definitions and UI text for S3 integration.
Full implementation requires boto3 dependency and data loading logic.
Refs: #4539
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Wire S3 config into training pipeline and prevent secrets persistence
- Pass s3_config from request into training_kwargs so it flows to training subprocess
- Add s3Config to NON_PERSISTED_STATE_KEYS to prevent AWS secrets from being
saved to localStorage
Addresses code review feedback on PR #5951.
* Exclude S3 config from database persistence to protect secrets
Filter out s3_config (which contains secret_access_key) from the
config_json stored in training_runs table, preventing AWS credentials
from being persisted to disk.
Addresses P1 security feedback on PR #5951.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Re-raise HTTPException in start_training and defer s3 DatasetSource widening for PR #5951
* Redact s3_config from W&B run config and accept camelCase S3 credential aliases for PR #5951
* feat(studio): implement S3 dataset loading end-to-end
Builds the actual S3 loader on top of the hardened #5951 foundation,
turning the 501-gated scaffold into a working dataset source.
Backend:
- Add core/training/s3_dataset.py: lists and downloads supported dataset
files (parquet/json/jsonl/csv) from an S3 bucket to a temp dir, using
IAM-role or access-key credentials. boto3 is imported lazily (optional dep).
- Wire s3_config into UnslothTrainer.load_and_format_dataset (downloads then
reuses the existing local-file path) and thread it through worker.py.
- Replace the 501 "not implemented" gate with a boto3-availability guard so
S3 works when boto3 is present and fails clearly when it is not.
- Add boto3 to studio.txt requirements.
- Add tests/test_s3_dataset.py (8 tests) covering download/filtering,
collisions, missing-boto3, and S3Config camelCase/IAM validation.
Frontend:
- Widen DatasetSource to include "s3"; add s3_config to the training payload
type and mapper; add an S3 validation branch and selectS3Source store action.
- Add s3-config-form.tsx (bucket/region/prefix/keys/IAM toggle) reusing the
existing studio.dataset.s3.* i18n strings.
- Add a Hugging Face / Local / Amazon S3 source toggle in dataset-section;
the S3 config card replaces the dataset combobox when S3 is selected.
- Fix DatasetPreviewDialog to accept the widened DatasetSource type.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix S3 dataset loader for PR #6222
* Fix S3 dataset edge cases for PR #6222
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix S3 IAM payload handling for PR #6222
* Block multimodal S3 datasets for PR #6222
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Ash <ash@MacBook-Pro.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: override chat template for unsloth/gemma-4-*-GGUF with bundled gemma-4.jinja
The chat templates baked into the shipped unsloth/gemma-4-*-GGUF quants predate
Google's gemma-4 chat-template PR #118 and lack the preserve_thinking flag, so
Studio cannot surface the "Preserve thinking" toggle for Gemma 4. Bundle the updated
template and override the embedded one at llama-server launch via --chat-template-file,
scoped to the gemma-4 GGUF family, so users do not need to re-download any quant.
- Add studio/backend/assets/chat_templates/gemma-4.jinja (PR #118 based;
preserve_thinking defaults false, the one deliberate divergence from upstream).
- Add core/inference/chat_templates.py: gemma-4 GGUF matcher plus an
effective-override resolver (explicit user template still wins).
- Wire the resolver into routes/inference.py ahead of the reload-dedup check and
both load_model calls so the live backend and the incoming request compare against
the same template text (no spurious reloads).
- Default preserve_thinking off in the launch-time chat_template_kwargs so direct
API callers match the UI default.
- Ship the asset via package-data and add unit tests.
* Studio: ship E2B/E4B edge variant of the bundled Gemma 4 template
Google ships two distinct gemma-4 chat templates: E2B and E4B omit the empty
"<|channel>thought<channel|>" block on enable_thinking=false, while the
12b/26B-A4B/31B family emits it (confirmed against google/gemma-4-E2B-it,
-E4B-it, -12b-it, -26B-A4B-it, -31B-it; the two families differ only in that
one block). The single PR #118 based template followed the larger-model
behavior, which is wrong for the E2B/E4B GGUFs this feature most targets.
- Add studio/backend/assets/chat_templates/gemma-4-edge.jinja: identical to
gemma-4.jinja minus the empty-thought-block, matching E2B/E4B behavior.
- Route unsloth/gemma-4-E2B-it-GGUF and -E4B-it-GGUF to the edge template;
12b/26B-A4B/31B keep gemma-4.jinja.
- Extend tests for the edge matcher, per-family routing, and the empty-thought
block difference (off for edge, on for standard).
* Studio: address review feedback on the gemma-4 template override
- Normalize owner-less shorthand model ids in the template matcher: a bare
"gemma-4-E2B-it-GGUF" is canonicalized to "unsloth/" the same way
ModelConfig.from_identifier does, so shorthand loads still get the override
(and the preserve_thinking capability) instead of falling back to the
embedded template.
- Scope the test's module stubs with unittest.mock.patch.dict instead of
sys.modules.setdefault, and only stub deps that are missing, so the global
module registry is not polluted for tests that run afterwards.
- Guard the Jinja render tests with pytest.importorskip("jinja2") so the suite
stays runnable in minimal Studio environments where jinja2 is not present.
- Add tests for shorthand resolution.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address 10-reviewer P1 findings on the gemma-4 template override
- /status no longer surfaces Studio's auto-applied bundled template as a
user-authored chat_template_override. The frontend adopts that field as
editable state and would otherwise re-send the gemma-4 template as an explicit
override for a later, unrelated model. /status now reports None when the live
override equals the model's auto-resolved bundled template.
- When a bundled family template is in effect, strip an inherited
--chat-template-file from llama_extra_args too (not only when the raw request
set chat_template_override). Otherwise a stale inherited template, appended
last, shadows the bundled one while Studio reports the bundled template's
capabilities.
- Write the temp chat-template file as UTF-8 explicitly, and keep the bundled
templates ASCII (replaced em dashes), so non-UTF-8 Windows locales cannot raise
UnicodeEncodeError or emit a mis-encoded template. Added an ASCII guard test.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: serve DiffusionGemma GGUFs with the on-device visual decoder
* Studio: render the DiffusionGemma denoising canvas live in chat with honest stats
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden DiffusionGemma runner resolution (Windows .exe, build/bin lookup, clear stale audio flag, safe PYTHONPATH, Linux-only pdeathsig)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: fix Downloaded model list disappearing and order it by last download
The chat model picker scan for cached GGUF and safetensors models aborted
whenever an auxiliary Hugging Face cache dir (such as ~/.cache/huggingface/hub)
was unreadable, returning an empty list. That hid the Downloaded section and
let already downloaded models appear under Recommended. Isolate each cache
probe so an inaccessible directory is skipped instead of failing the scan.
Also order Downloaded newest-first using cached blob mtimes (multi-quant repos
group by their most recent quant), keep the section visible while searching,
and make the per-quant downloaded check per-snapshot and mmproj aware so a
Recommended quant is never falsely marked downloaded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden gguf-variants scan and dedupe by newest timestamp
Guard f.stat() per file so a broken symlink or unreadable file in a
snapshot no longer aborts the downloaded check early, and match quant
labels case-insensitively. When the same repo is present in multiple
caches with equal size, keep the newest last_modified so Downloaded
ordering reflects the most recent copy.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: apply cache-scan guards to sibling endpoints found in review
Extend the inaccessible-cache guard and mmproj/stat hardening to the
parallel HF cache code paths flagged in review:
- list_local_models and the Hub inventory scan now skip an unreadable
auxiliary cache instead of returning 500.
- The GGUF download-progress endpoint excludes mmproj adapters and
guards f.stat() so one bad file does not zero a repo's progress.
- The offline snapshot scanner guards its is_dir() probes.
- The chat-only picker no longer renders a blank list when a search
matches only cached non-GGUF models.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* install.sh: persist ROCm-on-WSL drop-in even when rocminfo already works
_maybe_bootstrap_rocm_wsl calls _ensure_rocm_probe_env (which exports a
transient HSA_ENABLE_DXG_DETECTION + adds /opt/rocm/bin to PATH on the
installer process) right before the "rocminfo enumerates gfx1151 -> already
set up, return early" gate. On any reinstall over an existing /opt/rocm --
the common case, since the uninstaller keeps shared ROCm userspace but
removes /etc/profile.d/unsloth-rocm-wsl.sh -- that probe env makes rocminfo
succeed, so the gate returns 0 WITHOUT ever persisting the drop-in. The
transient env dies with the installer, so the next login shell (Studio,
llama-server) sees no GPU: torch cuda_avail=False, rocminfo finds nothing,
the llama.cpp ROCm prebuilt segfaults on a GPU it can't reach.
Factor the drop-in writer into _persist_rocm_wsl_dropin() and call it before
the early return so the persistent env is restored whenever librocdxg is
present. Idempotent (only writes when the drop-in is missing), gated on
librocdxg so it never fires on non-WSL/non-ROCDXG hosts, root-writes or
sudo-tees like before. The fast-path branch now reuses the same helper.
Reproduced on gfx1151 (Radeon 8060S) under dash (the curl|sh shell):
before the fix a reinstall left the drop-in absent and torch cuda_avail
False; after, the drop-in is persisted and a fresh login shell reports
cuda_avail True. Verified under both dash and bash, and idempotent on
re-run.
* Studio WSL: load system HIP before a prebuilt's bundled runtime (gfx1151)
The lemonade / published llama.cpp ROCm prebuilts bundle their own HIP
runtime (libamdhip64) built for bare-metal Linux. In WSL the GPU is reached
through the system ROCm's librocdxg bridge over /dev/dxg, which the bundled
runtime cannot drive -- it segfaults on the first GPU call. So:
- install_llama_prebuilt.py: the prebuilt's llama-quantize/llama-server
validation runs with the bundle dir first on LD_LIBRARY_PATH, segfaults
(empty stderr), and the install silently falls back to a CPU source build
(which on this host can't even build for GPU -- hipcc absent). The Strix
Halo WSL user ends up on CPU despite a working GPU.
- llama_cpp.py: even if a GPU prebuilt were kept, the serve-time launcher
put the bundle dir first too, so it would crash at load.
Fix: on a ROCDXG WSL host (gated on /dev/dxg + "microsoft" /proc/version +
a librocdxg-providing /opt/rocm), prepend the system ROCm lib dir to
LD_LIBRARY_PATH so the WSL-capable libamdhip64 + librocdxg load first, while
the bundle still supplies libggml-hip / librocblas with the gfx1151 kernels.
Set HSA_ENABLE_DXG_DETECTION=1 alongside. Added _wsl_system_rocm_lib_dirs()
to both modules (kept identical so a prebuilt that passed install validation
runs the same way at serve time). Strict no-op on bare-metal Linux, NVIDIA,
macOS, and Windows.
Verified on gfx1151 (Radeon 8060S) in WSL (ROCm 7.2.1 + librocdxg, Adrenalin
ROCDXG): before, the lemonade gfx1151 prebuilt segfaulted and the install
fell back to a broken CPU build; after, install_llama_prebuilt validates and
keeps the GPU prebuilt (source=published, prebuilt_fallback_used=False), and
Studio serves Qwen3-1.7B-GGUF at 53 tok/s with the model resident in GPU
memory (llama-server device_info: ROCm0 = AMD Radeon 8060S).
* tests: cover the WSL ROCDXG drop-in + system-HIP-ordering fixes
- _wsl_system_rocm_lib_dirs: no-op without /dev/dxg, on bare-metal Linux,
and on WSL without librocdxg; returns the system lib dir on a ROCDXG WSL
host.
- binary_env: prepends the system ROCm lib dir ahead of the bundle and sets
HSA_ENABLE_DXG_DETECTION on WSL; unchanged on bare-metal Linux.
- install.sh: _persist_rocm_wsl_dropin exists, is gated on librocdxg, and the
rocminfo-already-works early return calls it before returning.
- llama_cpp.py: the serve-time launcher prepends the WSL rocm dirs before the
bundle dir (mirrors binary_env).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten WSL ROCDXG fix comments (no logic change)
Condense the drop-in / system-HIP-ordering comments and docstrings added in
this PR. Verified comment-only via AST parse + py_compile + sh/bash -n, the
308-test rocm_support suite, and a dash functional re-run of the bootstrap
(drop-in still persisted, env still set).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio/responses): forward chat_template_kwargs enable_thinking to chat request
The /v1/responses translation in _build_chat_request dropped
chat_template_kwargs (e.g. {"enable_thinking": true}) sent via the
Responses extra-body, so reasoning control was silently ignored.
Lift enable_thinking onto the typed ChatCompletionRequest field,
mirroring openai_chat_completions, so both the non-streaming and
streaming Responses pass-through paths honor it.
Fixes#6198
Signed-off-by: Tai An <antai12232931@outlook.com>
* Fix/adjust Responses reasoning for PR #6202
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust reasoning none for PR #6202
* Fix/adjust structured reasoning for PR #6202
* Fix/adjust responses reasoning review findings for PR #6202
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust responses reasoning follow-ups for PR #6202
* Fix/adjust think parsing gate for PR #6202
---------
Signed-off-by: Tai An <antai12232931@outlook.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: Add Tensor-Parallel llama.cpp support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden Tensor-Parallel fallback and GPU selection
* Studio: reconcile split-mode extras and harden tensor-split planning
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reconcile split-mode extras in backend duplicate-load guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: preserve inherited non-tensor split modes on reload
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor cancellation in tensor fallback, preserve tensor mode on rollback, and don't raise an explicit small context
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reconcile split-mode in reload check and strip it on tensor downgrade
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Strip --tensor-split alongside --split-mode so inherited ratios don't override the tensor planner
An inherited or stale --tensor-split in llama_extra_args was appended after
Studio's computed --tensor-split and won last in llama.cpp, re-introducing the
asymmetric-GPU OOM tensor mode is meant to prevent. Group -ts/--tensor-split
into the split-mode shadow set so it is stripped on inherit and on the layer
fallback; parse_split_mode_override still keys on the mode value only.
* Drop quantized KV for the tensor attempt and report native max context
Tensor mode aborts on a quantized KV cache, so a user with q8_0/q4_1 etc. who
enabled Tensor Parallelism silently fell back to layer split. Clear the cache
type (and strip inherited/explicit --cache-type) for the tensor attempt only;
the layer fallback re-runs with tensor off and keeps the user's choice.
Also report max_available_ctx from the native context, not an explicit small
-c, so the context slider no longer warns too early in tensor mode.
* Reconcile inherited split-mode extras in the already-loaded check
When a same-model load omitted llama_extra_args, the tensor comparison resolved
the raw (None) request and treated an inherited --split-mode tensor server as a
mismatch, forcing a needless reload. Compare using the stored extras stripped
the same way the reload strips them.
* Pass tensor_parallel through compare-mode loads
The generalized compare path loaded each GGUF without tensor_parallel, so
compare ran layer split even with the toggle on and left the settings sheet
stale. Send the toggle and hydrate the loaded state from the response, matching
the main chat and recipe load paths.
* Add --tensor-parallel flag to unsloth studio run
The headless one-liner could only reach tensor mode by passing --split-mode
tensor as a raw llama.cpp extra. Add a first-class --tensor-parallel/
--no-tensor-parallel option that sets the tensor_parallel field on the
/api/inference/load payload, forwarded through the studio-venv re-exec like the
other polarity flags. Matches the web UI toggle and the API field.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* fix: allow absolute save_directory in export paths to prevent cross-drive copy failures
The GGUF export pipeline (and all other export flows) forced every
save_directory through resolve_export_dir(), which always resolved
the path under exports_root() — typically ~/.unsloth/studio/exports/
on the system drive (C: on Windows).
When a user selected an output directory on a different drive (E:):
1. The absolute path was rejected at the Pydantic validator level.
2. Even if it got through, resolve_export_dir would re-resolve it
under C:\Users\.unsloth\studio\exports\.
3. After GGUF conversion completed on E:, the relocation step would
try to move/copy the finished files to C:, causing:
- WinError 17 (cross-drive move failure when shutil.move falls
through to a cross-filesystem copy)
- WinError 112 (disk full on C:)
Fix both layers:
- _validate_save_directory: accept absolute paths (they represent an
explicit user choice of output location).
- resolve_export_dir, resolve_output_dir, resolve_tensorboard_dir:
return absolute paths as-is instead of forcing them under the
default root. Keep the existing safety checks (null bytes, '..'
segments) and fall through to resolve_under_root for relative paths.
Fixes: https://github.com/unslothai/unsloth/issues/6082
* refactor: centralize user path validation into _resolve_user_path helper
Addresses code review feedback: the null-byte, '..', and absolute-path
checks were duplicated across resolve_output_dir, resolve_export_dir,
and resolve_tensorboard_dir. Extract a single _resolve_user_path helper
that all three delegate to.
No behavioral change — pure consolidation.
* fix: address code review — contain destructive cleanup and scope absolute paths
Address all review feedback from gemini-code-assist:
1. P1: destructive subdirectory cleanup (export_gguf)
The flattening loop in export_gguf previously rmtree'd every
subdirectory under abs_save_dir. When targeting an existing user
directory on a different drive (#6082), this could nuke unrelated
subdirectories. Now snapshot existing subdirectories before the
export and only clean up dirs created during this run.
2. P2: keep scan/read endpoints contained
Only resolve_export_dir accepts absolute paths (export is a write
path where user picks location). Reverted resolve_output_dir and
resolve_tensorboard_dir to use resolve_under_root directly — these
are used by scan/read/training endpoints that must stay contained
under their respective roots.
3. Centralization feedback
Removed the _resolve_user_path helper since it's no longer needed
with the narrowed scope. resolve_export_dir has the absolute path
logic inline with a clear docstring.
* fix: skip pre-existing subdirs in GGUF flatten loop and clean stale export intermediates
Two issues caught in code review (chatgpt-codex-connector):
1. The flattening loop moved ALL .gguf files from ALL subdirectories
into abs_save_dir, including pre-existing unrelated user subdirs.
Now skip pre-existing subdirs entirely unless they are known
export-owned intermediates (model/, model_gguf/).
2. After a failed export, known export-owned subdirectories (model/,
model_gguf/) were snapshotted as pre-existing on retry and never
cleaned up. These are now always cleaned up regardless, since they
are known intermediates created by the export pipeline.
* fix: separate write vs read export paths, guard same-dir rmtree
Three issues caught in code review (chatgpt-codex-connector):
1. P1: scan endpoint containment
resolve_export_dir was changed to accept absolute paths, but it's
also used by scan/read endpoints (routes/models.py) that must stay
contained under exports_root(). Split into:
- resolve_export_dir: contained, used by scans
- resolve_export_write_dir: accepts absolute paths, used by export
backend only
2. P1: same-directory rmtree
When a non-PEFT checkpoint's gguf_dir resolves to the same path as
abs_save_dir (user selected the checkpoint's gguf output as their
export directory), shutil.rmtree(gguf_dir) would delete the user's
chosen output directory. Now skip relocation when both paths resolve
to the same location.
3. P1: pre-existing subdir flatten loop
Reverted _EXPORT_OWNED_SUBDIRS logic — 'model/' and 'model_gguf/'
are common directory names in shared model folders and don't prove
export ownership. Now only clean up subdirs that didn't exist before
the export started.
* fix: remove dead _EXPORT_OWNED_SUBDIRS and fix _export_details for absolute paths
Two fixes from review comments:
1. Remove unused _EXPORT_OWNED_SUBDIRS declaration (leftover from
previous iteration that was intentionally removed).
2. _export_details now returns the full absolute path when the export
target is outside exports_root(), instead of truncating to basename.
Users who export to E:\ can now see the full destination path in
the success dialog.
* fix: use unique tmp dir for GGUF intermediates to avoid overwriting user dirs
When exporting to an absolute destination that already contains a
model/ subdirectory (e.g. a shared models folder), the hard-coded
model_save_path would overwrite files in that unrelated directory.
Use _tmp_model_<uuid> as the intermediate path instead, so user
directories are never touched. The tmp dir is created as a new subdir
of abs_save_dir and cleaned up by the flatten loop after GGUF files
are relocated.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix GGUF local export paths for PR #6088
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address GGUF export follow-ups for PR #6088
* Clean GGUF temp dirs on export failure for PR #6088
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust export path tests for PR #6088
* Fix/adjust export path review findings for PR #6088
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust home export path handling for PR #6088
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
The --local / GGUF-only install resolves its Python deps from
no-torch-runtime.txt, installed with --no-deps. That file was missing the
RAG group that studio.txt declares (sqlite-vec, pymupdf, python-docx), so a
fresh `unsloth studio` came up with RAG disabled: rag_db.py cannot import
sqlite_vec and logs "RAG unavailable: sqlite-vec extension could not be
loaded", and the knowledge-base routes return 503. python-docx was also
absent, so DOCX ingestion failed.
Add the three RAG store and document-parsing deps with the same pins as
studio.txt so knowledge bases work out of the box on the no-torch path.
sentence-transformers (dense embeddings) was already present.
The post-install path cleared only the in-memory freshness caches and then
re-primed the 24h disk cache with a forced GitHub refresh. When that refresh
cannot reach GitHub, latest_published_release falls back to the last-good disk
value, so a still-fresh same-base mix tag cached before the swap (b9596-mix-aaa
vs the just-installed b9596-mix-bbb) is replayed and the prebuilt reads as
behind, surfacing a false update banner that points back at the build that was
just replaced.
Give reset_caches a drop_disk option and use it on the update path: with the
disk cache gone, an offline post-install refresh leaves latest as None and the
banner fails open (off) instead of lingering on the stale same-base value. The
no-arg form stays in-memory only. Adds regression coverage for the drop, the
default no-op, and the fail-open vs stale-replay contrast.
* Studio: tune llama.cpp env for data-center GPUs
Detect datacenter/professional NVIDIA GPUs at llama-server launch and set
the llama.cpp env flags that help them, gated so consumer GeForce, AMD/ROCm,
CPU and macOS are never touched.
- GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F=1 for any DC GPU (FP32 cuBLAS
accumulation). On a B200 this is ~0% throughput cost with identical
perplexity (7.3230 wikitext-2-raw, baseline and on), where on GeForce the
same flag costs real throughput, hence the gate.
- GGML_CUDA_P2P=1 and CUDA_SCALE_LAUNCH_QUEUES=4x for multi-GPU DC boxes.
Benchmarked on 6x B200: +33-51% prompt processing on tensor (row) split and
+8-16% on the default pipeline (layer) split, with no regression on the
other split or on token generation.
Detection uses torch device names (A100/A30/H100/H200/H800/GH200/B200/GB200/
GB300/L40/L4/RTX PRO 6000/RTX 6000 Ada). A mixed box with one consumer GPU in
the selection is treated as non-DC. All writes are setdefault so a user value
always wins, and UNSLOTH_DISABLE_DC_TUNING=1 turns the whole thing off.
37 unit tests cover detection, multi-GPU gating, user-override precedence, the
disable flag and fail-open on error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix data-center GPU detection false positives and physical-id mapping
Two issues in the data-center llama.cpp env tuning gate:
- _is_datacenter_gpu matched the marker allowlist as unbounded substrings, so
workstation/laptop parts "NVIDIA RTX A1000" and "NVIDIA RTX A3000" matched
"a100"/"a30" and were wrongly tuned as data-center GPUs (forcing FP32 cuBLAS
accumulation and the multi-GPU env, which carry a real cost on those cards).
Switch to a word-boundary regex.
- gpu_indices carries physical GPU ids (translated from torch ordinals by
_get_gpu_free_memory via CUDA_VISIBLE_DEVICES), but they were passed straight
into torch.cuda.get_device_properties, which expects mask-relative ordinals.
On a masked host (e.g. CUDA_VISIBLE_DEVICES=4,5,6,7) a selection like [4,5]
fell out of range and silently dropped the tuning, and on a mixed mask it could
probe the wrong GPU class. Build a physical-id to device-name map mirroring
_get_gpu_free_memory, then look up the selection by physical id.
Add regression tests for the A1000/A3000 false positives and for masked-host
physical-id selection (reordered and mixed-class masks included).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten data-center GPU tuning comments
Comment-only pass over the DC tuning block and its tests: shorten verbose
docstrings/comments, drop ones that restate the code, collapse multi-line
blocks. Keep the load-bearing rationale (physical-id vs ordinal mapping, the
word-boundary reason, the B200 benchmark numbers). No code change: verified
with comment_tools.py check --strip-docstrings (code unchanged, comments only).
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix step count mismatch when sequence packing is enabled
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Emit a single step-0 progress event and guard applyStatus totalSteps
Merge the two consecutive _update_progress calls before train() so the
step-0 gate in _on_progress fires once instead of twice, avoiding a
duplicate startup event and a null-metric step-0 row in training_metrics.
Apply the same positive-number guard to applyStatus that applyProgress
uses, so a stale or startup status poll can no longer overwrite the
packed step count with 0 or replace it with a stale total.
* Log debug message when train_dataset length is unavailable
The TypeError fallback for length-less datasets (e.g. streaming
IterableDataset) was silent, leaving no trace that the step estimate
came from the raw dataset rather than the packed one.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
IOReport energy counters can reset (sleep/wake, power gating), making a poll
delta negative. Return None for a negative total so the monitor shows -- for
that poll instead of a bogus negative wattage; it self-corrects next poll.
* Studio: Add inline confirmation (Allow/Always allow/Deny) for tool calls
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix race in tool-call confirmation gate
* Studio: gate built-in tool calls and harden the confirmation handshake
The Allow / Always allow / Deny controls only lived in the fallback tool
card, but the built-in tools (web search, python, terminal, code
execution, image generation) render with their own components and so
never showed the buttons. Those calls paused after tool_start with no way
to approve them, hanging until the 1 hour timeout. Only MCP tools, which
use the fallback renderer, actually worked.
Render the controls for every tool card by wrapping each registered tool
component (and the fallback) in thread.tsx with a shared
ToolConfirmationControls, so the gate applies uniformly.
Also make the handshake robust:
- The gate keys on a per-call approval_id minted by the backend and
echoed in tool_start, instead of session_id alone, so a stale or
concurrent confirmation can no longer resolve the wrong call.
- The approval slot is registered before tool_start is yielded, closing
the race where a fast click or an auto "Always allow" could reach the
backend before the waiter existed.
- The frontend resolves with the same session id the request was sent
with (plus the approval_id), fixing the new-thread mismatch where the
confirmation targeted a different session than the blocked stream.
- The confirm endpoint returns {resolved}; the UI keeps the buttons and
shows a retry hint until the backend confirms a match, instead of
hiding them on a failed or mistargeted post.
- The gate runs after the disabled-tool and duplicate-call checks, so a
call that will not execute is not put up for approval. A denied call is
still excluded from duplicate detection, so re-issuing and approving it
works.
- "Always allow" is scoped per session to match the backend gate.
Add backend tests for the approval registry, the SSE no-deadlock
handshake, and the loop integration (allow, deny, disabled, duplicate,
re-issue after deny).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Move "Confirm tool calls" to the Tools section
* Studio: Keep tool group open while a tool call awaits confirmation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix tool confirmation session scope for PR #5869
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix confirmation follow-ups for PR #5869
* Apply pre-commit formatting for PR #5869
* Fix confirmation cleanup for PR #5869
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden confirmation lookups for PR #5869
* Studio: make the tool-call confirmation decision immutable
resolve_tool_decision accepted a second confirmation for the same approval_id
and overwrote slot["decision"] in the window before the waiter reads it and
pops the slot, so a duplicate or out-of-order POST could flip an Allow to Deny
(and returned a misleading resolved:true). Reject once the slot's event is
already set so the first decision wins. Adds a regression test.
* Fix/adjust tool confirmations for PR #5869
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: show Apple GPU temperature and power in the GPU monitor (macOS)
The GPU monitor on Apple Silicon always showed -- for Temperature and
Power: the MLX branch of get_gpu_utilization() hardcoded None because
ioreg's AGXAccelerator PerformanceStatistics carries neither metric.
Add utils/hardware/apple.py, mirroring macmon's no-sudo approach:
- Temperature: average of the AppleSMC "Tg*" float keys via the
AppleSMCKeysEndpoint user client (ctypes/IOKit, macOS 14+).
- Power: IOReport "Energy Model" group, "GPU Energy" channels; each
poll diffs the energy counter against the previous poll's sample, so
the value is the average wattage over the polling window. The first
poll only sets the baseline and returns None.
Both readers latch to None on first failure and never raise, so
non-Mac platforms and locked-down hosts keep the previous behavior.
* Sample IOReport with the subscribed channels descriptor for PR #6187
IOReportCreateSubscription writes the channel descriptor that later samples
must use; sampling with the original requested group can return no Energy
Model entries on hosts that normalize the channel set, leaving power_draw_w
null after the baseline. Use the subscribed descriptor (matching macmon) and
fall back to the requested channels if the OS leaves it unset.
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: danielhanchen <23090290+danielhanchen@users.noreply.github.com>
* fix(studio): inherit llama_extra_args and honor --no-mmproj
Reloading the same GGUF from the UI without gguf_variant no longer drops
CLI pass-through args like --no-mmproj. Skip mmproj download and launch
when --no-mmproj is present in llama_extra_args.
Co-authored-by: Cursor <cursoragent@cursor.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): tighten GGUF llama_extra_args variant inheritance guard
Reject inherited CLI args when the request changes gguf_variant or when
omitted variant resolves differently from the stored extra_args source.
Co-authored-by: Cursor <cursoragent@cursor.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Treat --no-mmproj-auto and --mmproj-auto with last-wins parsing for PR #5902
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* fix: deduplicate lemonade ROCm prebuilt selection log
resolve_lemonade_rocm_choice() is called twice per install (direct
planner + resolve_upstream_asset_choice). The API fetch is already
memoised via _fetch_lemonade_release_cached but the selection log
lines were still emitted on both calls, printing the 'trying
lemonade-sdk ROCm prebuilt' banner and hash-manifest NOTE twice.
Add _lemonade_selection_logged set keyed on (gfx_target, asset_name)
and guard the two log() calls behind a membership check so they print
exactly once per process regardless of call count.
Also extend the _clear_lemonade_release_cache test fixture to clear
the new set between tests to prevent cross-test state bleed.
Fixes#6020
* fix: write log() output to stdout to avoid PowerShell NativeCommandError
On Windows, PowerShell treats any stderr output from a native process as
an error record and prefixes it with 'python.exe :' and sets the
ErrorId to NativeCommandError. Since log() wrote to sys.stderr, every
[llama-prebuilt] status line triggered this, making normal progress
output look like errors in the installer console.
Switch log() to sys.stdout. The download progress bar (DownloadProgress)
retains its stderr/tty logic unchanged -- that path is for interactive
terminal rendering, not status logging.
* fix: remove redundant 'or ""' in lemonade log_key
host.rocm_gfx_target is already guaranteed truthy by the early
return at the top of resolve_lemonade_rocm_choice. The fallback
was dead code.
* Keep resolver stdout machine-readable, route install logs to stdout
log() sending everything to stdout breaks the resolver modes: setup.sh
json.load()s the whole stdout, so one helper log line (network retry,
release-tag scan) corrupts the parse and silently drops back to building
"latest". Default log() to stderr and flip to stdout only on the install
path, where PowerShell otherwise renders stderr as NativeCommandError
noise. Also tighten the lemonade dedup comments.
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
On a 0.0.0.0 bind whose public ip:port is not reachable (cloud firewall),
the banner still printed "Secure link access via Cloudflare: <url>" right
after "is NOT reachable from the public internet", which reads as if the
tunnel might also be blocked. The Cloudflare quick-tunnel works regardless.
Thread the reachability probe result through a module-level _public_reachable
tri-state and, when the public probe definitively failed but the tunnel is
up, print "Also, the secure link access via Cloudflare works: <url>".
Reachable or undecided cases keep the existing wording.
* Studio: llama.cpp update banner redesign, About tab license info, inline system prompt editing, naming cleanup
- Redesign the llama.cpp update banner to match the chat composer surface
(borderless rounded card, composer shadow, Hellix Medium title), rename
actions to Update and add a 15 minute Remind me later snooze
- Keep the banner up until the user explicitly acts on it; drop the
outside click dismissal
- Add a Settings > General > Notifications toggle to disable the banner
for training-only setups (on by default)
- Rename the Help settings tab to About and add a License section
(Unsloth Studio AGPL-3.0, Unsloth Core Apache-2.0) linking to the
license files in this repo
- Make the run settings system prompt box an inline editable textarea;
the popup editor opens when the prompt overflows the box
- Pointer cursor on the preset dropdown chevron
- Dark mode toasts use the chat composer surface color
- Replace standalone Studio with Unsloth in user facing strings; keep
Unsloth Studio, LM Studio, Fine-tuning Studio, Recipe Studio and CLI
commands unchanged
* Studio: open the system prompt popup on box click, balance banner padding
- The system prompt box opens the Edit System Prompt dialog on click,
matching the pencil action
- Slightly more bottom padding on the llama.cpp update banner so the
spacing reads even next to the action pills
* Studio: replace unsloth studio update with the installer commands in update guidance
- The unsloth studio update command no longer works, so the About tab
update section now shows the one-line installer (curl or irm) for
PyPI and unknown installs, and git pull plus the local installer for
checkouts
- Add a short note that unsloth studio update is no longer supported
- Link the Installation, Updating and Windows install docs pages
- The package update banner now copies the platform installer command
instead of unsloth studio update
* Studio: rounder account menu, inline system prompt box with popup from the label
- Account menu corners go from 14px to 18px via a specific override,
since list menus pin border-radius globally
- llama.cpp banner bottom padding 22px
- System prompt is an inline editable textarea again; clicking the
System Prompt label opens the popup editor, and an overflowing
prompt opens it on box click
* Studio: show the standard install commands in the About update section
- Both one-line install commands (MacOS/Linux/WSL and Windows
PowerShell) are always shown, labeled like the docs, since running
them again updates an existing install
- Drop the unsloth studio update deprecation note
- Add the Mac install guide to the docs links
* Studio: clearer platform toggle and layout in the About update section
- Section heading is Update
- Platform picker is a pair of pill buttons, MacOS / Linux and Windows,
and only the selected platform's install command is shown
- Intro reads: To install or update Unsloth
- Local update heading separates checkout guidance from the standard
install command
* Studio: report GitHub branch instead of dev for source checkouts
A source checkout not on an exact release tag now shows
GitHub <branch> (e.g. GitHub main) as the Studio version in About.
Detached or unusual HEADs still fall back to dev.
* Studio: tighten the About update section copy and toggle styling
- Platform toggle buttons are borderless pills
- Shorter local update wording and restart note
- Docs links read Mac and Windows
* Studio: tighten line spacing in the sidebar account button
* Studio: fix vanishing compact MCP icon on hover, single line pill tooltips
- Compact caret pills (MCP, RAG) keep their icon on hover for inactive
pills too; the off switch hover rules hid the icon while compact mode
hid the X, leaving an empty slot
- Compact icon tooltips and single line compact tooltips render as full
pills; wrapped tooltips keep the 9px corners. TooltipContent measures
line count in a ref callback since Radix mounts portal content
without re-rendering the wrapper
- 1px gap between the name and Unsloth lines in the sidebar account
button
* Studio: Projects hover plus button, align recents with the label
- Hovering the Projects nav item reveals a plus button that opens the
New project dialog, with the same circular hover treatment as the
chat row actions
- Recent chat titles start at the same x as the Recents label
- The system prompt overflow lock only engages for a non-empty prompt
with a laid-out box, so a mis-measure cannot turn clicks into the
popup
* Clip system prompt overflow inside the rounded box
Wrap the inline system prompt textarea in a rounded overflow-hidden
surface so scrolled text and the scrollbar stay inside the box. The
focus ring moves to the wrapper via focus-within.
* Add updating progress bar to llama banner and shorten settings copy
While an update is applying, the banner action row becomes an
indeterminate progress bar that keeps animating under reduced motion,
matching the other loading indicators. Settings descriptions across
General, Profile, Appearance, Chat, Connections, API, and About are
trimmed without losing meaning.
* Address review: desktop update note, server platform detection, zh-CN keys
The About tab no longer shows terminal install commands in the desktop
app, where the bundled backend updates through the built-in updater;
it shows a short note and the docs links instead.
fetchDeviceType now sends the auth token to /api/health, which only
reports the server platform to authed callers, and caches only a
server-reported value. Copied install commands then match the host
platform rather than the browser when they differ (WSL, SSH).
zh-CN gains translations for the new notification and license keys,
the renamed About tab title, and the desktop update note.
* Real download progress for llama.cpp updates, prompt and sidebar polish
The update worker now streams the installer output and parses its
download percent lines into job progress, exposed via the update-status
API. The installer emits finer non-tty milestones when
UNSLOTH_PROGRESS_PERCENT_STEP is set; the worker requests 5 percent
steps. The banner renders a determinate bar from the reported fraction
and falls back to the sweep until the first percent arrives.
Also removes the focus ring on the inline system prompt box and
slightly shrinks the Projects hover plus icon.
* fix(rocm): stop overwriting ROCR_VISIBLE_DEVICES in apply_gpu_ids
ROCR_VISIBLE_DEVICES uses HSA agent-level indexing, not physical GPU
indices. Setting it to a bare integer breaks multi-GPU ROCm systems
where the parent already set ROCR_VISIBLE_DEVICES=0,1: narrowing to
1 causes torch.cuda.is_available() to return False in the training
worker, producing a misleading 'no HIP accelerator' error even on a
correctly configured ROCm host.
HIP_VISIBLE_DEVICES is sufficient for GPU selection on ROCm.
Leave ROCR_VISIBLE_DEVICES inherited from the parent environment.
* test(rocm): update apply_gpu_ids test to assert ROCR_VISIBLE_DEVICES is not overwritten
• fix: handle empty responses tool output
Normalize empty Responses `function_call_output.output` values before converting them into Chat Completions `role="tool"` messages. Empty strings, whitespace-only strings, and empty arrays now use the existing no-output sentinel, while non-empty text and content arrays are preserved.
Add regression coverage for empty tool outputs, image payloads outside `output`, content-array serialization, validator round trips, and preserving non-empty text.
---------
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: Tai An <antai12232931@outlook.com>
Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
When pinning GPUs for the llama-server child, the ROCm path set both
HIP_VISIBLE_DEVICES and ROCR_VISIBLE_DEVICES to the same physical
indices. These masks filter at different layers and stack:
ROCR_VISIBLE_DEVICES reduces the visible set at the HSA/ROCr layer and
re-indexes from 0, then HIP_VISIBLE_DEVICES indexes into that reduced
set. _select_gpus ranks by free VRAM and picks the most-free card, so a
single non-zero pin (e.g. "1") becomes out of range at the HIP layer,
HIP enumerates 0 devices, and the model silently runs on CPU
("ggml_cuda_init: failed to initialize ROCm: no ROCm-capable device is
detected").
Set only HIP_VISIBLE_DEVICES (which narrows correctly on its own) and
clear any inherited ROCR mask so it can't double up.
Verified on a 2x Radeon AI PRO R9700 (gfx1201) host, ROCm 7.1.1: the
same selected=[1] load that fell back to CPU (~7.7 tok/s) now runs on
the GPU (~78 tok/s).
Fixes#6175
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* studio: import MCP servers from a config file
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* import config' on the add-server form
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: defensively handle MCP config imports
* fix: address MCP import review follow-ups
* fix: preserve apostrophes in Windows MCP commands
* fix: preserve apostrophe-wrapped Windows MCP args
* fix: align Windows MCP parsing with list2cmdline
* fix: preserve explicit MCP remote transport intent
* fix: trim MCP remote URLs before transport checks
---------
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* fix(studio): surface live step with null loss through the SSE progress stream
The metric histories skip non-finite steps, so during a NaN stretch the
SSE live loop and final complete event replayed the last finite
step/loss pair. Follow the live progress step when it is ahead of the
history tail and report its loss honestly (null until recovery).
Completes the NaN honesty fix for the SSE consumer flagged in review.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Apply live-step handling to inactive streams and clear the UI loss on null for PR #6206
Fresh /progress connections after a finished run took the inactive branch
which still replayed the last finite step and loss pair; apply the same
live-step correction there. On the frontend, applyProgress kept the stale
currentLoss when a payload advanced the step with a null loss; clear it so
the display shows -- until the loss recovers. Widen the runtime state type
to number | null, which the view layer already handles.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving
A community report showed OpenCode failing tool calls every few minutes
against Studio's OpenAI-compatible API while the same GGUF was stable on
LM Studio. Root cause: Studio advertises the requested context length, but
llama-server can allocate less (memory-fit step on small GPUs, --parallel
slot split), so clients budget against a window that does not exist. Their
generations truncate mid tool call at the real wall (finish_reason=length
with cut JSON arguments) and eventually the prompt itself exceeds the real
window, returning a 400 that agentic clients treat as non-retryable.
Changes:
- After llama-server health, read default_generation_settings.n_ctx from
/props and adopt it whenever it is below Studio's computed context, with
a warning. The load response, status route, UI value, and the passthrough
max_tokens ceiling all become honest automatically.
- Expose context_length and max_context_length on /v1/models so clients can
budget against the enforced window.
- Accept empty role=tool content (commands with no output are routine in
agentic loops; OpenAI and llama-server both accept it) instead of a 400.
- Add context_overflow=truncate_middle (per request, or server-wide via
UNSLOTH_CONTEXT_OVERFLOW=truncate_middle): on exceed_context_size_error
the passthrough drops whole middle turn-groups (system prompt, first turn,
and recent turns kept; tool calls stay paired with their results), clips
oversized contents middle-out when group-dropping is not enough, clamps
max_tokens to the generation headroom, and retries. Default stays 'error'
with code=context_length_exceeded so clients running their own compaction
keep full control.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: allocate the requested context for real (kv-unified, fit-ctx floor)
Two launch-flag gaps caused the advertised vs allocated divergence at the
source:
- llama-server enables --kv-unified only when the slot count is auto; Studio
always passes --parallel N, which silently splits -c into per-slot windows
of -c/N. Pass --kv-unified when N > 1 so a single request can use the full
advertised window (same total KV memory, shared pool).
- with --fit on the fit step may set ctx as low as 4096; pass
--fit-ctx <requested> for explicit requests so fit offloads or fails into
the existing --fit off retry instead of silently shrinking the window.
Both flags are gated on --help capability probing so older builds keep the
current behavior, where the /props readback remains the backstop. Verified
live: -c 98304 --parallel 4 now serves per-slot n_ctx 98304 (was 24576),
48k-token requests pass through the passthrough, and the readback warning no
longer fires.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Studio's frontend exposes a Resume action and submits requests with
resume_from_checkpoint set to a previous run's output_dir. The CUDA
training paths in worker.py read this field from config and pass it to
trainer.train() (see lines 2729-2787 and 3108-3229). The MLX path
_run_mlx_training did neither: it never read config['resume_from_checkpoint']
and called trainer.train() with no args. The MLX trainer also did not
accept the kwarg, so even threading it through would have been a no-op.
With this PR + the unsloth-zoo companion PR adding the trainer-side
support (saves optimizer_state + trainer_state, accepts and applies
resume_from_checkpoint in MLXTrainer.train()), MLX Resume now works
end-to-end. Verified on M2 16GB with Qwen3-0.6B + unsloth/LaTeX_OCR:
loss at every post-resume step matches a fresh run bit for bit
(2.168627977371216 == 2.168627977371216 at step 6, etc).
Two lines: read the field near the other config.get() extractions in
_run_mlx_training, pass it as a kwarg at the trainer.train() call site.
Companion PR: unslothai/unsloth-zoo#751
When training produced a NaN or Inf loss event, the handler filtered the
value to None but never updated progress.loss — clients kept seeing the
last finite value as if everything were fine.
Now: on non-finite loss, clear progress.loss to None and log a one-shot
warning. Training continues (no phase=error, no _should_stop), matching
the expected behavior for a non-fatal numerical event.
Test: tests/test_training_nan_loss_handling.py with 6 cases covering
finite, NaN, +/-Inf, idempotency of the one-shot warning, and recovery
when a finite step follows a non-finite one.
* Studio: auto Cloudflare tunnel for 0.0.0.0 launches
Binding Studio to 0.0.0.0 for remote access often leaves the raw
http://<ip>:<port> URL unreachable (https-vs-http, blocked high ports,
closed cloud security groups). On a wildcard bind, auto-start a free
cloudflared quick tunnel and show its https://*.trycloudflare.com URL in
the startup banner:
Secure link access via Cloudflare: https://<random>.trycloudflare.com
- new studio/backend/cloudflare_tunnel.py: find or download+cache the
cloudflared binary (per-OS/arch GitHub release, safe .tgz extract),
start the tunnel, parse the URL, tear it down. Stdlib only; best-effort
and non-fatal throughout (a missing binary or offline box never blocks
or slows startup).
- run_server starts the tunnel for 0.0.0.0 only (skips loopback, api-only
and Colab), prints the line in the banner, and _graceful_shutdown stops
the child so it never orphans.
- --cloudflare/--no-cloudflare flag (default on) on `unsloth studio` and
`unsloth studio run`, forwarded through the re-exec into run_server.
- tests for the helper, the CLI flag forwarding, and the run.py defaults.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio cloudflare: send a User-Agent on the cloudflared download
GitHub's CDN can 403 the default Python-urllib User-Agent on release asset
downloads. Set an explicit UA and pin it with a test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio cloudflare: address review (opt-out for subcommands, tunnel teardown)
- reject --no-cloudflare placed before a subcommand (it would not reach the
subcommand), mirroring the --parallel guard
- register the tunnel before waiting for its URL so a shutdown during the wait
stops cloudflared instead of orphaning it
- tear the server + children down if `unsloth studio run` startup aborts
(health timeout, model-load error, Ctrl+C) before the wait loop
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Require a found ROCm DLL before forcing BNB_ROCM_VERSION in Studio paths
main.py previously set BNB_ROCM_VERSION=72 whenever HIP_PATH or ROCM_PATH
was set, and the training worker fell back to a blind 72 when DLL
detection found nothing. On a Windows machine with the AMD HIP SDK
installed but CUDA or CPU torch, that forces a ROCm backend onto a
non-ROCm bitsandbytes wheel, which raises at import. Both paths now only
write the override when a libbitsandbytes_rocm DLL actually exists (or a
seeded value is already present), matching the strict gates in
unsloth/import_fixes.py.
Also removes four redundant local import shutil statements in
unsloth/save.py that shadow the module-level import, the same pattern
that caused the UnboundLocalError fixed in #6149.
* Worker: gate the BNB override on a found ROCm DLL, preserving seeded marker
Review follow-ups: track _found_rocm_bnb in the worker like main.py so a
ROCm DLL with an unparsable name still gets the seeded or 72 fallback,
and skip the env write entirely when no DLL exists so a seeded value
keeps its sitecustomize marker and stays redetectable by later import
fixes.
* Studio: surface the llama.cpp update affordance when MTP is disabled
When a model asks for MTP (auto on an MTP model, or forced mtp / mtp+ngram)
but it gets disabled, the load already degrades gracefully and serves without
speculative decoding. Until now the UI gave no hint why, or that an update
would fix it.
Record why MTP was dropped on the backend (spec_fallback_reason): the probe
found no mtp token (binary_no_mtp), the spawn aborted with an outdated-arch /
context-build error such as a prebuilt that predates the Gemma drafter
(binary_outdated), or the current build could not run it, e.g. a CUDA kernel
limit (runtime_error). Expose it in the inference status. In the chat
Speculative Decoding section, show a short note and, for the two update-fixable
reasons, an inline Update llama.cpp button that reuses the existing update flow.
A runtime_error gets the note without an update push, since a newer build may
not fix it.
Backend tests cover the reason being set / cleared. Frontend typechecks.
* Address review: tighten the update hint to genuinely outdated binaries
Reserve binary_outdated (which surfaces the Update llama.cpp affordance) for an
unknown-architecture abort, which proves the prebuilt predates the model;
classify the generic memory/context build failures as runtime_error, where an
update may not help. Frontend: only append the "Update llama.cpp to enable it"
sentence when an update is actually available, so the text never points at an
action the UI is not offering.
The export route's load_checkpoint waits for the training subprocess to
exit by calling time.sleep(0.5) in a loop (up to 30s) inside an async
function. time.sleep blocks the whole event loop, so every other request
to the server stalls for that duration. Use await asyncio.sleep(0.5),
matching the async pattern already used elsewhere in this file
(asyncio.to_thread, await asyncio.sleep).
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* Studio: enable MTP for sub-3B Gemma separate-drafter GGUFs
The sub-3B auto-drop to ngram-mod was tuned for an embedded draft head
(Qwen), whose per-token cost regresses below 3B. Gemma ships the head as a
separate root mtp-*.gguf drafter, a tiny standalone model that is cheap
enough to win below 3B: B200 Q4_K_XL bench, draft-mtp n=2 vs spec-off,
gemma-4-E2B (2B) = 1.21x (accept ~0.65) while ngram-mod is 1.00x.
Exempt a separate drafter from the sub-3B gate everywhere the threshold is
applied: the resolver (_mtp_too_small), the auto-fit VRAM reserve, the
drafter auto-download decision, and the reload-skip mirror via a
has_separate_drafter flag on _auto_mode_drops_mtp. Embedded sub-3B heads
(Qwen) still drop to ngram-mod. A drafter the binary cannot build (older
prebuilt, or a CUDA kernel limit) still aborts the spawn and the load
retries once without speculative decoding.
Adds the full Qwen3.5 + Gemma-4 (regular and QAT) auto/off/forced resolver
matrix, plus explicit sub-3B exemption tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Always compare the separate drafter in the reload-skip mirror
The sub-3B wrapper around the drafter compare could skip it when the drafter
was deleted out from under a running sub-3B server (detected None, stored set),
leaving a stale launch. The resolved-path compare is cheap and already handles
every case, so drop the _auto_mode_drops_mtp guard (and its now-unused imports)
and always compare when the mode can use a drafter and the user does not own
--spec-type. Addresses review feedback on #6191.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: ignore unsupported env proxy during Studio startup
* fix: handle missing socksio env proxy at startup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Match printf logging style and inline the proxy predicate for PR #6102
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
