The same Gemma4ClippableLinear monkey-patch that exists in vision.py
for training is needed in loader.py for loading existing checkpoints
(used by export and inference).
Gemma4ClippableLinear wraps nn.Linear but does not subclass it, so
PEFT's LoRA injection fails with "Target module not supported".
The patch redirects PEFT to target the inner .linear child instead.
Applied only to the vision model PeftModel.from_pretrained path.
Temporary fix until PEFT adds native support (peft#3129).
* Studio: detect reasoning_effort and preserve_thinking in chat templates
Previously Studio's chat template sniffer only recognized Qwen's
enable_thinking and DeepSeek's thinking markers. For gpt-oss (Harmony
templates) and newer Qwen3.6 templates, the Think toggle was hidden or
could only be flipped on/off.
This change adds two new detections and corresponding UI controls:
1. reasoning_effort style (gpt-oss). When the chat template contains
reasoning_effort, the Think button becomes a Low / Medium / High
dropdown and the backend forwards {"reasoning_effort": <level>} in
chat_template_kwargs. Load-time --chat-template-kwargs flag is also
switched to the new style.
2. preserve_thinking kwarg (Qwen3.6). Independent of the reasoning
toggle. When the template mentions preserve_thinking, a new
Preserve Thinking on/off pill is shown next to Think. Off by
default, persisted via localStorage. When on, the backend adds
{"preserve_thinking": true} to chat_template_kwargs so past-turn
<think> blocks are kept in the prompt instead of being stripped.
Backend helper _request_reasoning_kwargs now merges all applicable
kwargs into a single chat_template_kwargs dict based on the model's
detected style and template capabilities. Inputs are validated with
Literal types in the Pydantic request model.
Tested end to end against cached GGUFs for unsloth/gpt-oss-20b-GGUF and
unsloth/Qwen3.6-35B-A3B-GGUF. Confirmed the llama-server startup
--chat-template-kwargs flag and per-request JSON body carry the
expected keys for all combinations.
* Studio: review pass and CI format fixes for reasoning-styles PR
Addresses review feedback and pre-commit CI:
- Preserve Thinking pill in shared-composer now gates on modelLoaded
only, matching the thread.tsx toggle. Previously the inline version
disabled whenever supports_reasoning was false.
- The non-GGUF already_loaded LoadResponse now emits reasoning_style
(and supports_preserve_thinking=False) so a reconnecting frontend
sees the correct style for an already-running gpt-oss safetensors
model.
- use-chat-model-runtime reconnect path now always clears
reasoningEnabled for models without reasoning support instead of
inheriting the previous model's state.
- _reasoning_default is now reset alongside the other reasoning flags
in both backend reset blocks.
- supports_reasoning description updated to mention reasoning_effort
alongside enable_thinking.
- Ran scripts/run_ruff_format.py on the touched Python files to
satisfy pre-commit.ci.
* Studio: detect reasoning flags on safetensors load + share Qwen param helper
Addresses bot review feedback:
- Extract the chat-template substring sniffer out of _read_gguf_metadata
into a module-level detect_reasoning_flags(template, model_id) helper.
Also runs on the safetensors / transformers load paths:
- POST /api/inference/load non-GGUF LoadResponse
- already_loaded non-GGUF early return
- GET /api/inference/status non-GGUF branch
The gpt-oss fallback via backend._is_gpt_oss_model() is preserved so
safetensors gpt-oss still surfaces reasoning controls even when no
chat_template is stored on the model record.
- Deduplicate the Qwen3 / Qwen3.5 / Qwen3.6 Think-toggle parameter
adjustment into a single features/chat/utils/qwen-params.ts. Both
the assistant-ui Think toggle (thread.tsx) and the shared composer
(shared-composer.tsx) now import the same helper. The superset that
applies presence_penalty=1.5 for Qwen3.5 and Qwen3.6 is now used by
both sites (thread.tsx previously did not apply it).
* Studio: fill missing reasoning flags on safetensors status + add always_on reset
Round 4 review fixes:
- routes/inference.py safetensors status response now populates
reasoning_always_on and supports_tools from detect_reasoning_flags.
Previously Pydantic defaulted both to False, so safetensors models
with always-on <think> templates or tool-calling templates were
silently losing those flags on /api/inference/status reconnect.
- routes/inference.py already_loaded safetensors branch now falls back
to backend._is_gpt_oss_model() when the chat template is missing,
matching the status-endpoint behaviour.
- Safetensors status endpoint log_source set to "Safetensors status"
so the emitted template-detection log lines are attributable.
- chat-runtime-store clearCheckpoint now also resets reasoningAlwaysOn
so switching from an always-on reasoning model to a non-always-on
one does not leave the Think button permanently locked on.
* Studio: skip reasoning kwargs when always-on; narrow non-GGUF advertisement
Round 5 addresses reviewer feedback:
- _request_reasoning_kwargs and the load-time --chat-template-kwargs
emission now skip when _reasoning_always_on is true. Templates with
hardcoded <think> tags do not consume enable_thinking / reasoning_effort
so sending them was noise.
- Non-GGUF (Unsloth / transformers) LoadResponse and InferenceStatusResponse
paths no longer advertise template-derived supports_reasoning /
reasoning_style / supports_preserve_thinking / supports_tools. The
transformers generation path does not yet forward chat_template_kwargs
to tokenizer.apply_chat_template, so exposing the UI controls on those
models was misleading. Only the gpt-oss Harmony case is kept
(reasoning_style = reasoning_effort) because it is handled via the
HarmonyTextStreamer at the tokenizer level. A follow-up PR can thread
chat_template_kwargs through the transformers path and re-enable the
broader detection.
- GGUF / llama-server paths keep the full detect_reasoning_flags output.
* add unsloth studio desktop app
* Fix review findings
- studio/src-tauri/tauri.conf.json: retarget updater to staging repo
(danielhanchen/unsloth-staging-2); switch to unslothai/unsloth on upstream merge.
- studio/src-tauri/linux/postremove.sh: drop the interactive read loop and the
/home/* iteration. Package maintainer scripts must stay non-interactive and
must not touch other users' data.
- studio/frontend/src/app/auth-guards.ts: honor tauriAutoAuth() boolean. Failed
auto-auth now redirects to /login; requireGuest/requirePasswordChangeFlow
only redirect to /chat when auth succeeds. The new early-return on failed
auth is intentional so the login / change-password flows remain reachable
when desktop auth is not yet established.
- studio/frontend/src/config/env.ts: keep fetched=false on health failure so
later calls retry instead of caching the client-side platform guess.
- studio/src-tauri/src/install.rs: pick the available system package manager
(apt-get, dnf, zypper, pacman); AppImage bundles run on non-Debian distros.
- studio/frontend/src/lib/open-link.ts + markdown-text/sources callers: return
boolean from openLink so callers only preventDefault on handled URLs; relative
hrefs now navigate natively.
- studio/frontend/src/features/settings/tabs/about-tab.tsx: fetch(apiUrl(...))
so the version request targets the backend port in desktop mode. The bare
/api/health predates the Tauri webview (blame: the earlier onboarding commit,
which ran with same-origin frontend/backend); in desktop mode the webview
origin is tauri://localhost so the bare path fails.
- install.ps1: gate the install_python_stack.py hotfix on a sentinel comment
instead of a content regex; append the sentinel after applying so reruns
are unambiguous.
- unsloth_cli/commands/studio.py _write_auth_secret: use the atomic mkstemp +
os.replace path on Windows too; chmod calls are wrapped in try/except OSError.
- studio/src-tauri/src/preflight.rs probe_existing_backends: fan out the health
probes concurrently; desktop-auth status still runs sequentially per candidate.
reqwest::Client is internally Arc-wrapped so the in-loop .clone() is a
refcount bump, not a deep clone; annotated inline.
- studio/src-tauri/src/preflight.rs run_cli_probe: wait() after kill() to reap
the child, matching probe_cli_capability.
- studio/src-tauri/src/process.rs + main.rs: add stop_backend_detached and use
it from the tray quit handler so the 5s graceful-wait does not block the
Tauri main loop. RunEvent::Exit keeps the synchronous safety-net call.
- studio/backend/main.py: drop the permissive localhost CORS regex in
api-only mode; the explicit allow_origins list is sufficient.
- .github/workflows/release-desktop.yml: drop max-parallel: 1 so platform
builds run in parallel, and lift releaseBody to an env var so the three
tauri-action invocations share one source of truth.
* Fix review findings (loop 2)
- studio/backend/auth/storage.py update_password: clear_desktop_secret()
alongside clear_bootstrap_password() so rotating the admin password
also revokes any previously provisioned .desktop_secret. Without this,
an old local desktop credential keeps minting fresh admin tokens via
/api/auth/desktop-login after a password rotation.
- studio/src-tauri/src/desktop_auth.rs provision_desktop_auth: wrap
cmd.output().await in tokio::time::timeout(30s). DESKTOP_AUTH_LOCK is
held across the whole desktop_auth flow, and previously a hanging
`unsloth studio provision-desktop-auth` subprocess would pin the lock
indefinitely and freeze every subsequent desktop_auth call.
* Add review tests
* Consolidate review tests
Merge review-added tests into the existing studio/backend/tests/test_desktop_auth.py
(the PR's authoritative desktop-auth test file). Drops three scaffolding files under
tests/python/ in favor of five focused tests next to the tests they extend:
- test_update_password_clears_desktop_secret (runtime)
- test_update_password_on_unknown_user_leaves_desktop_secret_intact (runtime)
- test_cli_provisioning_delegates_to_storage_create_desktop_secret (source-level)
- test_cli_connect_auth_db_reads_storage_db_path (source-level)
- test_desktop_auth_provision_has_bounded_timeout (Rust source-level)
* Revert auth-guards.ts Tauri branches to unconditional form
The review loop on PR 5144 introduced a regression: the isTauri branch of
requireAuth redirected to /login when tauriAutoAuth() returned false, and
requireGuest / requirePasswordChangeFlow silently fell through on the same
condition. The Tauri desktop app authenticates via a local auto-generated
secret; it must never surface /login or /change-password to the user. A
failed auto-auth should let the startup layer retry, not expose a password
form.
Restore the three Tauri branches to the author's original unconditional
form (requireAuth: return; requireGuest / requirePasswordChangeFlow: throw
redirect({to: '/chat'})). Keep the rest of the review fixes -- the
apiUrl() fetch wrapping, authRedirect helper, and fetchAuthStatus refactor
are all legitimate improvements and are preserved.
* Revert release-desktop.yml to author's version
The review loop's workflow-file tweaks (drop max-parallel: 1, lift releaseBody
to an env var) are cosmetic. OAuth tokens cannot push workflow-file changes,
and fine-grained PATs cannot honor maintainerCanModify on a third-party fork.
Reverting the workflow file to wasimysaid's version lets the push go through
without needing a classic PAT with both repo and workflow scopes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Daniel Han <unslothai@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
install.sh's normal-install branch passed the inherited parent-shell
environment to setup.sh without resetting STUDIO_LOCAL_INSTALL or
STUDIO_LOCAL_REPO. Consumers treat either as truthy:
studio/setup.sh:491 checks STUDIO_LOCAL_INSTALL != "1"
studio/install_python_stack.py:868 reads STUDIO_LOCAL_REPO, falsy
if empty
Net effect: if a user or developer shell has stale STUDIO_LOCAL_*
exports from a previous --local run, a subsequent 'normal' install or
desktop-managed install silently takes the local-dev path: version
checks are skipped and an editable overlay points at the stale repo.
Fix mirrors install.ps1:1082-1087 on Windows: set STUDIO_LOCAL_INSTALL=0
and STUDIO_LOCAL_REPO= explicitly in the env prefix for the non-local
branch so setup.sh sees a clean state regardless of parent exports.
Co-authored-by: Daniel Han <unslothai@gmail.com>
* fix: patch CONTROL type for special tokens in sentencepiece GGUF export (fixes#5070)
When converting a Gemma 3 fine-tune to GGUF via save_pretrained_gguf,
tokens like <start_of_turn> (id=105) and <end_of_turn> (id=106) are
already present in the sentencepiece model but are typed as NORMAL (1)
instead of CONTROL (3). llama.cpp only recognises CONTROL tokens when
parse_special=True is active, so these tokens get BPE-split during
chat inference and the model produces garbage output.
fix_sentencepiece_gguf now reads tokenizer.json's added_tokens list and,
for any token with "special": true whose ID falls within the existing
sentencepiece vocabulary, updates its type from NORMAL to CONTROL before
writing the patched tokenizer.model to disk. The same CONTROL type is
also applied when new tokens are appended for the out-of-range case, so
both code paths are consistent.
* Wire fix_sentencepiece_gguf into tokenizer save path and guard np.diff
- save.py: call fix_sentencepiece_gguf inside unsloth_tokenizer_save_pretrained
after _preserve_sentencepiece_tokenizer_assets. The helper was previously
unreferenced in the repo, so the PR's CONTROL-type patch never actually ran
during save_pretrained_gguf.
- tokenizer_utils.py: add an early-return guard for len(added_tokens_ids) < 2
before the existing np.diff contiguity check. np.diff on a single-element
array returns [] and .min() raises ValueError, which would discard the new
in-vocab CONTROL patch; the guard flushes tokenizer.model first. Guard is
inserted before the existing lines (diff = np.diff(...) and the min/max
check) so their blame is unchanged.
Dropped the separate refactor to fold the four duplicated "if patched > 0:
write tokenizer.model" blocks into a helper because doing so re-indents
lines whose blame is "Formatting & bug fixes"; the duplication
remains the author's pattern.
* Fix review findings: negative token_id guard and np.diff single-element
- tokenizer_utils.py:481: add 0 <= lower bound to the special_token_ids
bounds check. Previously a negative token_id from tokenizer.json passed
'token_id < sentence_piece_size' and Python's negative indexing wrapped
tokenizer_file.pieces[-1] to silently corrupt the last piece to CONTROL.
- tokenizer_utils.py:513: replace the loop-1 'if len < 2: return' guard
(which was too broad: it silently skipped vocab extension for single-entry
added_tokens.json) with a pre-pass that substitutes a trivially-contiguous
2-element sentinel for the contiguity check, then restores the original
array before the append loop. Lines 519 ('diff = np.diff(added_tokens_ids)')
and 520-529 (min/max/boundary checks and early-return write blocks) are
left literally unchanged so blame remains intact.
* Restore real added_tokens_ids before min boundary check
Move the '_real_added_tokens_ids' restore above the
'added_tokens_ids.min() != sentence_piece_size' check. With the previous
order the sentinel [sentence_piece_size, sentence_piece_size + 1] was
still in scope when the min check ran, so any single-entry added_tokens
.json with an out-of-range start id (e.g. 99 when sentence_piece_size=2)
bypassed the boundary check and fell through to the append loop.
* Scope fix_sentencepiece_gguf to GGUF export path only
Previously wired fix_sentencepiece_gguf into unsloth_tokenizer_save_pretrained,
which is the generic monkey-patch replacement for every tokenizer.save_pretrained
call. That caused the GGUF-specific mutation (and the unconditional protobuf
import in fix_sentencepiece_gguf) to run on every LoRA / merged 16-bit /
push_to_hub / torchao save, where it has no purpose and can abort the entire
save if the protobuf runtime is unavailable.
- save.py: remove fix_sentencepiece_gguf call from unsloth_tokenizer_save_pretrained.
- save.py: add the call inside unsloth_save_pretrained_gguf immediately before
save_to_gguf, wrapped in try/except so a protobuf import failure logs a
warning and lets GGUF conversion proceed rather than aborting the save.
* Broaden special-token retag to USER_DEFINED and narrow save.py except
- tokenizer_utils.py:483: the in-vocab retag previously only promoted NORMAL
pieces to CONTROL, but the real Gemma tokenizer (e.g. unsloth/functiongemma
-270m-it) stores <start_of_turn>/<end_of_turn> as USER_DEFINED (type 4).
Extend the predicate to cover both NORMAL and USER_DEFINED so tokens marked
"special": true in tokenizer.json are promoted regardless of their current
sentencepiece type. Only tokens explicitly flagged special are touched, so
non-special USER_DEFINED pieces are unchanged; already-CONTROL pieces stay
unchanged. The warning message is generalised accordingly.
- save.py:2294: narrow the except clause from Exception to ImportError. The
loop-3 try/except was added to tolerate a missing protobuf runtime; leaving
it broad also swallows OSError/PermissionError mid-write, which would ship
a corrupted tokenizer.model to save_to_gguf. ImportError still covers the
protobuf case while letting I/O errors propagate to the outer save handler.
* Harden fix_sentencepiece_gguf: widen except, protobuf fallback, revert USER_DEFINED widen, guard entry id
- save.py:2294: widen except from ImportError back to Exception. The loop-4
narrowing let JSONDecodeError / KeyError / OSError / PermissionError from
fix_sentencepiece_gguf abort the entire GGUF export, a regression vs
pre-PR behavior. The outer save_to_gguf try/except still covers GGUF-side
failures; any fix-side failure now logs a typed warning and lets
conversion proceed.
- tokenizer_utils.py:445: the direct 'from transformers.utils import
sentencepiece_model_pb2' raises TypeError ("Descriptors cannot be created
directly") on modern protobuf runtimes. Prepend a sys.modules.setdefault
pre-population using transformers.convert_slow_tokenizer.import_protobuf()
so the subsequent from-import finds a compatible module via the module
cache. The original import line is left verbatim at its place as the
final resolver.
- tokenizer_utils.py:483: revert loop-4 widening; retag only NORMAL pieces
to CONTROL. Retagging USER_DEFINED pieces caused a concrete tokenization
regression where an intentionally-USER_DEFINED in-vocab special token had
its sentencepiece encoding broken ('<user> hello' changed from [11, 3, 8]
to [11, 0, 12, 21, 0, 8]). The PR's stated scope is the NORMAL->CONTROL
Gemma case; USER_DEFINED handling is deferred.
- tokenizer_utils.py:475: defensive guard around entry["id"]. A malformed
added_tokens entry missing the "id" field or with a non-int id is now
skipped rather than raising KeyError / inserting garbage.
* Add review tests for sentencepiece GGUF fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: octo-patch <octo-patch@github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
`TestVisionCacheOnException::test_exception_result_cached` currently
patches `load_model_config` with `side_effect=OSError("network down")`
and asserts `assert_called_once()`. That assertion is impossible by
design: `_is_vision_model_uncached` in
`studio/backend/utils/models/model_config.py` intentionally returns
`None` for `OSError` so `is_vision_model` does not cache the fallback
and retries on the next call. The module docstring on
`_vision_detection_cache` itself spells this out:
Only definitive results (True/False from successful detection) are
cached; transient failures (network errors, timeouts) are NOT
cached so they can be retried.
The test has been failing identically on every downstream review run
against `unslothai/unsloth` main (e.g. `unsloth#5115`, `unsloth#5080`),
but the failure is not introduced by any of those PRs and does not
gate correctness.
Fix the collision by splitting the class into the two contracts the
code actually implements:
1. `test_permanent_exception_result_cached` keeps the original
intent ("exception falls back to False and that False is cached")
but uses `ValueError`, which is one of the exception types
`_is_vision_model_uncached` treats as permanent and caches. No
`huggingface_hub` import needed.
2. `test_transient_exception_not_cached` pins the opposite contract
with the original `OSError("network down")`: the call returns
False but the second invocation re-runs detection
(`call_count == 2`). This guards against a future regression
where somebody caches transient failures and then users with a
flaky network permanently see wrong detection for a model.
Both tests use `assert ... is False` on the public API and mock-count
assertions on `load_model_config`; no private helpers are touched.
* update gema4 chat templates
* udpate template
* update template for gemma4
* Add gemma4 chat template tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [WIP] Fast inference for qwen3.5
* fix tokenizer not saving properly
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* extend to VLM and clenaup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gate tokenizer.model saving
* fix for gated/private models
* Fix tokenizer save review findings
- save.py:261 restore dict-based _TOKENIZER_MODEL_CACHE so negative
results are cached; the set() in 0129fb5e regressed non-SentencePiece
tokenizer saves to a fresh HfApi.model_info call on every checkpoint.
Don't cache on exception so gated/private repos can retry later with a
valid token.
- save.py:282 guard `repo_info.siblings` with `or []`; huggingface_hub
types this Optional and returns None for empty or new repos, which
made any() raise TypeError out of save_pretrained.
- save.py:3487 split push_to_hub into local save + _preserve + push so
uploaded tokenizer_config.json/tokenizer.model include the fix rather
than the unfixed copies written before the upload.
- save.py:3352 call patch_saving_functions on tokenizers passed to
unsloth_save_pretrained_torchao to match the other three save
entrypoints; previously torchao saves skipped the preservation patch.
* Fix push_to_hub repo_id conflict and torchao token forwarding
- save.py:3493-3496 pop `repo_id` from kwargs (defaulting to
`save_directory`) before calling `self.push_to_hub(repo_id, **kwargs)`.
The previous `self.push_to_hub(save_directory, **kwargs)` passed
`save_directory` as the first positional `repo_id` while also
forwarding a user-supplied `repo_id` through kwargs, raising
`TypeError: got multiple values for argument 'repo_id'` on the
standard `save_pretrained(local_path, push_to_hub=True, repo_id=...)`
call shape. This regression was introduced by the earlier iteration
that split push_to_hub into an explicit second step.
- save.py:3314 forward `token=token` on the torchao non-PEFT
`tokenizer.save_pretrained(torchao_save_directory)` call so the
patched wrapper can reach gated repos when HF_TOKEN is not in the
environment. Left the sibling `unsloth_generic_save` call at 3063
untouched (blame points at an earlier full-finetuned
save_pretrained_merged fix and the token gap there is lower risk).
* Fix torchao tokenizer reload and push_to_hub repo_id default
- save.py:3283 after `auto_processor.from_pretrained(save_directory)`
re-runs `patch_saving_functions(tokenizer)` on the freshly loaded
tokenizer. The rebind at 3283 was overwriting the patched tokenizer
passed into `unsloth_save_pretrained_torchao`, so the subsequent
`tokenizer.push_to_hub` (3309) and `tokenizer.save_pretrained`
(3314) bypassed `_preserve_sentencepiece_tokenizer_assets` and left
`{save_directory}-torchao` without `tokenizer.model` / restored
`added_tokens_decoder`.
- save.py:3497 fall back to `os.path.basename(save_directory)` for
`repo_id` instead of the raw `save_directory`. The round-2 fallback
diverged from `transformers.PreTrainedTokenizerBase.save_pretrained`,
which defaults `repo_id = save_directory.split(os.path.sep)[-1]`;
nested local paths like `./out/my-repo` now resolve to `my-repo`
(the Hub id) instead of the full filesystem path.
* Revert tokenizer save_pretrained repo_id basename fallback
- save.py:3497 default `repo_id` back to `save_directory` as-is rather
than `os.path.basename(save_directory)`. The basename fallback (added
last iteration to match upstream transformers) stripped the user
namespace from the Unsloth convention `tokenizer.save_pretrained(
"user/repo", push_to_hub=True)`, redirecting the upload to
`{current_user}/repo`. save.py itself treats `save_directory` as the
repo id at 572, 593, 1723, 1779, 1836, 1844, 1858, and 3025, so the
wrapper should follow the same convention. Users who pass a nested
filesystem path with `push_to_hub=True` can supply explicit
`repo_id=...`.
* Guard processor.tokenizer recursion against None
save.py:3511 change `elif hasattr(model, "tokenizer")` to
`elif getattr(model, "tokenizer", None) is not None`. The previous
guard only checked attribute existence; a ProcessorMixin that sets
`tokenizer = None` (audio-only or manually constructed) would enter
the branch and crash inside the recursive patch_saving_functions on
`model.push_to_hub.__name__`.
* Add review tests for tokenizer save
* Consolidate review tests
Drop redundant assertion in test_patch_saving_functions_still_patches_non_none_tokenizer.
The hasattr check already proves the patch applied; the or-chained
repeat assertion added no signal.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Replace assistant-ui's shared-store autoscroll with independent per-pane state
- assistant-ui's useThreadViewportAutoScroll uses shared state that causes random scrolling issues on the compare page
- Replaced with independent per-pane autoscroll implementation
* Fix scroll-click scrolling for slow movements; polish sidebar animations and UI
- Fix middle-click scroll not responding to very slow movements
- Add animations to sidebar navigation icons; font adjustments
- Align app menu styling for consistency
- Fix rare lag on sidebar expand/collapse
- Smooth right sidebar (parameter tuning) animations and expand/collapse lag
* Address PR feedback: clean up unused code and polish sidebar
- Remove unused _SuggestionItem component and its dependencies (SuggestionPrimitive import, SUGGESTION_TOOLS, toolIconMap)
- Log delete-message errors to console before showing toast for better observability
- Further sidebar menu design adjustments
* Address PR feedback: auto-follow guard and NavItem cleanup
- Prevent auto-follow from breaking on short threads: gate wheel/touch detach on scrollTop > 0 so no-op upward gestures on non-scrollable viewports don't flip userDetachedRef
- Simplify NavItem: collapse identical-branch ternary and drop unused variant prop
* Address PR feedback: model selector consistency and nested scroll fix
- Adjust model selector to be consistent with the sidebar
- Fix auto-follow breaking when user scrolls inside nested scrollable regions (reasoning/tool panels): gate wheel/touch detach on innerScrollWillConsumeUpward(target) so bubbled events from inner scrollers don't flip viewport intent (Codex review)
* Filter layout-induced upward scroll deltas from auto-follow detach
Accumulator now counts distanceFromBottom growth instead of raw -scrollTop delta, so browser scroll-anchoring on auto-collapsing panels (reasoning, tool outputs) no longer falsely detaches auto-follow mid-stream.
---------
Co-authored-by: sneakr <hauzin@hotmail.com>
transformers >= 4.48's `_is_package_available(name)` returns a tuple
`(bool, version_or_None)`. TRL's `trl.import_utils` caches that tuple
directly in `_vllm_ascend_available`, `_llm_blender_available`,
`_deepspeed_available`, `_joblib_available`, etc. and the matching
`is_*_available()` accessors return the tuple unchanged. A non-empty
tuple is always truthy, so `if is_vllm_ascend_available():` (in
`trl/extras/vllm_client.py`) fires unconditionally and triggers
`from vllm_ascend.distributed.device_communicators.pyhccl import ...`,
which fails outside Huawei Ascend hosts and blocks
`from trl import GRPOConfig, GRPOTrainer`. The same shape blocks
`is_llm_blender_available()` -> `import llm_blender` in
`trl/trainer/judges.py`.
Add `fix_trl_vllm_ascend()` to `import_fixes.py` and call it from
`unsloth/__init__.py` before any `from .trainer import *` that would
eagerly `import trl`. The fix walks `trl.import_utils` once and coerces
every `_*_available` tuple to a bool; the existing accessors that just
return the cached value then naturally yield a bool and the `if`
checks behave.
* Studio: support images on /v1/messages (Anthropic-compat)
Translate Anthropic `image` content blocks (base64 and url sources) to
OpenAI `image_url` multimodal parts so the Anthropic endpoint reaches
llama-server's native vision path. Mirrors the `/v1/chat/completions`
vision behavior: 400 when the active GGUF isn't a vision model, and
embedded images are re-encoded to PNG (stb_image format coverage).
Server-side agentic loop is disabled when images are present, matching
the existing `not image_b64` gate on /v1/chat/completions.
Adds translator + normalizer unit tests.
* Studio: address gemini-code-assist review on /v1/messages image support
- Preserve interleaving of Anthropic text + image content blocks in the
translator (previously flattened all text first, then all images).
- Let _normalize_anthropic_openai_images return has_image so the route
skips the second scan it was doing.
- Use module-level base64/io in the helper instead of re-importing.
* Studio: forward standard OpenAI tools / tool_choice on /v1/responses
Mirrors the /v1/chat/completions client-side tool pass-through from #5099
so clients (OpenAI Codex CLI, OpenAI Python SDK, ...) that target the
Responses API receive structured function_call output items instead of
plain text with tool-call tokens leaking into content.
- ResponsesRequest: type tools/tool_choice properly, add parallel_tool_calls;
accept function_call and function_call_output input items for multi-turn
- Translate flat Responses tool / tool_choice shape to the nested Chat
Completions shape before forwarding to llama-server
- _normalise_responses_input: map function_call_output -> role="tool",
function_call -> assistant tool_calls (preserving call_id)
- Non-streaming: map returned tool_calls -> top-level function_call
output items keyed by call_id
- Streaming: emit response.output_item.added (function_call),
response.function_call_arguments.delta/.done, and response.output_item.done
per tool call while keeping the text message at output_index 0
- Pytest coverage: tools/tool_choice translation, multi-turn input mapping,
non-streaming tool_calls mapping, response round-trip
* Studio: merge system messages and close inner stream on /v1/responses
Fixes two issues surfacing when OpenAI Codex CLI drives /v1/responses
against a GGUF with a strict chat template (gpt-oss harmony, Qwen3, ...).
1. "System message must be at the beginning" upstream errors
Codex sends `instructions` AND a `role:"developer"` message in `input`,
producing two separate system-role messages. Strict templates raise
when a second system message exists or when one appears after a user
turn. _normalise_responses_input now hoists all instructions / system /
developer content into a single merged system message at the top of
the Chat Completions message list.
2. "async generator ignored GeneratorExit" / "Attempted to exit cancel
scope in a different task"
_responses_stream consumed the inner chat-completions body_iterator
without an explicit aclose() in a finally block. On client disconnect
(Codex frequently cancels mid-stream), Python 3.13 finalized the inner
async generator on a different task, tripping anyio's cancel-scope
check. Mirrored the same try/finally + aclose pattern used by the
/v1/messages, /v1/chat/completions, and /v1/completions passthroughs.
Tests: hoisting of instructions + developer, developer mid-conversation,
multiple system messages in input, no-system passthrough.
* Studio: accept Codex multi-turn shapes and fix cross-task stream close on /v1/responses
Two issues observed driving /v1/responses from OpenAI Codex CLI against a
GGUF backend.
1. 422 on every turn after the first
Codex replays prior assistant turns with
`content:[{"type":"output_text","text":...,"annotations":[],"logprobs":[]}]`
and carries forward `reasoning` items (o-series / gpt-5) between turns.
Our `ResponsesContentPart` union only accepted input_text / input_image,
and `ResponsesInputItem` only message / function_call / function_call_output,
so Pydantic failed the whole list and FastAPI returned
`"Input should be a valid string"` against the `str` branch of the
outer union.
- Add `ResponsesOutputTextPart` for assistant-replay content.
- Add `ResponsesUnknownContentPart` and `ResponsesUnknownInputItem`
as permissive catch-alls (drop during normalisation).
- Wire an explicit `Discriminator` so dispatch is deterministic and
the fallthrough reaches the catch-all instead of misreporting via
the outer `Union[str, list[...]]`.
- `_normalise_responses_input` now accepts output_text parts, flattens
single-part assistant text to a plain string (keeps legacy chat
templates happy), and silently drops reasoning / unknown items.
2. "async generator ignored GeneratorExit" / cross-task cancel scope
`_responses_stream` awaited `openai_chat_completions` in the parent
route-handler task, which opens the httpx client for the inner
passthrough on *that* task. The outer `StreamingResponse` then iterates
in a child task, so the asyncgen GC finalises the inner httpcore byte
stream on the child task, tripping anyio's "Attempted to exit cancel
scope in a different task". Move the `await` inside `event_generator`
so the httpx lifecycle stays within the single streaming child task,
and surface any HTTPException as a `response.failed` SSE frame.
Tests: assistant output_text replay, reasoning-item tolerance, unknown
content-part tolerance, end-to-end Codex-shape payload (developer + user +
reasoning + function_call + function_call_output + assistant output_text +
user), and single-part assistant flattening to plain string.
* Studio: call llama-server directly from streaming /v1/responses
The previous fix (running the inner await inside event_generator) was not
enough. Wrapping the existing `openai_chat_completions` pass-through still
stacks two async generators: when the outer generator is closed, the
innermost `HTTP11ConnectionByteStream.__aiter__` in httpcore doesn't
receive GeneratorExit before Python's asyncgen GC finalises it in a
sibling task, tripping "Attempted to exit cancel scope in a different
task" and "async generator ignored GeneratorExit" — the same Python 3.13
+ httpcore 1.0.x interaction already seen in PRs #4956, #4981, #5099.
Cure both pass-throughs had: a single same-task httpx lifecycle with
explicit `aiter_lines().aclose()` BEFORE `resp.aclose()` / `client.aclose()`
in the generator's finally block.
Apply it at the Responses layer by dropping the wrapper entirely for GGUF:
open httpx, consume `resp.aiter_lines()`, parse `chat.completion.chunk`,
emit Responses SSE events, close everything in finally — all in the
single StreamingResponse child task. Non-GGUF streaming is rejected with
a 400 (wrapping the transformers backend would re-introduce the
double-layer pattern and isn't a Codex-compatible path today anyway).
Also surfaces upstream httpx.RequestError / non-200 as a
`response.failed` SSE frame rather than a dropped stream now that the
request is dispatched after SSE headers have gone out.
* Studio: silence benign httpcore asyncgen GC warnings on Python 3.13
The streaming pass-throughs (/v1/chat/completions, /v1/messages,
/v1/responses, /v1/completions) all use the proven #4981 / #5099 pattern
— single-task httpx lifecycle with explicit aiter_lines().aclose() ahead
of resp.aclose() / client.aclose() in the generator's finally block.
That handles our own iterators correctly.
The residual noise ("async generator ignored GeneratorExit" /
"Attempted to exit cancel scope in a different task") comes from an
innermost HTTP11ConnectionByteStream.__aiter__ that httpcore creates
internally inside its pool. We hold no reference to it, so we cannot
aclose it ourselves. Python 3.13's asyncgen GC hook finalises it on the
finaliser task, its aclose path enters an anyio CancelScope shield, and
Python flags the cross-task exit. The response has already been
delivered with a 200 by then — it is purely log noise, not a functional
failure. Same interaction seen in modelcontextprotocol/python-sdk #831,
agno #3556, chainlit #2361, langchain-mcp-adapters #254.
Install a targeted sys.unraisablehook that swallows this specific tuple
— RuntimeError mentioning "cancel scope" or "GeneratorExit" plus an
object repr referencing HTTP11ConnectionByteStream — and defers to the
default hook for every other unraisable. Idempotent; guarded by a
sentinel attribute so repeated imports don't stack filters.
* Chatbox, scroll, and menu fixes
- Fixed chatbox auto-expand height for multi-line text on the compare page
- Fixed chatbox UI to be consistent across compare and new chat
- Fixed scrolling being enabled on pages with no content, which also triggered the scroll-to-bottom button
- Fixed scroll-to-bottom button to only appear after scrolling up a reasonable amount instead of instantly
- Added shutdown studio button to the menu for easier access
- Fixed pop-up menu width to match the user button width
(cherry picked from commit cd4e390dfa84fe311fae79a781b96cc0ef5970a9)
* fix: correct compare scroll viewport and clean up chat composer UI polish
* Dark theme refactor and sidebar/chat UI refinements
- Complete refactoring of dark theme
- Replaced square rounded-corner user profile image with a circular bordered one
- Replaced user profile icon with 'U' initial and renamed label from 'Studio' to 'User'
- Chat bubbles now have a pointy top-right edge
- Sidebar menu tab line color selection is now consistent across all menus
- Tab-selection color animation now also applies to recent chats
- Removed 'Compare' menu autoselect when a compare chat conversation is selected
- Fixed UI consistency in Compare to match New Chat
- Removed sidebar animation and tab line, replaced with rounded selection for consistency
- Further adjustments to sidebar UI
- Further adjustments to compare chat UI
* Fixed sidebar collapse/expand for recent chats and recent runs not being clickable
* Chatbox, scroll, and menu fixes
- Fixed chatbox auto-expand height for multi-line text on the compare page
- Fixed chatbox UI to be consistent across compare and new chat
- Fixed scrolling being enabled on pages with no content, which also triggered the scroll-to-bottom button
- Fixed scroll-to-bottom button to only appear after scrolling up a reasonable amount instead of instantly
- Added shutdown studio button to the menu for easier access
- Fixed pop-up menu width to match the user button width
* Sidebar, fonts, and chat UI refinements
- Replaced logo PNG with real font text for 'unsloth' and 'BETA' label
- Added Hellix font and applied it across menus and UI elements
- Lighter scrollbar in the sidebar compared to other areas of the app
- Adjusted chat font and chat bubble styling
- Adjusted app menu design to stay consistent with the sidebar
- Adjusted text style for 'New Chat' and repositioned content/chatbox
- Adjusted model selector and top area UI
- Fixed footer text from 'LLM's' to 'LLMs'
- Fixed active selection border color incorrectly appearing on page refresh and during general navigation
- Logo now defaults to 'New Chat' when clicked
* Sidebar, model selector, and mobile UI fixes
- Further adjustments to sidebar UI and logo
- Changed right bar icon
- Model selector adjustments
- Collapsed sidebar now matches the content area background
- Adjusted Hellix font spacing across pages
- Fixed sidebar icon overlap on mobile screens
* Adjust sidebar icons
* Adjust sidebar icons
* Fixed compare chat UI and scrolling issues
* Fixed inference settings icon behavior and context info positioning
- Fixed top right inference settings icon to move into sidepanel during expand/collapse, matching left sidebar behavior
- Adjusted context information element positioning
* Fix: textarea overflow in system prompt editor
* Code block redesign, font, and chat bubble adjustments
- Redesigned code block colors and theme
- Changed code block font to Fira Code
- Fixed scrollbar disappearing when expanding/collapsing tool calls in chats
- Adjusted chat bubble background color
* Fix chat bubble background color in dark theme
* fix: restore textarea auto-sizing and scope prompt editor sizing
* fix: add explicit textarea field sizing for prompt editor overflow
* fix: generate chat nonce on click instead of render
* fix: respect training lock on logo navigation
* Refactor compare page dual chat scrolling behavior
* Revert "Refactor compare page dual chat scrolling behavior"
This reverts commit d056ec09f2.
---------
Co-authored-by: sneakr <hauzin@hotmail.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* export: update GGUF quant list and ordering
* gguf: add Q2_K_L quantize flags for output and embeddings
* export: add live console logs for LoRA export flow
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: stream q2_k_l quantize logs and include subprocess error details
* fix: route Q2_K_L preset to q2_k ftype with q8_0 output+embeddings
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Trashing a thread mid-stream used to delete the Dexie rows while the
model kept generating, because the sidebar has no access to the
@assistant-ui aui context. Expose per-thread cancelRun() through the
chat runtime store and call it from deleteChatItem so trash behaves
like Stop → Trash. Covers compare pairs by cancelling each paired
thread.
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* fix(studio): forward OpenAI tools/tool_choice to llama-server (#4999)
Studio's /v1/chat/completions silently stripped standard OpenAI `tools`
and `tool_choice` fields, so clients using standard function calling
(opencode, Claude Code, Cursor, Continue, ...) never got structured
tool_calls back. Adds a client-side pass-through path mirroring the
existing Anthropic /v1/messages flow: when `tools` is present without
Studio's `enable_tools` shorthand, the request is forwarded to
llama-server verbatim so the client sees native id, finish_reason
("tool_calls"), delta.tool_calls, and accurate usage tokens.
Also wires Anthropic tool_choice forwarding: /v1/messages previously
accepted tool_choice on the request model but silently dropped it with
a warning. Translate the four Anthropic shapes to OpenAI format and
forward them so agentic clients can actually enforce tool use.
- ChatCompletionRequest: add tools, tool_choice, stop; extra="allow"
- ChatMessage: accept role="tool", optional tool_call_id / tool_calls /
name; content is now optional (assistant with only tool_calls)
- routes/inference.py: _openai_passthrough_stream /
_openai_passthrough_non_streaming helpers, routing branch in
openai_chat_completions, vision+tools via content-parts injection
- _build_passthrough_payload: tool_choice parameter (default "auto")
- anthropic_compat: anthropic_tool_choice_to_openai() translator
- tests/test_openai_tool_passthrough.py: Pydantic + translator unit tests
- tests/test_studio_api.py: 5 new E2E tests (non-stream, stream,
multi-turn, OpenAI SDK, Anthropic tool_choice=any regression)
* fix(studio): surface httpx transport errors from OpenAI passthrough
When the managed llama-server subprocess crashes mid-request, the
async pass-through helpers in routes/inference.py used to return a
bare 500 (non-streaming) or an "An internal error occurred" SSE chunk
(streaming) because _friendly_error only recognized the sync path's
"Lost connection to llama-server" substring -- httpx transport
failures (ConnectError / ReadError / RemoteProtocolError /
ReadTimeout) stringify differently and fell through to the generic
case.
- _friendly_error: map any httpx.RequestError subclass to the same
"Lost connection to the model server" message the sync chat path
emits. Placed before the substring heuristics so the streaming path
automatically picks it up via its existing except Exception catch.
- _openai_passthrough_non_streaming: wrap the httpx.AsyncClient.post
in a try/except httpx.RequestError and re-raise as HTTPException
502 with the friendly detail.
- tests/test_openai_tool_passthrough.py: new TestFriendlyErrorHttpx
class pinning the mapping for ConnectError, ReadError,
RemoteProtocolError, ReadTimeout, and confirming non-httpx paths
(context-size heuristic, generic fallback) are unchanged.
* fix(studio): close aiter_bytes/aiter_lines explicitly in passthroughs
The httpcore asyncgen cleanup fix in 5cedd9a5 is incomplete on Python
3.13 + httpcore 1.0.x: it switched to manual client/response lifecycle
but still used anonymous `async for raw_line in resp.aiter_lines():`
patterns in all three streaming paths. Python's async for does NOT
auto-close the iterator on break/return, so the aiter_lines /
aiter_bytes async generator remains alive, reachable only from the
surrounding coroutine frame. Once `_stream()` returns the frame is
GC'd and the orphaned asyncgen is finalized on a LATER GC pass in a
DIFFERENT asyncio task, where httpcore's
HTTP11ConnectionByteStream.aclose() enters anyio.CancelScope.__exit__
with a mismatched task and prints "Exception ignored in: <async
generator>" / "async generator ignored GeneratorExit" / "Attempted
to exit cancel scope in a different task" to the server log.
User observed this on /v1/messages after successful (status 200)
requests, with the traceback pointing at HTTP11ConnectionByteStream
.__aiter__ / .aclose inside httpcore.
Fix: save resp.aiter_lines() / resp.aiter_bytes() as a variable and
explicitly `await iter.aclose()` in the finally block BEFORE
resp.aclose() / client.aclose(). This closes the asyncgen inside the
current task's event loop, so the internal httpcore byte stream is
cleaned up before Python's asyncgen GC hook has anything orphaned to
finalize. Each aclose is wrapped in try/except Exception so nested
anyio cleanup noise can't bubble out.
Applied to all three streaming passthrough paths:
- _anthropic_passthrough_stream (/v1/messages client-side tool path)
- _openai_passthrough_stream (/v1/chat/completions client-side tool
path, new in this PR)
- openai_completions (/v1/completions bytes proxy from PR #4956)
* fix(studio): default ChatCompletionRequest.stream to false per OpenAI spec
OpenAI's /v1/chat/completions spec defaults `stream` to false, so
clients that omit the field (naive curl, minimal integrations) expect
a single JSON response back. Studio was defaulting to true, silently
switching those clients into SSE and breaking any parser that didn't
also handle streaming. ResponsesRequest and AnthropicMessagesRequest
already default to false correctly; only ChatCompletionRequest was
wrong.
Studio's own frontend always sets `stream` explicitly on every
chat-adapter / chat-api / runtime-provider call site, so the flip has
no UI impact. SDK users (OpenAI Python/JS SDK, opencode, Claude Code,
Cursor, Continue) also always pass `stream` explicitly, so they're
unaffected. The only clients feeling the change are raw-curl users
who were relying on the wrong default -- those get the correct OpenAI
behavior now.
Added a regression test pinning the default so it can't silently
flip back.
* fix(studio): reject images in OpenAI tool passthrough for text-only GGUFs
The new tool passthrough branch runs before _extract_content_parts,
skipping the existing not is_vision guard. Requests combining tools
with an image on a text-only tool-capable GGUF were forwarded to
llama-server, producing opaque upstream errors instead of the
pre-existing clear 400. Restore the guard inline at the dispatch
point, checking both legacy image_base64 and inline image_url parts.
* fix(studio): require tool_call_id on role=tool chat messages
Enforce the OpenAI spec rule that role="tool" messages must carry a
tool_call_id. Without it, upstream backends cannot associate a tool
result with the assistant's prior tool_calls entry and the request
fails in non-obvious ways through the passthrough path. Reject at the
request boundary with a 422 instead.
* fix(studio): harden OpenAI tool passthrough validation and error surfacing
Three related fixes called out by the PR review:
1. Preserve upstream status codes in the streaming passthrough. The
httpx request is now dispatched before the StreamingResponse is
constructed. Non-200 upstream responses and httpx RequestError
transport failures raise HTTPException with the real status
instead of being buried inside a 200 SSE error frame, so OpenAI
SDK clients see APIError/BadRequestError/... as expected.
2. Require non-empty content on user/system/tool messages. Per the
OpenAI spec, content may only be omitted on assistant messages
that carry tool_calls; enforce that at the request boundary so
malformed messages never reach the passthrough path.
3. Role-constrain tool-call metadata. tool_calls is only valid on
role=assistant, tool_call_id and name only on role=tool. Without
this, a user/system message with tool_calls would flip the
passthrough branch on and be forwarded to llama-server, surfacing
as an opaque upstream error.
* fix(studio): normalize image mode and passthrough JSON verbatim
Two Gemini-code-assist review findings on PR #5099:
1. Unconditionally convert decoded images to RGB before PNG encoding.
The prior code only handled RGBA, letting CMYK/I/F images crash
at img.save(format="PNG") and surface as opaque 400s. Applied to
both the passthrough helper and the non-passthrough GGUF path
that originally carried this pattern, keeping the two sites in
sync.
2. Return the upstream JSON body as raw bytes via Response rather
than parse-then-re-serialize with JSONResponse. Matches the
passthrough helper's "verbatim" contract and drops a redundant
round-trip.
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* unsloth gemma4 support files
* some fixes
* Fixing cache.empty() calls (#4813)
* Fixing cache.empty() calls
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Manan Shah <mananshah@Manans-MacBook-Pro.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix/gemma4 mlx (#4816)
* Fixing cache.empty() calls
* fixing for mlx versions
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Manan Shah <mananshah@Manans-MacBook-Pro.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* removed bidirectional check for 31b (#4839)
Co-authored-by: Manan17 <shahmanan170602@gmail.coml>
* Add Gemma 4 26B MoE support (MLX) (#4844)
* removed bidirectional check for 31b
* Change gemma4_text for moe
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Manan Shah <mananshah@Manans-MacBook-Pro.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(gemma4): cast RoPE offset to int before mx.arange() (#4901)
* fix(gemma4): cast RoPE offset to int before mx.arange()
* fix(gemma4): use zero-based arange + offset to avoid CPU-GPU sync
* qwen3.6 patches for multi-turn chat
* qwen3.6 script
* removing unnecessary scripts
* displaying errors for not installed packages
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Manan Shah <mananshah@Manans-MacBook-Pro.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Manan17 <shahmanan170602@gmail.coml>
Co-authored-by: Théophile Lafargue <138336683+eauchs@users.noreply.github.com>
* Add Qwen3.6 inference defaults for Studio
Add qwen3.6 family entry to inference_defaults.json with the
recommended sampling parameters from Qwen's documentation:
temperature=0.7, top_p=0.8, top_k=20, min_p=0.0,
presence_penalty=1.5, repetition_penalty=1.0.
Without this, Qwen3.6 models fall through to the generic qwen3
pattern which uses different defaults (temperature=0.6,
top_p=0.95, no presence_penalty).
* Add Qwen3.6-35B-A3B-GGUF to default model lists
* Add Qwen3.5/3.6 presence_penalty to thinking toggle and small-model disable logic
- Thinking toggle (on-load + button click) now sets presencePenalty: 1.5 for
Qwen3.5 and Qwen3.6 models (both thinking-ON and thinking-OFF states)
- Small-model thinking-disable check (<9B defaults to no-thinking) extended
from Qwen3.5-only to also cover Qwen3.6, in all 3 locations:
frontend on-load, frontend refresh, backend llama_cpp.py
* fix: multi-GPU inference crash for bnb 4-bit/8-bit models
When load_in_4bit or load_in_8bit is used with device_map="sequential"
and max_memory constraints that place weights across multiple GPUs (or
entirely on a non-default GPU like cuda:1), the bitsandbytes loading
path in transformers never calls dispatch_model. No AlignDevicesHook is
installed, and the first forward/generate call crashes with:
RuntimeError: Expected all tensors to be on the same device
This adds _attach_bnb_multidevice_hooks() which is called after
from_pretrained returns. It infers a device map from actual parameter
placements and calls dispatch_model(force_hooks=True) to install the
missing hooks. The function is a complete no-op for the common
single-GPU cuda:0 case.
Call sites: FastBaseModel.from_pretrained (vision.py) and
FastLlamaModel.from_pretrained (llama.py).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: align with PR #5053 final review improvements
- Add hook call to the bnb quantized loading branch in llama.py (the
primary load_in_4bit path), not just the non-fast-inference fallback
- Expand bnb detection: also check model.is_loaded_in_4bit,
model.is_loaded_in_8bit, model.quantization_method
- Pass explicit main_device and skip_keys to dispatch_model
- Use logger.info instead of print for the success message
- Use kwargs.get("load_in_8bit", False) at llama.py call sites
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* auth: default to chat
* settings: relaunch onboarding
* onboarding: return to launch page
* studio: stop auto guided tour
* ui: soften global radius
* cleanup: rename onboarding exit prop
* fix onboarding redirect safety
* Show real Unsloth version in settings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat(studio): replace navbar navigation with collapsible sidebar
Add an app-wide sidebar with hover-expand and pin-to-dock behavior.
Navigation items (Studio, Recipes, Export, Chat) move from the center
pill navbar to the sidebar. Chat threads and recipes render as
collapsible sub-lists. Navbar simplified to logo + update + close.
- Extend SidebarProvider with pinned/hovered state model
- New AppSidebar with animated active indicator, sloth profile menu,
theme toggle, guided tour, back/forward navigation
- Chat page refactored to URL-driven view state via search params
- Extract reusable hooks for chat thread and recipe sidebar data
- Guard startViewTransition for browser compatibility
- Wrap chat deletions in Dexie transaction for data integrity
* feat(studio): move logo to sidebar and make navbar overlay
- Sidebar is now full-height with logo in SidebarHeader
- Collapsed sidebar shows sticker.png, expanded shows full logo
- Navbar is absolute-positioned overlay (no layout space)
- Main content extends to top, aligning with navbar controls
* feat(studio): full-height sidebar with recents, edge-to-edge nav buttons
- Sidebar outside max-w-7xl, pinned to left edge
- Remove sidebar rounding, menu buttons rounded-md
- Nav buttons flush to sidebar edges with no left rounding
- Replace collapsible recipes/chat with flat nav items
- Add Recents section with chat history (1 item when not on chat, full on chat)
- New Chat as first nav item with PencilEdit02Icon
- Cursor pointer on all sidebar buttons
- Navbar temporarily hidden for screenshots
* fix(studio): fix chat scroll, action bar hover, collapsible recents
- Fix sticky composer by removing `relative` override on viewport footer
- Action bar buttons only show on hover (autohide=always)
- Remove floating border/shadow from action bar
- Add scroll space above composer for last message actions
- Back/forward buttons use router history (stay in-app)
- Recents section collapsible with chevron on chat route
- Set html/body/#root height for proper h-full chain
* fix(studio): address review feedback, clean up unused code
- Unhide navbar (was left hidden from screenshot)
- Remove unused imports: SidebarMenuSub*, BubbleChatIcon, ColumnInsertIcon
- Remove unused vars: recipeItems, activeRecipeId, canCompare, recipesOpen
- Include compare query id in active sidebar selection
- Use store type for contextUsage instead of inline type
- Simplify noop in sidebar.tsx
- Remove empty className prop
* feat(studio): add mobile sidebar, recent runs section, and misc UX fixes
* feat(studio): scaffold settings feature module with dialog store
* feat(studio): add tri-state theme store for settings
* feat(chat): add clear-all-chats and export-chat-history utils
* feat(studio): add settings dialog shell with tab rail
* feat(studio): add appearance tab with theme and sidebar pin
* feat(studio): add settings general tab with hf token, auto-title, reset prefs
* feat(studio): add settings chat tab with export and clear
* feat(studio): add api keys tab with list and revoke flow
* feat(studio): add create-key form and reveal dialog
* feat(studio): add usage examples panel to api keys tab
* feat(studio): add settings about tab with update and shutdown
* feat(studio): add settings dropdown item and cmd-comma shortcut
* feat(studio): remove legacy api-keys route and chat-sheet preference rows
* fix(studio): settings dialog a11y + polish pass
* feat(studio): inline api key reveal card replacing nested dialog
* fix(studio): hide revoked keys from settings list
* refactor(studio): strip navbar and hoist training unload guard
* feat(studio): explicit sidebar toggle, remove hover-open and pin icons
* fix(studio): use SidebarRight01Icon for collapsed sidebar open toggle
* fix(studio): address code review findings for settings dialog
* feat(studio): collapsible navigate group with standalone new-chat and compare
* fix(studio): chat-only standalone actions, use ColumnInsertIcon for compare
* fix(studio): sidebar new-chat/compare state reset and icon-mode collapsible
* feat(studio): add compact logo assets for sidebar header
* Fixed sidebar design
* fix(studio): sidebar delete icon hover contrast and sizing
* feat(studio): route-gate sidebar recents (chats off /studio, runs on /studio)
* feat(studio): add chat search store
* feat(studio): add chat search index hook with snapshot-on-open
* feat(studio): add chat search command dialog with global shortcut
* feat(studio): wire chat search into sidebar
* fix(studio): trim hf token on save, add show/hide toggle, commit on close
* revert(studio): restore original sidebar/border colors, brighten sidebar
* feat(studio): forward overlayClassName through CommandDialog
* fix(studio): wrap search dialog in Command context, redesign as flat 635px card
* fix(studio): reserve right padding on recent items so delete icon stops overlapping title
* fix(studio): skip hf token unmount-commit during reset-prefs reload
* chore(studio): drop unused icon import and unreachable runs navigate branch
* fix(studio): chat search index filters archived before limit, batches message query, picks up reasoning text
* fix(studio): keep CommandEmpty in tree so empty state renders correctly
* fix(studio): cap system prompt and chat template textareas so they scroll instead of growing
* fix(studio): attach chat-compare tour anchor to sidebar compare button
* fix(studio): persist system theme explicitly so next-themes does not clobber on reload
* fix(studio): auto-switch to history tab when selecting a recent run from sidebar
* UI overhaul: chatbox, scrollbar, sidebar, and compare view
UI Changes:
- Redesigned the Compare UI with general cleanup
- Redesigned the Chatbox UI
- Reduced the width of the user chat bubble for improved readability
- Narrowed the user chat box across the content page
- Adjusted thinking-box text color to be slightly darker
- Removed faded text effect from chat messages
- Removed faded text effect from the thinking box
- Added a small LLM chat safety note at the bottom of the chatbox
- Restyled the scrollbar
Layout & Behavior:
- Reworked the scrollbar to span the full height of the page (no top/bottom padding) and remain persistently visible when content is scrollable, rather than only on hover
- Reworked the Configuration sidebar to span full height — removed rounded corners and borders, with the scrollbar adjusted to match the full top-to-bottom layout
- Adjusted the top menu and bottom chatbox content areas to work correctly with the new full-page scroll behavior
- Made chat content match the chatbox width, with content sliding slightly behind the chatbox when scrolling
- Aligned chat text width with the chatbox for visual consistency, including how far the text extends behind the chatbox
Fixes:
- Fixed the chatbox not auto-expanding when typing multi-line input while bottom-positioned during an active chat (previously only worked before a chat had started)
- Fixed positioning and design of the user chat hover menu buttons to match the assistant chat box — now displayed below the chat bubble instead of on the left side
* Fix user message layout in thread component
* swap code icon
* fix compare layout
* fix compare pane flex
* Sidebar improvements and fixes
- Added scrolling support to the sidebar so menus and recent chats no longer get hidden
- Recent chats are now always visible in the sidebar, not hidden when in Studio, Recipes, or Export
- Recent chat is now deselected when selecting other navigations
- Fixed sidebar glitch where browser resize could make the sidebar and expand button disappear completely
- Fixed glitch where the open-sidebar hover tooltip appeared above the logo when clicking expand sidebar
- Reduced sidebar width on mobile to around 2/3 of the screen (was too wide)
- Made the close-sidebar hover tooltip consistent with the rest of the design
- Removed sidebar collapse/expand animation
- Small adjustment to chat width
* Fix route scrolling, polling, and theme sync issues
* Fix Studio page scrolling
---------
Co-authored-by: sneakr <hauzin@hotmail.com>
* Studio: Ollama support, recommended folders, Custom Folders UX polish
Backend:
- Add _scan_ollama_dir that reads manifests/registry.ollama.ai/library/*
and creates .gguf symlinks under <ollama_dir>/.studio_links/ pointing
at the content-addressable blobs, so detect_gguf_model and llama-server
-m work unchanged for Ollama models
- Filter entries under .studio_links from the generic models/hf/lmstudio
scanners to avoid duplicate rows and leaked internal paths in the UI
- New GET /api/models/recommended-folders endpoint returning LM Studio
and Ollama model directories that currently exist on the machine
(OLLAMA_MODELS env var + standard paths, ~/.lmstudio/models, legacy
LM Studio cache), used by the Custom Folders quick-add chips
- detect_gguf_model now uses os.path.abspath instead of Path.resolve so
the readable symlink name is preserved as display_name (e.g.
qwen2.5-0.5b-Q4_K_M.gguf instead of sha256-abc...)
- llama-server failure with a path under .studio_links or .cache/ollama
surfaces a friendlier message ("Some Ollama models do not work with
llama.cpp. Try a different model, or use this model directly through
Ollama instead.") instead of the generic validation error
Frontend:
- ListLabel supports an optional leading icon and collapse toggle; used
for Downloaded (download icon), Custom Folders (folder icon), and
Recommended (star icon)
- Custom Folders header gets folder icon on the left, and +, search,
and chevron buttons on the right; chevron uses ml-auto so it aligns
with the Downloaded and Recommended chevrons
- New recommended folder chips render below the registered scan folders
when there are unregistered well-known paths; one click adds them as
a scan folder
- Custom folder rows that are direct .gguf files (Ollama symlinks) load
immediately via onSelect instead of opening the GGUF variant expander
(which is for repos containing multiple quants, not single files)
- When loading a direct .gguf file path, send max_seq_length = 0 so the
backend uses the model's native context instead of the 4096 chat
default (qwen2.5:0.5b now loads at 32768 instead of 4096)
- New listRecommendedFolders() helper on the chat API
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: log silent exceptions and support read-only Ollama dirs
Replace silent except blocks in _scan_ollama_dir and the
recommended-folders endpoint with narrower exception types plus debug
or warning logs, so failures are diagnosable without hiding signal.
Add _ollama_links_dir helper that falls back to a per-ollama-dir hashed
namespace under Studio's own cache (~/.unsloth/studio/cache/ollama_links)
when the Ollama models directory is read-only. Common for system installs
at /usr/share/ollama/.ollama/models and /var/lib/ollama/.ollama/models
where the Studio process has read but not write access. Previously the
scanner returned an empty list in that case and Ollama models would
silently not appear.
The fallback preserves the .gguf suffix on symlink names so
detect_gguf_model keeps recognising them. The prior "raw sha256 blob
path" fallback would have missed the suffix check and failed to load.
* Address review: detect mmproj next to symlink target for vision GGUFs
Codex P1 on model_config.py:1012: when detect_gguf_model returns the
symlink path (to preserve readable display names), detect_mmproj_file
searched the symlink's parent directory instead of the target's. For
vision GGUFs surfaced via Ollama's .studio_links/ -- where the weight
file is symlinked but any mmproj sidecar lives next to the real blob
-- mmproj was no longer detected, so the model was misclassified as
text-only and llama-server would start without --mmproj.
detect_mmproj_file now adds the resolved target's parent to the scan
order when path is a symlink. Direct (non-symlink) .gguf paths are
unchanged, so LM Studio and HF cache layouts keep working exactly as
before. Verified with a fake layout reproducing the bug plus a
regression check on a non-symlink LM Studio model.
* Address review: support all Ollama namespaces and vision projector layers
- Iterate over all directories under registry.ollama.ai/ instead of
hardcoding the "library" namespace. Custom namespaces like
"mradermacher/llama3" now get scanned and include the namespace
prefix in display names, model IDs, and symlink names to avoid
collisions.
- Create companion -mmproj.gguf symlinks for Ollama vision models
that have an "application/vnd.ollama.image.projector" layer, so
detect_mmproj_file can find the projector alongside the model.
- Extract symlink creation into _make_symlink helper to reduce
duplication between model and projector paths.
* Address review: move imports to top level and add scan limit
- Move hashlib and json imports to the top of the file (PEP 8).
- Remove inline `import json as _json` and `import hashlib` from
function bodies, use the top-level imports directly.
- Add `limit` parameter to `_scan_ollama_dir()` with early exit
when the threshold is reached.
- Pass `_MAX_MODELS_PER_FOLDER` into the scanner so it stops
traversing once enough models are found.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: Windows fallback, all registry hosts, collision safety
_make_link (formerly _make_symlink):
- Falls back to os.link() hardlink when symlink_to() fails (Windows
without Developer Mode), then to shutil.copy2 as last resort
- Uses atomic os.replace via tmp file to avoid race window where the
.gguf path is missing during rescan
Scanner now handles all Ollama registry layouts:
- Uses rglob over manifests/ instead of hardcoding registry.ollama.ai
- Discovers hf.co/org/repo:tag and any other host, not just library/
- Filenames include a stable sha1 hash of the manifest path to prevent
collisions between models that normalize to the same stem
Per-model subdirectories under .studio_links/:
- Each model's links live in their own hash-keyed subdirectory
- detect_mmproj_file only sees the projector for that specific model,
not siblings from other Ollama models
Friendly Ollama error detection:
- Now also matches ollama_links/ (the read-only fallback cache path)
and model_identifier starting with "ollama/"
Recommended folders:
- Added os.access(R_OK | X_OK) check so unreadable system directories
like /var/lib/ollama/.ollama/models are not advertised as chips
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: filter ollama_links from generic scanners
The generic scanners (models_dir, hf_cache, lmstudio) already filter
out .studio_links to avoid duplicate Ollama entries, but missed the
ollama_links fallback cache directory used for read-only Ollama
installs. Add it to the filter.
* Address review: idempotent link creation and path-component filter
_make_link:
- Skip recreation when a valid link/copy already exists (samefile or
matching size check). Prevents blocking the model-list API with
multi-GB copies on repeated scans.
- Use uuid4 instead of os.getpid() for tmp file names to avoid race
conditions from concurrent scans.
- Log cleanup errors instead of silently swallowing them.
Path filter:
- Use os.sep-bounded checks instead of bare substring match to avoid
false positives on paths like "my.studio_links.backup/model.gguf".
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: drop copy fallback, targeted glob, robust path filter
_make_link:
- Drop shutil.copy2 fallback -- copying multi-GB GGUFs inside a sync
API request would block the backend. Log a warning and skip the
model when both symlink and hardlink fail.
Scanner:
- Replace rglob("*") with targeted glob patterns (*/*/* and */*/*/*)
to avoid traversing unrelated subdirectories in large custom folders.
Path filter:
- Use Path.parts membership check instead of os.sep substring matching
for robustness across platforms.
Scan limit:
- Skip _scan_ollama_dir when _generic already fills the per-folder cap.
* Address review: sha256, top-level uuid import, Path.absolute()
- Switch hashlib.sha1 to hashlib.sha256 for path hashing consistency.
- Move uuid import to the top of the file instead of inside _make_link.
- Replace os.path.abspath with Path.absolute() in detect_gguf_model
to match the pathlib style used throughout the codebase.
* Address review: fix stale comments (sha1, rglob, copy fallback)
Update three docstrings/comments that still referenced the old
implementation after recent changes:
- sha1 comment now says "not a security boundary" (no hash name)
- "rglob" -> "targeted glob patterns"
- "file copies as a last resort" -> removed (copy fallback was dropped)
* Address review: fix stale links, support all manifest depths, scope error
_make_link:
- Drop size-based idempotency shortcut that kept stale links after
ollama pull updates a tag to a same-sized blob. Only samefile()
is used now -- if the link doesn't point at the exact same inode,
it gets replaced.
Scanner:
- Revert targeted glob back to rglob so deeper OCI-style repo names
(5+ path segments) are not silently skipped.
Ollama error:
- Only show "Some Ollama models do not work with llama.cpp" when the
server output contains GGUF compatibility hints (key not found,
unknown architecture, failed to load). Unrelated failures like
OOM or missing binaries now show the generic error instead of
being misdiagnosed.
---------
Co-authored-by: Daniel Han <info@unsloth.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Fix review findings for PR #49
1. Sandbox fallback Jinja env in _VariantTokenizerProxy.apply_chat_template
(use SandboxedEnvironment, matching _derive_assistant_prefix_by_render)
2. Unwrap benign outer-If guards in _template_ends_with_toplevel_for so
templates like {% if messages %}{% for ... %}{% endfor %}{% endif %}
are still repairable (preserves Qwen3-Guard rejection via else-branch
and add_generation_prompt-name checks)
3. Preserve raw name_or_path in _VariantTokenizerProxy._source_path so
local-path detection works for dict/list variant tokenizers
4. Context-aware strict-mode messages: omit "will still load" and
"Set UNSLOTH_STRICT_CHAT_TEMPLATE=1" when already raising
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Older installers persisted the venv Scripts directory directly in the
User PATH registry. The shim approach from #4961 no longer writes that
entry, but on upgrade the old one survived and python.exe / pip.exe
from the unsloth venv continued winning resolution in every new shell.
Before creating the shim, read the current User PATH, filter out any
entry matching $VenvDir\Scripts (using the same symmetric raw+expanded
comparison as Add-ToUserPath), and write back if changed. No-op on
fresh installs where the legacy entry was never written.
Confirmed on a real Windows machine: `where.exe python` was returning
the venv interpreter first even after the shim PR merged.
Older installers persisted the venv Scripts directory directly in the
User PATH registry. The shim approach (added in this PR) no longer writes
that entry, but it also did not remove the old one. On upgrade, the
legacy entry survived and python.exe / pip.exe from the unsloth venv
continued winning resolution in every new shell, which is exactly the
hijack the shim was designed to prevent.
Before creating the shim, read the current User PATH, filter out any
entry matching $VenvDir\Scripts (using the same symmetric raw+expanded
comparison as Add-ToUserPath), and write back if changed. This runs
once per install and is a no-op on fresh installs where the legacy
entry was never written.
* Restrict flash attn to <=256 head dim. Consolidate attn impl checks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Consolidate the changes into single function
* safeguard for dict instead of object
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Chat-template repair: warn-by-default, AST classification, dict support
Follow-up hardening on top of PR #4426 (which fixed the #4150
RuntimeError for ChatML LoRA reloads).
Behavior changes:
- Warn-by-default instead of RuntimeError. When fix_chat_template cannot
repair a broken template, emit a warning and return the original.
Set UNSLOTH_STRICT_CHAT_TEMPLATE=1 to restore the pre-warn hard fail.
Fixes the UX where a missing `{% if add_generation_prompt %}` block on
a saved LoRA (typical after LlamaFactory / Axolotl re-serialize) would
block model loading entirely.
- Local path vs HF hub distinguished in the warning message. For local
paths the message points at the likely downstream tool; for HF IDs it
points at the upstream model maintainers. Previously both said "file a
bug report to the maintainers of <path>" even when <path> was the
user's own saves/ directory.
- Dict / list chat_template now handled. Hermes-3 ships with
{default, tool_use} and the previous code crashed with
AttributeError: 'dict' object has no attribute 'find' when entering
_fix_chat_template with a dict. Each variant is now fixed
independently; structure is preserved.
Internals:
- _find_end_position now matches all four Jinja whitespace-control
variants ({% %}, {%- %}, {% -%}, {%- -%}) and returns the rightmost
endfor/endif so multi-for templates aren't locked onto the first loop.
Previously {%- endfor -%} (both-side dash, used by Qwen3-Guard) was
silently bypassed.
- _has_add_generation_prompt_block uses Jinja AST via
jinja2.nodes.If/Name walks instead of substring matching, so
templates that hide the block behind comments or dash-style variants
are classified correctly.
- _template_ends_with_toplevel_for gates the GH#4150 ChatML repair on
the AST: only fires when the last structural top-level node is a For
(standard ChatML shape), ignoring trailing pure-whitespace output
nodes. Templates wrapped in an outer If (Qwen3-Guard) are now
explicitly skipped at the _fix_chat_template level as well, not just
at load_correct_tokenizer's name-based exemption.
- _validate_patched_template renders the patched template with and
without add_generation_prompt and confirms the patched output
responds to the flag by appending (not replacing) content. If
validation fails, the patch is discarded and we fall through to the
warn path.
Verified with an expanded regression suite in tests/:
- test_fix_chat_template_pr4426.py: 42/42 template-matrix cells
- test_load_correct_tokenizer_pr4426.py: 5/5 tokenizer loads
- test_chat_template_followups.py: 10/10 new follow-up tests
- test_mistral_pr4426.py: 5 Mistral variants byte-identical
- test_qwen_pr4426.py: 14 Qwen variants byte-identical
(Qwen1.5, Qwen2, Qwen2.5-Instruct/Coder/Math/VL, Qwen3,
Qwen3-Coder, QwQ, Qwen3-Guard-Gen)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard _validate_patched_template against read-only chat_template
If tokenizer.chat_template is a property or otherwise read-only, the
validation helper would crash with AttributeError when trying to
temporarily set the patched template. Catch the assignment failure and
return False (skip validation), and best-effort restore in the finally
block.
* Replace regex separator inference with render-diff; broaden repair to non-ChatML templates
The previous `_infer_assistant_separator` was a four-tier regex heuristic that
only worked on ChatML-shaped templates and forced a hard `<|im_start|>` /
`<|im_end|>` presence gate on Case 2 repair. This meant a Llama-3, Gemma, or
Phi-3 template stripped of its generation-prompt block by a downstream tool
(LlamaFactory, Axolotl, etc.) would still warn-and-return even though the
structural shape is identical to the ChatML case the PR already handles.
This replaces the regex with `_derive_assistant_prefix_by_render`: render the
template with two dialogs that differ only in assistant content, then
`os.path.commonprefix` on the tails captures the exact assistant-turn prefix
the template emits. The template itself is ground truth, so non-ChatML shapes
work as long as the assistant block is a literal the template emits once per
message.
Three guards keep the derivation safe:
A. both assistant renders extend the base render (no reordering);
B. the divergence point is exactly the content-insertion site (sentinel
follows the common prefix);
C. a user-role cross-check: if a render with a user sentinel also emits
the same prefix, role has no effect on output and we reject. A render
failure on [user, user] (e.g. Gemma's `raise_exception` alternation
check) is evidence that role matters; we accept.
Sentinels differ at character 0 so `commonprefix` cannot absorb them, and
trailing whitespace/comments after the last `{% endfor %}` are stripped
before probing (they would appear in base but not after the appended
assistant turn and break Guard A).
`_fix_chat_template` and `_repair_string_template` now thread an
`is_sharegpt` kwarg; `_fix_chat_template` retries once with
`is_sharegpt=True` if the first probe returns None (dual-probe fallback
for dict/list callers).
The ChatML `<|im_start|>` / `<|im_end|>` hard gate in Case 2 is dropped.
`_infer_assistant_separator` is deleted.
Verified via:
- tests/test_fix_chat_template_pr4426.py: 51/51 cells (new Llama-3,
Gemma, Phi-3 broken-template rows all repair FIX-OK)
- tests/test_load_correct_tokenizer_pr4426.py: 5/5
- tests/test_chat_template_followups.py: 18/18 (T11-T18 cover
non-ChatML repair + probe failure modes)
- tests/test_mistral_pr4426.py: 5/5 byte-identical
- tests/test_qwen_pr4426.py: 14/14 byte-identical (Qwen3-Guard AST
gate still rejects)
- tests/hermes3_lora_pr4426.py reload: patched template ends with
`<|im_start|>assistant\n`, inference returns sensible output.
- temp/sim/battery.py: 79/79 followup; vs baseline: 0 regressions,
9 improvements.
- Spot-check probe on real stripped tokenizers (Hermes-3, Phi-4,
Llama-3.2-1B, Gemma-3-1B): all derive the expected prefix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address reviewer findings: variant routing, positive-gate detection, comment-safe end scan
Resolves three reviewer findings on PR #5049 (`fix/chat-template-followups`):
Finding #1 [10/10]: dict/list variants now route through
`_fix_chat_template_for_tokenizer` via a new `_VariantTokenizerProxy`
adapter. Previously the dict/list branches called `_fix_chat_template`
directly, silently bypassing the warn/strict (`UNSLOTH_STRICT_CHAT_TEMPLATE`)
contract, the `no == yes` diagnostic, broken-existing-block detection,
and `_validate_patched_template` guard. The proxy swaps
`base.chat_template` to the variant string before each
`apply_chat_template` call so tokenizer globals (`bos_token`, custom
filters, `raise_exception`) remain available; if the base is read-only
it falls back to isolated Jinja rendering.
Finding #2 [1/10]: `_has_add_generation_prompt_block` now requires the
`If` body to contain at least one `Output` node (a new
`_if_body_emits_content` helper walks descendants). This distinguishes a
real generation-prompt block from a header guard like
`{% if not add_generation_prompt is defined %}{% set ... %}{% endif %}`
(body contains only `Assign`) which references the name but emits
nothing. Also dropped a now-redundant `"add_generation_prompt" not in
scrubbed` guard in `_fix_chat_template` Case 2 so header-guarded
templates still get repaired.
Finding #4 [1/10]: `_find_end_position` now replaces Jinja comments with
equal-length whitespace before scanning for `{% endfor %}` / `{% endif %}`
tokens. This prevents a trailing comment containing those tokens from
being picked as the real end tag. Positions in the padded string map 1:1
to positions in the original template.
Tests:
- tests/test_chat_template_followups.py: 21/21 (T19 strict-mode
dict variant, T20 header-guard repair, T21 comment-endfor trap
added; T4/T5 stubs updated with a working apply_chat_template
that routes through Jinja).
- tests/test_fix_chat_template_pr4426.py: 51/51 cells unchanged.
- tests/test_load_correct_tokenizer_pr4426.py: 5/5.
- tests/test_mistral_pr4426.py: 5/5 byte-identical.
- tests/test_qwen_pr4426.py: 14/14 byte-identical.
- temp/sim/battery.py: 79/79 followup; 0 regressions vs baseline.
- Phase 3 Hermes-3 broken-LoRA reload: inference still returns
`'The answer to the equation 2+2 is 4.'`.
- Spot-checks on Hermes-3 / Phi-4 / Llama-3.2-1B / Gemma-3-1B real
stripped templates: probe still derives the expected prefix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in chat-template helpers
Pure comment minimization across `_find_end_position`,
`_has_add_generation_prompt_block`, `_if_body_emits_content`,
`_derive_assistant_prefix_by_render`, `_fix_chat_template` Case 2,
and `_VariantTokenizerProxy`. No behavior change; same intent,
fewer lines. All 21 follow-up tests and the 51-cell Phase 1 matrix
still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Sandbox probe, fix is_sharegpt validator mismatch, reject negated gates
Three real bugs from the 10-agent Opus review:
1. Probe now uses `jinja2.sandbox.SandboxedEnvironment` instead of bare
`jinja2.Environment`. The probe renders at model-load time (before
the user calls `apply_chat_template`), so it was a new eager
code-execution surface that the base HF tokenizer loading does not
have. SandboxedEnvironment blocks attribute-chain exploits at
negligible cost.
2. `_repair_string_template` now tries validation with both
`is_sharegpt=False` and `is_sharegpt=True`. Previously, when
`_fix_chat_template` internally fell back to the other schema via
its dual-probe, the outer validation still used the caller's
original `is_sharegpt` -- rendering with the wrong message keys and
spuriously dropping a valid repair.
3. `_has_add_generation_prompt_block` now skips `If` nodes whose test
is a `Not` expression. A negated gate like
`{% if not add_generation_prompt %}{{ x }}{% endif %}` fires when
agp=False, so its emitting body is not a generation block -- but the
old code counted any Name reference regardless of polarity.
Cleanup: removed unused `self._label`, added `\r` escape in
generation-block literal, switched variant labels to `!r` formatting,
removed redundant `import os as _os`.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix jinja2.sandbox import and sandbox proxy fallback
Two critical findings from the 20-reviewer pass:
1. [20/20] The proxy read-only fallback used bare `jinja2.Environment`,
not sandboxed. All 20 reviewers independently reproduced marker-file
creation via `cycler.__init__.__globals__['os'].system(...)` during
`fix_chat_template()`. Fixed: fallback now uses
`from jinja2.sandbox import SandboxedEnvironment`.
2. [14/20] The render-diff probe did `import jinja2` then referenced
`jinja2.sandbox.SandboxedEnvironment`. `jinja2.sandbox` is a
submodule that is NOT auto-imported by `import jinja2` on Jinja 3.1.6.
This caused `AttributeError` (swallowed by `except Exception`),
making the entire Case 2 repair path silently return None in a clean
process. The 6 reviewers who saw it work had `jinja2.sandbox`
pre-imported by an earlier module in their process. Fixed: both the
probe and the proxy fallback now use
`from jinja2.sandbox import SandboxedEnvironment`.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Reduce inline comments from ~160 lines to ~25 across both files.
Keep one-line summaries of the "why"; drop multi-paragraph rationale
blocks that repeated information already captured in commit messages
and PR discussion.
* fix: replacing SetEnvironmentVariable with direct registry API
* apply reviews
* Use CreateSubKey for HKCU\Environment
* Store PATH backup under HKCU\Software\Unsloth
* Fix $backupKey registry handle leak in PATH backup block
Wrap $backupKey operations in try/finally so the handle is closed even
if GetValue or SetValue throws. The Add-ToUserPath helper already uses
this pattern for its registry key -- the backup block was the only
place missing it.
* Isolate WM_SETTINGCHANGE broadcast from PATH write error handling
Wrap the broadcast dummy-variable calls in their own try/catch so a
broadcast failure does not mask a successful registry PATH write.
Previously, if SetEnvironmentVariable threw after SetValue already
committed the new PATH, Add-ToUserPath would return $false and the
caller would skip Refresh-SessionPath.
* PATH helper polish: venv precedence, quoted entries, raw/expanded dedup
Three small follow-ups surfaced by a 10-reviewer pass against the rebased
PR head. None fix a regression vs main; each strictly improves the new
helpers.
Refresh-SessionPath / Refresh-Environment:
- Move $env:Path to the front of the merge so an activated venv keeps
precedence over machine/user PATH after a refresh. Pre-PR dropped
process-only entries entirely; post-PR kept them but at the back.
- Dedup on both raw and expanded forms so %USERPROFILE%\foo and the
already-expanded C:\Users\me\foo do not both survive.
Add-ToUserPath:
- Trim whitespace and surrounding double-quotes from each compared entry
so quoted PATH entries like "C:\Program Files\CMake\bin" deduplicate
against an unquoted directory of the same path.
* Back up User PATH inside Add-ToUserPath, before first mutation
Previously only studio/setup.ps1 took a one-time PATH backup, at script
top (line ~547). install.ps1 (the irm | iex entry point) had no backup,
so users who installed via that path had no recovery surface if anything
clobbered their PATH. The PR description's "one-time backup before any
modifications" promise only held for the studio installer flow.
Move the backup into Add-ToUserPath itself: just before the first actual
SetValue mutation, write the pristine raw PATH to
HKCU\Software\Unsloth\PathBackup if no backup already exists. This:
- Covers both entry points (install.ps1 and studio/setup.ps1).
- Captures the TRUE pristine PATH even when install.ps1 runs first and
studio/setup.ps1 runs afterwards (the script-top backup in setup.ps1
would otherwise see an already-modified PATH).
- Is idempotent: once a backup exists, subsequent calls preserve it.
- Skips when nothing would mutate (dedup match) or PATH is empty.
The script-top backup in studio/setup.ps1 is kept for defense in depth.
* Refresh PATH: venv-aware merge order
Reconcile two competing concerns about Refresh-SessionPath /
Refresh-Environment surfaced by separate review rounds:
- venv at the back -> activated venv loses precedence to system Python
- process at the front -> stale shims (old node, old python, etc.)
still on $env:Path can beat a freshly installed tool
New merge order:
1. Activated venv Scripts dir, only if $env:VIRTUAL_ENV is set
2. Machine PATH freshly read from registry
3. User PATH freshly read from registry
4. Current $env:Path as fallback
This way an explicitly-activated venv keeps priority while a tool the
script just installed wins over any stale entry that was already on
the inherited shell PATH. When no venv is active, fresh registry
entries take precedence as expected.
* Append to User PATH by default, close $envKey in finally
Add-ToUserPath gains a -Position Append|Prepend parameter defaulting to
Append so installing unsloth no longer prepends the bundled venv Scripts
directory ahead of the user's existing python / pip on new shells. The
four current call sites (install.ps1 launcher, studio/setup.ps1 CMake,
nvcc, Python user Scripts) all take the Append default because each one
that needs in-session precedence already does an inline $env:Path prepend
independently. This matches rustup / cargo / nvm / pyenv / uv behavior.
Also wrap the script-top $envKey.GetValue in a try/finally so the
registry handle is released even if the read throws. Matches the pattern
already used for $backupKey five lines below.
* Prepend cmake, nvcc, Python Scripts; keep venv Scripts appended
The previous commit switched Add-ToUserPath to append by default so that
installing unsloth would not silently hijack the user's system python /
pip. That was correct for the venv Scripts dir (which contains python.exe
and pip.exe alongside unsloth.exe), but wrong for the three studio/setup
call sites. Those persist cmake, the driver-compatible nvcc, and the
Python user Scripts dir for future shells, and in all three cases an
older tool already earlier in the user PATH would keep winning after the
install finished. The nvcc case is especially load-bearing: setup selects
a driver-compatible CUDA toolkit, then llama.cpp builds against whatever
wins PATH resolution, so a stale older nvcc produces broken builds.
Pass -Position 'Prepend' explicitly at the three setup.ps1 call sites
(cmake at line 754, nvcc bin at line 1025, Python user Scripts at line
1191). None of those directories holds python.exe, so prepending them
does not re-introduce the original hijack problem. Leave the install.ps1
venv Scripts call on the default Append with a comment explaining why.
* Symmetric dedup, Prepend reorders duplicates, unsloth shim dir
Address three separate findings surfaced by review:
1. Dedup asymmetry (Gemini high-priority): the existing dedup expanded
registry entries via ExpandEnvironmentVariables but did NOT expand the
new directory. Passing "%USERPROFILE%\foo" when "C:\Users\me\foo" was
already in PATH produced a duplicate. Expand both sides so the check
is symmetric.
2. -Position Prepend no-op on existing duplicates: the dedup loop
returned $false as soon as it saw a match, regardless of position.
That left a late-position duplicate in place instead of moving it to
the front, so "prepend the newly selected cmake/nvcc" did not always
beat an older copy earlier in PATH. Partition entries into kept and
dropped lists, then reinsert a single copy at the requested position.
Append still returns $false on any match so user-curated orderings
are not reshuffled. Prepend also returns $false when the only copy
is already at position 0 so we preserve the user's casing.
3. Stop adding the venv Scripts dir to User PATH entirely. That dir
holds python.exe and pip.exe alongside unsloth.exe, so neither
Prepend nor Append worked: prepend hijacked the user's system python
and pip, append made the freshly-installed unsloth.exe lose to any
older unsloth.exe earlier on PATH. Replace the Scripts-dir PATH add
with a dedicated shim directory that contains only unsloth.cmd, and
prepend that dir. The shim calls the venv's unsloth.exe by absolute
path so future pip upgrades inside the venv propagate automatically.
* Shim via hardlink, Append user Scripts, drop venv sysconfig fallback
Three follow-ups to the c0ab1ab shim commit, targeting concerns raised in
the second 20-reviewer pass:
1. Shim uses unsloth.exe (hardlink, copy fallback) instead of unsloth.cmd.
The batch-file approach had three distinct regressions:
- cmd.exe expanded %...% sequences inside user arguments, so prompts
like "What does 50% mean?" got mangled before reaching the CLI
- Git Bash / MSYS2 / POSIX-style shells on Windows do not resolve
bare-name lookups to .cmd files, so `unsloth` stopped working there
- Set-Content -Encoding ASCII replaced non-ASCII profile characters
with '?', so installs under C:\Users\Jörg\... wrote a broken shim
A hardlink (fallback: copy) of unsloth.exe is a native Windows
executable with no shell indirection. PATHEXT picks .exe before .cmd
in cmd.exe and PowerShell, Git Bash honors .exe natively, subprocess
callers hit it directly, and a hardlink stays in sync with the venv
on pip upgrades because both names point at the same inode.
2. studio/setup.ps1 Python user Scripts dir is added with default Append
instead of -Position Prepend. That directory holds every pip-installed
user console script (pip, pytest, huggingface-cli, and so on), not
just unsloth, so reordering it silently changed resolution order for
unrelated tools. The new install.ps1 shim at PATH position 0 already
guarantees `unsloth` resolves to the freshly installed copy, so the
Python user Scripts entry only needs to be present, not at the front.
3. The sysconfig lookup in studio/setup.ps1 no longer falls back to
sysconfig.get_path('scripts') when the nt_user scheme dir does not
exist. When setup.ps1 is invoked from an activated venv (a flow the
linked issue actually hits) that fallback returns the venv's Scripts
directory, which would then be added to the persisted User PATH and
re-introduce the python / pip hijack the shim dir is meant to avoid.
Stick strictly to the nt_user scheme; skip the block if it does not
exist on disk.
* Do not crash installer when unsloth.exe shim is locked
The shim update sequence at install.ps1:1095 did a bare Remove-Item /
New-Item HardLink / Copy-Item. Under the script's $ErrorActionPreference
a locked target (most commonly 'unsloth studio' still running while the
user re-invokes the installer) turns the Remove-Item failure into a
terminating error that aborts the install with no actionable message.
The existing shim is perfectly usable in that state, so there is no
reason to abort. Wrap the whole remove/link/copy sequence in a try/catch
that logs the probable cause (Studio still running), points at the fix
(close Studio and re-run), and lets the installer finish with the old
launcher still serving the command.
Also only emit the "added unsloth launcher to PATH" step line when the
launcher was actually (re)created AND the PATH entry was newly added --
previously the message fired even when the shim refresh silently failed,
which was confusing.
* Guard shim PATH entry on existence, use NullString for broadcast delete
Two follow-ups surfaced by the latest review pass:
1. Do not add the shim directory to User PATH when the launcher was not
actually created. Antivirus blocking unsloth.exe, a disk-full volume,
or restrictive filesystem permissions can make both the hardlink and
the copy fallback fail on a fresh install. In that case the existing
sequence would report "added unsloth launcher to PATH" warnings but
still prepend the empty $ShimDir to User PATH -- the user sees an
install that claims success but then cannot resolve `unsloth` in a
new shell. Gate Add-ToUserPath on Test-Path $ShimExe so the PATH
entry is only persisted when the launcher is really there.
2. Pass [NullString]::Value instead of $null to the broadcast-delete
call in Add-ToUserPath. On PowerShell 7.5 and later (running on .NET
9), a bare $null going into [Environment]::SetEnvironmentVariable
can be coerced to an empty string rather than a true .NET null,
which sets the dummy UnslothPathRefresh_XXXXXXXX variable to "" in
HKCU\Environment instead of deleting it. The leaked variable is
visible in System Properties and accumulates one entry per install
run. [NullString]::Value is a PowerShell-specific sentinel that
crosses the interop boundary as a real null and works on both PS 5.1
and PS 7.x. See PowerShell/PowerShell#24637 for the underlying issue.
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Fixes#4150.
Pre-PR, `_fix_chat_template` only patched templates where a trailing `{{ ... }}` expression followed the last `{% endfor %}`. ChatML templates (Hermes, Magnum, Phi-4, etc.) that end cleanly at `{% endfor %}` with no generation-prompt block were left unchanged, so the outer `fix_chat_template` raised:
```
RuntimeError: Unsloth: The tokenizer `...` does not have a
{% if add_generation_prompt %} for generation purposes.
```
This commonly shows up when a downstream tool (LlamaFactory, Axolotl) re-serializes the tokenizer during LoRA save and strips the generation-prompt block.
This PR adds a second branch to `_fix_chat_template` that fires when:
- the content after the last `{% endfor %}` is empty modulo Jinja `{# ... #}` comments,
- the scrubbed template contains `<|im_start|>` and `<|im_end|>`,
- and the scrubbed template does not already mention `add_generation_prompt`.
The assistant-turn separator is inferred from the template itself (preferring an explicit `'<|im_start|>assistant<sep>'` literal, then the unique `message['role'] + '<sep>'` from role concatenations, then `<|im_sep|>` for Phi-4-mini mixed-separator templates, then `\n`), so Phi-4-style templates are not silently corrupted with the wrong separator.
Verified against the existing chat-template corpus:
- Hermes-3, Magnum-v2, Phi-4-mini, Phi-4 multi-sep, ChatML with trailing whitespace, ChatML with trailing Jinja comment, dot-access `message.role`, split-literal `'<|im_start|>assistant'`: all repaired with the correct assistant prefix.
- Already-fixed ChatML templates: idempotent NOP.
- Trap templates with `<|im_start|>` only inside a Jinja comment: correctly not rewritten.
- Llama-3, Gemma-3, Qwen2.5 (non-ChatML): byte-identical.
- Mistral family (5 models including Mistral-Nemo, Mistral-Small-24B, Mixtral): byte-identical, protected both by the structural guard (no ChatML tokens) and the existing name-based exemption in `load_correct_tokenizer`.
- Qwen family (14 models including Qwen2.5, Qwen3, Qwen3-Coder, QwQ, VL, Math, Qwen3-Guard): byte-identical.
End-to-end reproduction: Hermes-3 LoRA SFT, save with stripped chat_template, reload. Pre-PR code path raises the RuntimeError above. Post-PR reload loads cleanly, patches the template at load time, and `apply_chat_template(add_generation_prompt=True)` produces the correct `<|im_start|>assistant\n` prefix.
* fix pass attn implementation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
pre-commit-ci[bot]