* Fix broken manual response-template markers in Studio's fallback table
Six template families in TEMPLATE_TO_RESPONSES_MAPPER shipped markers that
never match what their chat templates actually render, so the manual
train_on_completions path masked every assistant token and the run died on
the all-labels-masked safety net:
- mistral, llama: '[INST] ' / ' [/INST]' - the surrounding spaces fold into
the neighbouring tokens ('[INST]'/'[/INST]' are single special tokens in
Mistral v0.3, SentencePiece pieces in Llama-2), so the padded strings
never match. Now '[INST]' / '[/INST]'.
- starling: trailing space after 'GPT4 Correct Assistant:' folds into the
next content token. Now no trailing space.
- glm: '[gMASK]<sop>' renders once at text start, never before later user
turns, and '<think>' is generation scaffolding rendered as a lone
'</think>' on non-final turns. Now '<|user|>' / '<|assistant|>'.
- qwen3-thinking: '<think>' is stripped from non-final assistant turns
(Qwen3-Thinking-2507) and never rendered by QwQ. Now the bare assistant
header, matching the other qwen entries.
- zephyr: role tags are plain text and SentencePiece tokenizes them
differently at text start than after '</s>' + newline mid-conversation;
the markers need the leading newline anchor. Now '\n<|user|>\n' /
'\n<|assistant|>\n'.
Validated token-level on each family's representative tokenizer with a
two-turn fixture plus system message: user and system content fully masked,
every assistant turn trained, and the final EOS label never -100. The
fixed mistral, llama, starling and glm markers produce labels identical to
zoo auto-detection; qwen3-thinking differs only in one turn-separator
newline token. All 22 unchanged entries produce byte-identical labels to
before this change.
Adds tests/test_response_template_markers.py pinning the fixed and key
unchanged marker literals (dependency-free) plus token-level masking checks
that skip when tokenizers or unsloth_zoo are unavailable offline.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close tokenizer config handle and read it as UTF-8
Chat templates in tokenizer_config.json are rarely ASCII-only, so the
default locale codec could fail the GLM fallback loader on Windows.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
* Anchor the llama marker on <s> and harden the marker test
On transformers 5.x llama-2 tokenizes [INST] after <s> as a bare left
bracket while the standalone encoding gives the space-prefixed piece, so
the unanchored marker missed every turn boundary and later user turns
leaked into training; 4.57 masked this. Anchoring on <s>[INST] matches
both tokenizations, verified token-level under 4.57.6 and 5.5.0.
The test now unwraps the BatchEncoding that apply_chat_template returns
on 5.x before indexing, and the latent trailing spaces in the unreachable
unsloth and vicuna entries are dropped for table consistency.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* DeepSeek-V4: eager attention and trainable FP8 grouped experts
deepseek_v4 ships a custom attention that is not compatible with the sdpa
and flash paths, so add it to _EAGER_ONLY_PREFIXES to load with eager.
Its fused experts load as FP8GroupedLinear, whose forward calls a grouped
matmul kernel with no autograd formula, so loss.backward() fails during
finetuning. Patch the forward to dequantize the frozen fp8 weight and run a
differentiable grouped matmul while training, keeping the fused fp8 kernel
for inference.
* DeepSeek-V4: exclude sdpa/flash and stream fp8 grouped backward
Add deepseek_v4 to _SDPA_EXCLUDED_MODELS and _FLASH_EXCLUDED_MODELS so an
explicit attn_implementation=sdpa/flash request downgrades to eager instead of
raising (the model has no sdpa/flash kernel), matching the eager-only default.
Replace the FP8GroupedLinear training bmm with a custom autograd Function that
saves only the fp8 weight + scale rather than a full bf16 dequantized copy, so
no dequantized grouped weight is retained per layer, and unwrap tensor-parallel
shards before dequant. Bit-exact forward and grad with the previous path.
* FP8 grouped: consistent checkpointing math and block-size-aware dequant
Gate the differentiable training path on self.training rather than
torch.is_grad_enabled(), so a gradient-checkpointed segment runs the same bmm
math in its no-grad forward and its grad recompute instead of mixing the fused
fp8 kernel with bmm.
Dequantize with the layer's own block_size via _blockwise_weight_dequant_any_shape
so non-128 or rectangular fp8 blocks are scaled correctly instead of assuming
128x128.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
New releases of huggingface-hub (1.23.0) and openai (2.45.0) shifted or
added polling loops that the C2 polling/beaconing check flags, failing
all three pip scan-packages shards (studio 1, hf-stack 1, extras 3 new
CRITICAL findings) org-wide including on main.
Regenerated with scan_packages.py --write-baseline per CI shard (same
shard-to-requirements mapping and --with-deps as security-audit.yml)
and merged. All entries were manually reviewed at the resolved versions:
- huggingface-hub hf_api.py: create_repo 409-concurrency retry loop
body changed in 1.23.0; refreshed evidence hash. The loop POSTs to
the canonical Hub endpoint and retries only on a specific conflict
error. Benign client retry.
- openai beta/threads/runs/runs.py: create_and_poll run-status helper
refactored in 2.45.0 (Assistants deprecation annotations); refreshed
evidence hash. Documented polling helper against api.openai.com.
- openai beta/responses/responses.py: new beta websocket client whose
__aiter__ yields server events until the connection closes. New
entry; standard event-stream iterator, not beaconing.
- openai resources/responses/responses.py: evidence line number
refreshed only, hash unchanged.
The two dropped entries are the pre-refactor hashes of the same two
loops above; they no longer occur at the resolved versions. Verified
locally: all three shards exit 0 with 0 unsuppressed CRITICAL/HIGH
(hf-stack 120, studio 151, extras 99 suppressed).
* Auto-detect completion masking markers with template table fallback
Studio's train_on_completions previously relied only on the hardcoded
MODEL_TO_TEMPLATE_MAPPER / TEMPLATE_TO_RESPONSES_MAPPER tables and
silently disabled masking when a model was not in the table, so unmapped
models (LFM2-8B-A1B, DeepSeek, and others) trained on full sequences
without telling the user. Several mapped templates (glm, mistral, llama,
starling, zephyr, qwen3-thinking) also carried markers that mask every
assistant token, which made every row drop in the post-masking filter.
Both training callsites (CUDA trainer.py and MLX worker.py) now share
utils.datasets.completion_masking.apply_completion_masking:
- Try unsloth_zoo chat template auto-detection first; it raises loudly
when the template cannot be parsed and never masks the EOS token.
- gpt-oss models keep their manual markers so non-final assistant
<|end|> tokens stay trained, matching current behavior.
- If auto-detection raises, fall back to the template table exactly as
before.
- If the table also misses, emit an explicit user-visible warning that
completion masking could not be applied and full-sequence training
will occur, instead of a quiet log line.
The >30 percent dropped-rows safety net in trainer.py now guards the
auto path as well. Table consumers for inference and chat templates are
unchanged. Validated against one representative tokenizer for every
template in TEMPLATE_TO_RESPONSES_MAPPER plus the unmapped models:
no template regresses; unit tests cover the four decision paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restrict masking fallback to marker detection failures
The auto branch wrapped the whole train_on_responses_only call, so a real
failure while applying the masking (dataset map, tokenization) was treated
as a detection miss and training silently proceeded on full sequences.
Detect markers separately via get_chat_template_parts (test seam via
detect_fn), then apply them with errors propagating, matching the manual
path. Tokenizers with preset unsloth marker attrs skip detection and call
bare so zoo reuses the stored parts.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fail the run when applying completion masking raises
The helper already falls back internally on detection failures and returns
applied=False on a double miss, so an exception reaching the callsites is a
real failure applying the masking. Remove the callsite catches that
downgraded it to full-sequence training; the run now fails visibly instead.
Also use the explicit re-export alias form in utils/datasets/__init__.py for
the two new names, satisfying the import-hoist source lint.
* Import completion masking from its submodule
The import-hoist source lint counts only real name loads, so package-level
re-exports of the two new names cannot satisfy it. Import
apply_completion_masking from utils.datasets.completion_masking directly at
both callsites and leave utils/datasets/__init__.py untouched.
* Completion masking: gpt-oss renames and MLX raw/alpaca parity
Renamed or private gpt-oss checkpoints are name-detected as gpt-oss but miss
the exact-name table; default them to the gpt-oss template markers instead of
falling through to full-sequence training.
Gate the MLX masking call on not raw_text_mode and format_type != alpaca,
mirroring the CUDA path: raw/CPT text has no chat turns to mask and
Alpaca-rendered text lacks the tokenizer's chat markers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Define raw_text_mode outside the MLX feature-detect block
With an older zoo lacking the append_eos config field, the masking
gate referenced raw_text_mode before assignment. Hoist the assignment
above the feature detection so both consumers see it.
* Gate MLX masking on the formatter's resolved format
format_type auto can resolve to alpaca or raw text; the masking skip
checked only the requested value, so auto-detected Alpaca data got
chat-template markers applied to rendered prompt text. Track the
final_format returned by format_and_template_dataset and gate on it,
matching the CUDA path.
* Unwrap the mlx-lm TokenizerWrapper before marker checks
The wrapper delegates plain reads to the wrapped HF tokenizer but hides
underscore attrs, so preset unsloth markers were invisible and detection
relied on the loader's call patch. Unwrap to the real tokenizer first,
as the zoo MLX resolver does.
* Tighten masking comments
* gpt-oss: auto-detect markers first like every other template
The quantized and BF16 gpt-oss checkpoints ship a chat template without
the channel final header, so the pinned manual markers match nothing
there and masking trained zero tokens. Auto-detection derives markers
from whichever template the checkpoint ships and keeps the final
terminator trained; the manual gpt-oss markers remain the detection
failure fallback, including for renamed checkpoints.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: route models by CONFIG_MAPPING_NAMES instead of hardcoded tables
A model whose model_type is absent from an overlay's transformers cannot load
there, so a new MoE arch not yet in the tier tables gets routed to default and
fails (e.g. lfm2_moe, deepseek_v4). Add a static resolver that parses each
overlay's CONFIG_MAPPING_NAMES straight from source (AST only, no import, no
network, no trust_remote_code) and picks the lowest tier that ships the
model_type. Runs after the existing checks and only ever upgrades default, so
no existing routing changes and new archs no longer need a table edit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio router: harden the CONFIG_MAPPING_NAMES resolver
- Resolve the default tier map from the base install, skipping any .venv_t5_*
sidecar on sys.path, so an in-process 5.x activation cannot make a 5.x-only
model look loadable by 4.x.
- Do not cache an overlay whose sidecar dir is absent, so a later call re-reads
it once provisioned instead of serving a stale empty map.
- Also collect model types added via CONFIG_MAPPING_NAMES.update({...}) and
**{...} unpacking, not just the literal assignment (5.10 uses both).
- Wrap the AST walk in the try/except so a malformed source can never crash tier
resolution.
- Feed the mapping fallback from _load_config_json so a config served from the
hub cache during a transient outage still routes new architectures.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
LFM2-8B-A1B and any other lfm2_moe checkpoint were missing from the
transformers tier tables, so they fell through to the default 4.57.x
sidecar, which does not register lfm2_moe and errors with
"not supported yet in transformers==4.57.6". Only lfm2_vl was listed.
Add Lfm2MoeForCausalLM / lfm2_moe to the 5.3.0 tier (lfm2_moe is
registered in transformers 5.3.0). get_transformers_tier now returns
530 for LFM2-8B-A1B and the model loads and trains as expected.
* CI: retry transient HTTP timeouts in Studio smoke probes
The post() helper in the Studio inference smoke workflows does a single
urlopen with a 240s timeout against the local Studio server. On shared
runners this sporadically hits TimeoutError while the server is stalled,
failing the whole job for a transport hiccup; the same flake has recurred
across unrelated PRs on Linux and Windows (JSON/images and tool-calling
jobs) and passes on rerun.
Retry the probe up to 3 times on transport-level failures only
(TimeoutError, ConnectionError, non-HTTP URLError), 15s apart. HTTP
status errors still surface immediately, so genuine server failures are
unaffected. post_sse() is left unchanged: it has a 600s budget and has
not flaked.
* CI: retry only short probes so worst case fits the job budget
Some json-images calls pass timeout=600; three attempts there could spend
30 minutes in one step and hit the job's timeout-minutes instead of failing
with the Python error. Retry (3 attempts) only when timeout <= 300s, which
covers the observed flaky 180-240s probes; longer probes keep the pre-PR
single attempt.
* CI: give long smoke probes one capped retry
Round two of bounding the retries: timeout>300s probes previously got a
single attempt, so a transient stall in the 600s JSON-mode probes still
failed on first occurrence. Give them one retry with the attempt timeout
capped at 300s. Worst cases stay inside timeout-minutes: 240s probes
12.5 min, one 600s probe 15.25 min, the Windows JSON job's two long
probes 30.5 min against its 35 minute budget.
* Clarify in README that -H 0.0.0.0 starts a public Cloudflare tunnel
* Hedge tunnel URL wording and restore trusted-network caution
* Tighten the 0.0.0.0 tunnel note
* Drop trust-the-network caution from tunnel note
* Restore trusted-network note on the raw-bind sentence
* Use Cloudflare's quick tunnel terminology and consolidate the trust warning
* Studio: resolve the repo-root MTP drafter after the MTP/ GGUF rename
The Gemma 4 QAT GGUF repos renamed the higher-precision MTP/ subdir
copies from gemma-4-...-<quant>-MTP.gguf to mtp-gemma-4-...-<quant>.gguf,
so their basenames now start with the same mtp- prefix as the small
repo-root drafter (mtp-gemma-4-E4B-it.gguf).
The drafter selectors filtered candidates by a mtp- basename prefix and
took the first in sort order. With the new names the MTP/ copies also
match, and because MTP/ (uppercase) sorts before the lowercase root file,
selection flipped to the large BF16 copy under MTP/ instead of the root
drafter both functions document they should pick.
Restrict both selectors, and the companion byte estimate, to root-level
mtp-*.gguf so the MTP/ copies stay explicit-selection only:
- core/inference/llama_cpp.py _pick_mtp (loader auto-download)
- hub/utils/gguf_plan.py preferred_mtp_sibling (Hub variant plans)
- routes/inference.py _remote_gguf_companion_bytes (VRAM headroom)
Also reuse a drafter already in the local cache before downloading, so a
device that already holds a copy on disk does not re-fetch it.
Old-scheme names keep working (they have no root-level mtp- sibling to
mis-select). Adds regression tests for the new naming, both selection
paths, and the on-disk reuse.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate MTP drafter cache reuse to offline mode
Reuse the cached drafter only when HF is offline. Online, route back
through _download_companion_gguf/hf_hub_download so the current revision
is checked (etag) and a changed drafter is refetched, matching the
offline-only cross-snapshot reuse already used for the main GGUF. This
avoids pairing freshly downloaded weights with a stale cached draft.
Make the reuse tests offline and add an online-skips-reuse test.
* Studio: prefer a root MTP drafter across all cached snapshots
Offline reuse scanned snapshots one at a time and returned the first
snapshot that held any drafter, only preferring root within it. A newer
partial snapshot with just the MTP/ copy could shadow the small root
drafter in an older snapshot. Collect drafters across all snapshots and
prefer any repo-root file before an MTP/ copy.
* Studio: keep newest-first snapshot order when reusing cached drafters
Collecting root candidates and sorting by absolute snapshot path could
pick a drafter from an older snapshot. _iter_hf_cache_snapshots yields
newest first and the main GGUF is resolved in that order, so preserve it
(root still preferred over MTP/ copies) to avoid pairing a fresh main
weight with a stale drafter revision.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Restore dropped FP8 weight_scale_inv tensors on load
Some block-scale FP8 checkpoints (for example Qwen3.6-27B-FP8, issue #6200) load
with transformers leaving an mlp.gate_proj as a plain bf16 Linear instead of an
fp8 module. Its raw quantized values are read into the bf16 weight and the
weight_scale_inv is dropped as an unexpected key, so the weight is used un-scaled
and the base model is garbage (perplexity around 2 million).
After load, for every checkpoint weight_scale_inv whose live weight is not fp8,
dequantize the orphaned weight in place using the block scale from the checkpoint
index. Modules that were converted correctly keep an fp8 weight and are skipped,
so healthy checkpoints and single-file checkpoints are a no-op.
Verified on Qwen3.6-27B-FP8: 64 gate_proj scales restored, perplexity 2028902 to
8.9. No-op on Qwen3-8B-FP8 (all scales already live).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden FP8 weight_scale_inv restore from review
- Skip restore when the model has no fp8 weights, so an intentionally
dequantized load (load_in_16bit) is never re-scaled and corrupted.
- Thread revision, subfolder and cache_dir through the index and shard
downloads so scales come from the same snapshot as the weights.
- Cover unsharded single-file model.safetensors checkpoints (no index).
- Handle transposed block-scale layouts and skip on a true grid mismatch
instead of applying a wrong scale.
- Match text-only VLM loads where the language_model prefix was stripped.
- Restore on the FastLanguageModel text path too, not only vision.
- Handle a scalar weight_block_size; per-tensor error handling so one bad
tensor cannot abort the rest or hide a partial mutation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address second review round on FP8 scale restore
- Bound peak memory: dequantize block views in place with the fp32 scale
broadcast instead of materializing a full expanded scale and fp32 copy,
so a near-VRAM-limit load is not pushed into OOM by the repair.
- Restore on the sequence-classification load path too.
- Cover more VLM key remappings (language_model.model.* to
model.language_model.*) when matching modules.
- Skip the restore for variant loads (variant=...) rather than risk
applying default-checkpoint scales to variant weights.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align FP8 scale restore revision with the loaded weights and warn on disk-offloaded layers
In llama.py the CausalLM/SequenceClassification weight loads resolve model_name on its
default branch (revision is not forwarded there), so read the dropped weight_scale_inv
tensors from the same default branch instead of the requested revision, avoiding rescaling
default-branch weights with scales from another revision.
In loader_utils.py a disk-offloaded layer keeps its weight on the meta device until the
offload hook materializes it, so the scale cannot be applied in place. Skip such layers
explicitly and print a warning rather than silently leaving them unscaled.
* Tighten comments in the FP8 scale restore path
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
parse_direct_linux_release_bundle and direct_linux_release_plan are no
longer reached by any live code path. Fork Linux installs resolve through
_fork_manifest_release_plans -> _linux_published_attempts, and the upstream
(ggml-org) path uses direct_upstream_release_plan. The dead parser also
called _resolve_linux_bundle_profile, which no longer exists, so its CUDA
branch would raise NameError if ever executed.
Drop both functions and the obsolete TestDirectLinuxNvidiaCpuGate; its live
equivalent TestLinuxPublishedAttemptsNvidiaCpuGate already covers the
NVIDIA no-silent-CPU behaviour.
* scripts: refresh scan_packages allowlist baseline
Regenerate scripts/scan_packages_baseline.json against the current
resolved dependency set so the blocking pip scan-packages gate matches
what the scanner now finds. Refreshes evidence hashes for benign
findings whose code shifted lines (unsloth-zoo mlx loader, gguf/mlx
test /tmp fixtures) and adds two mainstream-library entries that were
newly surfaced (torch inductor codecache base64+subprocess compile
cache, torch testing common_utils socket import). Stale entries whose
matching code changed and no longer triggers are dropped.
All entries remain CRITICAL/HIGH findings manually judged benign;
matched on (package, file, check, evidence_hash).
* ci(security-audit): re-run scan when the allowlist baseline changes
The security-audit pull_request trigger listed the scanners but not
their allowlist baselines, so a baseline-only edit never re-ran the
scan that consumes it. A refreshed baseline could therefore merge
without CI confirming its evidence hashes match what the scanner finds.
Add scan_packages_baseline.json and scan_npm_packages_baseline.json to
the paths filter so baseline changes are validated on their own PR.
* Keep native RoPE scaling when extending context; carry rope_theta for linear
When max_seq_length exceeds a model's native window, the loader overwrote the
model's rope_scaling with linear scaling. For models that already ship a scaled
RoPE (llama3/yarn/longrope) that is far worse for long context, and on
transformers v5 the linear dict omitted rope_theta (v5 keeps it under
rope_parameters), so the rotary base fell back to 10000 and broke past ~8K tokens.
Keep the native scaling and just widen the window; only synthesize linear for
plain-RoPE models, and carry rope_theta so v5 keeps the real base.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Only preserve native llama3 when extending context; keep linear fallback otherwise
The patched attention constructor (patch_llama_rope_scaling) rebuilds only linear,
llama3 and longrope and its longrope branch reads a top-level
original_max_position_embeddings, so preserving yarn or a nested-only longrope config
would raise during construction on transformers <= 4.47.1. Keep only llama3 native;
yarn/longrope/other types fall back to the linear override, still carrying rope_theta.
* Correct long-context extension comment to match llama3-only preservation
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix fast_gemv crash on compressed-tensors FP8 models
Loading a compressed-tensors FP8 checkpoint (for example
unsloth/Llama-3.2-1B-Instruct-FP8-Block) with fast_inference=False and
running a forward crashed with 'Parameter object has no attribute absmax'
inside fast_gemv.
A compressed-tensors CompressedLinear exposes an already dequantized bf16
weight at forward time while keeping a weight_scale Parameter. The quant
state resolution in get_lora_parameters/get_lora_parameters_bias fell back
to that weight_scale, so a bf16 weight was routed into the bitsandbytes
fast_gemv/fast_dequantize path, which expects a bitsandbytes QuantState
with an absmax attribute.
Only fall back to weight_scale_inv/weight_scale when the weight is still
fp8. A decompressed bf16 weight then resolves to no quant state and flows
through the normal bf16 path, which already handles bias and the LoRA
backward. Real fp8 and bitsandbytes 4bit weights are unchanged.
* Skip the fast_gemv dispatch test before importing unsloth when bitsandbytes is absent
* Fix Windows installer torch index override
* Clear inherited uv index env vars for pinned installs in studio/setup.ps1 (#6898)
* Harden setup.ps1 index-var clearing to truly remove vars (#6898)
* Apply UV_DEFAULT_INDEX torch index fix to Linux/Mac install.sh (#6898)
* Neutralize all uv index env vars for pinned torch installs (#6898)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add Vulkan llama.cpp support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address gemini's feedback
* Studio: move the Vulkan VRAM probe into a standalone script
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Improve Vulkan probe error reporting
* Resolve llama-server symlink so Vulkan build is detected
* Drop unreachable Vulkan fallback in GPU free-memory dispatcher
* Skip the Intel GPU probe when NVIDIA or ROCm is present
* Reserve host RAM headroom for Vulkan integrated GPUs
* Add a `UNSLOTH_FORCE_VULKAN` environment variable
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor GGML_VK_VISIBLE_DEVICES, reserve discrete Vulkan VRAM headroom, and clear Intel GPU on --cpu-fallback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Route Intel and forced-Vulkan hosts to the upstream Vulkan prebuilt, add arm64 Vulkan, keep Vulkan out of RAG auto-detect
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Clear the fork release pin when routing a Vulkan host to the upstream repo
* Gate auto-Vulkan routing on no physical NVIDIA so hidden CUDA devices aren't used
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pin Vulkan launches with --device Vulkan<i> instead of the raw GGML_VK_VISIBLE_DEVICES index space
* Let user --device override the Vulkan pin, and gate direct Vulkan asset picks on no physical NVIDIA
* Update RAG auto-backend test mocks for the _resolve_auto binary and Vulkan probes
* Keep the add_dll_directory handle alive through the Vulkan probe DLL loads
* Revert RAG auto Vulkan guard, guard multi-backend Vulkan detection, and preserve forced Vulkan across updates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use getattr for RTLD_GLOBAL in the Vulkan probe CDLL mode
* Skip CUDA/ROCm APU and datacenter GPU tuning on Vulkan builds
On a Vulkan llama.cpp build gpu_indices are ggml compact ordinals, not
CUDA/ROCm physical ids, so _amd_apu_wants_unified_memory and
_apply_datacenter_env were reading the wrong device. On a mixed AMD APU
plus discrete GPU host that could raise a spurious system-RAM shortfall
and block a valid discrete-GPU load. Gate all three call sites on
not is_vulkan_backend; the Vulkan path already reserves iGPU host
headroom and the backend ignores GGML_CUDA_* anyway.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten Vulkan-guard comment in load_model
* Reduce comments in Vulkan support to be more succinct
* Resolve shell-wrapper llama-server entrypoint to the real lib dir
create_exec_entrypoint falls back to a #!/bin/sh wrapper at the install
root when it cannot symlink into build/bin. _find_llama_server_binary
returns that root entrypoint, but Path.resolve() does not follow a shell
wrapper, so _llama_lib_dir returned the install root and _is_vulkan_backend
missed libggml-vulkan.so -- silently skipping the Vulkan probe and --device
pin on an otherwise valid Vulkan install. Follow the wrapper's exec target
to build/bin. Regression test: test_shell_wrapper_entrypoint_resolves_to_real_lib_dir.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Studio /v1/messages: accept thinking and unknown content blocks
The Anthropic-compatible /v1/messages endpoint modeled a message's content as
Union[str, list[{text|image|tool_use|tool_result}]], so any other block type
made Pydantic reject the whole request with
`messages.N.content.str: Input should be a valid string`. Resuming a Claude
session commonly replays assistant turns that carry `thinking` (extended
thinking) blocks, and sometimes a null content for a tool-only turn, both of
which tripped this and returned a 400.
Accept them:
- Add a permissive AnthropicUnknownBlock fallback (any block whose type is not
one of the four known ones), so thinking/redacted_thinking/provider-specific/
future blocks validate. A validator keeps known types on their typed models,
so a malformed known block (e.g. a tool_use without id) still fails cleanly.
- Coerce a null message (and tool_result) content to "" so the converter's
`for block in content` stays safe.
The converter already drops block types it does not translate, so a thinking
block is not forwarded to the model.
* Studio /v1/messages: keep user content validation strict
Make the thinking/null leniency role-aware so it never silently drops real
user input. Assistant turns (replayed history) still accept unknown/thinking
blocks and coerce a null tool-only turn to empty. User turns keep the strict
boundary: a null user content is rejected, and a content block the converter
cannot translate is rejected instead of being dropped into an empty prompt.
Also remove an empty file committed by accident.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio /v1/messages: coalesce resumed user turns and tighten content checks
- The /v1/messages count and generation paths now coalesce the adjacent user
turns that dropping an empty or null assistant turn can leave behind, so a
strict GGUF chat template no longer 400s on non-alternating roles.
- A user content block with a non-string type (list / dict) is rejected as a
clean 400 instead of raising TypeError and escaping as a 500.
- The assistant null-to-empty coercion only applies to an explicit null; an
assistant turn that omits content entirely still fails required-field
validation instead of being silently coerced to an empty string.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio /v1/messages: tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* unsloth start: add --resume to persist and reopen agent sessions
`unsloth start <agent>` launches a coding agent whose home is a throwaway
temp dir wiped on exit, so codex/openclaw/hermes/pi (which relocate their
whole home there) cannot resume a conversation after you quit. opencode and
claude keep their session data in a fixed user dir, so they already resume.
Add an opt-in --resume/--no-resume flag: it routes the launch to the stable
Unsloth agents dir (the same one --no-launch already uses) so the session
survives the exit, never touching the user's own ~/.<agent>. A bare --resume
also reopens the last conversation via the agent's native flag (codex
`resume --last`, opencode/claude/pi `--continue`). The default is unchanged:
a plain launch still uses a temp dir and persists nothing.
Add a dispatch-only `resume` job to the Local Agent Guides CI that drives the
real launch path and asserts the split: codex/pi are wiped without --resume
and persist with it, while opencode/claude persist either way.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* unsloth start: rename --resume to --persist
The session flag collided with agents' own resume flags. `unsloth start
claude --resume <id>` used to forward `--resume <id>` straight to Claude
(which keeps its history in ~/.claude regardless), so a boolean --resume on
unsloth start would have swallowed the session id and turned it into a stray
prompt. Name the persistence flag --persist instead, so every agent's native
resume flag (claude --resume <id>, codex resume, opencode --continue, ...)
still passes through untouched. Behavior is otherwise identical: --persist
keeps a launched agent's session under the Unsloth agents dir, and a bare
--persist reopens the last conversation.
Add a regression test that `--resume <id>` passes through verbatim, and in the
CI resume experiment skip the redundant second pass for opencode/claude (they
persist either way, and a second CPU turn only risks a timeout).
* unsloth start: correct --persist help and drop the buggy auto-resume
Reword the --persist help to be accurate: claude and opencode keep sessions in
the user's own stores and resume regardless, so --persist only stabilizes the
otherwise-ephemeral relocated home of codex/openclaw/hermes/pi. Drop the
bare-launch auto-append of native resume tokens: it errored on a first launch
with no prior session, and was inconsistent between launch and no-launch.
--persist now only keeps the session dir; resume via the agent's own command
(e.g. `unsloth start codex --persist resume`), which now finds it.
In the CI resume experiment, fail the pass when the launched turn exits
non-zero, so a write-then-error is not misread as PERSISTED.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Silence torch._check_is_size FutureWarning and shim it if torch removes it
bitsandbytes 4-bit dequant calls torch._check_is_size, which torch
deprecated with a FutureWarning ("Use _check(i >= 0) instead") that prints
on every bnb-4bit load. Silence that warning in suppress_cuda_printf, and
add fix_torch_check_is_size so a future torch that removes _check_is_size
gets it shimmed to _check(i >= 0) (honoring the max bound) and bitsandbytes
keeps working.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten fix_torch_check_is_size docstring
Lead with what the shim does and drop the redundant line; two lines
instead of three, same intent.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Mirror dynamo/inductor config sets into defaults so torch 2.12 worker threads honor them
torch 2.12 stores config user overrides in ContextVars, so direct
assignments like torch._dynamo.config.recompile_limit = 1024 no longer
reach the autograd engine worker threads. Gradient checkpointing
recomputes fullgraph-compiled gpt-oss kernels inside backward on those
threads, which then read the default recompile limit of 8 and raise
FailOnRecompileLimitHit at step 0 of GRPO/SFT. Mirror direct config
assignments into the process-global entry defaults on torch >= 2.12,
restoring the torch <= 2.11 cross-thread semantics while leaving the
context-scoped config.patch API untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep config.patch thread-local when mirroring dynamo/inductor sets
config.patch(...) also assigns through ConfigModule.__setattr__, so the
default-mirror was leaking its scoped, thread-local writes into the
process-global entry default. Track patch enter/exit with a per-thread
depth counter (wrapping ConfigModule.patch) and skip mirroring while
inside a patch, so only genuine direct assignments restore the torch
2.11 cross-thread semantics and config.patch stays context-local.
* Also keep config.load_config thread-local when mirroring config sets
load_config restores a saved dynamo/inductor config by calling setattr
per key, which the default-mirror would otherwise leak process-wide just
like config.patch did. Wrap load_config with the same per-thread depth
counter (renamed to _scoped_depth) so both scoped writers skip the mirror
and stay context-local, while genuine direct assignments still restore the
torch 2.11 cross-thread default.
* Drop the pre-existing override replay from the config thread fix
The replay was redundant: this runs from _gpu_init before unsloth sets any
dynamo/inductor config, so the __setattr__ wrapper already mirrors every
later assignment (recompile_limit included). It could also read a value
that belonged to a config.patch context still active at import time and
write that thread-local override into the global default. Removing it keeps
the cross-thread fix and drops the now-unused _inductor.config import.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
When a coding agent is missing, `unsloth start <agent>` offers to run the
vendor's own installer (curl | bash, irm | iex, or npm) after an interactive
confirm. Those installers execute with the user's privileges and there is no
signature or hash check on the fetched content, so a blind "yes" is a
supply-chain risk if the delivery path is compromised.
Keep the auto-install convenience but make consent informed: before the prompt,
name the exact remote source the installer fetches (or the command it runs for a
package installer) and state that nothing verifies a signature or hash. Behavior
is otherwise unchanged: non-interactive stdin still never executes anything, and
the confirm still defaults to no.
* Fix FastSentenceTransformer Qwen embedding preprocessing
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Document Transformer.load embedding modality fix for #6881
* Harden #6881 fix and add forwards/backwards-compatible regression tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fall back to Transformer constructor on legacy sentence-transformers without Hub-capable load
* Mirror legacy sentence-transformers fallback in embedding-parity tripwire test
* Tighten #6881 comments and docstrings
* Skip embedding-parity test on CPU-only runners since FastSentenceTransformer requires CUDA
* Honor the transformer module's saved subfolder when loading
modules.json records a path for the Transformer module (root for
decoder embedders like Qwen3-Embedding, 0_Transformer for the classic
layout). Pooling/Normalize already load from their saved path; thread the
same path into Transformer.load as subfolder so config and tokenizer
resolve like stock ST. stays a no-op, so single-module models are
unchanged.
* Make embedding-parity test bf16-aware
fp16 overflows to NaN on bf16-native embedders such as EmbeddingGemma
(Gemma3), producing a false parity failure. Prefer bf16 when the GPU
supports it so the tripwire can guard the full documented embedding
matrix (Qwen3-Embedding, EmbeddingGemma, BGE-M3, all-MiniLM, GTE-ModernBERT),
not just fp16-safe models.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Retry the Studio UI shutdown re-login on transient goto timeout
The Chat UI Playwright smoke intermittently failed at the pre-shutdown
re-login: page.goto('/login') can hit a 60s TimeoutError on a slow runner
even while the server is healthy, and the surrounding except only tolerated
ERR_ABORTED / interrupted-navigation, so a plain timeout hard-failed the job.
Wrap the re-login goto/wait/fill/submit in the same 3-attempt retry the
change-password step already uses (recover_or_replace_page between tries,
per-attempt fail screenshots, wait_for_health pre-gate). The composer wait
stays outside the loop so a retry never re-navigates after login has set
tokens (which would redirect to /chat via the guest guard); it remains the
authoritative confirmation, so a genuinely broken login still fails.
* Catch transient login-request failures and preserve error listeners on recovery
Wait on the /api/auth/login POST inside the retry (via click_and_wait_for_response)
so a transient 4xx/5xx is retried in-loop instead of surfacing only at the
out-of-loop composer wait, matching the change-password step. When
recover_or_replace_page swaps in a fresh page, re-attach the pageerror/console
listeners so error tracking survives the replacement.
* Stabilize floating monitor drag
* Restore floating monitor exit animation
* Harden Windows Studio smoke checks
* Keep API menu badge removed
* Apply no-build-tools env overrides in-script
The runner does not apply step-level env keys containing parentheses,
so ProgramFiles(x86) kept its real value and Find-VsBuildTools still
detected VS through vswhere. Set the overrides inside each pwsh step
instead; child processes inherit them. The resolver step moves to pwsh
because bash cannot export a variable named ProgramFiles(x86).
* Reset chat UI session without a second browser context
macOS runs Chromium with --single-process, where closing the last
context tears down the whole browser, so the shutdown re-login died
with TargetClosedError on new_page. Clear cookies and swap pages
inside the same context instead, opening the replacement page before
closing the old one.
* Keep the no-build-tools Path filtered across session refreshes
install.ps1's Refresh-SessionPath and setup.ps1's Refresh-Environment
rebuild the session Path from the Machine and User registry scopes, so
the process-level filter could be undone mid-install and re-expose
CMake. Filter those scopes in the Prepare step with normalized dir
matching and restore them in cleanup.
* Drop stale localStorage auth tokens before re-login
Auth tokens live in localStorage, not cookies, and the login guest
guard redirects on their mere presence. Remove them during the session
reset so the /login navigation is deterministic instead of relying on
the tolerated redirect bounce.
* unstructured block removal
* Enhance unstructured block handling
* Restrict block cleanup to upload UIDs
* cleanup for seed block uploads
* upload cleanup queue for unstructured blocks in recipe studio
* Fix unstructured upload cleanup edge cases
* Fix unstructured upload import ownership
* Fix-unstructured-import-path-ownership
* Guard failed-delete restore against stale block in unstructured drop zone
* Drain queued upload cleanups when autosave is skipped
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: render thinking blocks for safetensors inference with prefilled <think> templates
Reasoning templates like Qwen3.6 end the generation prompt with an open
<think> tag. skip_prompt streaming drops it, so the frontend never sees
the opening tag and shows reasoning as plain text. Detect the prefill
and re-emit it at the start of the stream on the transformers and MLX
paths. Also stop stripping think tags in _clean_generated_text when a
tokenizer marks them special.
* Studio: guard think re-emit for special close tags, yield prefill early
Address review feedback:
- Guard: skip re-emitting the open <think> when the tokenizer marks </think>
as a special token, since skip_special_tokens would strip the model's close
tag and leave an unclosed block that swallows the answer. Falls back to
plain text (pre-fix behaviour) for those tokenizers.
- Yield the prefilled <think> before the first token so the thinking block
renders during prompt prefill instead of after the first generated token.
- Drop the now-unnecessary _clean_generated_text think-tag exemption; the
guard handles the special-token case at the source.
No mainstream reasoning model (Qwen3.6, Qwen3, DeepSeek-R1, QwQ, GLM-4.6)
marks think tags special, so behaviour is unchanged for them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lyxot <longyixing331@gmail.com>
PyPI release unsloth 2026.7.2 is now live. Bumps the pinned floor in
install.sh and install.ps1 from 2026.7.1 to 2026.7.2 for both unsloth and
unsloth-zoo across all 5 install commands (no-torch / reinstall / upgrade /
local / auto torch backend paths) so fresh installs resolve to the new wheel.
Follows the same pattern as #5716.
* Studio: allow CPU-only DiffusionGemma by granting the diffusion runner the CPU device
* Studio: mark CPU-only DiffusionGemma as non-GPU-resident for training VRAM preflight
* Studio: keep the CPU DiffusionGemma change minimal (revert VRAM-flag tweak; Metal hosts still hold unified memory)
* Studio: keep CPU DiffusionGemma fallback fully CPU-masked so a masked GPU host does not re-expose GPU 0
worker.py imports has_blackwell_gpu from utils.wheel_utils, but _load_worker_module
stubs utils.wheel_utils with a fixed name tuple that omitted it, so loading the worker
raised ImportError (cannot import name 'has_blackwell_gpu') and Backend CI could not
collect test_mlx_training_worker_config.py. Add the name to the stub so it matches
worker.py's imports.
* fix: Remove moot has_blackwell_gpu() function
Fixes unslothai/unsloth#6961. This function skipped flash-attn on Blackwell GPUs because no prebuilt wheel existed;
Dao-AILab now ships one and url_exists() already gates resolution.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: use torchao 0.17.0 for Blackwell
Fixes#6961. Torchao 0.16.0's cpp extensions are built against CUDA 12, so on a CUDA-13
torch (cu130 / Blackwell) they fail to load with "libcudart.so.12: cannot
open shared object file". Select 0.17.0 there instead: its cpp targets torch
2.11, so it is skipped cleanly rather than crashing. CUDA-12 / ROCm / CPU
torch 2.10 keeps 0.16.0 and its working kernels.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Condense torchao version-selection comments (no behavior change)
* Support torch 2.11 in the Studio installer via the torch2.10 prebuilt wheels
Map torch 2.11 to the torch2.10 prebuilt wheels for flash-attn, causal-conv1d,
and mamba through wheel_utils.prebuilt_wheel_torch_mm, applied in direct_wheel_url
(filename) and flash_attn_wheel_url (version). Those torch2.10 CUDA wheels load and
pass each project's own test suite on torch 2.11 (verified on B200), so a torch 2.11
environment gets the prebuilt accelerators instead of skipping or building from source.
Raise _CUDA_TORCH_PKG_SPEC to <2.12.0 (torchvision <0.27.0, torchaudio <2.12.0) so
the CUDA torch repair path can install torch 2.11, where torchao 0.17's cpp kernels
load cleanly. Add tests for the mapping.
* Keep has_blackwell_gpu as a False stub for future arch gating
* Restore has_blackwell_gpu as a return-False probe kept for future arch gating
Keep the nvidia-smi compute_cap detection and its two call sites, but short-circuit
with return False at the top so flash-attn is no longer skipped on Blackwell (sm_100+
now has prebuilt wheels and url_exists gates resolution). Drop the early return to
re-enable arch-based detection later.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* models: auto-target per-expert Linear MoE experts for LoRA (gpt-oss 4bit)
MoE checkpoints whose experts are stored as per-expert nn.Linear ModuleLists
could not receive expert LoRA. gpt-oss bnb-4bit is the canonical case: its
experts live at mlp.experts.gate_up_projs.<i> and mlp.experts.down_projs.<i> as
per-expert Linear4bit modules, not a fused nn.Parameter. The target_parameters
path only handles the fused nn.Parameter layout, and the plain
gate_proj/up_proj/down_proj leaf names do not match the per-expert indices, so
get_peft_model attached LoRA to attention only and left every expert frozen
(0 of 1536 on gpt-oss-20b) even though the grouped bnb-4bit training forward
exists.
Add get_moe_target_modules, the module-LoRA counterpart of
get_moe_target_parameters: it detects per-expert Linear ModuleLists under an
experts container and returns their suffix target_modules names
(gate_up_projs.<i> / down_projs.<i>). get_peft_model in both llama.py and
vision.py extends target_modules with these, handling the explicit leaf-list
form and the regex form (auto / all-linear / scoped). It is gated on the same
MLP-in-scope condition as the parameter path, so an attention-only request still
skips the experts.
Also gate get_moe_target_parameters on the fused parameter actually existing, so
a per-expert-Linear layout no longer produces a dead target_parameters path or a
misleading "Enabling LoRA on MoE parameters" line; those experts are handled
through target_modules instead.
Validated on gpt-oss-20b-unsloth-bnb-4bit (transformers 5.5.0): experts attach
(1536 modules, trainable 0.036 percent to 1.65 percent) across the default, None
and all-linear paths; training memorizes and the LoRA adapter reproduces exactly
after a cold reload in a fresh process. No regression: fused-parameter MoEs
(Qwen3-30B-A3B-4bit), non-MoE models, and attention-only requests are unaffected
(get_moe_target_modules returns an empty list).
Merging these per-expert adapters into a merged_16bit checkpoint is handled by a
companion unsloth-zoo change (saving_utils folds each per-expert delta into the
fused gate_up_proj / down_proj tensor). With both, the LoRA adapter and the
merged_16bit checkpoint reload the trained behavior identically.
* models: scope per-expert MoE targets, keep repeat get_peft_model idempotent, warn on old zoo
Address review of the per-expert Linear MoE targeting:
- Scope get_moe_target_modules to the requested projection leaves (gate/up map to
the gate_up ModuleList, down maps to the down ModuleList), so a narrowed request
such as target_modules=["down_proj"] no longer also trains gate_up_projs, matching
get_moe_target_parameters.
- Detect experts through a PEFT-wrapped base_layer as well, and recompute the
auto-added expert targets in the llama.py existing-adapter check, so a repeat
get_peft_model call with the same arguments stays idempotent instead of raising on
the saved expert targets.
- Warn when the installed unsloth_zoo cannot fold these per-expert experts into a
merged_16bit checkpoint (older releases keep the fused gate_up_proj / down_proj
tensors and drop the per-expert deltas), so the expert LoRA is not silently lost on
save_pretrained_merged; the fold lands in unsloth-zoo #885. The LoRA adapter itself
is unaffected.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio/hub): apply repo_id length limit per segment, not whole string
is_valid_repo_id() applied the 96-char limit to the full "namespace/repo_name"
string, so a repo with a valid (<=96 char) name but a long combined id was
falsely rejected. Match huggingface_hub.validate_repo_id by checking the length
per segment instead. Fixes#6946.
* Fix long repo id state filenames
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: source CPU llama.cpp prebuilts from the unslothai fork
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reject unknown Linux CPU arches and keep ROCm-tooling hosts off the CPU prebuilt
* Studio: extend the resolve-prebuilt ROCm-tooling guard to Windows
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: let ROCm-SDK-only CPU hosts take the fork CPU prebuilt
* Studio: accept windows-arm64 prebuilt kind and refresh stale fork-routing comments
* Studio: correct stale fork-routing comments and --resolve-prebuilt help
* Refresh stale ggml-org routing comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: keep transformers off sys.modules until the training worker activates the sidecar
The training worker (core/training/worker.py:run_training_process) decides the per-worker
Xet env flip during preflight by importing utils/hf_xet_fallback.py, which eagerly imported
unsloth_zoo at module load. unsloth_zoo's __init__ imports transformers, so the default
transformers 4.57.x was cached in sys.modules before activate_transformers_for_subprocess
prepended the 5.x sidecar to sys.path. Since activation only edits sys.path, the already
cached module won, and 5.x models failed to load their tokenizer or config:
- Qwen3.5 / GLM-4.7 (tokenizer_class TokenizersBackend): "Tokenizer class TokenizersBackend
does not exist or is not currently imported."
- gemma-4: "... is not supported yet in transformers==4.57.6."
Fix: load the shared unsloth_zoo backend lazily (only when a heavy download helper is first
used, which is after activation). child_should_disable_xet and the DEFAULT_* constants are
defined locally so importing the shim stays light. The download wrappers, the DownloadStallError
class, start_watchdog and get_hf_download_state resolve the shared backend on first use, and the
degraded no-unsloth_zoo fallback is preserved.
Tests:
- test_hf_xet_fallback.py: existing suite kept green via the restored _shared_* seam; the
GPU-init retry test now triggers the lazy load explicitly; new guard asserts importing
child_should_disable_xet does not import transformers/unsloth_zoo.
- test_training_worker_import_discipline.py: new invariant test that the worker preflight
imports leave transformers unimported, so this class of regression cannot return silently.
Runs in studio-backend-ci (CPU only, no network/GPU/weights).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: CPU-only guard that activation switches transformers to the model's sidecar version
Adds test_worker_activates_correct_transformers.py: runs the real worker preflight
(from utils.hf_xet_fallback import child_should_disable_xet) plus the real tier
detection and activate_transformers_for_subprocess for a transformers-5.x model
(Qwen3.5, tier 530), then asserts the in-process transformers actually switched to
the 5.x sidecar. A stale pre-activation import leaves 4.57.x pinned and fails the
assertion, which is exactly the TokenizersBackend regression (#6951).
Self-contained CUDA spoof (mirrors tests/_zoo_aggressive_cuda_spoof.py) forces
unsloth_zoo down its full, transformers-importing init path on a GPU-less runner;
without it unsloth_zoo degrades and never preloads transformers, masking the bug.
A one-line stub sidecar stands in for the 5.x venv, so no GPU, network, weights, or
real sidecar are needed. Passes on this fix, fails on buggy main.
* Studio: load the repo's canonical CUDA spoof in the correct-version guard
Load tests/_zoo_aggressive_cuda_spoof.py (the committed spoof the consolidated CI
already relies on) as the single source of truth so the guard matches CI and stays
robust on a CPU-only torch wheel, where a partial hand-rolled spoof could miss a
torch.cuda call and let the unsloth_zoo import raise (masking the bug). Falls back to
a minimal inline spoof for a standalone studio checkout. Verified: passes on this fix,
fails on buggy main, and the fallback path passes when the spoof file is absent.
* Studio: declare the lazily-resolved xet names so ruff F822 stays green
DownloadStallError, start_watchdog and get_hf_download_state are provided via the
module __getattr__ (PEP 562), so ruff F822 flagged them as undefined names in __all__
and the Source-lint / pre-commit checks went red. Add annotation-only declarations
(no value bound, so __getattr__ still resolves them lazily to the shared unsloth_zoo
backend) to mark them defined for the linter while keeping F822 active for the rest
of __all__.
* Studio: tighten comments on the sidecar-activation fix and its tests
* Studio: mirror the new MLX-dispatch preflight import in the import-discipline guard
The worker preflight now also runs 'from core.training.training import
is_apple_silicon_training_platform, should_use_mlx_training_backend' before it
activates the transformers sidecar. Add that import (guarded) to the guard's
preflight snippet so the invariant test stays a faithful mirror: a future change
that makes core.training.training pull transformers/unsloth_zoo eagerly would then
be caught too. Verified clean on the current tree (no leak).
---------
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* feat: detect installed coding agent CLIs in Studio settings
The API-keys panel only ever showed the "claude" flavor of the
`unsloth start` command, so anyone using Codex, OpenCode, OpenClaw,
Hermes, or Pi had to manually rewrite the copied command by hand.
Add a backend check that looks for each agent's CLI binary on PATH
(shutil.which, mirroring the pattern already used elsewhere in
studio/backend/utils) and expose it as GET /api/settings/coding-agents.
The API-keys panel now renders a picker for all six supported agents,
marks the ones it finds installed, and defaults to one of those instead
of always falling back to claude.
Includes unit tests for the detection helper.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address review feedback on coding-agent detection
Three fixes from PR review:
- detect_installed_coding_agents now treats a PATH lookup failure as
"not installed" instead of letting it bubble up and break the
settings endpoint; added a regression test for it.
- CodingAgentsResponse.agents is now typed as an immutable tuple
instead of a list built from one, matching CODING_AGENTS itself.
- Fixed a race in the API-keys panel: picking an agent while the
installed-CLI check is still in flight could get silently overwritten
once that check resolved. A ref now tracks whether the user has made
a manual choice, so the auto-detected default only applies before
that happens.
* Address Codex feedback: GGUF gating and remote-detection scope
- codex refuses to launch against a non-GGUF (transformers-backed) model
(unsloth_cli's _require_gguf_for_codex), so auto-defaulting to it produced
a copy-pasteable command that fails immediately whenever the loaded model
isn't GGUF. Add useActiveModelIsGguf() (looks up the active checkpoint in
the chat runtime store) and a correction effect that steers the auto-pick
away from codex unless the loaded model qualifies, without ever touching a
choice the user made by hand.
- Detection runs via shutil.which on the Studio backend host, which isn't
the same machine as the browser in a tunnel/remote session. Reword the
'installed'/'detected' copy to say so explicitly when the tunnel URL is
in use, instead of implying the check ran on the viewer's own device.
* Rework auto-default per review: loopback gating + inline GGUF check
Replaces the previous approach with the exact shape discussed on the PR:
- Export isLoopbackHost/normalizeHost from agent-command.ts. The detection
endpoint runs shutil.which on the Studio backend, which only describes the
browser's own machine when the base this panel targets resolves to
loopback. For a LAN or tunnel/remote base, gate the whole thing off --
don't mark anything as "detected" and don't let it drive the default --
instead of just relabeling the copy.
- Drop the separate GGUF-correction effect and useActiveModelIsGguf hook.
Read useChatRuntimeStore.getState().activeGgufVariant inline inside the
existing detection effect's .then() (so it doesn't need to sit in the
effect's deps), and pick the first detected agent that isn't codex unless
the loaded model is GGUF, leaving the existing default untouched when no
compatible agent is detected.
Verified both branches (loopback vs LAN/tunnel base, gguf vs non-gguf,
manual pick preserved, no-compatible-agent fallback) with a standalone
port of the .then() logic.
* Address latest Codex findings: stale detection, model swap, cache
- Clear detectedAgents (and skip the network call entirely) when the panel
leaves a loopback base, instead of leaving a previous loopback detection
result marked 'installed' for a command that now targets a LAN/tunnel/
remote host.
- Add a separate, network-free correction effect keyed on the live
activeGgufVariant: if codex was auto-picked while a GGUF model was loaded
and the user then switches to a transformers-backed model while this panel
stays mounted, steer away from codex instead of leaving a command that
unsloth_cli's _require_gguf_for_codex will now reject. Never touches a
manual pick.
- Drop coding-agents.ts's module-lifetime cache. Installed-CLI detection is
environment state, not a persisted setting, so a stale positive/negative
from before the user installed something (or reopened the tab) is worse
than one extra cheap local API call per mount; keep only the in-flight
de-dupe for concurrent callers.
Verified the correction-effect logic (gguf->non-gguf swap with/without a
fallback, still-gguf no-op, manual pick never overridden) with a standalone
port of the effect.
* Make the codex/GGUF auto-pick symmetric in both directions
The correction effect only steered away from codex when the model stopped
being GGUF; it never steered back toward codex if the model became GGUF
*after* a non-GGUF-gated fallback had already picked something else (e.g.
codex is the only detected CLI, a transformers model is loaded so the
selection correctly falls back to the claude default, then the user loads a
GGUF model while the panel stays mounted -- codex never gets reconsidered).
Consolidate into one effect that re-derives the preferred detected agent
from scratch whenever detectedAgents or activeGgufVariant changes, in either
direction, instead of only reacting to the codex-specific downgrade case.
The fetch effect now only populates detectedAgents/availableAgents; this
effect is the single source of truth for what gets auto-picked from that
list. Never overrides a manual choice.
Verified both transition directions plus the manual-pick-survives and
initial-detection cases with a standalone port of the derivation logic.
* Reset the auto-pick to the default when it stops being trustworthy
Two more real gaps from the latest Codex pass on d988f52:
- The unified derivation effect only handled the case where a *different*
detected agent could take over. If codex was the only detected agent and
auto-picked while a GGUF model was loaded, then the model stopped being
GGUF, 'preferred' came back undefined and the effect silently left the
selection on codex -- exactly the command unsloth_cli's
_require_gguf_for_codex now rejects. Fall back to DEFAULT_AGENT in that
case instead of leaving it untouched.
- Leaving a loopback base cleared detectedAgents (so the 'installed' badges
correctly disappear) but left whatever agent had been auto-picked from
that now-stale, server-side-only detection still selected. Reset to
DEFAULT_AGENT there too, unless the user picked by hand.
Introduces a shared DEFAULT_AGENT constant instead of repeating the "claude"
literal at each reset site. Verified all five cases (both new resets, both
manual-pick-survives variants, and the existing multi-detected-agent
fallback still preferring another compatible agent over resetting) with a
standalone port of the effects.
* Derive GGUF-ness from the actual loaded state, not just the variant string
activeGgufVariant only covers an HF-repo GGUF pick (a specific quant
variant string). A direct local .gguf file -- custom folder, LM
Studio, or drag-drop -- is just as much a GGUF the codex preflight
(unsloth_cli's _require_gguf_for_codex) would accept, but it never has
a "variant" to report, so it read as non-GGUF here even though
/api/inference/status correctly reports is_gguf: true for it. That
mismatch could leave a Codex-only install not auto-selected, or reset
an auto-picked Codex, for a model that actually supports it.
Combined activeGgufVariant with activeNativePathToken (covers the
drag-drop/picked-file case) and ggufContextLength (only ever populated
when the backend last reported is_gguf: true for the active model, see
applyActiveModelStatusToStore) so all three paths a model can be GGUF
through are covered, matching the same is_gguf-or-equivalent check
hasGgufSource already applies to a staged pick elsewhere in this
codebase.
* Clear stale native-path token on a non-GGUF status refresh
When a native (drag-dropped or picked) GGUF was loaded and the backend later
switches to a transformers model outside the UI load path, refresh() adopts the
new /api/inference/status via setCheckpoint and applyActiveModelStatusToStore.
Those reset activeGgufVariant and ggufContextLength but never clear
activeNativePathToken, so the isGguf OR stays true after the switch and a
Codex-only detection auto-selects unsloth start codex for a non-GGUF model its
preflight rejects.
Drop activeNativePathToken in applyActiveModelStatusToStore whenever the status
is non-GGUF. A real GGUF load reports is_gguf: true, so its token is preserved
(the load path owns it); only a non-GGUF status clears it.
* Add the AGPL-3.0 header to the new studio contract test
* Fix/adjust agent detection for PR #6909
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>
* fix: launch OpenClaw local TUI by default
* Fix/adjust OpenClaw launch paths for PR #6937
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Default OpenClaw to the local TUI only on a bare invocation
The first-arg startswith('-') branch rewrote passthrough globals into a broken
command: OpenClaw's grammar is openclaw [--dev] [--profile <name>] <command>, so
'unsloth start openclaw --profile test' became 'openclaw tui --local --profile
test', but tui does not accept --profile (or --dev), so the invocation failed.
A leading '--flag value' is ambiguous between a global (--profile test) and a tui
option (--message hi), so it cannot be reinterpreted safely. Default to the local
TUI only when no passthrough args are given, and forward everything else verbatim
so OpenClaw parses it under its own grammar. The bare-launch default (the point
of this change) is preserved; explicit subcommands and global flags pass through.
---------
Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* version-compat CI: fake CPU training runs for SFT/GRPO/DPO
Adds a runtime layer on top of the patch-run canary: actually runs
trainer.train() for a couple of steps on a CPU-only runner under the CUDA
spoof, wrapping a plain tiny HF model in the Unsloth-patched trainer. Exercises
the real train() loop (collation, generation, the injected
_get_per_token_logps_and_entropies, loss, backward, optimizer) so a TRL or
transformers change that breaks the loop at runtime -- not just the source
structure -- surfaces here. No GPU, no meaningful numerics.
Needs a chain of small CPU shims (eager torch.compile, dynamo suppress, cuda
tensor-alloc redirect to CPU, model.for_training/for_inference equivalents)
documented inline. Does not exercise Unsloth's Triton/GPU kernels (CPU can't).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* cpu fake-train: force adamw_torch + disable dynamo for CPU runner
On a real CPU-build torch runner (GitHub CI) two things bit that a CUDA-build
torch with GPUs hidden masked locally:
- The default optimizer is adamw_8bit (bitsandbytes), whose is_on_gpu() check
dies on CPU tensors. Force optim=adamw_torch in all three configs.
- import unsloth reinstalls the real torch.compile over the eager passthrough,
so the GRPO hot path (chunked_selective_log_softmax) actually compiles and
inductor picks the spoofed CUDA device, crashing on device props
(gcnArchName). Re-apply the eager passthrough after import and flip
torch._dynamo.config.disable so every @torch.compile runs eager at call time.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* cpu fake-train: write checkpoints under pytest tmp_path
Use pytest's tmp_path for each trainer's output_dir instead of a hardcoded
relative temp/ci_* path, so a local pytest run does not leave untracked dirs in
the repo tree and the tests are CWD-independent.
* version-compat CI: disable dynamo at process level for the fake-run job
Set TORCHDYNAMO_DISABLE / TORCH_COMPILE_DISABLE in the fake-run step env so
dynamo/inductor is off before conftest.py's early import unsloth, not only via
the per-test runtime shim. Defense in depth on the GPU-less runner: the GRPO
hot path never compiles regardless of when its functions were decorated.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
