Comment-only: condense the shim header, the _c10d_functional/_dtensor schema
notes, the is_win32_rocm parity note, and the export-page prune comment. No
code changes.
The availableFormats prune effect ran on the initial render before
/api/system/hardware resolves, when hasNvidia is false and the NVIDIA-only
compressed-tensors formats are transiently absent. On a fresh Export mount
with the module cache empty (cold start, or a remount during
refreshHardwareInfo), a running FP8/NVFP4 selection was pruned permanently,
since the later hardware response only adds formats back to availableFormats
and never restores selectedFormats. Gate the effect on hardware.loaded so it
prunes only against the authoritative capability set, matching the effect's
stated intent.
Address two review findings on the shim.
torchao's `from torch.distributed._tensor import DTensor` loads
torch.distributed.tensor._collective_utils, which runs
`register_fake("_dtensor::shard_dim_alltoall")` at import time. That op is
DEF'd only in C++ (Functional.cpp), so it is absent on distributed-less ROCm
wheels and register_fake raises "operator _dtensor::shard_dim_alltoall does
not exist", failing the acceptance `import torchao` and rolling the shim
back. Define the _dtensor schema via a second FRAGMENT library in the same
transaction. Schema verified against the live 2.9 dispatcher and the v2.11.0
Functional.cpp source.
Match the Studio is_win32_rocm() detector: fire on torch.version.hip OR a
"rocm" __version__ tag (AMD SDK wheels lack torch.version.hip but tag rocm).
Gating on hip alone left those wheels torchao-stubbed even though the export
gate disables torchao on them. The capability guards (distributed absent, no
_c10d_functional/_dtensor dispatcher ops) still rule out false positives.
Extend the native-present guard to the _dtensor namespace and add drift tests
pinning the _dtensor schema to the live dispatcher and the version-tag
detection.
Legacy Windows ROCm PyTorch wheels ship without the torch.distributed
C-extension, so importing torchao crashes at
`from torch._C._distributed_c10d import (...)` with
"No module named 'torch._C._distributed_c10d'". That makes torchao
completely unimportable and disables FP8/INT8 weight-only export.
fix_torchao_windows_rocm_import() installs a capability-gated compatibility
shim: it registers the version-exact _c10d_functional op schemas via a
FRAGMENT library plus a fake torch._C._distributed_c10d module, then runs
`import torchao` as an in-transaction acceptance test with atomic rollback.
It fires only on the exact broken config (win32 + torch.version.hip + real
torchao installed + distributed C-extension genuinely absent + no
_c10d_functional dispatcher ops) and is a strict no-op everywhere else,
including on fixed wheels (ROCm/TheRock#5694). Opt out with
UNSLOTH_DISABLE_TORCHAO_ROCM_SHIM=1.
Wired into _gpu_init before importing unsloth_zoo so the zoo torchao stub
self-disables once real torchao is importable. Adds drift tests that pin
the schema table to the live dispatcher and assert the no-op, guard,
FRAGMENT, and rollback invariants.
Normalize the requested alias through unsloth's torchao normalizer before the gate, so equivalent forms (portable_fp8, hyphen/space variants) also hit the clear Windows-ROCm rejection instead of being misclassified as compressed-tensors.
Remove frontend/backend detection drift: expose win32_rocm (mirrors is_win32_rocm(): hip OR a rocm build tag) in the export capability payload and have the Export UI read it, instead of re-deriving Windows ROCm from versions.rocm (unset on AMD SDK wheels).
Prune already-selected formats when the gate flips, so a stale torchao pick made before hardware resolves is not exported. Register the stub meta_path finder only once. Use an explicit None check for the stub sentinel. Tighten comments.
torch.distributed is unsupported on Windows ROCm, so real torchao cannot import and Studio import-stubs it; the stub's config classes return None, so TorchAoConfig(quant_type=None) crashed exports with 'quant_type must be either a string or an AOBaseConfig instance, got NoneType'.
Gate the portable torchao FP8/INT8 path off on Windows ROCm via a shared is_win32_rocm() helper (used by both the import stub and the export gate so they can't drift), and add an early defensive error in export_merged_model so a forced torchao request fails fast with a clear message instead of the cryptic crash. The Export UI also hides torchao and stops claiming it works there.
No change on Linux, macOS, or Windows CUDA (torchao is real); Windows ROCm users keep 16-bit merged and GGUF quantization.
* Handle linked instruction files in Bash cleanup
* Limit instruction cleanup to managed dependencies
* Make Bash cleanup test portable
* Run junction cleanup regression on Windows
* Keep instruction cleanup CI focused
* Studio: remove AGENTS.md from install artifacts
* Studio: prune CLAUDE.md from install artifacts
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Studio instruction cleanup edge cases
* Trim Studio cleanup comments
* Make Studio cleanup safe on PowerShell 5.1
* Fix Studio cleanup ownership boundaries
* Simplify Windows link detection
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: startup loading banner and mute the benign bitsandbytes ROCm warning
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: shorten startup banner wording
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix SyntheticDataKit.chunk_data emitting chunks over max_tokens
The multi-chunk path built boundaries from np.linspace(..., n_chunks), but
pairing boundaries[:-1] with boundaries[1:] turns N points into N-1 ranges,
so it produced one fewer, oversized chunk: every chunk exceeded max_tokens
and a document just over the threshold came back as a single unsplit chunk.
Use n_chunks + 1 points so exactly n_chunks ranges are emitted, each within
max_tokens.
Also base n_chunks on the non-overlapped span: consecutive chunks overlap by
overlap, so covering length needs ceil((length - overlap) / stride) chunks, not
ceil(length / stride). The looser count over-counted by one just past a stride
multiple (a 673-token doc became 3 chunks of ~267 instead of 2 of ~369),
emitting an extra redundant chunk. Coverage and overlap are unchanged and every
chunk still stays within max_tokens.
* Condense chunk_data comments and clarify over-split test for PR #7073
---------
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Fix broken manual response-template markers in Studio's fallback table
Six template families in TEMPLATE_TO_RESPONSES_MAPPER shipped markers that
never match what their chat templates actually render, so the manual
train_on_completions path masked every assistant token and the run died on
the all-labels-masked safety net:
- mistral, llama: '[INST] ' / ' [/INST]' - the surrounding spaces fold into
the neighbouring tokens ('[INST]'/'[/INST]' are single special tokens in
Mistral v0.3, SentencePiece pieces in Llama-2), so the padded strings
never match. Now '[INST]' / '[/INST]'.
- starling: trailing space after 'GPT4 Correct Assistant:' folds into the
next content token. Now no trailing space.
- glm: '[gMASK]<sop>' renders once at text start, never before later user
turns, and '<think>' is generation scaffolding rendered as a lone
'</think>' on non-final turns. Now '<|user|>' / '<|assistant|>'.
- qwen3-thinking: '<think>' is stripped from non-final assistant turns
(Qwen3-Thinking-2507) and never rendered by QwQ. Now the bare assistant
header, matching the other qwen entries.
- zephyr: role tags are plain text and SentencePiece tokenizes them
differently at text start than after '</s>' + newline mid-conversation;
the markers need the leading newline anchor. Now '\n<|user|>\n' /
'\n<|assistant|>\n'.
Validated token-level on each family's representative tokenizer with a
two-turn fixture plus system message: user and system content fully masked,
every assistant turn trained, and the final EOS label never -100. The
fixed mistral, llama, starling and glm markers produce labels identical to
zoo auto-detection; qwen3-thinking differs only in one turn-separator
newline token. All 22 unchanged entries produce byte-identical labels to
before this change.
Adds tests/test_response_template_markers.py pinning the fixed and key
unchanged marker literals (dependency-free) plus token-level masking checks
that skip when tokenizers or unsloth_zoo are unavailable offline.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close tokenizer config handle and read it as UTF-8
Chat templates in tokenizer_config.json are rarely ASCII-only, so the
default locale codec could fail the GLM fallback loader on Windows.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
* Anchor the llama marker on <s> and harden the marker test
On transformers 5.x llama-2 tokenizes [INST] after <s> as a bare left
bracket while the standalone encoding gives the space-prefixed piece, so
the unanchored marker missed every turn boundary and later user turns
leaked into training; 4.57 masked this. Anchoring on <s>[INST] matches
both tokenizations, verified token-level under 4.57.6 and 5.5.0.
The test now unwraps the BatchEncoding that apply_chat_template returns
on 5.x before indexing, and the latent trailing spaces in the unreachable
unsloth and vicuna entries are dropped for table consistency.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* DeepSeek-V4: eager attention and trainable FP8 grouped experts
deepseek_v4 ships a custom attention that is not compatible with the sdpa
and flash paths, so add it to _EAGER_ONLY_PREFIXES to load with eager.
Its fused experts load as FP8GroupedLinear, whose forward calls a grouped
matmul kernel with no autograd formula, so loss.backward() fails during
finetuning. Patch the forward to dequantize the frozen fp8 weight and run a
differentiable grouped matmul while training, keeping the fused fp8 kernel
for inference.
* DeepSeek-V4: exclude sdpa/flash and stream fp8 grouped backward
Add deepseek_v4 to _SDPA_EXCLUDED_MODELS and _FLASH_EXCLUDED_MODELS so an
explicit attn_implementation=sdpa/flash request downgrades to eager instead of
raising (the model has no sdpa/flash kernel), matching the eager-only default.
Replace the FP8GroupedLinear training bmm with a custom autograd Function that
saves only the fp8 weight + scale rather than a full bf16 dequantized copy, so
no dequantized grouped weight is retained per layer, and unwrap tensor-parallel
shards before dequant. Bit-exact forward and grad with the previous path.
* FP8 grouped: consistent checkpointing math and block-size-aware dequant
Gate the differentiable training path on self.training rather than
torch.is_grad_enabled(), so a gradient-checkpointed segment runs the same bmm
math in its no-grad forward and its grad recompute instead of mixing the fused
fp8 kernel with bmm.
Dequantize with the layer's own block_size via _blockwise_weight_dequant_any_shape
so non-128 or rectangular fp8 blocks are scaled correctly instead of assuming
128x128.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
New releases of huggingface-hub (1.23.0) and openai (2.45.0) shifted or
added polling loops that the C2 polling/beaconing check flags, failing
all three pip scan-packages shards (studio 1, hf-stack 1, extras 3 new
CRITICAL findings) org-wide including on main.
Regenerated with scan_packages.py --write-baseline per CI shard (same
shard-to-requirements mapping and --with-deps as security-audit.yml)
and merged. All entries were manually reviewed at the resolved versions:
- huggingface-hub hf_api.py: create_repo 409-concurrency retry loop
body changed in 1.23.0; refreshed evidence hash. The loop POSTs to
the canonical Hub endpoint and retries only on a specific conflict
error. Benign client retry.
- openai beta/threads/runs/runs.py: create_and_poll run-status helper
refactored in 2.45.0 (Assistants deprecation annotations); refreshed
evidence hash. Documented polling helper against api.openai.com.
- openai beta/responses/responses.py: new beta websocket client whose
__aiter__ yields server events until the connection closes. New
entry; standard event-stream iterator, not beaconing.
- openai resources/responses/responses.py: evidence line number
refreshed only, hash unchanged.
The two dropped entries are the pre-refactor hashes of the same two
loops above; they no longer occur at the resolved versions. Verified
locally: all three shards exit 0 with 0 unsuppressed CRITICAL/HIGH
(hf-stack 120, studio 151, extras 99 suppressed).
* Auto-detect completion masking markers with template table fallback
Studio's train_on_completions previously relied only on the hardcoded
MODEL_TO_TEMPLATE_MAPPER / TEMPLATE_TO_RESPONSES_MAPPER tables and
silently disabled masking when a model was not in the table, so unmapped
models (LFM2-8B-A1B, DeepSeek, and others) trained on full sequences
without telling the user. Several mapped templates (glm, mistral, llama,
starling, zephyr, qwen3-thinking) also carried markers that mask every
assistant token, which made every row drop in the post-masking filter.
Both training callsites (CUDA trainer.py and MLX worker.py) now share
utils.datasets.completion_masking.apply_completion_masking:
- Try unsloth_zoo chat template auto-detection first; it raises loudly
when the template cannot be parsed and never masks the EOS token.
- gpt-oss models keep their manual markers so non-final assistant
<|end|> tokens stay trained, matching current behavior.
- If auto-detection raises, fall back to the template table exactly as
before.
- If the table also misses, emit an explicit user-visible warning that
completion masking could not be applied and full-sequence training
will occur, instead of a quiet log line.
The >30 percent dropped-rows safety net in trainer.py now guards the
auto path as well. Table consumers for inference and chat templates are
unchanged. Validated against one representative tokenizer for every
template in TEMPLATE_TO_RESPONSES_MAPPER plus the unmapped models:
no template regresses; unit tests cover the four decision paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restrict masking fallback to marker detection failures
The auto branch wrapped the whole train_on_responses_only call, so a real
failure while applying the masking (dataset map, tokenization) was treated
as a detection miss and training silently proceeded on full sequences.
Detect markers separately via get_chat_template_parts (test seam via
detect_fn), then apply them with errors propagating, matching the manual
path. Tokenizers with preset unsloth marker attrs skip detection and call
bare so zoo reuses the stored parts.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fail the run when applying completion masking raises
The helper already falls back internally on detection failures and returns
applied=False on a double miss, so an exception reaching the callsites is a
real failure applying the masking. Remove the callsite catches that
downgraded it to full-sequence training; the run now fails visibly instead.
Also use the explicit re-export alias form in utils/datasets/__init__.py for
the two new names, satisfying the import-hoist source lint.
* Import completion masking from its submodule
The import-hoist source lint counts only real name loads, so package-level
re-exports of the two new names cannot satisfy it. Import
apply_completion_masking from utils.datasets.completion_masking directly at
both callsites and leave utils/datasets/__init__.py untouched.
* Completion masking: gpt-oss renames and MLX raw/alpaca parity
Renamed or private gpt-oss checkpoints are name-detected as gpt-oss but miss
the exact-name table; default them to the gpt-oss template markers instead of
falling through to full-sequence training.
Gate the MLX masking call on not raw_text_mode and format_type != alpaca,
mirroring the CUDA path: raw/CPT text has no chat turns to mask and
Alpaca-rendered text lacks the tokenizer's chat markers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Define raw_text_mode outside the MLX feature-detect block
With an older zoo lacking the append_eos config field, the masking
gate referenced raw_text_mode before assignment. Hoist the assignment
above the feature detection so both consumers see it.
* Gate MLX masking on the formatter's resolved format
format_type auto can resolve to alpaca or raw text; the masking skip
checked only the requested value, so auto-detected Alpaca data got
chat-template markers applied to rendered prompt text. Track the
final_format returned by format_and_template_dataset and gate on it,
matching the CUDA path.
* Unwrap the mlx-lm TokenizerWrapper before marker checks
The wrapper delegates plain reads to the wrapped HF tokenizer but hides
underscore attrs, so preset unsloth markers were invisible and detection
relied on the loader's call patch. Unwrap to the real tokenizer first,
as the zoo MLX resolver does.
* Tighten masking comments
* gpt-oss: auto-detect markers first like every other template
The quantized and BF16 gpt-oss checkpoints ship a chat template without
the channel final header, so the pinned manual markers match nothing
there and masking trained zero tokens. Auto-detection derives markers
from whichever template the checkpoint ships and keeps the final
terminator trained; the manual gpt-oss markers remain the detection
failure fallback, including for renamed checkpoints.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: route models by CONFIG_MAPPING_NAMES instead of hardcoded tables
A model whose model_type is absent from an overlay's transformers cannot load
there, so a new MoE arch not yet in the tier tables gets routed to default and
fails (e.g. lfm2_moe, deepseek_v4). Add a static resolver that parses each
overlay's CONFIG_MAPPING_NAMES straight from source (AST only, no import, no
network, no trust_remote_code) and picks the lowest tier that ships the
model_type. Runs after the existing checks and only ever upgrades default, so
no existing routing changes and new archs no longer need a table edit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio router: harden the CONFIG_MAPPING_NAMES resolver
- Resolve the default tier map from the base install, skipping any .venv_t5_*
sidecar on sys.path, so an in-process 5.x activation cannot make a 5.x-only
model look loadable by 4.x.
- Do not cache an overlay whose sidecar dir is absent, so a later call re-reads
it once provisioned instead of serving a stale empty map.
- Also collect model types added via CONFIG_MAPPING_NAMES.update({...}) and
**{...} unpacking, not just the literal assignment (5.10 uses both).
- Wrap the AST walk in the try/except so a malformed source can never crash tier
resolution.
- Feed the mapping fallback from _load_config_json so a config served from the
hub cache during a transient outage still routes new architectures.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
LFM2-8B-A1B and any other lfm2_moe checkpoint were missing from the
transformers tier tables, so they fell through to the default 4.57.x
sidecar, which does not register lfm2_moe and errors with
"not supported yet in transformers==4.57.6". Only lfm2_vl was listed.
Add Lfm2MoeForCausalLM / lfm2_moe to the 5.3.0 tier (lfm2_moe is
registered in transformers 5.3.0). get_transformers_tier now returns
530 for LFM2-8B-A1B and the model loads and trains as expected.
* CI: retry transient HTTP timeouts in Studio smoke probes
The post() helper in the Studio inference smoke workflows does a single
urlopen with a 240s timeout against the local Studio server. On shared
runners this sporadically hits TimeoutError while the server is stalled,
failing the whole job for a transport hiccup; the same flake has recurred
across unrelated PRs on Linux and Windows (JSON/images and tool-calling
jobs) and passes on rerun.
Retry the probe up to 3 times on transport-level failures only
(TimeoutError, ConnectionError, non-HTTP URLError), 15s apart. HTTP
status errors still surface immediately, so genuine server failures are
unaffected. post_sse() is left unchanged: it has a 600s budget and has
not flaked.
* CI: retry only short probes so worst case fits the job budget
Some json-images calls pass timeout=600; three attempts there could spend
30 minutes in one step and hit the job's timeout-minutes instead of failing
with the Python error. Retry (3 attempts) only when timeout <= 300s, which
covers the observed flaky 180-240s probes; longer probes keep the pre-PR
single attempt.
* CI: give long smoke probes one capped retry
Round two of bounding the retries: timeout>300s probes previously got a
single attempt, so a transient stall in the 600s JSON-mode probes still
failed on first occurrence. Give them one retry with the attempt timeout
capped at 300s. Worst cases stay inside timeout-minutes: 240s probes
12.5 min, one 600s probe 15.25 min, the Windows JSON job's two long
probes 30.5 min against its 35 minute budget.
* Clarify in README that -H 0.0.0.0 starts a public Cloudflare tunnel
* Hedge tunnel URL wording and restore trusted-network caution
* Tighten the 0.0.0.0 tunnel note
* Drop trust-the-network caution from tunnel note
* Restore trusted-network note on the raw-bind sentence
* Use Cloudflare's quick tunnel terminology and consolidate the trust warning
* Studio: resolve the repo-root MTP drafter after the MTP/ GGUF rename
The Gemma 4 QAT GGUF repos renamed the higher-precision MTP/ subdir
copies from gemma-4-...-<quant>-MTP.gguf to mtp-gemma-4-...-<quant>.gguf,
so their basenames now start with the same mtp- prefix as the small
repo-root drafter (mtp-gemma-4-E4B-it.gguf).
The drafter selectors filtered candidates by a mtp- basename prefix and
took the first in sort order. With the new names the MTP/ copies also
match, and because MTP/ (uppercase) sorts before the lowercase root file,
selection flipped to the large BF16 copy under MTP/ instead of the root
drafter both functions document they should pick.
Restrict both selectors, and the companion byte estimate, to root-level
mtp-*.gguf so the MTP/ copies stay explicit-selection only:
- core/inference/llama_cpp.py _pick_mtp (loader auto-download)
- hub/utils/gguf_plan.py preferred_mtp_sibling (Hub variant plans)
- routes/inference.py _remote_gguf_companion_bytes (VRAM headroom)
Also reuse a drafter already in the local cache before downloading, so a
device that already holds a copy on disk does not re-fetch it.
Old-scheme names keep working (they have no root-level mtp- sibling to
mis-select). Adds regression tests for the new naming, both selection
paths, and the on-disk reuse.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate MTP drafter cache reuse to offline mode
Reuse the cached drafter only when HF is offline. Online, route back
through _download_companion_gguf/hf_hub_download so the current revision
is checked (etag) and a changed drafter is refetched, matching the
offline-only cross-snapshot reuse already used for the main GGUF. This
avoids pairing freshly downloaded weights with a stale cached draft.
Make the reuse tests offline and add an online-skips-reuse test.
* Studio: prefer a root MTP drafter across all cached snapshots
Offline reuse scanned snapshots one at a time and returned the first
snapshot that held any drafter, only preferring root within it. A newer
partial snapshot with just the MTP/ copy could shadow the small root
drafter in an older snapshot. Collect drafters across all snapshots and
prefer any repo-root file before an MTP/ copy.
* Studio: keep newest-first snapshot order when reusing cached drafters
Collecting root candidates and sorting by absolute snapshot path could
pick a drafter from an older snapshot. _iter_hf_cache_snapshots yields
newest first and the main GGUF is resolved in that order, so preserve it
(root still preferred over MTP/ copies) to avoid pairing a fresh main
weight with a stale drafter revision.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Restore dropped FP8 weight_scale_inv tensors on load
Some block-scale FP8 checkpoints (for example Qwen3.6-27B-FP8, issue #6200) load
with transformers leaving an mlp.gate_proj as a plain bf16 Linear instead of an
fp8 module. Its raw quantized values are read into the bf16 weight and the
weight_scale_inv is dropped as an unexpected key, so the weight is used un-scaled
and the base model is garbage (perplexity around 2 million).
After load, for every checkpoint weight_scale_inv whose live weight is not fp8,
dequantize the orphaned weight in place using the block scale from the checkpoint
index. Modules that were converted correctly keep an fp8 weight and are skipped,
so healthy checkpoints and single-file checkpoints are a no-op.
Verified on Qwen3.6-27B-FP8: 64 gate_proj scales restored, perplexity 2028902 to
8.9. No-op on Qwen3-8B-FP8 (all scales already live).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden FP8 weight_scale_inv restore from review
- Skip restore when the model has no fp8 weights, so an intentionally
dequantized load (load_in_16bit) is never re-scaled and corrupted.
- Thread revision, subfolder and cache_dir through the index and shard
downloads so scales come from the same snapshot as the weights.
- Cover unsharded single-file model.safetensors checkpoints (no index).
- Handle transposed block-scale layouts and skip on a true grid mismatch
instead of applying a wrong scale.
- Match text-only VLM loads where the language_model prefix was stripped.
- Restore on the FastLanguageModel text path too, not only vision.
- Handle a scalar weight_block_size; per-tensor error handling so one bad
tensor cannot abort the rest or hide a partial mutation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address second review round on FP8 scale restore
- Bound peak memory: dequantize block views in place with the fp32 scale
broadcast instead of materializing a full expanded scale and fp32 copy,
so a near-VRAM-limit load is not pushed into OOM by the repair.
- Restore on the sequence-classification load path too.
- Cover more VLM key remappings (language_model.model.* to
model.language_model.*) when matching modules.
- Skip the restore for variant loads (variant=...) rather than risk
applying default-checkpoint scales to variant weights.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align FP8 scale restore revision with the loaded weights and warn on disk-offloaded layers
In llama.py the CausalLM/SequenceClassification weight loads resolve model_name on its
default branch (revision is not forwarded there), so read the dropped weight_scale_inv
tensors from the same default branch instead of the requested revision, avoiding rescaling
default-branch weights with scales from another revision.
In loader_utils.py a disk-offloaded layer keeps its weight on the meta device until the
offload hook materializes it, so the scale cannot be applied in place. Skip such layers
explicitly and print a warning rather than silently leaving them unscaled.
* Tighten comments in the FP8 scale restore path
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
parse_direct_linux_release_bundle and direct_linux_release_plan are no
longer reached by any live code path. Fork Linux installs resolve through
_fork_manifest_release_plans -> _linux_published_attempts, and the upstream
(ggml-org) path uses direct_upstream_release_plan. The dead parser also
called _resolve_linux_bundle_profile, which no longer exists, so its CUDA
branch would raise NameError if ever executed.
Drop both functions and the obsolete TestDirectLinuxNvidiaCpuGate; its live
equivalent TestLinuxPublishedAttemptsNvidiaCpuGate already covers the
NVIDIA no-silent-CPU behaviour.
* scripts: refresh scan_packages allowlist baseline
Regenerate scripts/scan_packages_baseline.json against the current
resolved dependency set so the blocking pip scan-packages gate matches
what the scanner now finds. Refreshes evidence hashes for benign
findings whose code shifted lines (unsloth-zoo mlx loader, gguf/mlx
test /tmp fixtures) and adds two mainstream-library entries that were
newly surfaced (torch inductor codecache base64+subprocess compile
cache, torch testing common_utils socket import). Stale entries whose
matching code changed and no longer triggers are dropped.
All entries remain CRITICAL/HIGH findings manually judged benign;
matched on (package, file, check, evidence_hash).
* ci(security-audit): re-run scan when the allowlist baseline changes
The security-audit pull_request trigger listed the scanners but not
their allowlist baselines, so a baseline-only edit never re-ran the
scan that consumes it. A refreshed baseline could therefore merge
without CI confirming its evidence hashes match what the scanner finds.
Add scan_packages_baseline.json and scan_npm_packages_baseline.json to
the paths filter so baseline changes are validated on their own PR.
* Keep native RoPE scaling when extending context; carry rope_theta for linear
When max_seq_length exceeds a model's native window, the loader overwrote the
model's rope_scaling with linear scaling. For models that already ship a scaled
RoPE (llama3/yarn/longrope) that is far worse for long context, and on
transformers v5 the linear dict omitted rope_theta (v5 keeps it under
rope_parameters), so the rotary base fell back to 10000 and broke past ~8K tokens.
Keep the native scaling and just widen the window; only synthesize linear for
plain-RoPE models, and carry rope_theta so v5 keeps the real base.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Only preserve native llama3 when extending context; keep linear fallback otherwise
The patched attention constructor (patch_llama_rope_scaling) rebuilds only linear,
llama3 and longrope and its longrope branch reads a top-level
original_max_position_embeddings, so preserving yarn or a nested-only longrope config
would raise during construction on transformers <= 4.47.1. Keep only llama3 native;
yarn/longrope/other types fall back to the linear override, still carrying rope_theta.
* Correct long-context extension comment to match llama3-only preservation
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix fast_gemv crash on compressed-tensors FP8 models
Loading a compressed-tensors FP8 checkpoint (for example
unsloth/Llama-3.2-1B-Instruct-FP8-Block) with fast_inference=False and
running a forward crashed with 'Parameter object has no attribute absmax'
inside fast_gemv.
A compressed-tensors CompressedLinear exposes an already dequantized bf16
weight at forward time while keeping a weight_scale Parameter. The quant
state resolution in get_lora_parameters/get_lora_parameters_bias fell back
to that weight_scale, so a bf16 weight was routed into the bitsandbytes
fast_gemv/fast_dequantize path, which expects a bitsandbytes QuantState
with an absmax attribute.
Only fall back to weight_scale_inv/weight_scale when the weight is still
fp8. A decompressed bf16 weight then resolves to no quant state and flows
through the normal bf16 path, which already handles bias and the LoRA
backward. Real fp8 and bitsandbytes 4bit weights are unchanged.
* Skip the fast_gemv dispatch test before importing unsloth when bitsandbytes is absent
* Fix Windows installer torch index override
* Clear inherited uv index env vars for pinned installs in studio/setup.ps1 (#6898)
* Harden setup.ps1 index-var clearing to truly remove vars (#6898)
* Apply UV_DEFAULT_INDEX torch index fix to Linux/Mac install.sh (#6898)
* Neutralize all uv index env vars for pinned torch installs (#6898)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add Vulkan llama.cpp support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address gemini's feedback
* Studio: move the Vulkan VRAM probe into a standalone script
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Improve Vulkan probe error reporting
* Resolve llama-server symlink so Vulkan build is detected
* Drop unreachable Vulkan fallback in GPU free-memory dispatcher
* Skip the Intel GPU probe when NVIDIA or ROCm is present
* Reserve host RAM headroom for Vulkan integrated GPUs
* Add a `UNSLOTH_FORCE_VULKAN` environment variable
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor GGML_VK_VISIBLE_DEVICES, reserve discrete Vulkan VRAM headroom, and clear Intel GPU on --cpu-fallback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Route Intel and forced-Vulkan hosts to the upstream Vulkan prebuilt, add arm64 Vulkan, keep Vulkan out of RAG auto-detect
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Clear the fork release pin when routing a Vulkan host to the upstream repo
* Gate auto-Vulkan routing on no physical NVIDIA so hidden CUDA devices aren't used
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pin Vulkan launches with --device Vulkan<i> instead of the raw GGML_VK_VISIBLE_DEVICES index space
* Let user --device override the Vulkan pin, and gate direct Vulkan asset picks on no physical NVIDIA
* Update RAG auto-backend test mocks for the _resolve_auto binary and Vulkan probes
* Keep the add_dll_directory handle alive through the Vulkan probe DLL loads
* Revert RAG auto Vulkan guard, guard multi-backend Vulkan detection, and preserve forced Vulkan across updates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use getattr for RTLD_GLOBAL in the Vulkan probe CDLL mode
* Skip CUDA/ROCm APU and datacenter GPU tuning on Vulkan builds
On a Vulkan llama.cpp build gpu_indices are ggml compact ordinals, not
CUDA/ROCm physical ids, so _amd_apu_wants_unified_memory and
_apply_datacenter_env were reading the wrong device. On a mixed AMD APU
plus discrete GPU host that could raise a spurious system-RAM shortfall
and block a valid discrete-GPU load. Gate all three call sites on
not is_vulkan_backend; the Vulkan path already reserves iGPU host
headroom and the backend ignores GGML_CUDA_* anyway.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten Vulkan-guard comment in load_model
* Reduce comments in Vulkan support to be more succinct
* Resolve shell-wrapper llama-server entrypoint to the real lib dir
create_exec_entrypoint falls back to a #!/bin/sh wrapper at the install
root when it cannot symlink into build/bin. _find_llama_server_binary
returns that root entrypoint, but Path.resolve() does not follow a shell
wrapper, so _llama_lib_dir returned the install root and _is_vulkan_backend
missed libggml-vulkan.so -- silently skipping the Vulkan probe and --device
pin on an otherwise valid Vulkan install. Follow the wrapper's exec target
to build/bin. Regression test: test_shell_wrapper_entrypoint_resolves_to_real_lib_dir.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Studio /v1/messages: accept thinking and unknown content blocks
The Anthropic-compatible /v1/messages endpoint modeled a message's content as
Union[str, list[{text|image|tool_use|tool_result}]], so any other block type
made Pydantic reject the whole request with
`messages.N.content.str: Input should be a valid string`. Resuming a Claude
session commonly replays assistant turns that carry `thinking` (extended
thinking) blocks, and sometimes a null content for a tool-only turn, both of
which tripped this and returned a 400.
Accept them:
- Add a permissive AnthropicUnknownBlock fallback (any block whose type is not
one of the four known ones), so thinking/redacted_thinking/provider-specific/
future blocks validate. A validator keeps known types on their typed models,
so a malformed known block (e.g. a tool_use without id) still fails cleanly.
- Coerce a null message (and tool_result) content to "" so the converter's
`for block in content` stays safe.
The converter already drops block types it does not translate, so a thinking
block is not forwarded to the model.
* Studio /v1/messages: keep user content validation strict
Make the thinking/null leniency role-aware so it never silently drops real
user input. Assistant turns (replayed history) still accept unknown/thinking
blocks and coerce a null tool-only turn to empty. User turns keep the strict
boundary: a null user content is rejected, and a content block the converter
cannot translate is rejected instead of being dropped into an empty prompt.
Also remove an empty file committed by accident.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio /v1/messages: coalesce resumed user turns and tighten content checks
- The /v1/messages count and generation paths now coalesce the adjacent user
turns that dropping an empty or null assistant turn can leave behind, so a
strict GGUF chat template no longer 400s on non-alternating roles.
- A user content block with a non-string type (list / dict) is rejected as a
clean 400 instead of raising TypeError and escaping as a 500.
- The assistant null-to-empty coercion only applies to an explicit null; an
assistant turn that omits content entirely still fails required-field
validation instead of being silently coerced to an empty string.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio /v1/messages: tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* unsloth start: add --resume to persist and reopen agent sessions
`unsloth start <agent>` launches a coding agent whose home is a throwaway
temp dir wiped on exit, so codex/openclaw/hermes/pi (which relocate their
whole home there) cannot resume a conversation after you quit. opencode and
claude keep their session data in a fixed user dir, so they already resume.
Add an opt-in --resume/--no-resume flag: it routes the launch to the stable
Unsloth agents dir (the same one --no-launch already uses) so the session
survives the exit, never touching the user's own ~/.<agent>. A bare --resume
also reopens the last conversation via the agent's native flag (codex
`resume --last`, opencode/claude/pi `--continue`). The default is unchanged:
a plain launch still uses a temp dir and persists nothing.
Add a dispatch-only `resume` job to the Local Agent Guides CI that drives the
real launch path and asserts the split: codex/pi are wiped without --resume
and persist with it, while opencode/claude persist either way.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* unsloth start: rename --resume to --persist
The session flag collided with agents' own resume flags. `unsloth start
claude --resume <id>` used to forward `--resume <id>` straight to Claude
(which keeps its history in ~/.claude regardless), so a boolean --resume on
unsloth start would have swallowed the session id and turned it into a stray
prompt. Name the persistence flag --persist instead, so every agent's native
resume flag (claude --resume <id>, codex resume, opencode --continue, ...)
still passes through untouched. Behavior is otherwise identical: --persist
keeps a launched agent's session under the Unsloth agents dir, and a bare
--persist reopens the last conversation.
Add a regression test that `--resume <id>` passes through verbatim, and in the
CI resume experiment skip the redundant second pass for opencode/claude (they
persist either way, and a second CPU turn only risks a timeout).
* unsloth start: correct --persist help and drop the buggy auto-resume
Reword the --persist help to be accurate: claude and opencode keep sessions in
the user's own stores and resume regardless, so --persist only stabilizes the
otherwise-ephemeral relocated home of codex/openclaw/hermes/pi. Drop the
bare-launch auto-append of native resume tokens: it errored on a first launch
with no prior session, and was inconsistent between launch and no-launch.
--persist now only keeps the session dir; resume via the agent's own command
(e.g. `unsloth start codex --persist resume`), which now finds it.
In the CI resume experiment, fail the pass when the launched turn exits
non-zero, so a write-then-error is not misread as PERSISTED.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Silence torch._check_is_size FutureWarning and shim it if torch removes it
bitsandbytes 4-bit dequant calls torch._check_is_size, which torch
deprecated with a FutureWarning ("Use _check(i >= 0) instead") that prints
on every bnb-4bit load. Silence that warning in suppress_cuda_printf, and
add fix_torch_check_is_size so a future torch that removes _check_is_size
gets it shimmed to _check(i >= 0) (honoring the max bound) and bitsandbytes
keeps working.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten fix_torch_check_is_size docstring
Lead with what the shim does and drop the redundant line; two lines
instead of three, same intent.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Mirror dynamo/inductor config sets into defaults so torch 2.12 worker threads honor them
torch 2.12 stores config user overrides in ContextVars, so direct
assignments like torch._dynamo.config.recompile_limit = 1024 no longer
reach the autograd engine worker threads. Gradient checkpointing
recomputes fullgraph-compiled gpt-oss kernels inside backward on those
threads, which then read the default recompile limit of 8 and raise
FailOnRecompileLimitHit at step 0 of GRPO/SFT. Mirror direct config
assignments into the process-global entry defaults on torch >= 2.12,
restoring the torch <= 2.11 cross-thread semantics while leaving the
context-scoped config.patch API untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep config.patch thread-local when mirroring dynamo/inductor sets
config.patch(...) also assigns through ConfigModule.__setattr__, so the
default-mirror was leaking its scoped, thread-local writes into the
process-global entry default. Track patch enter/exit with a per-thread
depth counter (wrapping ConfigModule.patch) and skip mirroring while
inside a patch, so only genuine direct assignments restore the torch
2.11 cross-thread semantics and config.patch stays context-local.
* Also keep config.load_config thread-local when mirroring config sets
load_config restores a saved dynamo/inductor config by calling setattr
per key, which the default-mirror would otherwise leak process-wide just
like config.patch did. Wrap load_config with the same per-thread depth
counter (renamed to _scoped_depth) so both scoped writers skip the mirror
and stay context-local, while genuine direct assignments still restore the
torch 2.11 cross-thread default.
* Drop the pre-existing override replay from the config thread fix
The replay was redundant: this runs from _gpu_init before unsloth sets any
dynamo/inductor config, so the __setattr__ wrapper already mirrors every
later assignment (recompile_limit included). It could also read a value
that belonged to a config.patch context still active at import time and
write that thread-local override into the global default. Removing it keeps
the cross-thread fix and drops the now-unused _inductor.config import.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
When a coding agent is missing, `unsloth start <agent>` offers to run the
vendor's own installer (curl | bash, irm | iex, or npm) after an interactive
confirm. Those installers execute with the user's privileges and there is no
signature or hash check on the fetched content, so a blind "yes" is a
supply-chain risk if the delivery path is compromised.
Keep the auto-install convenience but make consent informed: before the prompt,
name the exact remote source the installer fetches (or the command it runs for a
package installer) and state that nothing verifies a signature or hash. Behavior
is otherwise unchanged: non-interactive stdin still never executes anything, and
the confirm still defaults to no.
* Fix FastSentenceTransformer Qwen embedding preprocessing
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Document Transformer.load embedding modality fix for #6881
* Harden #6881 fix and add forwards/backwards-compatible regression tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fall back to Transformer constructor on legacy sentence-transformers without Hub-capable load
* Mirror legacy sentence-transformers fallback in embedding-parity tripwire test
* Tighten #6881 comments and docstrings
* Skip embedding-parity test on CPU-only runners since FastSentenceTransformer requires CUDA
* Honor the transformer module's saved subfolder when loading
modules.json records a path for the Transformer module (root for
decoder embedders like Qwen3-Embedding, 0_Transformer for the classic
layout). Pooling/Normalize already load from their saved path; thread the
same path into Transformer.load as subfolder so config and tokenizer
resolve like stock ST. stays a no-op, so single-module models are
unchanged.
* Make embedding-parity test bf16-aware
fp16 overflows to NaN on bf16-native embedders such as EmbeddingGemma
(Gemma3), producing a false parity failure. Prefer bf16 when the GPU
supports it so the tripwire can guard the full documented embedding
matrix (Qwen3-Embedding, EmbeddingGemma, BGE-M3, all-MiniLM, GTE-ModernBERT),
not just fp16-safe models.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Retry the Studio UI shutdown re-login on transient goto timeout
The Chat UI Playwright smoke intermittently failed at the pre-shutdown
re-login: page.goto('/login') can hit a 60s TimeoutError on a slow runner
even while the server is healthy, and the surrounding except only tolerated
ERR_ABORTED / interrupted-navigation, so a plain timeout hard-failed the job.
Wrap the re-login goto/wait/fill/submit in the same 3-attempt retry the
change-password step already uses (recover_or_replace_page between tries,
per-attempt fail screenshots, wait_for_health pre-gate). The composer wait
stays outside the loop so a retry never re-navigates after login has set
tokens (which would redirect to /chat via the guest guard); it remains the
authoritative confirmation, so a genuinely broken login still fails.
* Catch transient login-request failures and preserve error listeners on recovery
Wait on the /api/auth/login POST inside the retry (via click_and_wait_for_response)
so a transient 4xx/5xx is retried in-loop instead of surfacing only at the
out-of-loop composer wait, matching the change-password step. When
recover_or_replace_page swaps in a fresh page, re-attach the pageerror/console
listeners so error tracking survives the replacement.
* Stabilize floating monitor drag
* Restore floating monitor exit animation
* Harden Windows Studio smoke checks
* Keep API menu badge removed
* Apply no-build-tools env overrides in-script
The runner does not apply step-level env keys containing parentheses,
so ProgramFiles(x86) kept its real value and Find-VsBuildTools still
detected VS through vswhere. Set the overrides inside each pwsh step
instead; child processes inherit them. The resolver step moves to pwsh
because bash cannot export a variable named ProgramFiles(x86).
* Reset chat UI session without a second browser context
macOS runs Chromium with --single-process, where closing the last
context tears down the whole browser, so the shutdown re-login died
with TargetClosedError on new_page. Clear cookies and swap pages
inside the same context instead, opening the replacement page before
closing the old one.
* Keep the no-build-tools Path filtered across session refreshes
install.ps1's Refresh-SessionPath and setup.ps1's Refresh-Environment
rebuild the session Path from the Machine and User registry scopes, so
the process-level filter could be undone mid-install and re-expose
CMake. Filter those scopes in the Prepare step with normalized dir
matching and restore them in cleanup.
* Drop stale localStorage auth tokens before re-login
Auth tokens live in localStorage, not cookies, and the login guest
guard redirects on their mere presence. Remove them during the session
reset so the /login navigation is deterministic instead of relying on
the tolerated redirect bounce.
* unstructured block removal
* Enhance unstructured block handling
* Restrict block cleanup to upload UIDs
* cleanup for seed block uploads
* upload cleanup queue for unstructured blocks in recipe studio
* Fix unstructured upload cleanup edge cases
* Fix unstructured upload import ownership
* Fix-unstructured-import-path-ownership
* Guard failed-delete restore against stale block in unstructured drop zone
* Drain queued upload cleanups when autosave is skipped
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
