PageTable reserves batch_idx=0 as a no-op slot, so passing
``max_batch_size=N`` to ``PageTable.__init__`` yields only N-1
user-allocatable slots. When a user requests bs=N concurrent sequences,
the Nth sequence gets stranded in the ``waiting`` queue until the first
N-1 sequences finish, then runs serially. For bs=8 × 64 tokens this
split the decode into 63 iterations at B=7 followed by 63 at B=1 —
throughput collapsed from a predicted ~800 tok/s to 143 tok/s.
Fix: bump the internal PageTable capacity by 1 so the user's
max_batch_size maps to that many concurrent slots. All batch-indexed
buffers (input_pos_buffer, block_mask_logical, _linear_conv_states,
_linear_recurrent_states, _linear_caches) size accordingly. Slot 0
remains internally reserved; user-facing ``self.max_batch_size``
unchanged.
Throughput (Qwen3.5-4B, 128 new tokens, post-warmup):
| bs | pre-fix tok/s | post-fix tok/s | change |
|---:|---:|---:|---:|
| 1 | 191.2 | 191.5 | same |
| 2 | 338.1 | 339.4 | same |
| 4 | 548.6 | 542.0 | same |
| 8 | 143.5 | **804.7** | **5.6x** |
MoE Qwen3.6-35B-A3B bs=8 × 64: 38.2 → **305.2 tok/s** (8x).
Parity at bs=4 × 16 tokens: was 3/4 exact (P1 matched 13/16), now 4/4
exact — the serialization also exposed an extra bf16-drift path.
Add static per-layer conv/recurrent state buffers for the Gated DeltaNet
sub-layers and a batched decode path that gather/scatters through them.
The step is captured into a CUDA graph per bucket size; generate()
replays the matching graph when the active batch size exactly matches
a captured bucket, else falls back to the eager batched path.
Throughput (Qwen3.5-4B, 128 new tokens, post-warmup, B200):
| bs | pre-capture tok/s | post-capture tok/s | speedup |
|---:|---:|---:|---:|
| 1 | 20.1 | 191.2 | 9.5x |
| 2 | 20.5 | 338.1 | 16x (extrapolated) |
| 4 | 20.9 | 548.6 | 26x |
bs=8 is currently a regression (143 tok/s vs 549 at bs=4) — likely FLA
kernel sub-linear scaling in B. Tracked as follow-up.
Parity (flex-capture vs flex-eager-batched, bs=4 x 16 tokens greedy):
3/4 prompts token-exact, 1 prompt matches 13/16 tokens (bf16 drift).
Extend the flex inference engine to cover Qwen3.5 / Qwen3.6 hybrid
linear-attention architectures. 75% of layers are Gated DeltaNet
(linear_attention) and 25% are standard full_attention; Flex Attention
only expresses softmax attention so the engine dispatches per layer:
- linear_attention layers: HF's Qwen3_5GatedDeltaNet.forward runs as-is.
With flash-linear-attention installed it dispatches to FLA's Triton
kernels (``chunk_gated_delta_rule`` for prefill,
``fused_recurrent_gated_delta_rule`` for decode). Per-layer
conv_state + recurrent_state live in a DynamicCache seeded from
config.layer_types.
- full_attention layers: the standard ``Qwen3_5Attention.forward`` is
swapped for a flex_attention + PageTable forward (same pattern as
flex_qwen3_llama). Handles attn_output_gate (q_proj outputs 2x dim,
chunk into query + gate, attn_output *= sigmoid(gate)), partial
rotary (factor=0.25), per-head Q/K RMSNorm before RoPE.
New file: unsloth/inference/flex_qwen3_5.py defines
``FlexQwen3_5Inference`` + per-layer walker + patched attention
forward. Wired into flex_engine._detect_arch (new ``qwen3_5`` /
``qwen3_5_moe`` branches gated before plain ``qwen3``) and the
dispatch map. Added to the bind_peft_model MoE shortcut so the
double-copy LoRA pattern works on the MoE variant.
vision.py: ``qwen3_5`` / ``qwen3_5_moe`` added to the flex allowlist
for fast_inference=True + UNSLOTH_FAST_INFERENCE=1.
Smoke: Qwen/Qwen3.5-4B (dense, 4.6B, 32 layers / 8 full-attn) and
Qwen/Qwen3.6-35B-A3B (MoE, 35B, 40 layers / 10 full-attn / 256 experts
top-8) both load and generate coherent text on 3 chat prompts × 16
tokens. Outputs match the vLLM and HF-naive paths in style
("Thinking Process:..." instruct-tuned output); P0 / P2 bit-match HF
naive at 16 tokens on the 4B, P1 diverges after ~9 tokens (typical
bf16 greedy drift on high-entropy prompts).
Throughput (B200, bf16, 64 new tokens, with one warmup pass):
* Qwen3.5-4B flex bs=1: 20.1 tok/s bs=4: 20.9 tok/s
(vs vLLM 22.8 / 23.0 — within 10% on equivalent greedy decode,
eager on both)
Follow-ups:
* CUDA graph capture for the decode step — currently deferred because
the per-seq DeltaNet FLA call makes naive graph capture non-trivial.
Would need batched FLA kernels or per-graph-slot state management.
* Packed-batch prefill — current impl runs one prefill per incoming
request. Multi-prompt batching is possible by splitting the
DeltaNet call into per-seq chunks but doesn't parallelise cleanly.
* torch.compile on the walker.
Add qwen3_5 (Qwen3.6-27B and 4B dense family) and qwen3_5_moe
(Qwen3.6-35B-A3B MoE family) to VLLM_SUPPORTED_VLM.
Also force enforce_eager=True + compilation_config=0 for these archs.
Same FX splitter bug as gemma4 — vLLM's -O3 inductor compile hits
`_decompose_size_nodes "Tried to erase Node size_N but it still had N
users"` on the hybrid DeltaNet / full-attention forward graph. Eager
preserves correctness at a small throughput cost; revert when the
upstream splitter handles these shape-dependent erasures.
Pair with unsloth-zoo patch_qwen3_5_vllm_lora_support which wires
get_expert_mapping / supports_lora onto the multimodal wrapper.
Throughput harness for the flex engine (UNSLOTH_FAST_INFERENCE=1) across
a batch-size sweep. Captures CUDA graphs per bucket and reports
post-warmup tok/s. Pairs with tests/gemma4_fast_inference_parity.py
(HF-naive parity) and tests/gemma4_fast_bench.py (vLLM nightly path) so
the three engines can be compared on the same prompt pool and new-token
budget.
Measured on B200 bf16 greedy, 64 new tokens, bs=16:
E2B 645 tok/s (28x HF naive 22.9)
E4B 492 tok/s (25x HF naive 19.7)
31B 198 tok/s (15x HF naive 13.5)
26B-A4B 391 tok/s (24x HF naive 16.2, 5.7x vLLM nightly 68.1)
Regression test covering all four Gemma 4 variants (E2B / E4B dense, 31B
dense, 26B-A4B MoE) through FastLanguageModel(fast_inference=True). Runs
HF naive first, frees the GPU, then runs vLLM — loading vLLM first in the
same process leaves global state that perturbs the subsequent HF run's
numerics, producing false divergences. HF-first ordering gives bitwise
matches on all variants:
E2B 62/62
E4B 62/62
31B 29/29
26B-A4B MoE 34/34
Exits non-zero on any divergence so CI can gate on bitwise parity.
Requires vLLM nightly (>= 2026-04-17 for vllm#39291 Gemma 4 LoRA) and the
unsloth-zoo#603 vLLM Gemma 4 runtime patches.
Add "gemma4" to VLLM_SUPPORTED_VLM so FastLanguageModel(fast_inference=True)
stops rejecting unsloth/gemma-4-E2B-it / gemma-4-26b-a4b-it at the
vision-model arch gate. vLLM nightly (vllm#39291, merged 2026-04-17)
registers Gemma4ForConditionalGeneration as SupportsLoRA; the
unsloth-zoo#603 vLLM patches (patch_gemma4_vllm_lora_support +
patch_gemma4_vllm_k_eq_v_support) wire LoRA + attention_k_eq_v support
on top of that.
Also force compilation_config=0 and enforce_eager=True for any gemma4
variant. vLLM's -O3 backend currently hits "Tried to erase Node size_N
but it still had N users" in _decompose_size_nodes when splitting the
Gemma 4 FX graph (both dense E2B audio path and MoE 26B-A4B language
path). Until the upstream FX splitter grows support for these size()
users, eager execution preserves correctness — bitwise token parity
with HF naive confirmed at 25/25, 4/4, 5/5 on 3 chat prompts × 32
tokens.
Extends flex fast-inference to `unsloth/gemma-4-26B-A4B-it` (30 layers,
128 experts top-k 8, H=2816, ~3.8B active of 25.2B). Mirrors the
FlexGptOssInference / FlexMoEInference template with Gemma 4-specific
wiring:
- Dual dense MLP + MoE per decoder layer (Gemma4TextMLP alongside
Gemma4TextExperts; outputs summed before the residual add, then
multiplied by the per-layer `layer_scalar` buffer).
- Per-layer sliding-window dispatch (25 sliding @ 1024 tokens, 5 full
attention) via twin BlockMask built once per generate() entry.
- Two-tier RoPE: sliding layers use rope_theta=10K with full head_dim
rotation; full-attn layers use rope_type=proportional with theta=1M
and partial_rotary_factor=0.25 (the inv_freq's zero-padded tail makes
the generic rotate_half a no-op on the unrotated dims, so a single
rotary helper covers both).
- Per-head Q/K/V RMSNorm applied before RoPE / KV write.
- attention_k_eq_v=True on full-attn layers (v_proj is None): value is
the raw k_proj output, followed only by v_norm (with_scale=False).
- Rebind Gemma4TextExperts.forward to forward_native_grouped_mm so
decode uses the grouped_mm backend (the slow Python loop in the stock
forward is neither fast nor CUDA-graph-capturable). The routing
weights already include per_expert_scale via Gemma4TextRouter.forward,
so no extra folding is needed.
- CUDA graph capture and UNSLOTH_FLEX_COMPILE_WALKER=1 inherit from the
MoE template (single bucket ladder, single pool across buckets).
Arch detection:
- `_detect_arch` distinguishes dense vs MoE Gemma 4 via
`text_config.num_experts > 1` (same class name covers both variants).
- `bind_peft_model` extends the MoE no-deepcopy shortcut to gemma4_moe.
Validation (B200, bf16, 3 chat prompts, 64 tokens):
- Dense 31B sanity check: coherent completion via existing
FlexGemma4Inference.
- Flex (cudagraph) vs HF naive: 24/24, 2/2, 4/4 tokens bitwise match.
- Merge parity: 7/7 cases bitwise (rank 16 / 64, 1 + 2 adapters, bf16
+ fp32, E=128, 2I=1408, H=2816 / H, I=704).
- Throughput bs=8/16/32/48: 510 / 1030 / 1564 / 916 tok/s vs HF naive
134 tok/s at bs=8 (3.8x-11.7x).
- GRPO smoke: DAPO-Math-17k, seed 3407, max_steps=5 — stable (see
follow-up comment on PR).
Out of scope: E2B/E4B KV-shared + per-layer-input variants (guarded
with NotImplementedError); bnb-4bit stacked experts (no such class
ships for Gemma 4 today).
Adds three things for a complete exactness audit:
- ``--backend flex_eager`` — monkeypatches capture_decode_cudagraph to
a no-op so the engine takes the eager branch, same as we did in
flex_moe_parity.py. Lets us compare flex-capture vs flex-eager.
- ``--lora_path`` — attaches a LoRA to either the flex or HF path
before the greedy decode so we can check the merge
(refresh_moe_lora_merge_from_pristine) against HF's peft forward.
- Default ``--max_new_tokens`` bumped 24 → 64 for stricter matching.
Result summary (B200, unsloth/gpt-oss-20b-BF16, 3 chat prompts):
- flex (capture) vs HF: 64/64 on all 3 prompts.
- flex eager vs HF: 41/64 + 64/64 + 64/64 (bf16 tile-selection drift
on prompt 0, not a bug; cudagraph capture happens to pick the same
tile as HF eager).
- flex (capture) + LoRA vs HF + LoRA: 64/64 + 48/64 + 64/64.
LoRA used is rank-16 with B=0 random-init-A so the merge is
mathematically a no-op — tests the wrapper + merge plumbing without
changing semantics. The merge formula itself was bitwise-verified in
tests/flex_moe_merge_parity.py (commit 835b346) on both standard and
transposed orientations at Qwen3 MoE shapes; transposed is what
gpt-oss uses.
FlexGptOssInference mirrors FlexMoEInference with three arch-specific
pieces:
1. Attention sinks via flex_attention(return_lse=True) + post-softmax
sigmoid(lse - sinks[h]) scaling. Same math as
unsloth_zoo.flex_attention.attention_sink.
2. Per-layer sliding window: the walker passes both a full and a
sliding-128 BlockMask; each attention forward picks one based on
self.sliding_window.
3. gpt-oss rotary: first/second-half split with head_dim/2-sized
cos/sin, not the Llama-style rotate_half on full-dim.
Reuses:
- refresh_moe_lora_merge_from_pristine for stacked-expert LoRA merge
(transposed orientation branch covers gpt-oss's (E, H, 2I) layout)
- forward_native_grouped_mm — the GptOssExperts branch already handles
the interleaved gate/up split + gate * sigmoid(gate * 1.702) activation
bnb-4bit: GptOssExpertsBnb4bit uses an nn.ModuleList per-expert loop
that can't be CUDA-graph-captured. __init__ detects it and disables
capture; decode still benefits from paged KV + flex_attention.
Also fixes a pre-existing deepcopy recursion in _LazyFlexEngineSentinel
that surfaced when the flex inference deepcopy ran on a model still
holding the sentinel (vision.py path doesn't pre-seed the inference
copy).
Bench on B200 (unsloth/gpt-oss-20b-BF16, max_new_tokens=64, chat):
| bs | HF naive | flex walker |
|---:|---------:|------------:|
| 8 | 258.9 | 832.1 |
| 32 | — | 1595.3 |
| 48 | — | 2146.4 |
3.2x over HF naive at bs=8. Parity: 24/24 greedy-token match vs HF on
3 chat prompts; cudagraph capture replay bitwise-matches eager decode.
Chat-tuned models (gpt-oss) return gibberish on raw-string prompts. Added
an opt-in --chat_template flag that:
- vLLM bench: switches to llm.chat(messages, ...) with harmony-aware
templating through vLLM's chat_utils (raw apply_chat_template +
llm.generate still produced gibberish on gpt-oss bf16).
- HF naive bench: wraps each prompt in the tokenizer's chat template
before tokenization, matching what the model was trained on.
- --user_prompt {i} lets the per-prompt content vary for batched
throughput runs.
Also adds --enforce_eager to the vLLM bench as a diagnostic knob (to
A/B cudagraph vs eager decode paths when a model misbehaves).
Standalone test that verifies refresh_moe_lora_merge_from_pristine
produces the same W_inf as the textbook reference
``W_ref[e] = W_pristine[e] + scaling * B[e] @ A[e]`` for both standard
(E, 2I, H) and transposed (E, H, 2I) stacked expert layouts, single and
multi-adapter, fp32 + bf16. All 8 cases bitwise-match.
Also benchmarks the batched torch.baddbmm path against (a) a per-expert
torch.addmm loop of equivalent arithmetic and (b) the dense-layer
torch.addmm used by refresh_lora_merge_from_pristine. At Qwen3-30B-A3B
MoE shapes on B200: baddbmm is 5-9.6x faster than the addmm loop; its
per-expert amortized cost is 3.6-7.1x cheaper than the dense addmm
baseline — baddbmm amortizes kernel launch and grid setup across
E=128 experts in one kernel call.
Mirrors the flex_moe_bench workload using vllm.LLM directly (no unsloth).
Supports bf16, 4bit (bitsandbytes on-the-fly), and FP8 (via the -FP8
checkpoint variant). Adds --enable_lora + --lora_path for LoRA serving
comparison.
DeepGEMM isn't installed in this env and vLLM's FP8 warmup crashes the
engine-core subprocess without it, so the script sets VLLM_USE_DEEP_GEMM=0
before importing vllm.
VRAM is queried via nvidia-smi because vLLM's allocator bypasses
torch.cuda.memory_reserved.
Adds ``--compile_opts coord_descent`` that sets
``torch._inductor.config.coordinate_descent_tuning = True`` alone.
Tested; it regresses 4bit at bs=48 from 3383 → 2147 tok/s so
``coord_descent`` alone is not worth shipping. Keeping the option
in the bisection bench so the regression stays reproducible.
The default ``graph_bs = [1, 2, 4, 8] + range(16, max_bs+1, 16)``
matches the dense FlexInference pattern and is optimal for
power-of-2 batch sizes. For workloads that routinely hit
intermediate sizes (e.g. bs=24, 40, 56), capturing dedicated
buckets improves replay efficiency ~5-10% on those sizes
(they otherwise round up to the next power-of-2 bucket).
Measurements on Qwen3-30B-A3B 4bit, bs=24/40/56 with
compile_walker=True:
| bs | default ladder (round-up) | UNSLOTH_FLEX_GRAPH_BS=8,16,24,32,40,48,56,64 |
|----|------------------------------:|---------------------------------------------:|
| 24 | (rounds to 32, ~1800 tok/s) | 1836 tok/s |
| 40 | (rounds to 48, ~2750 tok/s) | 2772 tok/s |
| 56 | (rounds to 64, ~2700 tok/s) | 3797 tok/s |
The fine ladder costs a few extra capture calls at startup and
slightly larger CUDA graph pool footprint. Power-of-2 bucket
throughput is marginally lower with the fine ladder (memory pool
is shared across more captures), so it's opt-in via env var.
Invalid values fall back to the default ladder with a log.
Optional ``FlexMoEInference(compile_walker=True)`` / env var
``UNSLOTH_FLEX_COMPILE_WALKER=1`` wraps the decode walker
(``call_moe_model_with_flex_kwargs``) with
``torch.compile(fullgraph=False, dynamic=False)`` before the CUDA
graph capture kicks in. Inductor fuses the layernorm + residual +
router pointwise ops, and the compiled kernels end up recorded
inside the captured graph. Net: ~2x decode tok/s on the grouped_mm
path with no change in VRAM, no correctness regression, and no
user-facing API change unless the flag is set.
Numbers on Qwen3-30B-A3B-Instruct-2507, B200, 128 new tokens,
bs sweep, median of 2 timed rounds after 1 warmup:
| precision | bs | baseline (v2) | + compile_walker | speedup |
|-----------|---:|--------------:|-----------------:|--------:|
| 4bit | 16 | 699 | 1347.8 | 1.93x |
| 4bit | 32 | 1243 | 2423.9 | 1.95x |
| 4bit | 48 | 1735 | 3383.2 | 1.95x |
| 4bit | 64 | 1523 | 2981.9 | 1.96x |
| bf16 | 16 | — | 1401.1 | — |
| bf16 | 32 | — | 2864.0 | — |
| bf16 | 48 | — | **3911.1** | — |
Peak throughput: 3911 tok/s at bf16 bs=48 — 39x the pure-HF naive
baseline on the same workload (101.1 tok/s with
``AutoModelForCausalLM`` + eager attn, left-padded, no unsloth).
At 4bit bs=48, 51x the pure-HF naive baseline (66.7 tok/s).
GRPO 5-step validation (Qwen3_MoE_GRPO.py --backend flex
--max_steps 5 on DAPO-Math-17k):
| precision | baseline (v2) | + compile_walker | speedup |
|-----------|--------------:|-----------------:|--------:|
| 4bit | 548.3s | 434.5s | 1.26x |
| bf16 | 451.8s | **407.4s** | 1.10x |
Peak VRAM unchanged (130-133 GB). Loss / KL stable on both, no
NaN, rewards pegged at -7.5 (base-model artifact; orthogonal).
Parity (greedy 32 tokens × 3 prompts at bf16 and 4bit via
``FLEX_MOE_COMPILE_WALKER=1 tests/flex_moe_parity.py``):
flex-captured with the compile wrap matches flex-captured without
the compile wrap on 6/6 prompts with no gibberish, and matches pure
``transformers.AutoModelForCausalLM`` 32/32 on 5 of 6 (prompt ×
precision) pairs (the one divergence is a tie-break logit boundary
on an open-ended continuation — both coherent English).
Bisection of a few torch.compile flag sets against the default at
bs=32 4bit (max_batch_size=32):
| config | tok/s |
|------------------------------------------------------------|-------:|
| default (``torch.compile(fullgraph=False, dynamic=False)``)| 1581.5 |
| + max_autotune + coord_descent + aggressive_fusion | 1704.9 |
| + ``freezing=True`` | 935.7 |
``freezing=True`` is a regression on this path; shipping with the
default config only. The other flags are +7.8% at this size but
at large bs (48+) the max_autotune variant timed out during
compile (>40 min) so the default stays the ship-target for now.
Other attention backends don't help on B200 today:
- pure HF with ``attn_implementation="sdpa"``: cuDNN Frontend error
("No valid execution plans built") on sm_100 + torch 2.11.
- ``flash_attention_2`` 2.8.3: works, but kernels compiled for
sm_80/sm_90 only — slower than eager on B200 (46.7 / 67.9 tok/s
vs eager 66.7 / 101.1 at 4bit / bf16).
- ``flash_attention_3``: ``no kernel image for execution on the
device`` — sm_100 kernels not yet in flash_attn_interface.
- FA4 / ``flash_attention_4``: works standalone but transformers'
integration hard-codes ``flash_attn_with_kvcache = None`` for it,
so it can't service decode. Prefill-only, out of scope here.
New tests:
- ``tests/flex_moe_micro_bench.py``: tight probe that loads the
model once, sweeps batch sizes, prints a sample completion per
bucket (catches gibberish early). Supports ``--compile_mode
{off, walker, walker_fullgraph}`` and ``--compile_opts
{stock, unsloth_O3, inference_freeze}``.
- ``tests/flex_moe_bench.py``: add ``--backend hf_naive`` which
imports pure ``transformers`` (no ``import unsloth``) for the
reference HF baseline, with ``HF_ATTN_IMPL`` env var to switch
between eager / sdpa / flash_attention_{2,3,4}.
- ``tests/flex_moe_parity.py``: add ``FLEX_MOE_COMPILE_WALKER=1``
env var to exercise the compile wrap through the parity harness.
Qwen3-30B-A3B decode on bs=8/64-new-tokens jumps from 55.6 tok/s
(eager) to 374.3 tok/s at 4bit / 421.6 tok/s at bf16 — 4-7x over pure
Hugging Face generate (66.7 / 101.1 tok/s) on B200. Output is bit-exact
against eager decode on all 32 greedy tokens across 3 prompts.
FlexMoEInference.capture_decode_cudagraph now mirrors the dense
FlexInference capture pattern: reserve a scratch page per batch_idx
slot, allocate static input_ids / batch_idx / outputs buffers, capture
one graph per bucket in [1, 2, 4, 8] + range(16, max_bs+1, 16), share
the CUDA memory pool across buckets, erase scratch pages at the end.
_decode_step replays on subsequent calls: zero graph buffers (except
outputs), copy live input_ids / batch_idx into the pinned buffers,
graph.replay(), slice outputs. Matches flex_qwen3_llama.py:725-801
verbatim.
Capture is gated on select_moe_backend() == "grouped_mm". Other
backends (unsloth_triton, native_torch) print a warning and leave
cudagraph_captured = False so generate() silently falls back to eager.
The legacy NotImplementedError rationale — torch.where + Python
for-loop over experts — only applied to native_torch; grouped_mm's
bincount + cumsum + argsort + torch._grouped_mm + index_add_ path is
fully capture-friendly on H100/B200.
FlexEngine.__init__ drops the blanket capture_cudagraph=False force
for qwen3_moe arch since capture correctness is now backend-aware.
tests/flex_moe_bench.py gains a hf_naive backend that imports pure
transformers (no unsloth patches) for a clean "fast inference vs naive
HF" comparison: flex 374.3 vs 66.7 tok/s at 4bit = 5.6x, flex 421.6
vs 101.1 tok/s at bf16 = 4.2x.
tests/flex_moe_parity.py adds greedy-decode parity across three
prompts with flex-captured / flex-eager / HF-pure. Results at bf16:
flex-captured matches flex-eager 32/32 on all three prompts (CUDA
graph is numerically identical to eager), and matches HF pure
transformers 32/32 on 2/3 prompts with the remaining divergence at a
tie-break logit boundary.
Pairs with an unsloth-zoo PR dropping @torch.compiler.disable on the
sparse MoE block and replacing torch.bincount with a capture-safe
scatter_add_ — without those, capture trips on a CPU→CUDA scalar copy
inside bincount(minlength=python_int).
Adds tests/flex_moe_bench.py: 2-round median decode throughput bench
comparing flex (FlexMoEInference) and HF generate on the same
(n_prompts, max_new_tokens, precision) workload. Writes
async_task_outputs/qwen3_moe_grpo_bench/bench_decode_{backend}_{precision}.json.
Also defensively unpacks self.mlp(hidden_states) in
Qwen3MoeDecoderLayer_fast_forward's training branch:
unsloth_zoo.temporary_patches.qwen3_moe.sparse_moe_block_forward
returns a plain tensor for transformers 5.x stacked experts, but the
decoder wrapper unpacked a 2-tuple. The inference branch was already
fixed in the previous commit; the training branch hit the same
ValueError under plain HF generate (no _flag_for_generation).
Bench numbers (Qwen3-30B-A3B, 4bit, rank 16 LoRA, bs=8, 64 new tokens,
B200):
| backend | median tok/s | peak VRAM (GB) | median wall (s) |
|---------|--------------|----------------|------------------|
| HF | 80.5 | 57.2 | 6.36 |
| Flex | 55.6 | 116.0 | 9.21 |
Flex is correctness-complete but not yet performance-competitive on
MoE decode at bs=8. Two structural reasons:
- MoE decode runs eager (forward_moe_backend uses bincount + Python
expert loops which are not CUDA-graph capturable), so flex loses
its main dense-model advantage.
- FlexEngine deep-copies the HF model for the rollout copy, doubling
weight residency. For Qwen3-30B-A3B at bf16 that is ~60 GB extra.
The pristine-base third copy is skipped for Qwen3 MoE (see the
first commit of this series) but the inference deep-copy remains.
Follow-ups (not blockers for correctness):
- torch.compile(dynamic=True) on call_moe_model_with_flex_kwargs to
recover some of the CUDA-graph throughput without requiring graph
capture.
- Evaluate flex's scaling vs HF generate at bs=32 / bs=64, where
paged-KV reuse should dominate per-prompt cost.
- A quantised-only inference copy (4bit forward, fp32 LoRA injection)
so the flex path fits inside 2x 4bit weight residency (~34 GB)
instead of the current post-dequantisation footprint.
Four pre-existing integration blockers hit while wiring Qwen3 30B A3B
MoE + GRPO end-to-end. All four must be fixed for the smoke to run; none
depend on the FlexMoEInference work itself but they surface because
fast_inference=True is the first path where the full MoE + GRPO chain
gets exercised on current transformers 5.x.
1. unsloth/import_fixes.py + unsloth/__init__.py: port fix_trl_vllm_ascend
from #5129 onto this branch so `from trl import GRPOConfig,
GRPOTrainer` works without installing vllm_ascend. transformers 4.48
changed _is_package_available to a tuple, and TRL's module-level
_*_available caches remain truthy on "not installed" hosts, which
then triggers an unconditional `import vllm_ascend` on GRPO import.
2. unsloth/models/qwen3_moe.py FastQwen3MoeModel.pre_patch: do NOT
overwrite Qwen3MoeSparseMoeBlock.forward with the legacy
Qwen3MoeSparseMoeBlock_fast_forward. That fast_forward expects a
flat self.gate_proj attribute which no longer exists on transformers
5.x stacked-expert MoE blocks (self.gate + self.experts).
unsloth_zoo's patch_qwen3_moe installs the correct
sparse_moe_block_forward at TEMPORARY_PATCHES init time; the
override here stomps on that with a broken function and causes
AttributeError('gate_proj') during training.
3. unsloth/models/qwen3_moe.py Qwen3MoeDecoderLayer_fast_forward (inference
path): replace the direct call to Qwen3MoeSparseMoeBlock_fast_forward
with self.mlp(...) so the class-level (unsloth_zoo-patched) forward
runs instead of the broken legacy path. Unpack the (hidden_states,
router_logits) tuple defensively in case a downstream patch returns
a plain tensor.
4. unsloth/models/llama.py LlamaModel_fast_forward_inference_custom:
delegate MoE MLP to decoder_layer.mlp(X) and skip
mlp_fast_forward_inference when the block has no gate_proj /
up_proj / down_proj attribute. Without this, every
model.generate(...) call on a Qwen3 MoE model (e.g. TRL GRPO's
use_vllm=False rollout) crashes in fast_swiglu_inference trying to
read Qwen3MoeSparseMoeBlock.gate_proj.
Smoke-B parity (max_steps=20, 4bit, num_generations=2, seed 3407,
Qwen3-30B-A3B, DAPO-Math-17k, GPUs 2 and 3):
| metric | naive (fast_inference=False) | flex (fast_inference=True) |
|---------------------|------------------------------|----------------------------|
| train_runtime (s) | 2745.97 | 2746.63 |
| peak VRAM (GB) | 66.4 | 136.1 |
| loss step 1 / step 20 | 1192 / 4.65e-06 | 3.41 / 4.01e-06 |
| KL step 1 / step 20 | 1.19e6 / 0.00465 | 3.41e3 / 0.00401 |
Both runs exit cleanly, both converge KL from a huge initial spike
down to ~0.004 by step 20, both see identical reward saturation at
-7.5 (base model needs the SFT format-priming step the reference
Qwen3_(4B)-GRPO.ipynb does before GRPO; our smoke skipped that for
speed). Shapes track each other; absolute magnitudes differ because
the flex deep-copy is allocated from a different stream and the PEFT
adapter is initialised in a different order.
Four integration fixes wired up while bringing Qwen3-30B-A3B-Instruct-2507
green end-to-end on UNSLOTH_FAST_INFERENCE=1:
1. unsloth/models/llama.py patch_peft_model: transformers 5.x reports
model_type as "qwen3_moe" (with underscore); the PR's check was
"qwen3moe" and fell through to NotImplementedError.
2. unsloth/models/llama.py patch_peft_model dense MLP patching: the
fused gate/up/down LoRAMLP swap walks layer.mlp.gate_proj, which is
a Qwen3MoeSparseMoeBlock for MoE and has no gate_proj attribute.
Skip the swap when the MLP does not expose the dense trio; MoE
LoRA is wired through unsloth_zoo/moe_utils anyway.
3. unsloth/inference/flex_qwen3_llama.py flex attention forward:
bnb-4bit Linear compute produces fp32 k / v even under autocast,
which makes the paged KV index_put_ refuse the mixed dtype (bf16
cache, fp32 update). Cast k / v to self._paged_cache.k_cache.dtype
before update. Also benefits the dense path.
4. unsloth/inference/flex_engine.py bind_peft_model: for Qwen3 MoE the
ParamWrapper keeps LoRA un-merged on the stacked expert tensors, so
the training model's expert weights ARE the pristine source. Skip
the pristine-base deep-copy for arch=="qwen3_moe" and point
refresh_moe_lora_merge_from_pristine at the training base directly.
Avoids a third 30-60 GB residency on 30B-A3B.
5. unsloth/inference/flex_moe.py call_moe_model_with_flex_kwargs: lock
activations to the embed dtype across layernorm + MoE MLP; RMSNorm
+ bnb-4bit compute promote activations to fp32 along the MoE path
under autocast. Also force-restore Qwen3MoeSparseMoeBlock.forward
to the stock or unsloth_zoo version if FastQwen3MoeModel.pre_patch
clobbered it with a legacy Qwen3MoeSparseMoeBlock_fast_forward that
expects a flat self.gate_proj (which does not exist on transformers
5.x stacked-expert MoE blocks).
Adds tests/flex_moe_smoke.py: generates 32 tokens twice (cold + warm),
records first-call / warm-call tokens/s, peak VRAM, arch, impl. Writes
async_task_outputs/qwen3_moe_grpo_bench/smoke_A_{4bit,bf16}.json.
Measured on a single B200 (sm_100), Qwen3-30B-A3B-Instruct-2507 +
LoRA rank 16 + grouped_mm MoE backend:
| precision | t_load (s) | peak VRAM (GB) | cold tok/s | warm tok/s |
|-----------|------------|----------------|------------|------------|
| 4bit | 27.0 | 123.4 | 3.7 | 6.9 |
| bf16 | 33.5 | 125.9 | 3.7 | 6.7 |
Both completions coherent ("the lazy dog. ...").
FastLanguageModel.from_pretrained(model_name=\"unsloth/Qwen3-30B-A3B-...\",
fast_inference=True) was silently routing Qwen3MoeForCausalLM to the
dense FlexInference path because _detect_arch matched the \"qwen3\"
substring first. That path works for attention but drops all MoE LoRA
adapters at rollout time: refresh_lora_merge_from_pristine walks
named_modules for LoraLayer instances and calls
base_model.get_submodule(name).weight.data, which does not see the
stacked nn.Parameter tensors on Qwen3MoeExperts.gate_up_proj /
down_proj. Decode capture also breaks on bincount + Python expert
loops inside forward_moe_backend.
Adds unsloth/inference/flex_moe.py:
- call_moe_model_with_flex_kwargs: Qwen3 MoE decoder walker. Identical
to the dense walker for the attention half. Unpacks the mlp(...)
return for both stock HF (plain tensor) and Unsloth's patched
Qwen3MoeSparseMoeBlock_fast_forward (tuple of
(hidden_states, router_logits)).
- FlexMoEInference: API-compatible with FlexInference so the arch
dispatch is a one-line change. cudagraph_captured is permanently
False; capture_decode_cudagraph raises NotImplementedError so a stray
capture_cudagraph=True fails loudly rather than producing silently
wrong output.
- refresh_moe_lora_merge_from_pristine: batched torch.baddbmm LoRA
fuse over stacked 3D expert tensors. Handles both standard
(E, 2*I, H) and transposed (E, H, 2*I) orientations via a runtime
shape check against the flat lora_A / lora_B shapes. In-place write
so flex prefill and paged-KV replay see refreshed values.
Wires the new class into the engine:
- flex_engine._detect_arch: check \"qwen3moe\" / \"qwen3_moe\" BEFORE
the dense \"qwen3\" substring (Qwen3MoeForCausalLM contains both).
- flex_engine.FlexEngine.__init__: route arch==\"qwen3_moe\" to
FlexMoEInference and force self.capture_cudagraph = False so the
MoE expert loops are never captured.
- inference.__init__: export FlexMoEInference.
- models/loader.py: uncomment the qwen3_moe branch so FastQwen3MoeModel
applies training-side patches before FlexEngine wraps the model.
Flex attention / paged KV / block-mask / sampling-param shim / vLLM
shim / sleep-wake are reused verbatim.
Follow-up to 35231d4f (initial flex backend wiring).
Sleep/wake:
- sleep_mode.py adds kv_cache_pool / weight_pool context managers
backed by vLLM's CuMemAllocator. Activated when UNSLOTH_VLLM_STANDBY=1
and vLLM is importable; no-op otherwise so TRL's unconditional
sleep / wake_up calls stay valid.
- FlexEngine routes the inference deep-copy + per-layer PagedKVCache
through the pools. Captured CUDA graphs survive a sleep -> wake
round-trip because cuMem keeps the GPU virtual addresses stable.
- 4-bit single-copy path drops only the KV cache; level 2 warns and
falls back to level 1.
- tests/flex_sleep_mode_smoke.py covers sleep / wake memory deltas,
captured-graph survival, and the no-op path.
Lazy batch sizing:
- FlexEngine's max_batch_size drives fixed-shape page tables, the
input_pos_buffer, the block_mask_logical build, and the CUDA-graph
bucket list at __init__ time. There is no post-init resize, so
picking it at from_pretrained time forces over- or under-shoot.
- build_flex_engine() defers construction until the GRPO rollout shape
is known. install_flex_sentinel() attaches a _LazyFlexEngineSentinel
to model.vllm_engine so hasattr(model, "vllm_engine") keeps working
between from_pretrained and the first build; fast_generate triggers
a floor build on first call.
- rl.py injects _build_flex_from_args(model, args) before both
self.llm = model.vllm_engine rewrite sites (pre-TRL-0.18
sampling_params prefix and >=0.18 colocate LLM replacement). Sizes
the engine from max(pdbs * spg, pdbs * spg * ngen) derived from the
GRPO args. No-op on non-flex models, so the injection is safe for
every TRL backend.
- Precedence: user's max_batch_size kwarg is a floor; the GRPO target
overrides only when strictly larger, with a warning naming both.
- First-build only: post-build growth raises RuntimeError pointing the
user back to max_batch_size= in from_pretrained. The pristine
inference deep-copy is consumed on first build and gemma4 shell
extraction mutates its module tree, so a safe rebuild would require
a second deep-copy.
- tests/flex_lazy_batch_smoke.py (unit, stubbed FlexEngine) covers
default, GRPO bump, user-floor, and post-build-refused cases.
tests/flex_lazy_live_smoke.py exercises sentinel + build against a
live Qwen3-0.6B-Base.
All eight files introduced on this branch now carry the SPDX AGPLv3
header used by the MoE kernels. flex_paged_attention.py keeps its
BSD 3-Clause attribution to attention-gym alongside the new header.
Setting UNSLOTH_FAST_INFERENCE=1 makes
FastLanguageModel.from_pretrained(..., fast_inference=True) return the
flex attention + paged KV + CUDA-graph backend instead of vLLM for
Qwen3, Llama-3, and Gemma-4-E2B-it. Default "0" keeps the existing vLLM
path.
The flex engine lives in a new unsloth.inference subpackage so the
FastLanguageModel wiring and any external caller can reach it without
going through scripts/. The user-facing GRPO notebooks for Qwen3,
Llama-3, and Gemma-4 run end-to-end with no code edits, just the env
var.
New subpackage
- unsloth/inference/flex_paged_attention.py: paged KV cache and
block-mask helpers.
- unsloth/inference/flex_qwen3_llama.py: FlexInference for Qwen3 and
Llama-3.2. tokenize() skips pre-populated input_ids so the engine
can accept vLLM-style token-id prompts.
- unsloth/inference/flex_gemma4.py: FlexGemma4Inference. Text-shell
extraction from Gemma4ForConditionalGeneration, per-layer-type
sliding-window masks, KV sharing for layers 15-34.
- unsloth/inference/flex_engine.py: FlexEngine with the vLLM LLM
surface. Arch dispatch, deep-copy of the HF model for colocate
rollout so training forward stays intact, lazy pristine-base copy
on first bind_peft_model, torch.amp.autocast("cuda", dtype=...)
wrapping for the whole generate path, auto-tune for Triton block
sizes per GPU capability and head_dim band. FA4 default OFF
because the sm_100 FLASH backend crashes on short prompts; users
can opt in with fa4_prefill=True.
- unsloth/inference/vllm_shim.py: LoRARequest, RequestOutput,
CompletionOutput dataclasses plus save_lora / load_lora that
mirror unsloth_zoo.vllm_utils line for line so the TRL GRPO patch
sees the same attribute shape.
Wiring
- unsloth/models/loader.py: UNSLOTH_FAST_INFERENCE=1 bypasses the
vLLM import guard in FastLanguageModel.from_pretrained (L356) and
FastModel.from_pretrained (L982).
- unsloth/models/llama.py: the elif not fast_inference load branch
now also handles the flex case, pops max_batch_size from kwargs
(FlexEngine-only), snapshots the model into an inference copy
BEFORE Unsloth's post-patch so flex-attention patching lands on a
clean HF layout, and after the tokenizer loads constructs
FlexEngine and attaches .vllm_engine / .fast_generate /
.fast_generate_batches. FastLlamaModel.get_peft_model calls
engine.bind_peft_model(peft) right after patch_peft_fast_inference
so state_dict() reads the training LoRA tensors.
- unsloth/models/vision.py: same pattern for FastBaseModel. Gemma-4
is allowed through the vision-model gate when the flex backend is
selected; the engine extracts the text shell from
Gemma4ForConditionalGeneration internally.
- unsloth/models/_utils.py: patch_peft_fast_inference picks
unsloth.inference.vllm_shim.save_lora / load_lora when the engine
is a FlexEngine, so the flex path never imports
vllm.lora.request. fast_inference_setup skips patch_vllm() when
UNSLOTH_FAST_INFERENCE=1.
Smoke tests (B200 sm_100)
- Qwen3-4B-Base bf16 (no LoRA and with get_peft_model LoRA)
- Qwen3-4B-Base fp16
- Llama-3.2-3B-Instruct bf16
- gemma-4-E2B-it bf16
all load through FastLanguageModel + UNSLOTH_FAST_INFERENCE=1 and
generate coherent completions.
Batched steady-state throughput via FastLanguageModel (autocast ON,
n_prompts=8 max_new_tokens=64 max_batch_size=16 max_seq_length=1024):
| Model | Flex CLI | Integration | Delta |
|----------------------------|-----------|-------------|-------|
| Qwen3-4B-Base | 819 tok/s | 595 tok/s | -27% |
| Llama-3.2-3B-Instruct | 1163 | 1357 | +17% |
Qwen3's regression is outside the 10% budget; most likely cause is
q_norm / k_norm being promoted to fp32 under autocast. Tracked for
a follow-up; the integration is functional and GRPO-usable today.
Note
The flex engine source files live here in unsloth/inference/. The
CLI benchmark drivers in scripts/benchmarks/ land in a separate PR
(#5108) which also carries the Gemma-4 and Llama-3 regression runs.
This PR depends only on what is already on main.
Known follow-ups
- FA4 on sm_100 needs a more robust enable rule before it can be
default-on.
- Memory: 2x base-model VRAM without LoRA, 3x with LoRA.
- FlexEngine.sleep() / .wake_up() are no-op stubs; real CPU-offload
parity with UNSLOTH_VLLM_STANDBY is a separate PR.
* Studio: forward standard OpenAI tools / tool_choice on /v1/responses
Mirrors the /v1/chat/completions client-side tool pass-through from #5099
so clients (OpenAI Codex CLI, OpenAI Python SDK, ...) that target the
Responses API receive structured function_call output items instead of
plain text with tool-call tokens leaking into content.
- ResponsesRequest: type tools/tool_choice properly, add parallel_tool_calls;
accept function_call and function_call_output input items for multi-turn
- Translate flat Responses tool / tool_choice shape to the nested Chat
Completions shape before forwarding to llama-server
- _normalise_responses_input: map function_call_output -> role="tool",
function_call -> assistant tool_calls (preserving call_id)
- Non-streaming: map returned tool_calls -> top-level function_call
output items keyed by call_id
- Streaming: emit response.output_item.added (function_call),
response.function_call_arguments.delta/.done, and response.output_item.done
per tool call while keeping the text message at output_index 0
- Pytest coverage: tools/tool_choice translation, multi-turn input mapping,
non-streaming tool_calls mapping, response round-trip
* Studio: merge system messages and close inner stream on /v1/responses
Fixes two issues surfacing when OpenAI Codex CLI drives /v1/responses
against a GGUF with a strict chat template (gpt-oss harmony, Qwen3, ...).
1. "System message must be at the beginning" upstream errors
Codex sends `instructions` AND a `role:"developer"` message in `input`,
producing two separate system-role messages. Strict templates raise
when a second system message exists or when one appears after a user
turn. _normalise_responses_input now hoists all instructions / system /
developer content into a single merged system message at the top of
the Chat Completions message list.
2. "async generator ignored GeneratorExit" / "Attempted to exit cancel
scope in a different task"
_responses_stream consumed the inner chat-completions body_iterator
without an explicit aclose() in a finally block. On client disconnect
(Codex frequently cancels mid-stream), Python 3.13 finalized the inner
async generator on a different task, tripping anyio's cancel-scope
check. Mirrored the same try/finally + aclose pattern used by the
/v1/messages, /v1/chat/completions, and /v1/completions passthroughs.
Tests: hoisting of instructions + developer, developer mid-conversation,
multiple system messages in input, no-system passthrough.
* Studio: accept Codex multi-turn shapes and fix cross-task stream close on /v1/responses
Two issues observed driving /v1/responses from OpenAI Codex CLI against a
GGUF backend.
1. 422 on every turn after the first
Codex replays prior assistant turns with
`content:[{"type":"output_text","text":...,"annotations":[],"logprobs":[]}]`
and carries forward `reasoning` items (o-series / gpt-5) between turns.
Our `ResponsesContentPart` union only accepted input_text / input_image,
and `ResponsesInputItem` only message / function_call / function_call_output,
so Pydantic failed the whole list and FastAPI returned
`"Input should be a valid string"` against the `str` branch of the
outer union.
- Add `ResponsesOutputTextPart` for assistant-replay content.
- Add `ResponsesUnknownContentPart` and `ResponsesUnknownInputItem`
as permissive catch-alls (drop during normalisation).
- Wire an explicit `Discriminator` so dispatch is deterministic and
the fallthrough reaches the catch-all instead of misreporting via
the outer `Union[str, list[...]]`.
- `_normalise_responses_input` now accepts output_text parts, flattens
single-part assistant text to a plain string (keeps legacy chat
templates happy), and silently drops reasoning / unknown items.
2. "async generator ignored GeneratorExit" / cross-task cancel scope
`_responses_stream` awaited `openai_chat_completions` in the parent
route-handler task, which opens the httpx client for the inner
passthrough on *that* task. The outer `StreamingResponse` then iterates
in a child task, so the asyncgen GC finalises the inner httpcore byte
stream on the child task, tripping anyio's "Attempted to exit cancel
scope in a different task". Move the `await` inside `event_generator`
so the httpx lifecycle stays within the single streaming child task,
and surface any HTTPException as a `response.failed` SSE frame.
Tests: assistant output_text replay, reasoning-item tolerance, unknown
content-part tolerance, end-to-end Codex-shape payload (developer + user +
reasoning + function_call + function_call_output + assistant output_text +
user), and single-part assistant flattening to plain string.
* Studio: call llama-server directly from streaming /v1/responses
The previous fix (running the inner await inside event_generator) was not
enough. Wrapping the existing `openai_chat_completions` pass-through still
stacks two async generators: when the outer generator is closed, the
innermost `HTTP11ConnectionByteStream.__aiter__` in httpcore doesn't
receive GeneratorExit before Python's asyncgen GC finalises it in a
sibling task, tripping "Attempted to exit cancel scope in a different
task" and "async generator ignored GeneratorExit" — the same Python 3.13
+ httpcore 1.0.x interaction already seen in PRs #4956, #4981, #5099.
Cure both pass-throughs had: a single same-task httpx lifecycle with
explicit `aiter_lines().aclose()` BEFORE `resp.aclose()` / `client.aclose()`
in the generator's finally block.
Apply it at the Responses layer by dropping the wrapper entirely for GGUF:
open httpx, consume `resp.aiter_lines()`, parse `chat.completion.chunk`,
emit Responses SSE events, close everything in finally — all in the
single StreamingResponse child task. Non-GGUF streaming is rejected with
a 400 (wrapping the transformers backend would re-introduce the
double-layer pattern and isn't a Codex-compatible path today anyway).
Also surfaces upstream httpx.RequestError / non-200 as a
`response.failed` SSE frame rather than a dropped stream now that the
request is dispatched after SSE headers have gone out.
* Studio: silence benign httpcore asyncgen GC warnings on Python 3.13
The streaming pass-throughs (/v1/chat/completions, /v1/messages,
/v1/responses, /v1/completions) all use the proven #4981 / #5099 pattern
— single-task httpx lifecycle with explicit aiter_lines().aclose() ahead
of resp.aclose() / client.aclose() in the generator's finally block.
That handles our own iterators correctly.
The residual noise ("async generator ignored GeneratorExit" /
"Attempted to exit cancel scope in a different task") comes from an
innermost HTTP11ConnectionByteStream.__aiter__ that httpcore creates
internally inside its pool. We hold no reference to it, so we cannot
aclose it ourselves. Python 3.13's asyncgen GC hook finalises it on the
finaliser task, its aclose path enters an anyio CancelScope shield, and
Python flags the cross-task exit. The response has already been
delivered with a 200 by then — it is purely log noise, not a functional
failure. Same interaction seen in modelcontextprotocol/python-sdk #831,
agno #3556, chainlit #2361, langchain-mcp-adapters #254.
Install a targeted sys.unraisablehook that swallows this specific tuple
— RuntimeError mentioning "cancel scope" or "GeneratorExit" plus an
object repr referencing HTTP11ConnectionByteStream — and defers to the
default hook for every other unraisable. Idempotent; guarded by a
sentinel attribute so repeated imports don't stack filters.
* Chatbox, scroll, and menu fixes
- Fixed chatbox auto-expand height for multi-line text on the compare page
- Fixed chatbox UI to be consistent across compare and new chat
- Fixed scrolling being enabled on pages with no content, which also triggered the scroll-to-bottom button
- Fixed scroll-to-bottom button to only appear after scrolling up a reasonable amount instead of instantly
- Added shutdown studio button to the menu for easier access
- Fixed pop-up menu width to match the user button width
(cherry picked from commit cd4e390dfa84fe311fae79a781b96cc0ef5970a9)
* fix: correct compare scroll viewport and clean up chat composer UI polish
* Dark theme refactor and sidebar/chat UI refinements
- Complete refactoring of dark theme
- Replaced square rounded-corner user profile image with a circular bordered one
- Replaced user profile icon with 'U' initial and renamed label from 'Studio' to 'User'
- Chat bubbles now have a pointy top-right edge
- Sidebar menu tab line color selection is now consistent across all menus
- Tab-selection color animation now also applies to recent chats
- Removed 'Compare' menu autoselect when a compare chat conversation is selected
- Fixed UI consistency in Compare to match New Chat
- Removed sidebar animation and tab line, replaced with rounded selection for consistency
- Further adjustments to sidebar UI
- Further adjustments to compare chat UI
* Fixed sidebar collapse/expand for recent chats and recent runs not being clickable
* Chatbox, scroll, and menu fixes
- Fixed chatbox auto-expand height for multi-line text on the compare page
- Fixed chatbox UI to be consistent across compare and new chat
- Fixed scrolling being enabled on pages with no content, which also triggered the scroll-to-bottom button
- Fixed scroll-to-bottom button to only appear after scrolling up a reasonable amount instead of instantly
- Added shutdown studio button to the menu for easier access
- Fixed pop-up menu width to match the user button width
* Sidebar, fonts, and chat UI refinements
- Replaced logo PNG with real font text for 'unsloth' and 'BETA' label
- Added Hellix font and applied it across menus and UI elements
- Lighter scrollbar in the sidebar compared to other areas of the app
- Adjusted chat font and chat bubble styling
- Adjusted app menu design to stay consistent with the sidebar
- Adjusted text style for 'New Chat' and repositioned content/chatbox
- Adjusted model selector and top area UI
- Fixed footer text from 'LLM's' to 'LLMs'
- Fixed active selection border color incorrectly appearing on page refresh and during general navigation
- Logo now defaults to 'New Chat' when clicked
* Sidebar, model selector, and mobile UI fixes
- Further adjustments to sidebar UI and logo
- Changed right bar icon
- Model selector adjustments
- Collapsed sidebar now matches the content area background
- Adjusted Hellix font spacing across pages
- Fixed sidebar icon overlap on mobile screens
* Adjust sidebar icons
* Adjust sidebar icons
* Fixed compare chat UI and scrolling issues
* Fixed inference settings icon behavior and context info positioning
- Fixed top right inference settings icon to move into sidepanel during expand/collapse, matching left sidebar behavior
- Adjusted context information element positioning
* Fix: textarea overflow in system prompt editor
* Code block redesign, font, and chat bubble adjustments
- Redesigned code block colors and theme
- Changed code block font to Fira Code
- Fixed scrollbar disappearing when expanding/collapsing tool calls in chats
- Adjusted chat bubble background color
* Fix chat bubble background color in dark theme
* fix: restore textarea auto-sizing and scope prompt editor sizing
* fix: add explicit textarea field sizing for prompt editor overflow
* fix: generate chat nonce on click instead of render
* fix: respect training lock on logo navigation
* Refactor compare page dual chat scrolling behavior
* Revert "Refactor compare page dual chat scrolling behavior"
This reverts commit d056ec09f2.
---------
Co-authored-by: sneakr <hauzin@hotmail.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* export: update GGUF quant list and ordering
* gguf: add Q2_K_L quantize flags for output and embeddings
* export: add live console logs for LoRA export flow
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: stream q2_k_l quantize logs and include subprocess error details
* fix: route Q2_K_L preset to q2_k ftype with q8_0 output+embeddings
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Trashing a thread mid-stream used to delete the Dexie rows while the
model kept generating, because the sidebar has no access to the
@assistant-ui aui context. Expose per-thread cancelRun() through the
chat runtime store and call it from deleteChatItem so trash behaves
like Stop → Trash. Covers compare pairs by cancelling each paired
thread.
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* fix(studio): forward OpenAI tools/tool_choice to llama-server (#4999)
Studio's /v1/chat/completions silently stripped standard OpenAI `tools`
and `tool_choice` fields, so clients using standard function calling
(opencode, Claude Code, Cursor, Continue, ...) never got structured
tool_calls back. Adds a client-side pass-through path mirroring the
existing Anthropic /v1/messages flow: when `tools` is present without
Studio's `enable_tools` shorthand, the request is forwarded to
llama-server verbatim so the client sees native id, finish_reason
("tool_calls"), delta.tool_calls, and accurate usage tokens.
Also wires Anthropic tool_choice forwarding: /v1/messages previously
accepted tool_choice on the request model but silently dropped it with
a warning. Translate the four Anthropic shapes to OpenAI format and
forward them so agentic clients can actually enforce tool use.
- ChatCompletionRequest: add tools, tool_choice, stop; extra="allow"
- ChatMessage: accept role="tool", optional tool_call_id / tool_calls /
name; content is now optional (assistant with only tool_calls)
- routes/inference.py: _openai_passthrough_stream /
_openai_passthrough_non_streaming helpers, routing branch in
openai_chat_completions, vision+tools via content-parts injection
- _build_passthrough_payload: tool_choice parameter (default "auto")
- anthropic_compat: anthropic_tool_choice_to_openai() translator
- tests/test_openai_tool_passthrough.py: Pydantic + translator unit tests
- tests/test_studio_api.py: 5 new E2E tests (non-stream, stream,
multi-turn, OpenAI SDK, Anthropic tool_choice=any regression)
* fix(studio): surface httpx transport errors from OpenAI passthrough
When the managed llama-server subprocess crashes mid-request, the
async pass-through helpers in routes/inference.py used to return a
bare 500 (non-streaming) or an "An internal error occurred" SSE chunk
(streaming) because _friendly_error only recognized the sync path's
"Lost connection to llama-server" substring -- httpx transport
failures (ConnectError / ReadError / RemoteProtocolError /
ReadTimeout) stringify differently and fell through to the generic
case.
- _friendly_error: map any httpx.RequestError subclass to the same
"Lost connection to the model server" message the sync chat path
emits. Placed before the substring heuristics so the streaming path
automatically picks it up via its existing except Exception catch.
- _openai_passthrough_non_streaming: wrap the httpx.AsyncClient.post
in a try/except httpx.RequestError and re-raise as HTTPException
502 with the friendly detail.
- tests/test_openai_tool_passthrough.py: new TestFriendlyErrorHttpx
class pinning the mapping for ConnectError, ReadError,
RemoteProtocolError, ReadTimeout, and confirming non-httpx paths
(context-size heuristic, generic fallback) are unchanged.
* fix(studio): close aiter_bytes/aiter_lines explicitly in passthroughs
The httpcore asyncgen cleanup fix in 5cedd9a5 is incomplete on Python
3.13 + httpcore 1.0.x: it switched to manual client/response lifecycle
but still used anonymous `async for raw_line in resp.aiter_lines():`
patterns in all three streaming paths. Python's async for does NOT
auto-close the iterator on break/return, so the aiter_lines /
aiter_bytes async generator remains alive, reachable only from the
surrounding coroutine frame. Once `_stream()` returns the frame is
GC'd and the orphaned asyncgen is finalized on a LATER GC pass in a
DIFFERENT asyncio task, where httpcore's
HTTP11ConnectionByteStream.aclose() enters anyio.CancelScope.__exit__
with a mismatched task and prints "Exception ignored in: <async
generator>" / "async generator ignored GeneratorExit" / "Attempted
to exit cancel scope in a different task" to the server log.
User observed this on /v1/messages after successful (status 200)
requests, with the traceback pointing at HTTP11ConnectionByteStream
.__aiter__ / .aclose inside httpcore.
Fix: save resp.aiter_lines() / resp.aiter_bytes() as a variable and
explicitly `await iter.aclose()` in the finally block BEFORE
resp.aclose() / client.aclose(). This closes the asyncgen inside the
current task's event loop, so the internal httpcore byte stream is
cleaned up before Python's asyncgen GC hook has anything orphaned to
finalize. Each aclose is wrapped in try/except Exception so nested
anyio cleanup noise can't bubble out.
Applied to all three streaming passthrough paths:
- _anthropic_passthrough_stream (/v1/messages client-side tool path)
- _openai_passthrough_stream (/v1/chat/completions client-side tool
path, new in this PR)
- openai_completions (/v1/completions bytes proxy from PR #4956)
* fix(studio): default ChatCompletionRequest.stream to false per OpenAI spec
OpenAI's /v1/chat/completions spec defaults `stream` to false, so
clients that omit the field (naive curl, minimal integrations) expect
a single JSON response back. Studio was defaulting to true, silently
switching those clients into SSE and breaking any parser that didn't
also handle streaming. ResponsesRequest and AnthropicMessagesRequest
already default to false correctly; only ChatCompletionRequest was
wrong.
Studio's own frontend always sets `stream` explicitly on every
chat-adapter / chat-api / runtime-provider call site, so the flip has
no UI impact. SDK users (OpenAI Python/JS SDK, opencode, Claude Code,
Cursor, Continue) also always pass `stream` explicitly, so they're
unaffected. The only clients feeling the change are raw-curl users
who were relying on the wrong default -- those get the correct OpenAI
behavior now.
Added a regression test pinning the default so it can't silently
flip back.
* fix(studio): reject images in OpenAI tool passthrough for text-only GGUFs
The new tool passthrough branch runs before _extract_content_parts,
skipping the existing not is_vision guard. Requests combining tools
with an image on a text-only tool-capable GGUF were forwarded to
llama-server, producing opaque upstream errors instead of the
pre-existing clear 400. Restore the guard inline at the dispatch
point, checking both legacy image_base64 and inline image_url parts.
* fix(studio): require tool_call_id on role=tool chat messages
Enforce the OpenAI spec rule that role="tool" messages must carry a
tool_call_id. Without it, upstream backends cannot associate a tool
result with the assistant's prior tool_calls entry and the request
fails in non-obvious ways through the passthrough path. Reject at the
request boundary with a 422 instead.
* fix(studio): harden OpenAI tool passthrough validation and error surfacing
Three related fixes called out by the PR review:
1. Preserve upstream status codes in the streaming passthrough. The
httpx request is now dispatched before the StreamingResponse is
constructed. Non-200 upstream responses and httpx RequestError
transport failures raise HTTPException with the real status
instead of being buried inside a 200 SSE error frame, so OpenAI
SDK clients see APIError/BadRequestError/... as expected.
2. Require non-empty content on user/system/tool messages. Per the
OpenAI spec, content may only be omitted on assistant messages
that carry tool_calls; enforce that at the request boundary so
malformed messages never reach the passthrough path.
3. Role-constrain tool-call metadata. tool_calls is only valid on
role=assistant, tool_call_id and name only on role=tool. Without
this, a user/system message with tool_calls would flip the
passthrough branch on and be forwarded to llama-server, surfacing
as an opaque upstream error.
* fix(studio): normalize image mode and passthrough JSON verbatim
Two Gemini-code-assist review findings on PR #5099:
1. Unconditionally convert decoded images to RGB before PNG encoding.
The prior code only handled RGBA, letting CMYK/I/F images crash
at img.save(format="PNG") and surface as opaque 400s. Applied to
both the passthrough helper and the non-passthrough GGUF path
that originally carried this pattern, keeping the two sites in
sync.
2. Return the upstream JSON body as raw bytes via Response rather
than parse-then-re-serialize with JSONResponse. Matches the
passthrough helper's "verbatim" contract and drops a redundant
round-trip.
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* unsloth gemma4 support files
* some fixes
* Fixing cache.empty() calls (#4813)
* Fixing cache.empty() calls
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Manan Shah <mananshah@Manans-MacBook-Pro.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix/gemma4 mlx (#4816)
* Fixing cache.empty() calls
* fixing for mlx versions
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Manan Shah <mananshah@Manans-MacBook-Pro.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* removed bidirectional check for 31b (#4839)
Co-authored-by: Manan17 <shahmanan170602@gmail.coml>
* Add Gemma 4 26B MoE support (MLX) (#4844)
* removed bidirectional check for 31b
* Change gemma4_text for moe
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Manan Shah <mananshah@Manans-MacBook-Pro.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(gemma4): cast RoPE offset to int before mx.arange() (#4901)
* fix(gemma4): cast RoPE offset to int before mx.arange()
* fix(gemma4): use zero-based arange + offset to avoid CPU-GPU sync
* qwen3.6 patches for multi-turn chat
* qwen3.6 script
* removing unnecessary scripts
* displaying errors for not installed packages
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Manan Shah <mananshah@Manans-MacBook-Pro.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Manan17 <shahmanan170602@gmail.coml>
Co-authored-by: Théophile Lafargue <138336683+eauchs@users.noreply.github.com>
* Add Qwen3.6 inference defaults for Studio
Add qwen3.6 family entry to inference_defaults.json with the
recommended sampling parameters from Qwen's documentation:
temperature=0.7, top_p=0.8, top_k=20, min_p=0.0,
presence_penalty=1.5, repetition_penalty=1.0.
Without this, Qwen3.6 models fall through to the generic qwen3
pattern which uses different defaults (temperature=0.6,
top_p=0.95, no presence_penalty).
* Add Qwen3.6-35B-A3B-GGUF to default model lists
* Add Qwen3.5/3.6 presence_penalty to thinking toggle and small-model disable logic
- Thinking toggle (on-load + button click) now sets presencePenalty: 1.5 for
Qwen3.5 and Qwen3.6 models (both thinking-ON and thinking-OFF states)
- Small-model thinking-disable check (<9B defaults to no-thinking) extended
from Qwen3.5-only to also cover Qwen3.6, in all 3 locations:
frontend on-load, frontend refresh, backend llama_cpp.py
* fix: multi-GPU inference crash for bnb 4-bit/8-bit models
When load_in_4bit or load_in_8bit is used with device_map="sequential"
and max_memory constraints that place weights across multiple GPUs (or
entirely on a non-default GPU like cuda:1), the bitsandbytes loading
path in transformers never calls dispatch_model. No AlignDevicesHook is
installed, and the first forward/generate call crashes with:
RuntimeError: Expected all tensors to be on the same device
This adds _attach_bnb_multidevice_hooks() which is called after
from_pretrained returns. It infers a device map from actual parameter
placements and calls dispatch_model(force_hooks=True) to install the
missing hooks. The function is a complete no-op for the common
single-GPU cuda:0 case.
Call sites: FastBaseModel.from_pretrained (vision.py) and
FastLlamaModel.from_pretrained (llama.py).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: align with PR #5053 final review improvements
- Add hook call to the bnb quantized loading branch in llama.py (the
primary load_in_4bit path), not just the non-fast-inference fallback
- Expand bnb detection: also check model.is_loaded_in_4bit,
model.is_loaded_in_8bit, model.quantization_method
- Pass explicit main_device and skip_keys to dispatch_model
- Use logger.info instead of print for the success message
- Use kwargs.get("load_in_8bit", False) at llama.py call sites
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* auth: default to chat
* settings: relaunch onboarding
* onboarding: return to launch page
* studio: stop auto guided tour
* ui: soften global radius
* cleanup: rename onboarding exit prop
* fix onboarding redirect safety
* Show real Unsloth version in settings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat(studio): replace navbar navigation with collapsible sidebar
Add an app-wide sidebar with hover-expand and pin-to-dock behavior.
Navigation items (Studio, Recipes, Export, Chat) move from the center
pill navbar to the sidebar. Chat threads and recipes render as
collapsible sub-lists. Navbar simplified to logo + update + close.
- Extend SidebarProvider with pinned/hovered state model
- New AppSidebar with animated active indicator, sloth profile menu,
theme toggle, guided tour, back/forward navigation
- Chat page refactored to URL-driven view state via search params
- Extract reusable hooks for chat thread and recipe sidebar data
- Guard startViewTransition for browser compatibility
- Wrap chat deletions in Dexie transaction for data integrity
* feat(studio): move logo to sidebar and make navbar overlay
- Sidebar is now full-height with logo in SidebarHeader
- Collapsed sidebar shows sticker.png, expanded shows full logo
- Navbar is absolute-positioned overlay (no layout space)
- Main content extends to top, aligning with navbar controls
* feat(studio): full-height sidebar with recents, edge-to-edge nav buttons
- Sidebar outside max-w-7xl, pinned to left edge
- Remove sidebar rounding, menu buttons rounded-md
- Nav buttons flush to sidebar edges with no left rounding
- Replace collapsible recipes/chat with flat nav items
- Add Recents section with chat history (1 item when not on chat, full on chat)
- New Chat as first nav item with PencilEdit02Icon
- Cursor pointer on all sidebar buttons
- Navbar temporarily hidden for screenshots
* fix(studio): fix chat scroll, action bar hover, collapsible recents
- Fix sticky composer by removing `relative` override on viewport footer
- Action bar buttons only show on hover (autohide=always)
- Remove floating border/shadow from action bar
- Add scroll space above composer for last message actions
- Back/forward buttons use router history (stay in-app)
- Recents section collapsible with chevron on chat route
- Set html/body/#root height for proper h-full chain
* fix(studio): address review feedback, clean up unused code
- Unhide navbar (was left hidden from screenshot)
- Remove unused imports: SidebarMenuSub*, BubbleChatIcon, ColumnInsertIcon
- Remove unused vars: recipeItems, activeRecipeId, canCompare, recipesOpen
- Include compare query id in active sidebar selection
- Use store type for contextUsage instead of inline type
- Simplify noop in sidebar.tsx
- Remove empty className prop
* feat(studio): add mobile sidebar, recent runs section, and misc UX fixes
* feat(studio): scaffold settings feature module with dialog store
* feat(studio): add tri-state theme store for settings
* feat(chat): add clear-all-chats and export-chat-history utils
* feat(studio): add settings dialog shell with tab rail
* feat(studio): add appearance tab with theme and sidebar pin
* feat(studio): add settings general tab with hf token, auto-title, reset prefs
* feat(studio): add settings chat tab with export and clear
* feat(studio): add api keys tab with list and revoke flow
* feat(studio): add create-key form and reveal dialog
* feat(studio): add usage examples panel to api keys tab
* feat(studio): add settings about tab with update and shutdown
* feat(studio): add settings dropdown item and cmd-comma shortcut
* feat(studio): remove legacy api-keys route and chat-sheet preference rows
* fix(studio): settings dialog a11y + polish pass
* feat(studio): inline api key reveal card replacing nested dialog
* fix(studio): hide revoked keys from settings list
* refactor(studio): strip navbar and hoist training unload guard
* feat(studio): explicit sidebar toggle, remove hover-open and pin icons
* fix(studio): use SidebarRight01Icon for collapsed sidebar open toggle
* fix(studio): address code review findings for settings dialog
* feat(studio): collapsible navigate group with standalone new-chat and compare
* fix(studio): chat-only standalone actions, use ColumnInsertIcon for compare
* fix(studio): sidebar new-chat/compare state reset and icon-mode collapsible
* feat(studio): add compact logo assets for sidebar header
* Fixed sidebar design
* fix(studio): sidebar delete icon hover contrast and sizing
* feat(studio): route-gate sidebar recents (chats off /studio, runs on /studio)
* feat(studio): add chat search store
* feat(studio): add chat search index hook with snapshot-on-open
* feat(studio): add chat search command dialog with global shortcut
* feat(studio): wire chat search into sidebar
* fix(studio): trim hf token on save, add show/hide toggle, commit on close
* revert(studio): restore original sidebar/border colors, brighten sidebar
* feat(studio): forward overlayClassName through CommandDialog
* fix(studio): wrap search dialog in Command context, redesign as flat 635px card
* fix(studio): reserve right padding on recent items so delete icon stops overlapping title
* fix(studio): skip hf token unmount-commit during reset-prefs reload
* chore(studio): drop unused icon import and unreachable runs navigate branch
* fix(studio): chat search index filters archived before limit, batches message query, picks up reasoning text
* fix(studio): keep CommandEmpty in tree so empty state renders correctly
* fix(studio): cap system prompt and chat template textareas so they scroll instead of growing
* fix(studio): attach chat-compare tour anchor to sidebar compare button
* fix(studio): persist system theme explicitly so next-themes does not clobber on reload
* fix(studio): auto-switch to history tab when selecting a recent run from sidebar
* UI overhaul: chatbox, scrollbar, sidebar, and compare view
UI Changes:
- Redesigned the Compare UI with general cleanup
- Redesigned the Chatbox UI
- Reduced the width of the user chat bubble for improved readability
- Narrowed the user chat box across the content page
- Adjusted thinking-box text color to be slightly darker
- Removed faded text effect from chat messages
- Removed faded text effect from the thinking box
- Added a small LLM chat safety note at the bottom of the chatbox
- Restyled the scrollbar
Layout & Behavior:
- Reworked the scrollbar to span the full height of the page (no top/bottom padding) and remain persistently visible when content is scrollable, rather than only on hover
- Reworked the Configuration sidebar to span full height — removed rounded corners and borders, with the scrollbar adjusted to match the full top-to-bottom layout
- Adjusted the top menu and bottom chatbox content areas to work correctly with the new full-page scroll behavior
- Made chat content match the chatbox width, with content sliding slightly behind the chatbox when scrolling
- Aligned chat text width with the chatbox for visual consistency, including how far the text extends behind the chatbox
Fixes:
- Fixed the chatbox not auto-expanding when typing multi-line input while bottom-positioned during an active chat (previously only worked before a chat had started)
- Fixed positioning and design of the user chat hover menu buttons to match the assistant chat box — now displayed below the chat bubble instead of on the left side
* Fix user message layout in thread component
* swap code icon
* fix compare layout
* fix compare pane flex
* Sidebar improvements and fixes
- Added scrolling support to the sidebar so menus and recent chats no longer get hidden
- Recent chats are now always visible in the sidebar, not hidden when in Studio, Recipes, or Export
- Recent chat is now deselected when selecting other navigations
- Fixed sidebar glitch where browser resize could make the sidebar and expand button disappear completely
- Fixed glitch where the open-sidebar hover tooltip appeared above the logo when clicking expand sidebar
- Reduced sidebar width on mobile to around 2/3 of the screen (was too wide)
- Made the close-sidebar hover tooltip consistent with the rest of the design
- Removed sidebar collapse/expand animation
- Small adjustment to chat width
* Fix route scrolling, polling, and theme sync issues
* Fix Studio page scrolling
---------
Co-authored-by: sneakr <hauzin@hotmail.com>
* Studio: Ollama support, recommended folders, Custom Folders UX polish
Backend:
- Add _scan_ollama_dir that reads manifests/registry.ollama.ai/library/*
and creates .gguf symlinks under <ollama_dir>/.studio_links/ pointing
at the content-addressable blobs, so detect_gguf_model and llama-server
-m work unchanged for Ollama models
- Filter entries under .studio_links from the generic models/hf/lmstudio
scanners to avoid duplicate rows and leaked internal paths in the UI
- New GET /api/models/recommended-folders endpoint returning LM Studio
and Ollama model directories that currently exist on the machine
(OLLAMA_MODELS env var + standard paths, ~/.lmstudio/models, legacy
LM Studio cache), used by the Custom Folders quick-add chips
- detect_gguf_model now uses os.path.abspath instead of Path.resolve so
the readable symlink name is preserved as display_name (e.g.
qwen2.5-0.5b-Q4_K_M.gguf instead of sha256-abc...)
- llama-server failure with a path under .studio_links or .cache/ollama
surfaces a friendlier message ("Some Ollama models do not work with
llama.cpp. Try a different model, or use this model directly through
Ollama instead.") instead of the generic validation error
Frontend:
- ListLabel supports an optional leading icon and collapse toggle; used
for Downloaded (download icon), Custom Folders (folder icon), and
Recommended (star icon)
- Custom Folders header gets folder icon on the left, and +, search,
and chevron buttons on the right; chevron uses ml-auto so it aligns
with the Downloaded and Recommended chevrons
- New recommended folder chips render below the registered scan folders
when there are unregistered well-known paths; one click adds them as
a scan folder
- Custom folder rows that are direct .gguf files (Ollama symlinks) load
immediately via onSelect instead of opening the GGUF variant expander
(which is for repos containing multiple quants, not single files)
- When loading a direct .gguf file path, send max_seq_length = 0 so the
backend uses the model's native context instead of the 4096 chat
default (qwen2.5:0.5b now loads at 32768 instead of 4096)
- New listRecommendedFolders() helper on the chat API
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: log silent exceptions and support read-only Ollama dirs
Replace silent except blocks in _scan_ollama_dir and the
recommended-folders endpoint with narrower exception types plus debug
or warning logs, so failures are diagnosable without hiding signal.
Add _ollama_links_dir helper that falls back to a per-ollama-dir hashed
namespace under Studio's own cache (~/.unsloth/studio/cache/ollama_links)
when the Ollama models directory is read-only. Common for system installs
at /usr/share/ollama/.ollama/models and /var/lib/ollama/.ollama/models
where the Studio process has read but not write access. Previously the
scanner returned an empty list in that case and Ollama models would
silently not appear.
The fallback preserves the .gguf suffix on symlink names so
detect_gguf_model keeps recognising them. The prior "raw sha256 blob
path" fallback would have missed the suffix check and failed to load.
* Address review: detect mmproj next to symlink target for vision GGUFs
Codex P1 on model_config.py:1012: when detect_gguf_model returns the
symlink path (to preserve readable display names), detect_mmproj_file
searched the symlink's parent directory instead of the target's. For
vision GGUFs surfaced via Ollama's .studio_links/ -- where the weight
file is symlinked but any mmproj sidecar lives next to the real blob
-- mmproj was no longer detected, so the model was misclassified as
text-only and llama-server would start without --mmproj.
detect_mmproj_file now adds the resolved target's parent to the scan
order when path is a symlink. Direct (non-symlink) .gguf paths are
unchanged, so LM Studio and HF cache layouts keep working exactly as
before. Verified with a fake layout reproducing the bug plus a
regression check on a non-symlink LM Studio model.
* Address review: support all Ollama namespaces and vision projector layers
- Iterate over all directories under registry.ollama.ai/ instead of
hardcoding the "library" namespace. Custom namespaces like
"mradermacher/llama3" now get scanned and include the namespace
prefix in display names, model IDs, and symlink names to avoid
collisions.
- Create companion -mmproj.gguf symlinks for Ollama vision models
that have an "application/vnd.ollama.image.projector" layer, so
detect_mmproj_file can find the projector alongside the model.
- Extract symlink creation into _make_symlink helper to reduce
duplication between model and projector paths.
* Address review: move imports to top level and add scan limit
- Move hashlib and json imports to the top of the file (PEP 8).
- Remove inline `import json as _json` and `import hashlib` from
function bodies, use the top-level imports directly.
- Add `limit` parameter to `_scan_ollama_dir()` with early exit
when the threshold is reached.
- Pass `_MAX_MODELS_PER_FOLDER` into the scanner so it stops
traversing once enough models are found.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: Windows fallback, all registry hosts, collision safety
_make_link (formerly _make_symlink):
- Falls back to os.link() hardlink when symlink_to() fails (Windows
without Developer Mode), then to shutil.copy2 as last resort
- Uses atomic os.replace via tmp file to avoid race window where the
.gguf path is missing during rescan
Scanner now handles all Ollama registry layouts:
- Uses rglob over manifests/ instead of hardcoding registry.ollama.ai
- Discovers hf.co/org/repo:tag and any other host, not just library/
- Filenames include a stable sha1 hash of the manifest path to prevent
collisions between models that normalize to the same stem
Per-model subdirectories under .studio_links/:
- Each model's links live in their own hash-keyed subdirectory
- detect_mmproj_file only sees the projector for that specific model,
not siblings from other Ollama models
Friendly Ollama error detection:
- Now also matches ollama_links/ (the read-only fallback cache path)
and model_identifier starting with "ollama/"
Recommended folders:
- Added os.access(R_OK | X_OK) check so unreadable system directories
like /var/lib/ollama/.ollama/models are not advertised as chips
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: filter ollama_links from generic scanners
The generic scanners (models_dir, hf_cache, lmstudio) already filter
out .studio_links to avoid duplicate Ollama entries, but missed the
ollama_links fallback cache directory used for read-only Ollama
installs. Add it to the filter.
* Address review: idempotent link creation and path-component filter
_make_link:
- Skip recreation when a valid link/copy already exists (samefile or
matching size check). Prevents blocking the model-list API with
multi-GB copies on repeated scans.
- Use uuid4 instead of os.getpid() for tmp file names to avoid race
conditions from concurrent scans.
- Log cleanup errors instead of silently swallowing them.
Path filter:
- Use os.sep-bounded checks instead of bare substring match to avoid
false positives on paths like "my.studio_links.backup/model.gguf".
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: drop copy fallback, targeted glob, robust path filter
_make_link:
- Drop shutil.copy2 fallback -- copying multi-GB GGUFs inside a sync
API request would block the backend. Log a warning and skip the
model when both symlink and hardlink fail.
Scanner:
- Replace rglob("*") with targeted glob patterns (*/*/* and */*/*/*)
to avoid traversing unrelated subdirectories in large custom folders.
Path filter:
- Use Path.parts membership check instead of os.sep substring matching
for robustness across platforms.
Scan limit:
- Skip _scan_ollama_dir when _generic already fills the per-folder cap.
* Address review: sha256, top-level uuid import, Path.absolute()
- Switch hashlib.sha1 to hashlib.sha256 for path hashing consistency.
- Move uuid import to the top of the file instead of inside _make_link.
- Replace os.path.abspath with Path.absolute() in detect_gguf_model
to match the pathlib style used throughout the codebase.
* Address review: fix stale comments (sha1, rglob, copy fallback)
Update three docstrings/comments that still referenced the old
implementation after recent changes:
- sha1 comment now says "not a security boundary" (no hash name)
- "rglob" -> "targeted glob patterns"
- "file copies as a last resort" -> removed (copy fallback was dropped)
* Address review: fix stale links, support all manifest depths, scope error
_make_link:
- Drop size-based idempotency shortcut that kept stale links after
ollama pull updates a tag to a same-sized blob. Only samefile()
is used now -- if the link doesn't point at the exact same inode,
it gets replaced.
Scanner:
- Revert targeted glob back to rglob so deeper OCI-style repo names
(5+ path segments) are not silently skipped.
Ollama error:
- Only show "Some Ollama models do not work with llama.cpp" when the
server output contains GGUF compatibility hints (key not found,
unknown architecture, failed to load). Unrelated failures like
OOM or missing binaries now show the generic error instead of
being misdiagnosed.
---------
Co-authored-by: Daniel Han <info@unsloth.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Fix review findings for PR #49
1. Sandbox fallback Jinja env in _VariantTokenizerProxy.apply_chat_template
(use SandboxedEnvironment, matching _derive_assistant_prefix_by_render)
2. Unwrap benign outer-If guards in _template_ends_with_toplevel_for so
templates like {% if messages %}{% for ... %}{% endfor %}{% endif %}
are still repairable (preserves Qwen3-Guard rejection via else-branch
and add_generation_prompt-name checks)
3. Preserve raw name_or_path in _VariantTokenizerProxy._source_path so
local-path detection works for dict/list variant tokenizers
4. Context-aware strict-mode messages: omit "will still load" and
"Set UNSLOTH_STRICT_CHAT_TEMPLATE=1" when already raising
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Older installers persisted the venv Scripts directory directly in the
User PATH registry. The shim approach from #4961 no longer writes that
entry, but on upgrade the old one survived and python.exe / pip.exe
from the unsloth venv continued winning resolution in every new shell.
Before creating the shim, read the current User PATH, filter out any
entry matching $VenvDir\Scripts (using the same symmetric raw+expanded
comparison as Add-ToUserPath), and write back if changed. No-op on
fresh installs where the legacy entry was never written.
Confirmed on a real Windows machine: `where.exe python` was returning
the venv interpreter first even after the shim PR merged.
Older installers persisted the venv Scripts directory directly in the
User PATH registry. The shim approach (added in this PR) no longer writes
that entry, but it also did not remove the old one. On upgrade, the
legacy entry survived and python.exe / pip.exe from the unsloth venv
continued winning resolution in every new shell, which is exactly the
hijack the shim was designed to prevent.
Before creating the shim, read the current User PATH, filter out any
entry matching $VenvDir\Scripts (using the same symmetric raw+expanded
comparison as Add-ToUserPath), and write back if changed. This runs
once per install and is a no-op on fresh installs where the legacy
entry was never written.
danielhanchen