* fix: developer to api
* fix: help svg and Unsloth text
* svg fix
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* Fix FastSentenceTransformer compatibility with sentence-transformers 5.4
* Support varied Transformer init signatures
Detect Transformer.__init__ parameters and build init kwargs accordingly so trust_remote_code and other args are passed using the correct names. Instead of unconditionally using model_args/config_args, the code now inspects the constructor to decide between model_kwargs/config_kwargs vs model_args/config_args and also sets processor_kwargs or tokenizer_args when present. Initializes Transformer with constructed transformer_kwargs (including max_seq_length) to improve compatibility with different Transformer implementations.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden SentenceTransformer path and module checks
* Scrub .github/workflows for staging push (matches staging base)
* Guard auto_model write in FastSentenceTransformer._apply_torch_compile
On sentence-transformers >=5.4 Transformer.auto_model is a read-only
@property backed by self.model, so a direct assignment raises
AttributeError. The two get_peft_model paths already guard the write
with isinstance(getattr(type(...), "auto_model", None), property);
the auto-compile path missed the same guard, which broke the default
trainer path whenever max_steps >= _compile_threshold.
* Add tests for FastSentenceTransformer property guards
* Tighten FastSentenceTransformer redirect lifecycle tests
Drop a duplicate assertion-less case, remove dead AST extraction helper,
and trim unused imports. The remaining six tests cover substitution on
match, restoration on constructor exception, passthrough for unrelated
names, pathlib.Path normalisation, trailing slash handling, and the
no-identifier guard.
* Sync .github/workflows with upstream author branch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Avoid sharing trust_remote_code kwargs dict across constructor buckets
In FastSentenceTransformer._create_transformer_module, the same
trust_remote_code_kwargs dict was being assigned to model_kwargs,
config_kwargs, and processor_kwargs (or model_args / config_args /
tokenizer_args) on the Transformer constructor. transformers'
from_pretrained code paths (configuration_utils, auto_factory,
processing_auto, etc.) call kwargs.pop("trust_remote_code", ...) on
the dict they receive, which would drain the shared object and silently
strip trust_remote_code from the other buckets. Pass an independent
copy to each bucket so subsequent buckets and any pass-through
auxiliary loads still see trust_remote_code.
* Wire do_lower_case and return_dict through Transformer init for ST 5.4
In FastSentenceTransformer._create_transformer_module:
- When Transformer.__init__ accepts do_lower_case (ST 5.4+), pass
the unsloth tokenizer's do_lower_case as a constructor kwarg. The
existing post-init attribute assignment alone is too late: ST 5.4's
__init__ uses do_lower_case to install a Lowercase normalizer on
tokenizer.backend_tokenizer.normalizer, which is not re-applied if
we only set the attribute after construction. The post-init line
is preserved untouched for older ST versions.
- Add return_dict to the manually completed model_forward_params set
so wrapped models with forward(*args, **kwargs) signatures keep ST's
forced dict-like output safety net. ST 5.4's own __init__ unions the
forward signature with the same set plus return_dict; the previous
override silently dropped it.
* Preserve flash-attention forward keys when wrapping ST 5.4 Transformer
Sentence-transformers 5.4's Transformer.__init__ calls
_can_flatten_inputs() during construction, which augments
self.model_forward_params with cu_seq_lens_q, cu_seq_lens_k,
max_length_q, max_length_k, seq_idx whenever feature-extraction with
text modality, the torch backend, flash-attention 2, and varlen
flash-attn support are all available. The post-init override of
transformer_module.model_forward_params used to replace the attribute
outright, silently dropping those keys so ST's preprocess() filter
stripped flash-attn kwargs before reaching model.forward.
Snapshot the constructor-populated set first, leave the existing
overwrite intact for the forward-signature plus tokenizer keys, and
union the snapshot back in so flash-attn forwarding keeps working on
ST 5.4. For older sentence-transformers releases the attribute is
absent and getattr returns an empty set, leaving behavior unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Update VRAM estimator to cater to broader model configs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix attn backend check, better support for MoE etc
* Studio: tighten VRAM estimator structured-shape and attention paths
- Conservative attention fallback: when resolve_attention_implementation
fails, charge the quadratic non-flash activation path instead of
silently keeping the optimistic flash_attention_2 default.
- Resolve attention on a shallow config copy so _set_attn_impl does not
mutate the cached config returned by _load_config_for_gpu_estimate.
- Use getattr for AutoModelForCausalLM._model_mapping to avoid raising
on private-attribute renames in transformers.
- Treat sdpa as O(n) linear attention; PyTorch SDPA dispatches to flash
or memory-efficient backends, only eager needs the quadratic term.
- Per-layer activation accounting: structured archs (head_dim,
layer_types, attention_k_eq_v, num_kv_shared_layers, double-wide MLP)
now flow into compute_activation_bytes via _text_linear_dims, instead
of using the legacy hidden_size//num_attention_heads KV/MLP shape.
- Exclude MLA configs (q_lora_rank set) from the structured-shape path
so q_lora low-rank projection formulas keep applying when head_dim is
also present.
- _build_text_module_elements emits a single MLA self_attn aggregate
using _compute_attn_elements when q_lora_rank is set, avoiding the
~10% overcount that fed into _compute_skipped_quantizable_elements.
- Restrict _module_path_matches to known text-tower prefixes so VLM
skip names like vision_tower.model.layers.<i>.self_attn.q_proj no
longer falsely shadow the text alias model.layers.<i>.self_attn.q_proj.
- Pick up enable_moe_block from the config and add the per-layer dense
MLP alongside the MoE experts in compute_total_params and
compute_lora_params (Gemma4-style parallel dense + MoE block).
- Single-pass structured layer accounting in _compute_layer_elements,
removing the duplicate _text_linear_dims walks.
- Drop the now-zero (activations - activations_computed) shard term in
VramBreakdown.min_gpu_vram and the stale comment that referred to it.
- attention_implementation typed as Optional[str] to match call sites
that pass None.
- Inline rationale comments on DOUBLE_QUANT_4BIT_FACTOR and
NON_FLASH_ATTENTION_FACTOR pointing at VRAM_ESTIMATION.md.
* Studio: extend parallel-MoE accounting + non-prefix dense layer support
- Apply enable_moe_block / moe_has_dense_mlp symmetrically: activation
per-layer MLP size in _layer_qkv_mlp_sizes now adds the parallel dense
MLP for MoE layers, matching the weight and LoRA accounting added in
the prior commit. Skip-quantizable mapping in _build_text_module_elements
now registers both mlp.experts and per-projection mlp.{name} entries
for MoE layers when the parallel dense block is present, so an
llm_int8_skip_modules entry like "model.layers.N.mlp" covers both.
- Track dense layer indices as a tuple (dense_layer_indices) extracted
from first_k_dense_replace or decoder_sparse_step + mlp_only_layers,
and dispatch dense-vs-MoE accounting through _is_dense_mlp_layer. The
prior count-based path silently mis-bucketed layers when mlp_only_layers
was non-prefix (e.g. [3, 5] on an 8-layer model). num_dense_layers is
derived from len(dense_layer_indices) for backward compatibility.
- Drop the redundant ">0" check in _is_kv_shared_layer so configs with
num_kv_shared_layers == num_hidden_layers (every layer shared) are
correctly recognized as shared.
- Refresh VRAM_ESTIMATION.md section 5 to note that sdpa joins
flash_attention_2 in the linear activation path; refresh the
VramBreakdown.activations_computed comment now that the activation
floor is gone.
* Studio: Gemma4 PLE accounting, flex_attention, KV-share guard restore
- Add flex_attention to LINEAR_ATTENTION_IMPLS. Unsloth's
resolve_attention_implementation returns "flex_attention" when
HAS_FLASH_ATTENTION is False and the model class supports flex; PyTorch
FlexAttention is a memory-efficient kernel, not a quadratic eager
attention path. Without this, activation estimates over-charge ~36x.
- Restore the `> 0` guard in _is_kv_shared_layer. Transformers Gemma4
(modeling_gemma4.py:1031, modular_gemma4.py:863, :926) uses
`layer_idx >= first_kv_shared_layer_idx > 0`, so configs that mark
every layer as KV-shared raise on construction. Reverting the
unconditional acceptance avoids producing a detailed estimate for a
shape the actual model code rejects.
- Extend the parallel dense MLP path (`enable_moe_block`) in
_build_text_module_elements: when the arch is non-structured, use
arch.intermediate_size for the dense gate/up/down dims instead of
_text_linear_dims (which returns moe_intermediate_size via
_get_mlp_size). Prior code under-counted skipped quantizable elements
for the parallel dense block by up to 8x on GLM-style configs.
- Add Gemma4 per-layer-input (PLE) module accounting:
per_layer_model_projection (one global Linear) plus per-layer
per_layer_input_gate and per_layer_projection are added to the
quantizable text-linear total in _compute_layer_elements;
post_per_layer_input_norm and per_layer_projection_norm flow into
the non-quantizable bucket. compute_lora_params adds the same three
Linear modules to the all-linear total. References:
transformers_versions/5.7.0/.../gemma4/modular_gemma4.py:1077-1083,
:1247-1253.
- VRAM_ESTIMATION.md section 5 now lists flex_attention alongside sdpa
and flash_attention_2 as linear-memory backends.
* Studio: shared-expert variants, mlp_layer_types dispatch, PLE skip, all-linear str, deepcopy resolver
Five targeted estimator corrections:
- _compute_dense_layer_indices now reads `mlp_layer_types` ahead of
`first_k_dense_replace` / `decoder_sparse_step`. Transformers Exaone-MoE,
Laguna, Hy_v3, GLM-MoE-DSA, GLM4-MoE-Lite, Ernie4_5_VL_MoE etc. ship the
per-position list and may omit the prefix-style fields entirely.
- _build_text_module_elements registers per_layer_input_gate /
per_layer_projection (per layer) and per_layer_model_projection (global)
in the canonical element map and alias map. The PLE element count was
added to total_quantizable in a prior commit but skip-module matching
against names like model.layers.0.per_layer_input_gate produced 0-byte
delta. Layer aggregate text.layers.<i> now sums all layer modules so
prefix skip names cover the PLE pieces too.
- _targets_all_linear coerces a bare string `"all-linear"` to `["all-linear"]`
before set comparison; the previous set comprehension iterated chars.
PEFT LoraConfig.target_modules accepts the bare-string convention.
- ModelArchConfig gains `shared_expert_intermediate_size`. extract_arch_config
reads `n_shared_experts` / `num_shared_experts` aliases and infers
`n_shared_experts=1` when only `shared_expert_intermediate_size` is set.
_compute_moe_mlp_elements and the structured + non-structured LoRA paths
size the shared expert with its own intermediate (Qwen3.5-MoE: 512 vs
routed moe_intermediate_size).
- _determine_attention_impl_for_gpu_estimate uses copy.deepcopy so the
resolver does not mutate nested text_config on the cached source.
PreTrainedConfig._attn_implementation setter walks `sub_configs` and the
prior shallow copy still touched the inner objects.
* Studio: extend MoE/PLE/KV-share accounting to activation and skip-alias paths
Five activation-path corrections plus two LoRA / skip-alias corrections so
that shared-expert, per-layer-input, and KV-shared-layer support is symmetric
across weights, LoRA, skip-quantizable, and activation paths.
- _layer_qkv_mlp_sizes: include shared-expert FFN in mlp_size (live shared
expert per token alongside routed experts) and keep K/V activation memory
for KV-shared layers; only the WEIGHT path uses has_k/has_v from
_layer_attention_dims.
- _per_layer_activation_bytes / compute_activation_bytes: account for
per_layer_input_gate (hd-sized) and per_layer_projection (pli-sized) per
layer plus the global per_layer_model_projection [B,S,L,PLI] tensor when
hidden_size_per_layer_input is set.
- _build_text_module_elements: split mlp.experts into routed and
mlp.shared_expert canonical entries; register layers.<i>.experts alias for
Gemma4 enable_moe_block layouts and mlp.shared_experts (plural) alias for
Exaone-MoE / Laguna / GLM4-MoE-Lite shared-expert variants.
- _compute_moe_mlp_elements: split into _compute_routed_moe_elements and
_compute_shared_moe_elements; only count shared_expert_gate (hd->1 Linear
per shared expert) when shared_expert_intermediate_size is set, which is
the Qwen2-MoE / Qwen3.5-MoE discriminator. Other shared-expert families
(Exaone-MoE, HY-V3, GLM4-MoE-Lite, Laguna) lack the gate.
- compute_lora_params: when target_modules='all-linear' bare keyword, drop
routed and shared MoE expert LoRA contributions. PEFT's all-linear targets
nn.Linear only; Unsloth's get_moe_target_parameters expands MoE expert
nn.Parameter LoRA only when target_modules contains explicit
gate_proj/up_proj/down_proj/gate_up_proj names.
- _per_layer_input_lora_params: thread target_modules through and add the
per-PLE-module contribution when the corresponding name appears, not only
under all-linear.
* Studio: top-k MoE activations, ERNIE list configs, suffix skips, multimodal full bytes
Six estimator corrections aligning the detailed accounting paths with real
training behavior:
- _layer_qkv_mlp_sizes scales the MoE-layer mlp_size by num_experts_per_tok
so the active routed-expert intermediate tensors are charged for activations.
Adds num_experts_per_tok to ModelArchConfig and extracts it from
num_experts_per_tok / top_k_experts (Gemma4 alias) in extract_arch_config.
- compute_lora_params splits routed and shared MoE LoRA contributions so that
bare target_modules='all-linear' zeroes routed (nn.Parameter expert tensors,
which Unsloth's get_moe_target_parameters does NOT enable for the bare
keyword) but keeps shared-expert LoRA (regular nn.Linear MLPs that
Unsloth's get_peft_regex DOES match).
- extract_arch_config gains a _first_scalar helper for ERNIE-style
moe_intermediate_size = [routed, shared] lists, plus moe_num_experts and
moe_num_shared_experts attribute aliases. When moe_intermediate_size is a
pair and shared_expert_intermediate_size is unset, the second element is
treated as the shared-expert intermediate.
- estimate_required_model_memory_gb's detailed branch retains
max(0, model_size_bytes - compute_total_params(arch) * 2) on top of the
arch-derived breakdown.model_weights so multimodal models (vision/audio
towers) and partially-modeled families (Gemma3n AltUp/Laurel etc.) do not
silently drop bytes that the safetensors total includes.
- _module_path_matches accepts a tail-only match when the skip entry is
shorter than the alias path. Transformers' BNB quantizer suffix-matches
short skip entries like ['q_proj'] / ['lm_head'] against full module
paths; the previous len(skip) < len(alias) early-return missed those.
- _per_layer_input_lora_params drops the all_linear branch and only counts
PLE LoRA when the user explicitly names per_layer_input_gate /
per_layer_projection / per_layer_model_projection. Unsloth's
get_peft_regex requires module names to contain a component tag
(mlp/attn/...); PLE module names lack any tag, so all-linear training
does not attach LoRA to them.
* Studio: full-FT extra optimizer/gradient inflation, MoE top-k aliases, ERNIE position dispatch, sibling experts aggregate
When the safetensors total exceeds the text-arch fp16 estimate (multimodal
vision/audio towers, partially-modeled families), only inflate the model
weights line for adapter methods but extend optimizer + gradient bytes
under full fine-tuning, where the extra params are trainable.
DBRX exposes top-k routing as moe_top_k and Hunyuan-V1-MoE as moe_topk;
neither is aliased to num_experts_per_tok via attribute_map, so probe both
when extracting arch config.
ERNIE 4.5 MoE / VL MoE configs declare MoE layers via
moe_layer_start_index / moe_layer_end_index / moe_layer_interval (with -1
meaning the last layer); add the position-style dispatch alongside the
existing mlp_layer_types / first_k_dense_replace / decoder_sparse_step
paths.
When moe_has_dense_mlp is set (Gemma4 enable_moe_block) the routed experts
live as a sibling of self.mlp at layers.<i>.experts in the actual model
layout; keep the layer mlp aggregate to the dense path and add a separate
experts aggregate so a skip module model.layers.<i>.mlp does not collapse
the routed experts as well.
* Studio: extend MoE family extraction (Llama4 / DBRX / Hunyuan / ERNIE) and align dense vs routed MLP widths
- Llama4: pick up `config.moe_layers` (auto-populated from
interleave_moe_layer_step) so dense layer indices reflect the actual
is_moe_layer dispatch.
- Llama4: add a separate `dense_intermediate_size` derived from
`intermediate_size_mlp` (used for the dense feed_forward path) and keep
`intermediate_size` for the routed/shared expert width. Auto-attach one
shared expert per MoE layer when the dense-vs-MoE width split is present.
- DBRX: walk the `ffn_config` sub-config when extracting MoE attrs
(moe_num_experts / moe_top_k / ffn_hidden_size). Without this DBRX is
misclassified as a dense arch.
- Hunyuan: normalize layer-wise `moe_topk` (and the canonical
`num_experts_per_tok` lookup it shadows via attribute_map) through a
worst-case scalar so the int(...) cast cannot crash on list values.
- ERNIE 4.5 MoE: switch the start/end/interval dispatch to the model's
`(layer_idx + 1) % interval == 0` modulo gate so MoE layers match the
decoder when interval > 1.
- ERNIE 4.5 VL MoE: drop the heuristic that read
`moe_intermediate_size[1]` as the shared expert width; in VL configs [1]
is the vision-routed width and shared experts are sized from [0].
- estimate_fp16_model_size_bytes: prefer the larger of config-derived and
local-weight bytes so the multimodal extra_bytes correction can fire
for local VLM directories.
* Add tests for VRAM estimator extensions
* Studio: trim verbose comments in VRAM estimator
Collapse multi-paragraph rationale blocks to 1-3 lines stating the single
load-bearing fact. Fix one inverted "fall through ... last" comment whose
claim disagreed with the surrounding code.
* Consolidate added tests into existing test_vram_estimation.py and test_gpu_selection.py
Move Llama4 / DBRX / ERNIE arch-extraction tests into test_vram_estimation.py
as TestLlama4ArchExtraction / TestDbrxFfnConfigExtraction /
TestErniePhaseModuloDispatch / TestErnieVlSharedExpertWidth classes. Move
estimate_fp16_model_size_bytes prefer-larger-of-config-or-local tests into
test_gpu_selection.py as TestEstimateFp16ModelSizeBytesPrefersLocalWeights.
Drop one redundant Llama4 num_dense_layers assertion already covered by the
moe_layers dispatch test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* fix KVCache estimates for gemma4 style sliding window models
Signed-off-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: add per-arch SWA pattern fallback + n_kv_heads mirror for PR #5225
The pattern-aware SWA estimator added in this PR only fires when the
GGUF carries `<arch>.attention.sliding_window_pattern`. Today's
Gemma-2 / Gemma-3 / Gemma-3n / gpt-oss / Phi-3 GGUFs ship
`attention.sliding_window` but not the pattern field (llama.cpp's
converter strips it), so the new branch is bypassed and we fall back to
the legacy 1/4-global heuristic on the most popular SWA arches in our
catalogue (gemma3 alone has 6+ variants in the unsloth/* top 30 by
downloads, plus gpt-oss-20b/120b).
Two additions on top of this PR:
1. `_SWA_PATTERN_DEFAULTS_BY_ARCH` table keyed by GGUF arch name. When
the GGUF reports a sliding window but no pattern, we synthesise the
pattern from the architecture's canonical period (gemma2=2,
gemma3=6, gemma3n=5, gpt_oss=2, phi3=1, cohere2=4). Periods sourced
from a survey of the top 150 unsloth/* HF configs against
`text_config.layer_types` and `Gemma*Config.sliding_window_pattern`.
2. Mirror `_n_kv_heads_by_layer` into the scalar `_n_kv_heads` (using
max as a conservative upper bound) when the head_count_kv array is
read. Without this, any non-SWA estimator path (GQA, legacy) on a
Gemma-4-style model falls through to `n_heads`, which can be many
times larger than the real per-layer KV head count. Also let
`_can_estimate_kv` accept the array directly as belt-and-suspenders.
End-to-end check on `unsloth/gemma-3-270m-it-Q4_K_M.gguf` (18 layers,
sliding_window=512, no pattern field): the parser now resolves the
pattern to period=6 (3 global, 15 SWA), matching the actual
Gemma3TextConfig default. KV estimate at 32k context drops from
141 MB (legacy 1/4) to 108 MB (per-layer), a 23% reduction that
directly translates into more headroom for `_fit_context_to_vram` and
fewer cases where the slider lands on the 4096 floor.
Tests: extended `test_kv_cache_estimation.py` with
`TestArchSwaPatternDefaults` covering the six tabled arches, an
unknown-arch negative, explicit-pattern precedence, and a
no-sliding-window negative; updated `test_array_fields_parsed` to
reflect the new mirror semantics; updated
`test_end_to_end_synthetic_swa` to use the period=6 expectation. All
102 tests in the kv-cache / context-fit / max-context suites pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten arch SWA table to verified non-regressing entries
Audit of every unsloth/* HF model (1334 repos, all config.json fetched
in scripts/survey_all_unsloth.py) plus end-to-end checks against five
real GGUFs (gemma-3-270m, gemma-3-1b, qwen2.5-0.5b, phi-3.5-mini,
falcon-h1-0.5b, granite-4.1-8b) confirms:
* Pure-GQA arches (llama, qwen3, mistral3, glm4, llama4, ...) and the
qwen2 family with use_sliding_window=False all reach Path 4 GQA
cleanly. The llama.cpp converter strips `attention.sliding_window`
for qwen2/qwen2_vl/qwen2_5_vl when use_sliding_window=False, so the
SWA path never fires for them. Verified on Qwen2.5-0.5B-GGUF: no
sliding_window field in metadata.
* MLA arches (deepseek_v3/v32/v4, glm4_moe_lite, glm_moe_dsa, kimi_k25)
emit `kv_lora_rank` -> Path 1 fires correctly.
* Hybrid Mamba/Attn arches that emit both ssm.* and
full_attention_interval (qwen3_5, qwen3_5_moe, qwen3_next) -> Path 2
fires correctly.
Two table changes:
1. Drop the `phi3` entry. Phi-3 GGUFs emit
`phi3.attention.sliding_window=262144` but never emit
`attention.key_length`/`value_length`, so the SWA path is gated
off and the estimator falls to the legacy formula. The huge
sliding_window also means SWA layers and global layers cache
identical numbers of tokens at any practical context, so a fallback
would be a no-op anyway. The previous `phi3: 1` entry was also
semantically wrong: period=1 with the (i+1)%N!=0 rule produces
all-global, not the all-SWA you'd want for Phi-3.
2. Document the audit findings in the table comment, including the
two arches we deliberately skip (phi3, qwen2*) and the one
architecture family that is not a regression vs. main but is also
not yet optimal (mistral v0.1/v0.2 all-SWA every-layer cannot be
expressed with the period sentinel).
Tests: added `test_non_swa_arch_uses_full_attention_path` parametrized
over llama / qwen2 / qwen3 / mistral / mistral3 / glm4 / llama4 to
pin the invariant that pure-GQA arches never receive a synthetic SWA
pattern. Removed phi3 from the parametrize list of
`test_arch_default_pattern_applied`. All 108 tests pass.
Known separately tracked (not addressed here): falcon-h1 GGUFs ship
ssm.* + key_length but no full_attention_interval, so the hybrid
path 2 cannot fire and the estimator falls to GQA path 4, which
counts every block as an attention layer. Affects 8 unsloth/* repos
(~500 downloads). Same gap exists for granitemoehybrid-class GGUFs.
Worth a follow-up that adds either a HYBRID_ATTENTION_INTERVAL_BY_ARCH
table or a tensor-name probe.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: make SWA pattern resolver dynamic so new models work without code changes
The static `_SWA_PATTERN_DEFAULTS_BY_ARCH` dict only covered
architectures we knew about at PR-merge time. New SWA models would
need a code change here every time a new arch shipped, which doesn't
scale. Replaced with a 4-tier resolver so any newly-released model
that lands on Hugging Face with a normal `config.json` is covered
automatically:
Tier 0 (parser) -- explicit GGUF metadata if the converter emits it
(BOOL array or scalar period). Already supported.
Tier 1 (cache) -- $UNSLOTH_STUDIO_HOME/swa_cache.json. Populated by
previous Tier 3 fetches. Survives restarts.
Tier 2 (bootstrap) -- `_BOOTSTRAP_SWA_DEFAULTS` shipped with Studio.
Same five entries as the old static table
(gemma2/3/3n/gpt_oss/cohere2). Lets fully-offline
installs keep working for popular SWA arches
with zero network.
Tier 3 (HF fetch) -- pulls `config.json` from the GGUF's source HF
repo and reads `sliding_window_pattern` (int) or
`text_config.layer_types` (string array). Result
is cached to Tier 1 so subsequent loads are
offline-fast. Disabled by
`UNSLOTH_STUDIO_OFFLINE=1`. Network errors and
missing repos fall through silently.
Tier 4 (caller) -- legacy 1/4-global SWA estimate (unchanged).
The GGUF parser now also extracts a handful of `general.*` keys
(`source.huggingface.repository`, `source.url`, `source.repo_url`,
`base_model.0.repo_url`, `base_model.0.organization` + `.name`,
`organization` + `basename`) so the resolver has source-repo
candidates to try.
End-to-end smoke against a brand-new arch (`never_seen_before_arch`,
not in the bootstrap dict) pointing at `google/gemma-3-1b-it`:
resolver fetched the HF config, derived period=6, materialised the
26-layer mask with 4 global layers (indices 5/11/17/23), and wrote
`{"never_seen_before_arch": 6}` to the on-disk cache. Next load hits
Tier 1 with no network.
Tests: added `TestDynamicSwaResolver` with 10 tests covering each
tier (period derivation, aperiodic mask handling, URL parsing,
bootstrap precedence, cache precedence, HF fetch + persistence,
candidate fallback, offline env knob, network failure). All 118
tests in the kv-cache / context-fit / max-context suites pass.
The `_SWA_PATTERN_DEFAULTS_BY_ARCH` name was retired in favour of
`_BOOTSTRAP_SWA_DEFAULTS` to make the tier semantics explicit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: add Tier 2.5 transformers introspection to SWA resolver
Slots a new tier between bootstrap and HF fetch that asks the
locally-installed `transformers` package directly. Two strategies, in
order, both offline-friendly:
a. Default-instantiate the matching `Config` class via
`CONFIG_MAPPING[arch]()` and read `sliding_window_pattern` /
`text_config.layer_types`. Drills into `text_config` for
multimodal wrappers.
b. `inspect.getsource(cfg_class)` regex-parse for
`sliding_window_pattern: int = N` defaults. Catches configs
whose constructor raises (missing required args), or where the
default is bound only in the __init__ signature. Walks
`cfg_class.sub_configs["text_config"]` too so multimodal wrappers
that delegate to a TextConfig still get inspected.
Resolver chain is now 5 tiers: GGUF metadata, on-disk cache,
bootstrap defaults, transformers introspection, HF Hub fetch, legacy
fallback. Tier 2.5 results are persisted to the same on-disk cache as
Tier 3 so subsequent loads skip the import overhead.
`_arch_aliases` normalises hyphen vs underscore variants (`falcon-h1`
vs `falcon_h1`) since GGUF and HF disagree for a handful of arches.
Cross-version verification (probe at `temp/swa_probe/`):
```
arch transformers 4.57.6 transformers 5.7.0
gemma3 6 6
gemma2 2 2
cohere2 4 4
gpt_oss 2 2
gemma3n 5 5
gemma4 ARCH-MISSING 6 <- new arch picked up automatically
falcon_h1 None [True]*32 <- per-layer mask used verbatim
phi3 None None
mistral None None
qwen2 1 (all-global) 1
llama None None
deepseek_v3 None None
```
The `gemma4` and `falcon_h1` rows are the headline: a brand-new arch
that lands in transformers (gemma4 is 5.x-only) is supported by the
resolver the moment a user upgrades the package, with zero edits to
this file. Same applies to any future arch with a `Config` class.
Tests: added `TestTransformersIntrospection` with 6 cases covering
arch-alias normalisation, real-arch resolution against the live
transformers, inspect.getsource fallback when default-init raises,
graceful behaviour when transformers is unavailable, unknown-arch
returns None, and Tier-2.5-before-Tier-3 ordering. Also adjusted the
existing Tier 3 failure test to mock Tier 2.5 out so it specifically
exercises the network-failure path. All 124 tests in the kv-cache /
context-fit / max-context suites pass.
Updated the module-level resolver comment from "4-tier" to "5-tier"
to document the new tier.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: consolidate verbose comments and docstrings in SWA resolver
Net -345 lines: -210 in llama_cpp.py, -357/+111 in
test_kv_cache_estimation.py. No behaviour change; only comments,
docstrings, and one tests-only `_SWA_FIELDS` helper to remove the
copy-pasted GGUF metadata dict from each resolver test.
Code changes only delete or shorten:
* 5-tier resolver header collapsed from a 38-line block diagram to
a 9-line summary; the rest is the function bodies.
* Bootstrap dict per-arch comments collapsed to one-line `Config`
references.
* `_swa_cache_path`, `_save_swa_cache`, `_period_from_layer_types`,
`_arch_aliases`, `_swa_entry_from_config_obj`,
`_resolve_swa_pattern`, `_resolve_swa_entry_from_transformers`
docstrings stripped to one line or removed when the body is
self-evident.
* Tier-by-tier inline comments inside `_resolve_swa_pattern` removed
(function body reads top-to-bottom in tier order).
* Path-3 SWA estimator comment shortened from a 12-line tier
breakdown to 3 lines.
* Parser fallback comment block (originally explained the resolver
in-line) trimmed to two lines pointing at the resolver.
* `_can_estimate_kv` legacy-clause comment shortened to one line.
* GGUF `general.*` WANTED block comment shortened to one line.
Test changes:
* Per-test docstrings dropped where the test name and body already
explain intent.
* Class-level docstrings reduced to one line.
* Common GGUF field dict factored to module-level `_SWA_FIELDS`.
* Multi-line URL/list assertions collapsed to one-liners.
All 124 tests pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: account for SWA cache double-buffering in path 3 estimate
Cross-check against llama.cpp ground truth (running llama-server with
--parallel 1 and reading the `llama_kv_cache: size = X MiB ( N cells,
M layers, ... )` log lines) showed the SWA path under-counted by
~20% on Gemma-3-shaped models:
GGUF pred MiB actual MiB ratio
gemma-3-270m-it-Q4_K_M 31.50 39.00 0.81
gemma-3-1b-it-Q2_K 43.00 54.00 0.80
Root cause: llama.cpp double-buffers the SWA cache so it can keep the
current and next windows during the shift, allocating
`2 * sliding_window` cells per SWA layer (capped at n_ctx). My formula
was using `min(n_ctx, sliding_window)` instead of
`min(n_ctx, 2 * sliding_window)`. Verified directly:
llama_kv_cache_iswa: creating SWA KV cache, size = 1024 cells
llama_kv_cache: size = 15.00 MiB (1024 cells, 15 layers, ...)
with `gemma3.attention.sliding_window = 512` -> 2 * 512 = 1024 cells.
Fix: introduce `swa_cells = min(n_ctx, 2 * swa)` in path 3 and use
that for both the per-layer-pattern branch and the legacy
1/4-global fallback.
Re-run after the fix:
GGUF pred MiB actual MiB ratio
gemma-3-270m-it-Q4_K_M 39.00 39.00 1.000
gemma-3-1b-it-Q2_K 54.00 54.00 1.000
qwen2.5-0.5b-instruct-q4_k_m 96.00 96.00 1.000
Phi-3.5-mini-instruct-Q4_K_M 3072.00 3072.00 1.000
Falcon-H1-0.5B-Instruct-Q4_K_M 144.00 144.00 1.000
granite-4.1-8b-Q3_K_M 1280.00 1280.00 1.000
All 5 paths now match llama.cpp's actual allocation exactly under
single-sequence inference (Studio's default).
Tests: updated `test_gemma3`, `test_gpt_oss`,
`test_gemma4_per_layer_swa_metadata`, `test_ctx_smaller_than_window`,
`test_odd_layer_count`, and `test_end_to_end_synthetic_swa` to use
the doubled SWA cell count. All 124 tests in the kv-cache /
context-fit / max-context suites pass.
* studio: tolerate truncated GGUF input so resolver fallback still runs
Wraps each iteration of the GGUF KV-pair loop in a try/except that
breaks out cleanly on `struct.error` or `UnicodeDecodeError`, instead
of letting the outer try eat the exception and skip the SWA resolver
fallback at the end.
The motivating use case is reading the GGUF metadata via an HF Hub
HTTP byte-range fetch. The first ~128 KiB of a typical GGUF contains
all the metadata we need (arch, block_count, attention.*, sliding
window, ssm, MLA fields, plus the tokenizer config) -- but for models
with large tokenizer vocabs (Gemma 3 has 262144 tokens) the tokenizer
arrays spill past the 128 KiB boundary. The truncation used to bubble
out as `unpack requires a buffer of 8 bytes`, abandoning the resolver
fallback and leaving us with no SWA pattern (so the SWA path fell
through to the legacy 1/4 estimate).
Verified end to end against `unsloth/gemma-3-1b-it-GGUF`:
Range-fetch first 128 KiB of `gemma-3-1b-it-Q2_K.gguf` over HTTP
(HTTP 206 Partial Content), parse:
arch = gemma3
block_count = 26
attention.sliding_window = 512
sliding_window_pattern = set (4 global) <- via Tier 2 bootstrap
KV @ ctx=8192 = 54.00 MiB <- matches llama.cpp
ground truth
This means Studio can preview KV-cache requirements (and therefore
auto-context fit) for any HF GGUF without downloading the weights.
All 124 tests pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: thread llama-server KV flags through the estimator
Adds keyword-only knobs to _estimate_kv_cache_bytes and _fit_context_to_vram
that mirror the llama-server CLI options that change KV memory:
--swa-full (swa_full) SWA layers cache the full n_ctx
instead of 2 * sliding_window cells
--parallel N (n_parallel) number of server slots
--kv-unified (kv_unified) single shared KV buffer; when off,
multiplies KV by n_parallel
--ctx-checkpoints (ctx_checkpoints) per-slot SWA snapshots, each one
sliding-window of state per SWA layer
--kv-offload (kv_on_gpu) when off, KV lives in CPU RAM and is
not subtracted from the VRAM budget
Defaults preserve the previous behavior (swa_full=False, n_parallel=1,
kv_unified=True, ctx_checkpoints=0, kv_on_gpu=True) so existing call sites
are unaffected. All five paths (MLA, hybrid, SWA pattern, SWA fallback,
GQA, legacy) now apply the per-slot replication factor; the SWA paths
also honor swa_full and add the checkpoint term when applicable.
Tests: TestServerFlags (17 cases) covers every flag, the no-op cases, the
swa_full + ctx_checkpoints interaction, slot multiplication on each path,
and the kv_on_gpu shortcut in _fit_context_to_vram.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: account for shared_kv_layers (Gemma 3n / Gemma 4)
Gemma 3n and Gemma 4 set <arch>.attention.shared_kv_layers in the GGUF
metadata (convert_hf_to_gguf.py lines 7578 and 7712). The trailing N
layers of the model reuse KV from earlier layers and don't allocate
their own cache, so n_layers in the per-layer formulas overcounted by
exactly that many blocks. For google/gemma-3n-E4B-it (35 layers, 15
shared), this puts ~43% of the KV estimate back on the table.
Changes:
- _read_gguf_metadata parses <arch>.attention.shared_kv_layers into
self._shared_kv_layers; init / unload / reparse all reset it.
- _estimate_kv_cache_bytes computes n_layers_kv = max(1, n_layers -
shared_kv_layers) and substitutes it for n_layers in:
Path 1 (MLA), Path 3 (SWA pattern loop bound and the no-pattern
fallback), Path 4 (GQA), Path 5 (legacy). Path 2 (hybrid) keeps
n_layers since hybrid + shared_kv combined isn't a thing today and
the semantics would need to specify which attention layers are
shared.
- max(1, ...) floor protects against pathological GGUFs where shared
>= n_layers.
- Composes naturally with --swa-full, --kv-unified / --parallel,
--ctx-checkpoints, and the per-layer SWA pattern from the dynamic
resolver. When the field is unset (every other arch) the math is
byte-identical to before.
Tests: TestSharedKVLayers (13 cases) covers each path's drop, the
no-op-when-unset case, the floor at one layer, composition with the
server-flag knobs, and lifecycle reset. test_end_to_end_synthetic_shared_kv_round_trip
exercises the full GGUF parse -> estimate path on a synthetic gemma3n_text
blob. Existing TestLifecycle tests extended to cover the new field.
Full suite: 132 passing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: only stub httpx in tests when the real lib is missing
The unit suite stubs httpx unconditionally so tests can run on a minimal
Python install. Surfaced during a fresh-venv simulation: when the stub
is installed via setdefault on a system that DOES have httpx,
huggingface_hub.errors fails to import HTTPError / Response at module
load time, which the transformers introspection tier swallows via its
bare except. Result: TestTransformersIntrospection passes in venvs
where httpx happened to be imported first (workspace) and silently
fails in venvs where it doesn't (fresh uv venv).
Switch to "only stub when real lib unavailable", and round out the stub
with HTTPError, RequestError, and Response so any test environment
without httpx still gets a complete enough surface for huggingface_hub
to import.
* studio: per-layer-type --parallel N memory accounting for SWA
Empirical verification against llama-server (see
workspace_5/temp/sim_pr5225/probe_parallel_full_matrix.py and
verify_parallel_matches_server.py) showed the prior whole-cache
slot_factor multiplication in _estimate_kv_cache_bytes was wrong for
n_parallel > 1. The actual rule, verified bit-exact across the full
(parallel x ctx) grid for both SWA and pure-GQA models:
* non-SWA layers: total cells = n_ctx, partitioned across slots
(per-slot ctx = n_ctx / parallel). Total memory
is CONSTANT in n_parallel.
* SWA layers: per-slot cells = 2 * sliding_window (clamped at
n_ctx and at per_slot_ctx when ctx is split among
many slots). Total memory grows LINEARLY in
n_parallel.
* --kv-unified: no measurable difference to total memory; both
modes yield the same byte total in measured cases.
Retained as accepted-but-ignored kwarg for API
forward-compat.
Closed form (Path 3 with per-layer pattern):
total_kv = sum_global_layers(n_ctx * n_kv * (k+v) * bpe)
+ parallel * sum_swa_layers(
min(2*sliding_window, n_ctx, n_ctx//parallel)
* n_kv_layer * (k_swa + v_swa) * bpe
)
+ parallel * checkpoint_extra_per_slot (when ctx_checkpoints > 0)
Changes to _estimate_kv_cache_bytes:
- Path 3 (SWA pattern): accumulate global_bytes and swa_bytes_per_slot
separately; final result = global_bytes + slots * (swa_bps + cp_bps).
- Path 3 (no-pattern fallback): same split using the 1/4-global heuristic.
- Paths 1 / 2 / 4 / 5: drop the slot_factor multiplication. Non-SWA
caches don't scale with --parallel.
- swa_full=True: SWA cells = per_slot_ctx (was n_ctx), so slots
cancels out and total stays constant. Matches llama-server's
--swa-full --parallel N output exactly.
Production wiring fix in start():
- Seven internal calls to _estimate_kv_cache_bytes / _fit_context_to_vram
used the default n_parallel=1, even though load_model accepts the
caller's n_parallel value (forwarded to llama-server via --parallel
on the command line). Pass n_parallel through all seven so VRAM
budgeting is correct when an operator sets parallel slots above 1.
Studio's default ships at 1 so production today is unaffected;
this completes the wiring for operators who tune it.
Tests:
- TestParallelSWAScaling (10 new cases): closed-form invariants per
path, swa_full + parallel collapse, kv_unified no-op proof,
per-slot SWA cell clamping, and the empirical Gemma-3 270m formula
(24 + parallel * 15 MiB at ctx=8192) baked from the verifier.
- TestServerFlags: rewrote 4 assertions and renamed 2 to reflect the
per-layer rule; non-SWA paths now correctly assert constancy.
- TestSharedKVLayers::test_composes_with_n_parallel: rewrote to assert
only the SWA portion of the unshared layers scales.
Backward compatibility: at n_parallel=1 the output is bit-identical to
before this change (verified across 120,960 sweep combinations and the
141-test suite in both workspace and fresh-uv-venv environments).
Verifier output at --parallel in {1,2,4,8} x ctx in {4096,8192,16384}
shows ratio 1.000 against llama-server for both SWA and pure-GQA
models (24/24 cells exact match).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: accept new estimator kwargs in load-time test stubs
`test_llama_cpp_context_fit.py` and `test_llama_cpp_max_context_threshold.py`
patch `_estimate_kv_cache_bytes` with constant per-token stubs and then
call the real `_fit_context_to_vram`. After 29dcf96e threaded the
llama-server flag kwargs (`swa_full`, `n_parallel`, `kv_unified`,
`ctx_checkpoints`) through `_fit_context_to_vram`, the production method
forwards them to the stubbed estimator and the old positional-only stubs
raise `TypeError`.
These two suites exercise the load-time fit decision and the max-context
threshold property with a constant per-token KV cost; SWA / parallel-slot
accounting is intentionally out of scope, so the stubs absorb the new
kwargs and ignore them. No production change.
Restores both files to fully passing: 15/15 in `test_llama_cpp_context_fit`
and 8/8 in `test_llama_cpp_max_context_threshold`. Combined with the
existing 141/141 in `test_kv_cache_estimation`, the three KV-cache test
modules are 164/164 green.
---------
Signed-off-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Pin Studio GGUF export to local llama.cpp convert script
setdefault UNSLOTH_LLAMA_CPP_SCRIPTS_DIR=LLAMA_CPP_DEFAULT_DIR before
save_pretrained_gguf so the convert_hf_to_gguf.py used at conversion
time matches the pinned llama-quantize binary and gguf-py installed
under ~/.unsloth/llama.cpp. Without this, the script is pulled from
upstream master and can drift past the binary's gguf API, causing
intermittent export failures.
setdefault preserves any explicit user override; validation of the
path lives in unsloth_zoo's _resolve_local_convert_script (warns and
falls back to network on a bad value).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scrub .github/workflows for staging push (matches staging base)
* Pin GGUF convert script for hub-only export path
Hoist the UNSLOTH_LLAMA_CPP_SCRIPTS_DIR setdefault and the
unsloth_zoo.llama_cpp import out of the if save_directory: block so
push_to_hub_gguf also runs with the pin. The worker passes
save_directory="" for hub-only exports, which previously skipped the
local branch and left the convert script fetched from master.
* Trim GGUF convert script pin rationale comment
Collapse 7 lines of rationale into 3 lines stating the load-bearing
facts: pin matches llama-quantize binary, set before both branches
because hub-only export has empty save_directory.
* Sync .github/workflows with upstream author branch
* Scrub .github/workflows for staging push (matches staging base)
* Warn when unsloth_zoo is too old to honor UNSLOTH_LLAMA_CPP_SCRIPTS_DIR
Studio's GGUF export sets UNSLOTH_LLAMA_CPP_SCRIPTS_DIR before
save_pretrained_gguf and push_to_hub_gguf so unsloth_zoo can prefer the
local pinned convert_hf_to_gguf.py. The resolver only exists in the
companion unsloth_zoo change; on older zoo builds permitted by the
current dependency floor, the env var is silently ignored and the
converter is still downloaded from llama.cpp master.
Probe for the resolver and emit a one-time warning so operators know the
pin is inactive and can upgrade unsloth_zoo.
* Combine the GGUF script-pin imports into one guarded block and warn once
Both LLAMA_CPP_DEFAULT_DIR and the resolver probe come from
unsloth_zoo.llama_cpp; older zoo wheels (e.g. 2026.1.4) lack
LLAMA_CPP_DEFAULT_DIR, so the previous unguarded import could crash the
GGUF export path on environments installed with --no-deps or a manually
pinned zoo. Move the constant import alongside the resolver probe inside
a single try/except ImportError so a missing symbol degrades to the
warning instead of a hard crash, matching the graceful-degradation
intent the probe was added for.
The compatibility warning previously fired on every export call because
'from X import Y' re-raises ImportError on every invocation when Y is
absent. Gate emission on a module-level flag so operators see it once
per process instead of once per export.
* Add Studio GGUF export script-pin test coverage
Consolidate tests for the UNSLOTH_LLAMA_CPP_SCRIPTS_DIR env-var pin in
ExportBackend.export_gguf into a single behavior-named module:
- AST-asserts the module-level _LLAMA_CPP_SCRIPTS_WARNING_EMITTED flag,
the merged try-block importing both LLAMA_CPP_DEFAULT_DIR and
_resolve_local_convert_script, and the warn-once gate inside the
ImportError handler.
- Behaviorally verifies setdefault preserves explicit user overrides,
assigns the default when unset, fires the compatibility warning at
most once across multiple export calls, and degrades to a warning
(without setting the env var) when LLAMA_CPP_DEFAULT_DIR itself is
missing on an older unsloth_zoo.
* Sync .github/workflows with upstream author branch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* fix: preserve bf16 GGUF file when explicitly requested in quantization list
When users request multiple quantization methods including the base format
(e.g., ["q4_k_m", "bf16"]), the bf16 GGUF serves as both the intermediate
conversion and a user-requested output. The cleanup step unconditionally
deleted this file, losing the explicitly requested bf16 output.
Only delete the intermediate base GGUF when the user did not request it.
Fixes#4932
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: keep reverse() outside conditional deletion to preserve VLM ordering
Address review feedback: the reverse() call must always execute when
quants_created is True to maintain correct [text_model, mmproj] ordering
for VLMs. Only the file deletion should be conditional on whether the
user requested the base format.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Ricardo-M-L <ricardoporsche001@icloud.com>
* fix: move preserved base GGUF away from list boundaries for correct example commands
Address review from @Datta0: when the base format (e.g. bf16) is kept
in all_saved_locations, it could end up at [-1], causing the VLM example
command to use bf16 as --mmproj instead of the actual projector file.
Move the preserved base file to index 1 (after the primary quantized
model, before mmproj) so [0] and [-1] remain correct for example
commands.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Ricardo-M-L <ricardoporsche001@icloud.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: use % 8 instead of // 8 in FP8 weight shape check
weight.shape[X] // 8 != 0 is True for any non-zero dimension, causing
incorrect fallback to dequantization for small weights. Using % 8
correctly checks non-divisibility: weights not divisible by 8 should
dequantize, while those with % 8 == 0 stay on the fast kernel path.
* Preserve sharded base GGUF files during cleanup
convert_to_gguf can return multiple base text shards plus an mmproj
entry when llama.cpp splits the output. The previous cleanup only
removed/repositioned base_gguf=initial_files[0]:
- when the base format is NOT in quantization_method, sibling shards
were left both in all_saved_locations and on disk as orphans
- when the base IS preserved, the reverse + insert(1, base_gguf)
step left a sibling base shard at all_saved_locations[-1] for VLMs,
so the example llama-mtmd-cli command ended up with --mmproj
pointing at a text shard instead of the projector
Treat every initial file whose basename does not contain "-mmproj"
as part of the base set, then remove/unlink or reposition all of
them together. Drop the redundant frozenset() construction at both
call sites and the dead `base_gguf in all_saved_locations` clause
in the reorder guard.
* Apply bias in FP8 dequant fallback and dedupe full-precision flag
unsloth/kernels/fp8.py:
FbgemmFp8Linear_matmul.forward had a dequant fallback that called
torch_matmul without adding bias. The fast row-wise branch and the
block FP8 branch both apply `output = output + bias if bias is not
None else output` immediately after the matmul; the fallback now
matches. This silently dropped bias for any FP8 layer routed to the
fallback (Qwen 2.5 VL gate/up_proj 3420x1280, transposed-weight
backward dispatch, and the small-shape cases newly routed here by
the recent `% 8` divisibility fix).
unsloth/save.py:
preserved_base inside the cleanup block and want_full_precision below
it computed the identical expression `first_conversion in
quantization_method`. Hoist want_full_precision above the cleanup
block, reuse it for the not-preserved deletion and the preserved
reposition, and assign True directly in the GPT-OSS branch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: Ricardo-M-L <ricardoporsche001@icloud.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Add process-level tool_policy state for unsloth run
* Apply tool_policy override at chat/completions, /messages, and tool pass-through gates
* Add pure resolver for unsloth run --enable-tools/--disable-tools
* Wire --enable-tools/--disable-tools into unsloth run
* Color tool-policy notices and confirmation prompt in Claude orange
* Always show tool-status notice; print URL + API key in silent mode
* Treat any non-loopback bind as external; forward --yes after parent prompt
* Fix tool_policy double-module bug: import via state.tool_policy to share global with routes
* chore: change API Keys settings to API Access
* chore: replace key with plug svg
* chore: replace API access with developer and related SVG
* chore: rename API keys menu to developer
* fix: focus active settings tab on open
* fix: prevent settings autofocus on open
* Revert "fix: prevent settings autofocus on open"
This reverts commit 4b64d73eda.
* chore: always show API usage examples and docs links
* chore: add colon to API setup docs label
* settings: move help links, add API shortcut, and update API copy
* Studio: forward unknown CLI args directly to llama-server
`unsloth studio run --model X --top-k 20 --chat-template-file foo.jinja`
now passes the unknown flags through to the llama-server subprocess.
Adds a denylist for flags Studio manages (port, -m, -c, --api-key, -ngl,
--flash-attn, --no-context-shift, --jinja, GPU-fit, model-identity, ...)
that returns HTTP 400 on collision. HTTP callers can supply the same
list via LoadRequest.llama_extra_args.
* Studio: accept `--model org/repo:variant` shorthand in `unsloth studio run`
Mirrors llama.cpp's `-hf <repo>:<quant>` and ollama's pull syntax so
`unsloth studio run --model unsloth/gpt-oss-20b-GGUF:UD-Q4_K_XL` is
equivalent to `--model unsloth/... --gguf-variant UD-Q4_K_XL`. Local
paths and Windows drive letters are preserved verbatim. If both an
embedded variant and an explicit `--gguf-variant` are given and they
disagree, the command fails with a clear error.
* Studio: register `unsloth run` as alias for `unsloth studio run`
Top-level `unsloth run --model ...` is now equivalent to
`unsloth studio run --model ...`. Same context_settings, so unknown
flags continue to pass through to llama-server.
* Studio: let users override soft-managed llama-server flags from CLI
Trims the denylist to flags Studio fundamentally cannot share with
the user (model identity, --host/--port/--path/--api-prefix,
--api-key, --ssl-*, --webui, --models-*). Soft-managed flags --
-c/--ctx-size, --parallel, --flash-attn, --no-context-shift,
--jinja, -ngl, -t/--threads, --fit* -- now pass through and override
Studio's auto-set version via llama.cpp's last-wins CLI parsing.
Lets users tune their run on the spot:
unsloth run --model X -c 131072 --parallel 1 --threads 32
* Studio: accept `-hf` / `-hfr` / `--hf-repo` as aliases for `--model`
Matches llama-server's `-hf <repo>:<quant>` spelling so users coming
from llama.cpp can use the same flag. Typer claims the aliases before
the pass-through validator runs, so the HTTP-API denylist on those
flags is unaffected.
unsloth run -hf unsloth/gpt-oss-20b-GGUF:UD-Q4_K_XL
* feat(studio): add Tauri native GGUF intake
* feat(studio): polish native GGUF intake
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): load backend helpers during local setup
* fix(studio): acquire native load lease before unload
* Studio: harden native path lease verification and Tauri intake
- Wrap path.resolve(strict=True) and Path.stat() in NativePathLeaseError so a deleted or unmounted GGUF returns 400 instead of leaking the full filesystem path through the generic load_model/validate_model handler.
- Re-apply _reject_network_or_device_path to the resolved canonical path for defense in depth after symlink resolution.
- Replace try/except ValueError pattern in the device-path guard with Path.is_relative_to; the previous shape silently swallowed NativePathLeaseError (which subclasses ValueError) so /dev,/proc,/sys were never actually rejected.
- Broaden the lease redaction regex and dict-key check (Python and Rust diagnostics) to cover both native_path_lease and nativePathLease so the camelCase form emitted by Tauri/frontend payloads is also redacted.
- Hoist the redact_native_paths import to module top in loggers/handlers; the recursive filter no longer pays a per-record import lookup.
- Persist activeNativePathToken in the chat runtime store so the rollback branch can mint a fresh lease and reload the previous native GGUF when a new load fails after unload; clear it in clearCheckpoint and overwrite it on each successful load.
- use-native-drop: read options through a ref so the Tauri onDragDropEvent listener is registered once and stays attached across option changes; reject ambiguous multi-file drops up front instead of silently registering only the first GGUF.
- pick_native_model: use an async pick_file with a tokio oneshot channel instead of blocking_pick_file so the Tokio worker is not held for the duration of the OS dialog.
- registerNativeModelPath: drop the duplicate sourceKind argument; the Rust command parameter is source_kind.
- install_python_stack: insert the script directory (studio/) on sys.path; the previous insert pointed at studio/backend/ which does not satisfy `from backend.utils.wheel_utils import ...`.
* install_python_stack: keep _BACKEND_DIR on sys.path
Restore the studio/backend insertion. Although the immediately following `from backend.utils.wheel_utils import (...)` is satisfied by studio/ already being on sys.path[0] when invoked as `python studio/install_python_stack.py`, wheel_utils itself runs `from utils.native_path_leases import ...`, which requires studio/backend/ to be importable. Without the backend insertion, the existing tests/python/test_install_python_stack.py collection fails with ModuleNotFoundError: No module named 'utils'.
* Studio: tighten native path lease lifecycle and Tauri intake IPC
- register_native_model_path now hardcodes NativePathSourceKind::Drop on the Rust side and the frontend stops sending source_kind. The previous JS payload (source_kind only) never reached the Rust deserializer because Tauri's default ArgumentCase::Camel maps the Rust parameter source_kind to the JS key sourceKind, so drag/drop registration silently failed. Hardcoding the source kind also keeps audit metadata trustworthy on this command.
- Add native_path_secret_removed_for_child_start context manager and wrap multiprocessing.Process.start() at the inference, export, training, and data-recipe job spawn sites. The previous wrapper-only scrub left UNSLOTH_STUDIO_NATIVE_PATH_LEASE_SECRET visible to spawn-platform import-time worker code. The wrapper run_without_native_path_secret stays as defense-in-depth inside the child.
- Stop passing exc_info=True from the native-grant load/validate error logs in routes/inference.py. The structlog filter_sensitive_data processor runs before the renderer, so ConsoleRenderer formatted tracebacks bypassed redaction; the redacted str(e) preserves the message text.
- Replace the os.path.normcase string equality on the resolved canonical path with Path.samefile (with a normcase fallback) so Windows leases that differ only in extended-length \\?\ prefix or short-name spelling are accepted.
- Wrap consumeNativePathToken in its own try/catch in the chat runtime rollback. If the previous native-model token has aged out of TOKEN_TTL we now surface a clear modelsError instead of silently swallowing the rollback inside the outer catch.
- Reject non-ASCII lease strings in _split_lease and convert UnicodeEncodeError / binascii.Error / ValueError raised by _b64decode into NativePathLeaseError so verify_native_path_lease never escapes raw exceptions to the route handler.
- Tighten dropStateForPaths to mark multi-file payloads invalid so the overlay matches the post-fix drop handler that rejects the same payload.
- Replace the one-shot fetch in useNativePathLeasesSupported with a delayed-retry loop so the picker/drop becomes available once the backend is up rather than staying disabled for the rest of the session after a transient failure.
- Drop the unused setActiveNativePathToken setter; the value is set via setState directly in use-chat-model-runtime.
- Add a toast on auto-load failure in use-native-drop so a collapsed model selector does not hide the error.
- Burn the lease nonce before _validate_current_stat so a stat-failed lease is single-use even if a later state change happens to match the original size/mtime.
* Studio: cache lease secret, harden native path stat checks, polish intake UX
- Cache the decoded UNSLOTH_STUDIO_NATIVE_PATH_LEASE_SECRET on first verify and validate that it is base64-decodable and at least 32 bytes. Subsequent _decode_secret calls return from the cache and never touch os.environ, so concurrent /api/inference/load and /api/health requests no longer race with native_path_secret_removed_for_child_start scrubbing the env. native_path_leases_supported now wraps _decode_secret so the health flag matches what verify_native_path_lease actually accepts.
- Replace path.is_file()/is_dir() + path.stat() with os.lstat() in _validate_current_stat and explicitly reject S_ISLNK; size and mtime checks now refer to the link itself, closing the same-size+same-mtime symlink-swap window that the prior follow-symlink stat() left open.
- Add an issued_at_ms < expires_at_ms sanity check in _validate_payload to reject internally inconsistent (HMAC-protected) lease payloads.
- Sort _NATIVE_PATH_REDACTIONS by length (descending) before iterating in redact_native_paths so a longer registered path is replaced before a shorter prefix path; otherwise logs containing /foo/X.gguf.bak after only /foo/X.gguf was registered would leak the .bak suffix.
- classify_existing_path now re-checks the canonical path with symlink_metadata after canonicalize, so a regular file that is replaced with a symlink in the small canonicalize window is rejected at registration.
- ModelSelector renders the local file picker as its own block (not in the eject ternary), so a user with an active model can still replace it via the picker rather than only via drag/drop.
- useNativePathLeasesSupported caps the readiness probe at MAX_READINESS_POLLS (60 = ~5 minutes) and aborts the in-flight fetch on unmount via AbortController, so a permanently-disabled backend stops generating sustained traffic and hot-reload no longer leaks open connections.
- useChooseNativeModel returns a stable useCallback closure and guards the OS dialog with a useRef so rapid double-clicks cannot open multiple dialogs and orphan Rust tokens.
- Branch the multi-file drop toast: if no GGUF was present we say "Only .gguf model files can be dropped here." and otherwise "Drop a single .gguf model file." so users dropping non-GGUF attachments get an accurate explanation.
* native_path_leases: lstat the signed canonical path before resolving
The earlier change to lstat inside _validate_current_stat operates on grant.canonical_path, which is the post-resolve target. If the user atomically replaces the originally-signed file with a symlink to a different file of identical size and mtime, path.resolve(strict=True) follows the symlink, samefile returns True (both ends share the new inode), and the lstat in _validate_current_stat sees the regular target file rather than the symlink, so the swap goes undetected.
Add an os.lstat on the signed canonical path before path.resolve(strict=True), and reject S_ISLNK there. The lstat in _validate_current_stat stays as defense-in-depth for swaps that occur strictly between resolve and stat.
* Studio: scrub native lease secret before mp.Queue spawn and tighten lease lifecycle
- Move _CTX.Queue / _CTX.Event / _CTX.Process construction inside native_path_secret_removed_for_child_start at the inference, export, training and data-recipe spawn sites. The first Queue creation lazily spawns Python's multiprocessing.resource_tracker child, so when it ran outside the scrub context the tracker process inherited the lease secret. Reproduced via the proc filesystem environ entry; the wrapped order keeps the tracker clean.
- native_path_secret_removed_for_child_start now refcounts entries: the env var is popped on the first entry and restored only when the last context exits. Concurrent training/inference/export starts no longer serialize on the env lock across the entire proc.start yield, while still guaranteeing the env stays empty for the duration of every overlapping spawn.
- run_without_native_path_secret now also nulls the module-level cached lease secret. With the existing spawn-only multiprocessing context the cache is irrelevant in practice, but a future fork caller would otherwise inherit the in-memory secret even though the env var was scrubbed.
- filter_sensitive_data now applies the native lease key check on the top-level event_dict, not only on nested dicts, so a logger call that includes a lease value as a top-level keyword field actually redacts it (the bare value does not match the prefix-anchored regex).
- chat-page loadNativeModelIntent now passes intent.id to clearModelIntent so a second drag-drop during an in-flight first auto-load is not wiped from the chip area when the first resolves.
- Bump useNativePathLeasesSupported's MAX_READINESS_POLLS from 60 to 720 so first-run installs that compile llama.cpp from source or download large CUDA wheels (well past 5 minutes) don't permanently disable the native picker.
* native_path_leases: serialize first-decode against scrub context
_decode_secret used a separate _SECRET_INIT_LOCK from the env scrub's _NATIVE_PATH_ENV_LOCK, so the very first decode (before the cache is populated) could race a concurrent native_path_secret_removed_for_child_start and read os.environ during the env-empty window, raising "Native path grants require the managed desktop backend." Subsequent calls hit the cache and were already safe.
Acquire _NATIVE_PATH_ENV_LOCK around the env read inside _SECRET_INIT_LOCK and fall back to _SCRUB_SAVED_SECRET when the scrub has temporarily popped the env var. Lock ordering (init then env) is consistent with no other caller, so no deadlock.
* Studio: surface native model load errors and harden native path label cache
- Native model load and validate now bubble up the actual exception (with
paths redacted) and apply the same friendly-error rewrite the non-native
path uses, so users see "CUDA OOM", "trust_remote_code required", etc.
instead of a generic "Failed to load native model: <label>".
- run_without_native_path_secret now also nulls _SCRUB_SAVED_SECRET so a
forked grandchild that imports native_path_leases cannot recover the
secret via the scrub-aware fallback in _decode_secret.
- _NATIVE_PATH_LABELS now has its own 10000-entry cap independent of the
100-entry redaction list, so display_label_for_native_path no longer
falls back to returning the raw canonical path after 101 native paths
in one session. Redaction list keeps the 100-entry cap for log-scan
performance.
- _validate_payload now also rejects null bytes in display_label, which
is echoed back in HTTP responses and log lines.
* Studio: harden native path lease validation and chained native rollback
- child_env_without_native_path_secret now copies os.environ under
_NATIVE_PATH_ENV_LOCK so a concurrent scrub-context env pop cannot
raise RuntimeError: dictionary changed size during iteration in a
background hardware scan or other env reader.
- _validate_payload and grant construction route every signed numeric
field (version, issued_at_ms, expires_at_ms, size_bytes, modified_ms)
through new _required_int / _optional_int helpers that wrap raw int()
ValueError into NativePathLeaseError. The single upstream catcher
produces 400 instead of 500 for malformed signed payloads.
- verify_native_path_lease now runs _validate_current_stat before
_consume_nonce, so a transient stat error on the canonical path no
longer permanently burns the nonce. Concurrent verifies still
serialize through _consume_nonce, so single-use is preserved.
- Chained native model rollback now restores activeNativePathToken in
the chat runtime store after a successful rollback loadModel. Without
this, a second consecutive failed switch could not re-roll-back
because the store token had been overwritten by the failed attempt.
- validate_model now applies the same not_supported_hints friendly
rewrite to native model errors that load_model already does, so a
native .gguf that fails validation with an upstream "is not supported"
message gets the same actionable wording as the non-native branch.
* Studio: harden native path log redaction, status disclosure, and chip lifecycle
- structlog processor chain now runs format_exc_info before
filter_sensitive_data so traceback strings are produced (and then
redacted) rather than passed through as untouched (type, value, tb)
tuples that the JSON or console renderer formats after the redaction
filter has already finished.
- native_path_secret_removed_for_child_start clears _CACHED_LEASE_SECRET
in addition to popping the env var, so a fork during the scrub window
cannot inherit the cached bytes via the parent's heap. Parent verify
calls during the window keep working through the existing scrub-aware
fallback in _decode_secret.
- load_model's except ValueError handler now redacts native paths and
uses the native model log label when native_grant_backed is true.
Previously a ValueError raised after lease verification (e.g. from
ModelConfig.from_identifier or downstream GGUF parsing) returned the
raw exception string in the HTTP response body.
- llama_cpp_backend now records the native display label at GGUF load
time, and /api/inference/status prefers it over the redaction store.
After a Python backend restart the redaction store is empty; the
attribute keeps the friendly label, and an absolute model_identifier
with no other label source falls back to the basename so the canonical
path no longer appears in active_model.
- reveal_path_token uses native "reveal and select" commands on macOS
(open -R) and Windows (explorer /select,) so the file is highlighted
in the file manager. Linux keeps the existing parent-directory open.
- Native model rollback that fails because the previous token cannot be
consumed now throws a rollback-specific Error, and the outer empty
catch was replaced with one that re-throws the rollback error. The
rollback-specific message now reaches the user instead of being
overwritten by the original load error message.
- NativeModelChip tracks the Rust token's expiresAtMs on a single
setTimeout, disables the Load button at expiry, and relabels it
"Select again" with an explanatory tooltip so users do not click into
a guaranteed-failure path after the 15-minute TTL elapses.
* Studio: tighten native artifact policy, mmproj sibling check, and intake UX
- is_open_safe_artifact no longer grants Open for directories. Reveal
already handles directory navigation, so the change closes the
attack surface where a macOS .app artifact could be launched via
open_path_token + open::that_detached.
- Display labels are sanitized in classify_existing_path. Control
characters in filenames (newlines, tabs, NUL et al.) are replaced
with spaces and the label is trimmed and capped, so a file named
with embedded newlines cannot inject forged log lines or scramble
the UI status panel.
- validate_entry_path skips the size_bytes/modified_ms equality check
when the operation is Reveal or Open. Cloud-sync agents (Dropbox,
iCloud Drive, OneDrive) routinely rewrite extended-attribute
metadata which bumps mtime, and the user expects Reveal/Open to
remain available for files in synced folders.
- llama_cpp_backend gains a _native_grant_backed flag at GGUF load
success. /api/inference/status only applies the absolute-path
basename fallback when that flag is true, so a non-native absolute
local GGUF still reports its canonical model_identifier and unload
by identifier keeps working.
- Native vision GGUFs now run through _validate_native_mmproj_companion
before llama-server starts: the companion mmproj must be a regular
file, not a symlink, and must live in the same resolved directory as
the granted GGUF. This stops a hostile sibling or symlinked mmproj
from being loaded under a single-file lease.
- Chained native rollback restructured: the rollback loadModel + state
+ refresh runs inside its own try/catch that swallows so the outer
throw error surfaces the ORIGINAL load failure. The native-token
consume-failure case still throws the rollback-specific message
early, before the inner block runs, so its actionable guidance is
preserved.
- Loading-model state and the duplicate-load guard in the chat runtime
hook now compare both the model id and the native path token. Two
drops or picks with the same basename in different folders no longer
silently dedup; the second token is honored.
- chat-page loadNativeModelIntent awaits selectModel before clearing
the pending intent. If selectModel returns early via dedup or
throws, the chip and its token stay so the user can retry instead
of losing the selection.
- NativeModelChip's Reveal button is disabled when the lease has
expired (Rust would reject it anyway), and the Load button label
reads "Expired" instead of "Select again" so the disabled element
no longer promises an action it cannot perform.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Studio bound to 0.0.0.0 by default and the installer silently auto-started
a server at end of install, exposing it on the network without consent and
contradicting the privacy-first / local-only guarantee.
- studio/backend/run.py: run_server() and argparse --host default to 127.0.0.1
- unsloth_cli/commands/studio.py: studio_default() and run() --host default to 127.0.0.1
- install.sh: drop -H 0.0.0.0 from generated launcher template; replace silent
auto-start with a [Y/n] prompt; add cloud/network note to manual hint
- install.ps1: drop -H 0.0.0.0 from PowerShell launcher template; replace
silent auto-start with a Read-Host [Y/n] prompt; add cloud/network note
- studio/setup.sh: drop -H 0.0.0.0 from launch hint; add cloud/network note
- README.md: simplify launch examples to `unsloth studio -p 8888`; note
-H 0.0.0.0 is available for cloud/LAN use
Tests:
- studio/backend/tests/test_host_defaults.py
- tests/studio/test_cli_studio_defaults.py
- tests/sh/test_install_host_defaults.sh
* Fix check for libcurl hearders in install.sh
Checking for `dpkg` has no relation to libcurl headers at all. If the package is installed, then an executable `curl-config` is available in Ubuntu/Debian as it can be seen here:
https://ubuntu.pkgs.org/26.04/ubuntu-main-amd64/libcurl4-openssl-dev_8.18.0-1ubuntu2_amd64.deb.html
This also fix the installation in ArchLinux, provided the needed packages are installed previously as shown in the error message when a package is missing.
* Fix check for libcurl hearders in studio/setup.sh
Use the same check as install.sh.
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* UI: polish studio spacing and profile input radius
* chore: use standard rounded radius for profile input
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* feat: add checkpoint resume for stopped training runs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix:add resume checkpoint helpers
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: use checkpoint parent as resume output dir
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: save optimizer and scheduler state on stop-and-save
Use Trainer._save_checkpoint instead of save_state so resume restores
optimizer momentum and LR-schedule position via the checkpoint-NNN/
subdir written by HF's official path.
* fix: clean up resume training history and startup progress
* fix: preserve resume output dirs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: tighten resume run lookup
* fix: remove stale output-dir lookup
* fix: preserve startup download progress
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* feat: enable deleting fine-tuned chat models
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: tighten fine-tuned model delete guards
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden fine-tuned model deletion edge cases
* fix: reject gguf export delete without variant; surface 503 on backend probe failure
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: narrow loaded-model delete guard by gguf_variant
* fix: clear inference state when cancelling model load
* fix: verify deletion completed and prune empty parent dirs
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
* qwen3.6 unsloth studio support
* Add qwen3.6 causal-conv1d detection
* Update model_mappings.py
moved qwen3.6-27B to thinking train on completion template
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: add --local to setup.sh + overlay unsloth-zoo from git main
setup.sh now accepts --local, which exports STUDIO_LOCAL_INSTALL=1 and
STUDIO_LOCAL_REPO=$REPO_ROOT. install_python_stack.py overlays unsloth-zoo
from git main on top of the editable unsloth checkout in both local_repo
branches (no-torch and with-torch).
The Colab notebook now invokes ./studio/setup.sh --local so the cloned
repo is used in editable mode and unsloth-zoo tracks main, matching the
behavior of install.sh --local on a VM. install.sh --local is unchanged:
it still sets SKIP_STUDIO_BASE=1, which short-circuits the local_repo
branches in install_python_stack.py, so the overlay is not run twice.
* studio: make --local overlays visible + guard empty arg parsing
- setup.sh: gate the --local flag loop on $# > 0 (defensive against any
shell that surfaces unset $@ under set -u) and emit a substep when local
mode is detected so the user can confirm the flag was parsed.
- install_python_stack.py: emit explicit _step lines before each overlay
pip_install in both local_repo branches so overlays appear in the static
log instead of being overwritten by the in-place progress bar.
When --local is passed, also overlay unsloth-zoo from the upstream main
branch (--no-deps --reinstall-package) on top of the PyPI install. This
keeps the editable unsloth checkout paired with the latest unreleased
unsloth-zoo, mirroring the existing -e $_REPO_ROOT --no-deps overlay.
Applied to all four --local paths in install.sh (migrated, fresh no-torch,
fresh with-torch, auto-torch fallback) and the three corresponding paths
in install.ps1.
* MROPE for VLM GRPO
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: pluesclues <136766175+pluesclues@users.noreply.github.com>
* fix _scan_ollama_dir to handle ollama cloud models correctly
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Fix DPO trainer multi process hang
* Fix datacollator error
* further dpo vision changes
* cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden DPO vision row processing and source rewrites
- dpo_trainer_vision_signature_columns: also match TRL 0.22.x layout
(image_sizes followed by ref_chosen_logps), so vision keys are not
stripped via remove_unused_columns on the originally-affected version.
- dpo_trainer_concatenated_inputs: fall back to inserting after the
image_sizes block when no token_type_ids anchor follows it.
- Apply the same vision model_kwargs forwarding rewrite to
_compute_loss_liger via dpo_trainer_compute_loss_liger so the Liger DPO
path does not drop pixel_position_ids/image_position_ids/
mm_token_type_ids when args.use_liger_loss is true.
- dpo_trainer_vision_process_row:
- guard chosen/rejected EOS append with tokenizer.eos_token_id is not None
- use features.get("images") and features.get("prompt") to match the
existing get on line 164 and avoid KeyError on rows without those keys
- drop the torch.is_tensor gate so list-form pixel_position_ids/
image_position_ids returned without return_tensors are still aliased
- skip the loop entry for image_position_ids when it was already
promoted to pixel_position_ids, so the output dict no longer carries
both keys with identical data
- dpo_trainer_data_collator_vision_keys: switch from pad_sequence to
trl.trainer.utils.pad with padding_side='left' (matches the DPO
collator's prompt left-pad) and padding_value=-1 for *_position_ids
keys (sentinel for padded patches), 0 otherwise. Skip the key when not
every example carries it. Falls back to pad_sequence if trl.pad is
unavailable or the tensor rank is too high.
- dpo_trainer_prepare_dataset: keep TRL's writer_batch_size=10 when
popping num_proc; removing it defaults to 1000 and reintroduces the
vision OOM risk that writer_batch_size=10 was set to avoid.
* DPO vision row: keep upstream-facing keys and fix patch padding
- dpo_trainer_vision_process_row: no longer aliases image_position_ids
to pixel_position_ids. Each upstream-emitted vision key is forwarded
under its own name. Gemma4 ForConditionalGeneration.forward accepts
image_position_ids directly and renames it to pixel_position_ids only
at the vision-tower call site, so aliasing in the row helper hid the
kwarg the model actually consumes.
- dpo_trainer_vision_process_row: extract pixel_values via "in"
membership instead of unconditional indexing. With the missing-images
path returning [] to the processor, modern processors no longer emit
a pixel_values key, and the previous indexing raised KeyError.
- dpo_trainer_data_collator_vision_keys: pick padding_side per key
family. *_position_ids tensors are patch-aligned to pixel_values
(TRL's DataCollatorForPreference right-pads pixel_values), so pad
them right with the -1 sentinel; mm_token_type_ids is token-aligned
to prompt_input_ids (left-padded by TRL), so pad it left with 0.
* DPO vision: handle multi-image prompts and arbitrary-rank collator pad
- dpo_trainer_vision_process_row: when a prompt is missing vision
placeholders, insert one placeholder per missing image instead of
always inserting a single token. Multi-image rows now satisfy the
processor's token-vs-image count check rather than under-inserting
and tripping the placeholder/feature mismatch.
- dpo_trainer_data_collator_vision_keys: drop the dim()<=2 gate around
trl.trainer.utils.pad. trl.pad handles arbitrary rank correctly,
while the previous fallback to torch.nn.utils.rnn.pad_sequence
raised RuntimeError on rank-3 patch-position tensors with mismatched
non-leading dimensions. The pad_sequence path remains as a degraded
fallback only when trl.pad is unavailable or raises.
* DPO vision row: support scalar images and align prompt-aligned aux ids
- dpo_trainer_vision_process_row: type-aware normalization of the
features['images'] column instead of a truthiness/len check that
raised on single image objects (PIL.Image has no __len__) and on
numpy ndarrays (truthiness ambiguous). Lists/tuples count as their
length, scalar image objects count as one, None counts as zero, and
the original value is forwarded to the processor.
- dpo_trainer_vision_process_row: when max_prompt_length truncates
prompt_input_ids, also slice token_type_ids and mm_token_type_ids
by the same [-max_prompt_length:] suffix. Those keys are 1:1 token
aligned to prompt_input_ids (Gemma 4 vision attention keys off
mm_token_type_ids per modular_gemma4.py), so leaving them at the
original length silently misaligned the multimodal mask.
* DPO vision row: stop synthesizing vision-token placeholders
Pass features['prompt'] and features['images'] straight to the
processor without inserting any extra placeholder tokens. The previous
helper used processing_class.image_token, which is the right prompt
placeholder for Gemma 4 but the wrong one for Gemma 3 (whose prompt
placeholder is boi_token while image_token is the inner expansion
target). Synthesizing that token also broke multi-image rows: text
ended up with N placeholders while the row helper only forwarded the
first image's pixel_values via the standard [0] indexing that mirrors
upstream TRL process_row, so token vs image-feature counts diverged.
Removing the synthesis matches stock TRL behavior; users provide the
correct placeholders for their processor in the prompt.
* Add tests for DPO vision row processor passthrough
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* fix: clipped model selector text descenders
* Studio: Fix image-only chat requests failing validation (#5212)
* fix: allow image-only chat messages
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: deduplicate empty content validation coverage
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix descender clipping in sidebar user account section
Replace `leading-none` with `leading-tight` on the parent div wrapping
`displayTitle` and the "Studio" label inside `SidebarMenuButton`. The
child spans use `truncate` (overflow: hidden), so `line-height: 1`
clipped descenders (g, p, q, y, j) on user names. Same root cause and
fix as the model selector trigger.
* Add tests for studio text descender clipping
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Patch checkpoint reload init functions to strip unsupported args
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Try adding attrs back if possible
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* import_fixes: harden peft weight converter compatibility shim
Three small fixes to patch_peft_weight_converter_compatibility:
- Restore peft_config=None default and @functools.wraps on the
build_peft_weight_mapping wrapper so the upstream coordinated
signature (weight_conversions, adapter_name, peft_config=None) is
preserved. Without the default, callers using the documented
two-argument form raise TypeError after import unsloth.
- Serialize the temporary class-init patch/restore behind a
threading.RLock. The previous unsynchronized window let two
concurrent build_peft_weight_mapping calls (e.g. dynamic LoRA
serving) re-expose the original distributed_operation TypeError
when one thread restored a class while another was still inside
original_build.
- Hand _patch_weight_converter_ctors a caller-owned accumulator
list and append in place. If signature inspection ever raises
mid-loop, the finally block now sees the partial list and
restores already-patched classes instead of leaving them with
the compat init permanently installed.
* Add tests for peft weight converter compatibility shim
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: fix 4 failing studio_unit_tests on main
Three of the failing tests had drifted from production:
1. test_health_response_reports_desktop_capability_fields stubbed
`routes` with a SimpleNamespace that omitted `inference_studio_router`,
so importing studio.backend.main raised ImportError. Add the missing
router stub.
2. test_local_recipe_token_preserves_desktop_marker and
test_local_recipe_token_keeps_web_marker_absent decoded the local
provider's api_key as a JWT, but _inject_local_providers now mints
a unified sk-unsloth-* internal API key (not a forwarded JWT), so
jwt.decode raised "Not enough segments". Renamed and rewrote both
tests to validate the API-key contract: starts with
storage.API_KEY_PREFIX and authenticates via get_current_subject as
the real admin user. The web vs desktop distinction is irrelevant
at this layer because the unified API-key path does not carry
session flags.
The fourth failure was a real production bug:
3. test_github_validate_skips_live_access_with_honest_note expected
github-seed validation to return valid=True per
_GITHUB_VALIDATE_NOTE ("GitHub access and rate limits are checked
when the run starts"). The validate route called
build_config_builder which lazy-imports the optional data_designer
module; when it is missing, the bare except blocked the recipe.
Catch ImportError specifically and treat it as a deferred check,
matching the documented intent.
Verified all 4 tests pass and the rest of studio/backend/tests still
pass (608 total, with the only remaining failures being environment
specific: 4 GPU-aware tests on a no-GPU host and 1 Anthropic-API
smoke test, both unrelated).
* Studio: fix 3 test_gpu_selection route tests after load_model signature change
`routes/inference.load_model` gained a `fastapi_request: Request`
positional argument (used to read `app.state.llama_parallel_slots`
inside the GGUF path), but the three TestRouteErrors cases that
exercise the early validation path were not updated and failed with
`TypeError: load_model() missing 1 required positional argument:
'fastapi_request'`.
Pass a SimpleNamespace mock that satisfies the attribute path the
production code reads. The validation under test fires before the
mock is consumed, but supplying the realistic shape protects against
regressions if the validation order changes.
Affected tests:
- test_inference_route_rejects_gpu_ids_for_gguf
- test_inference_route_returns_400_for_invalid_gpu_ids
- test_inference_route_returns_400_for_uuid_parent_visibility_gpu_ids
* Studio: address review feedback on validate.py ImportError handling
Two reviewers flagged the ImportError bypass added in b0d33cf:
- chatgpt-codex-connector[bot]: catching bare ImportError marks recipes
as valid even when build_config_builder fails for unrelated import
problems (broken internal imports, missing transitive deps after a
version bump), hiding real regressions until run start.
- gemini-code-assist[bot]: silent pass discourages troubleshooting;
the deferred-validation case should be logged at debug level.
Tighten the bypass to ModuleNotFoundError where the missing module name
starts with "data_designer". Other ImportErrors propagate to the outer
handler and surface as validation failures, restoring the visibility
the reviewers asked for. Add a debug-level log entry that names the
missing module so operators can trace why validation deferred.
* UX: Refine chat preset and group built-in presets
* fix: reuse built-in preset names and unify GGUF state reads
* fix: built-in chat preset save and refresh behavior
* Add chat preset invariant tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: decouple chat presets from model-specific settings
Limit chat preset compare/apply/save behavior to temperature, topP, topK, minP, repetitionPenalty, presencePenalty, maxTokens, and systemPrompt.
Preserve legacy stored preset data on load for backwards compatibility, but stop treating model-specific settings such as checkpoint, trustRemoteCode, and maxSeqLength as part of preset identity.
Also align legacy prompt migration dedupe with the new preset semantics and add invariant coverage for preset-owned config comparisons.
* fix: detect built-in preset edits from param changes
* fix: correct built-in preset dirty state and speculative select values
* fix: preserve default preset sync and keep qwen think pristine
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* UX: single chat header error placement and selector alignment
* fix: centre model dropdown chevron
* fix: revert view mode single
* fix: allow model selector label to truncate in narrow headers
* fix(studio): use py.exe to detect supported Python on Windows
Description:
The previous detection looked at `python --version` on PATH and
hard-failed if the resolved Python wasn't 3.11-3.13. On systems
where Python 3.14 sits ahead of 3.13 in PATH order, this aborted
the installer even though a supported interpreter was installed.
Prefer the py.exe launcher and probe `py -3.13`, `py -3.12`,
`py -3.11` in turn. Fall back to `python --version` only when py.exe
is absent, and surface a clearer error when no supported version
can be found via either path.
* Studio: consolidate Windows studio overlay into single Tauri-gated block
Replace the in-file sentinel hotfix and the unconditional file-copy
overlay with a single block gated on $TauriMode. Hash-compare makes
re-runs no-ops, removing the sentinel-clobbering bug that occurred
when the second copy path overwrote the marker without re-adding it.
Non-Tauri --local installs no longer need a copy overlay: the
editable install above (uv pip install -e $RepoRoot --no-deps) makes
_PACKAGE_ROOT in unsloth_cli/commands/studio.py resolve to the repo
source tree via PEP 660 __file__-relative resolution, so
`unsloth studio setup` finds the local setup.ps1 and
install_python_stack.py without any file copying.
Plain PyPI installs invoked from a checked-out repo directory are
also no longer silently overlaid from cwd.
* fix(studio): work around uv space-in-path truncation on Windows
uv 0.11.x truncates `-c <path>` and `-r <path>` arguments at the
first space, breaking installs on Windows when the venv or repo
sits under a path containing spaces (e.g. C:\Users\First Last\...).
Pass paths through GetShortPathNameW to convert to 8.3 short form
before handing them to uv. Plain pip is unaffected and keeps the
original long path. No-op on Linux/Mac (gated on IS_WINDOWS and
on the path actually containing a space).
* Refactor Python stack overlay logic in install.ps1
Refactor overlay logic for Python stack installation and improve handling of missing target directories.
* Update Python installation logic in setup.ps1
* Studio: add github_repo seed reader and GitHub Support Bot recipe
Adds a first-party Data Designer seed reader that scrapes GitHub issues,
pull requests, and commits from one or more repositories via the GraphQL
API, and a learning recipe (GitHub Support Bot) that turns those rows into
synthetic support Q&A pairs for fine-tuning.
Backend (new plugin studio/backend/plugins/data-designer-github-repo-seed):
* GitHubRepoSeedSource config: repos, token (falls back to GH_TOKEN /
GITHUB_TOKEN env var), item_types (issues / pulls / commits),
per-resource limit (0 means all), max_comments_per_item.
* Rate-limit-aware GraphQL client (GitHubClient + RepoScraper) shared
across repos; flattens each item into a uniform row with columns
item_type, repo, number, title, body, state, author, created_at,
closed_at, url, labels, comments.
* Registered via the data_designer.plugins entry point.
Frontend:
* New seed_github block variant so the seed node card shows
"GitHub repositories" instead of the generic "Document file"
placeholder, with its own icon and inline summary (repo count +
item-type list).
* Rewritten seed dialog github_repo form: repos textarea pre-filled with
unslothai/unsloth + unslothai/unsloth-zoo, password input for the GH
token, items-per-repo number with an "All" toggle, and the noisier
options (item types, max comments, include comments) tucked under an
Advanced collapsible.
* Local model auto-load on Run: if a recipe uses an is_local provider
and the inference server is not already serving that model, the
executions hook calls /api/inference/load first. Removes the "open
/chat to load a model" prerequisite that users kept tripping on.
* Honor the recipe's run.rows value in the Run dialog (previously the
store reset to 5 regardless of what the template shipped).
Recipe (studio/frontend/src/features/data-recipes/learning-recipes/
github-support-bot.json):
* Defaults to the Local Model provider + unsloth/gemma-4-E2B-it-GGUF.
* Scrapes unslothai/unsloth and unslothai/unsloth-zoo, issues and pulls,
up to 100 items per resource.
* Two LLM blocks: normalized_question (llm-text) rewrites each thread
into a clean support question, support_answer (llm-structured)
produces JSON with answer / diagnosis_questions / cites / confidence.
* Run defaults to 10 rows for a quick smoke test.
Verified end-to-end on a running Studio: card renders, source-data
dialog is pre-populated, All toggle disables the limit input, the
recipe executes and produces rows against a loaded local GGUF.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: improve GitHub recipe support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: speed up GitHub scraper and harden the support-bot recipe
Addresses a perf issue found while demoing the github_repo seed reader:
Scraper is too slow at scale. The PRs GraphQL query pulls deeply nested
fields (reviewThreads, reviews, commits, timelineItems, etc.) so the
page size was pinned at 3 to stay under GitHub's node-count ceiling. 100
PRs meant 34 serial round trips. Added lighter query variants
(PRS_PAGE_QUERY_LIGHT, ISSUES_PAGE_QUERY_LIGHT) that drop the fields the
Studio flatten layer does not use (it only reads title, body, state,
author, labels, comments). With the light query PR pages can safely go
to 25 per page and issues to 50. The plugin scraper now passes
light=True to RepoScraper so Studio always uses the fast path; the heavy
query remains available for other callers.
Recipe defaults are now demo-ready with production knobs called out:
- max_parallel_requests: 1 and max_tokens: 800 so small local models
stay stable when running the support_answer structured column.
- support_answer prompt trimmed to 80-200 words so gemma-4-E2B GGUF can
actually comply with the schema. The canonical 150-300 word codex
prompt is still documented in the node3 markdown note for
production upgrades.
* Studio: rename GitHub recipe to 'GitHub Scraper' and add Easy mode
Changes the recipe framing from a single-purpose 'Support Bot' pipeline
to a general-purpose scraper that produces {user_request,
grounded_response} training pairs. Aligns with the canonical
github_data_gatherer dataset (11 enrichment tasks mirrored in pr_requests_20
/ issue_requests_20 on the input side and explain_pr / issue_fix_plan /
issue_solution on the output side).
Recipe JSON changes:
- columns[0] renamed normalized_question -> user_request, prompt now
inverts a GitHub thread into a realistic user ask instead of
normalising it.
- columns[1] renamed support_answer -> coauthor_response, emits
{response, followups, cites, task, confidence} and branches on
issue vs PR thread type.
- Notes rewritten to document the 11-task catalog and the canonical
production prompt to paste in for a full dataset backfill.
Frontend: Easy mode for github_repo recipes. The drag-and-drop canvas is
hidden behind an 'Advanced' tab; Easy mode is the default for any recipe
whose seed_source_type is github_repo. The Easy form reuses the existing
GithubRepoSeedForm (promoted to exported), adds a rows input bound to
previewRows, a model field bound to the model_config, and a single Run
button that calls runPreview() directly (no modal). Non-github recipes
see the same Editor / Runs tabs as before.
View mode persists per-recipe-id in localStorage under
recipe-studio:view-mode:<recipeId>.
* Studio: auto-detect server GH_TOKEN and widen Easy-mode detection
The GitHub seed form now fetches /api/data-recipe/seed/github/env-token
on mount and, when the server exposes a GH_TOKEN / GITHUB_TOKEN env var
and the token field is blank, shows a small 'Using server env var' badge
and swaps the placeholder text. The token value itself is never returned
to the UI.
Widens Easy-mode detection in recipe-studio-page.tsx so that recipes
saved before ui.seed_source_type was persisted also get the Easy tab:
falls back to recipe.seed_config.source.seed_type, which is always
present for github_repo seeds.
* fix: polish GitHub recipe UI
* Studio: default llama-server --threads to -1 (auto)
Previously we passed --threads only when the caller set an explicit
value, which meant llama-server fell back to its internal default.
That default has varied across llama.cpp builds (some versions use
hardware concurrency including hyperthreads, which hurts throughput on
CPU-heavy inference). Always passing --threads -1 pins the behaviour
to llama.cpp's auto-detect (physical cores).
Caller-supplied n_threads still wins when non-None.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch Easy mode to Runs pane on run start
Easy mode had no progress island or canvas overlay, so after clicking Run
the only visible state was the button label flipping to "Running..." while
the screen otherwise stayed identical. This reads as stuck even though the
job is progressing.
Wire an onExecutionStart callback from recipe-studio-page.tsx through to
useRecipeExecutions so that when a run is kicked off from easy mode, the
page flips to the executions view where the Runs sidebar, progress bar,
rate/ETA panel, and live log are rendered. Advanced/editor mode keeps its
existing behavior and stays on the canvas (it already has the floating
ExecutionProgressIsland).
* fix: clean up GitHub scraper layout
* Studio: forward llm-structured output_format as llama-server response_format
Local GGUF runs of llm-structured columns used to generate the full
max_tokens budget before the prompt-level "return JSON in a ```json
fence" instruction got parsed. Small models (e.g. gemma-4-E2B-it)
routinely broke format, so each row took ~65s and frequently failed
with "No parsable JSON structure within ```json markdown fence".
For any local-provider model_config referenced by an llm-structured
column, clone the model_config and inject response_format into the
clone's inference_parameters. Uses llama.cpp server's flat shape
(tools/server/README.md):
{"type": "json_schema", "schema": <output_format>}
Not the OpenAI-nested form; data_designer's OpenAI adapter forwards
response_format verbatim via facade._COMPLETION_REQUEST_FIELDS, and
llama-server's documented schema path expects the flat variant.
The clone is per (model_alias, column) so:
- llm-text / llm-judge columns that share the same alias keep
free-form sampling.
- Each structured column gets its own schema, so columns with
different output_formats don't collide.
Effect on gemma-4-E2B-it demos: every row parses cleanly, and the
model terminates immediately after the closing brace instead of
running to max_tokens. Net wall-clock is usually faster even though
grammar-constrained sampling is slightly slower per token.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: flip Easy to Runs pane before validation scrape, not after
Previously onExecutionStart fired inside runExecution, which runs AFTER
validateRecipe() -- and validation re-invokes the seed reader. For the
github_repo reader that is a full GraphQL scrape, so the user sat on a
"Running..." button with an otherwise unchanged Easy form for 10-15s
before anything moved.
Call onExecutionStart at the top of runWithValidation, right after we
have a payload to send. The view flips immediately; ensureLocalModelLoaded
+ validateRecipe now run against the Runs pane instead of a frozen Easy
form. runExecution still calls onExecutionStart downstream, but the
callback is idempotent (the page's easy -> executions guard skips the
second call), so no behaviour change for runs that pass validation.
If validation fails the toast + runErrors path still fires; the Easy
form's error banner still reads runErrors when the user switches back.
* Studio: unify data-recipe workflow auth on sk-unsloth-* keys
The previous commit (a61b4cc9) assumed storage.create_api_key(..., internal=True)
and storage.revoke_internal_api_key(key_id) existed, but those helpers were
only in the working tree, never committed. Recipe runs in local-model mode
were therefore crashing with 500 when _inject_local_providers tried to mint
a workflow key. This commit ships the missing pieces.
auth/storage.py:
- api_keys schema gains is_internal INTEGER DEFAULT 0 (with a guarded
ALTER TABLE migration so existing auth.db files upgrade in place).
- create_api_key takes an internal=False kwarg; internal keys are flagged
so they can be hidden from user-facing listings.
- list_api_keys takes include_internal=False so UIs never see workflow keys.
- New revoke_internal_api_key(key_id): id-only revoke for keys minted by
non-user subjects (the JobManager does not know a username).
core/data_recipe/jobs/manager.py:
- JobManager.start accepts internal_api_key_id and stores it on Job so
lifecycle handlers can revoke eagerly.
- _handle_event revokes on EVENT_JOB_COMPLETED / _ERROR / _CANCELLED.
- _pump_loop subprocess-died fallback also retires the key so a crashed
worker cannot leak a live sk-unsloth-* beyond its TTL.
- Revocation is best-effort (swallow exceptions) -- the 24h TTL is the
safety net if storage hiccups.
core/data_recipe/jobs/types.py:
- Job dataclass gains internal_api_key_id: int | None = None.
Replaces the bespoke 24h JWT path that jobs.py used to mint for local
providers. One mint/revoke/verify surface for every API key the server
issues, and revocation is now eager (seconds, not 24h) instead of TTL-only.
* Studio: plug workflow-key leak on unexpected create_job errors
Review follow-up on the sk-unsloth-* workflow-key lifecycle in
create_job. Previously the revoke handlers wrapped mgr.start(...) but
only caught RuntimeError and ValueError, and get_job_manager() sat
outside the try block entirely. Any other exception type (TypeError
from a mismatched kwarg, OSError from the queue write, etc.) would
bubble up to FastAPI and leave the minted key live until its 24h TTL.
Fix: one try block covers both get_job_manager() and mgr.start(), with
a trailing except Exception that revokes and re-raises. The
RuntimeError -> 409 and ValueError -> 400 paths are unchanged so
specific client-facing status codes still surface. Revocation is still
best-effort (_revoke_internal_api_key_safe swallows errors) because we
never want revoke failures to mask the original crash.
Severity is low -- the key can't bootstrap longer access and the 24h
TTL bounds the window -- but the reviewer's point stands: eager revoke
on every failure path is the right invariant.
* Studio: nest response_format under extra_body so pydantic accepts it
The previous commit dropped response_format at the top level of a cloned
model_config's inference_parameters, which BuilderConfig rejected with:
ValidationError: Extra inputs are not permitted [type=extra_forbidden]
data_designer.model_configs.1.inference_parameters.response_format
data_designer's BaseInferenceParams is a pydantic model with extra=forbid
and only a fixed set of fields (temperature, top_p, max_tokens,
max_parallel_requests, timeout, extra_body). The pass-through path for
anything the schema doesn't know about is `extra_body`, which the
OpenAI SDK spreads into the chat-completions request body at the top
level -- which is exactly where llama-server reads response_format from.
Inject under extra_body (merging with any existing extra_body contents)
so the clone validates. llama-server still receives
{"type": "json_schema", "schema": <output_format>} at the top level of
the request body, which is the flat shape llama.cpp's server expects.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: forward response_format to llama-server and fence-wrap the reply
Two-part fix for the llm-structured data-recipe path:
(1) The /v1/chat/completions proxy was dropping response_format. The
route's passthrough branch only triggered on tools / tool messages, so
requests carrying a JSON schema fell into the non-passthrough GGUF path
which calls generate_chat_completion (no response_format kwarg). The
schema never reached llama-server, so guided decoding was a no-op and
the model emitted free-form text that happened to parse a fraction of
the time. Widen the passthrough trigger and teach _build_passthrough_payload
to forward response_format so llama-server's GBNF grammar actually runs.
Guided decoding does not require supports_tools, so split the condition:
a request is now passthrough-routed if it carries tools/tool messages
(existing behavior) OR carries response_format (new). The vision guard,
streaming fork, and tools-choice defaulting are unchanged.
(2) data_designer's llm-structured parser looks for a ```json ... ```
markdown fence and discards anything else. Guided decoding emits only
the JSON object (the GBNF grammar has no fence tokens), so a
100%-valid schema-constrained run still ended up 0 ok / N failed with
"No parsable JSON structure within ```json markdown fence". In
_openai_passthrough_non_streaming, wrap each choice's content in the
expected fence when the caller asked for guided decoding. Already-fenced
content is left alone so other clients that prefer raw JSON are not
affected; the wrap is scoped to requests that carried response_format.
Net effect on the GitHub Support Bot recipe on a local GGUF: schema
actually binds during sampling, content arrives wrapped in the fence
data_designer expects, and generation terminates immediately after the
closing brace instead of running out to max_tokens.
* Studio: Easy mode runs a full run, capped at the user's row count
Easy mode used to call runPreview, which produces a test run: no
artifact persisted, reduced progress tracking, and framed in the Runs
pane as "Test run". The whole point of the form is to let a user kick
off a real dataset build with one click, so wire it to runFull instead
and bind the Rows input to fullRows (not previewRows).
runFull requires a non-empty fullRunName. The Easy form has no run-name
input, so seed a default on mount whenever Easy is active and
fullRunName is still empty. Uses `<recipe name> <iso-timestamp>` so
each Easy run gets a stable-ish default that still sorts chronologically
in the Runs pane. User can override it from the Advanced run dialog
before clicking Run.
Rename GithubScraperEasyView's rows props from previewRows/setPreviewRows
to rows/setRows so the view stays agnostic to which hook state the page
chooses to bind. Loading indicator now follows fullLoading.
* Studio: clamp GitHub scrape page size and memoize the materialization
Two wins for the "before Generating fires" gap on small previews:
(1) scrape_{issues,prs,commits} hardcoded per_page (50 / 25 / 100) and
only checked the trial limit AFTER the page was written, so a 1-row
Easy run still asked GitHub for a full 50-issue + 25-PR page, wrote
them all to JSONL, and then stopped because total_new already exceeded
the trial cap. Cap per_page at min(page_cap, trial_limit) so
github_limit=1 actually asks for first:1.
(2) GitHubRepoSeedReader.get_dataset_uri used to scrape fresh on every
invocation. data_designer calls the seed reader multiple times per
recipe job (validation, preview, per-column sampling), so a 2-repo
Easy preview ran the full GraphQL scrape three times back-to-back,
burning ~15s of dead air before any LLM generation began.
Added a module-level in-process cache keyed on
(repos, item_types, limit, include_comments, max_comments_per_item,
sha256(token)[:16]) that stores the JSONL path of the first
materialization. Subsequent calls with the same signature return the
cached path, guarded by a staleness check that drops the entry if the
file was tmp-cleaned. Raw token values never land in the key.
Net effect on a 1-row Easy run, 2 repos, limit=1: 2 GraphQL round
trips instead of ~12, and the first-to-Generating gap collapses from
~15s to roughly 2-3s.
* Studio: make Easy mode Rows input editable instead of snapping to 1
The Rows to generate input used type="number" with value bound directly
to the rows state and an onChange that coerced any non-positive parse
result back to 1. The moment the user pressed backspace to clear the
field, the parent re-rendered with value=1 and the caret jumped, making
it impossible to change the value without arrowing the browser's +/-
spinner.
Switch to a text input with inputMode="numeric" and pattern="[0-9]*"
(so mobile still shows a numeric keyboard, and the browser drops the
spinner buttons the user did not want). Add a local rowsText buffer so
the field can hold transient empty / partial digit strings while
editing without fighting the parent state; the canonical rows value
only advances when the buffer parses to a valid integer in [1, 10000],
and onBlur clamps back to 1 or 10000 if the user left it out of range.
No behavior change for valid numeric edits - the downstream runFull()
still sees a clean positive integer.
* Studio: expand dataset cells horizontally by column on click
Click a long cell to expand that whole column. Click again to collapse.
Replaces the prior row-level vertical expansion which made it hard to
compare cells across columns. State is scoped per execution and per
column; the row itself is no longer a click target.
* Studio: force expanded dataset column to grow wide enough to read
* Studio: disable thinking for local recipe inference and plumb the kwarg
Reasoning-capable models (gemma-3n, qwen3.5, etc.) emit a
<think>...</think> preamble ahead of the answer by default, which
roughly doubles the generated token count per row on a local GGUF
and pushes the actual answer past data_designer's json-fence regex
on llm-structured columns. Recipes want the terse answer, not the
scratchpad.
Two halves of the fix:
(1) routes/data_recipe/jobs.py: when _inject_local_providers walks
the recipe's model_configs to point them at the local endpoint, also
stash chat_template_kwargs={"enable_thinking": false} under each
config's inference_parameters.extra_body. OpenAI SDK spreads
extra_body into the top-level request body, so llama-server and the
Studio /v1/chat/completions route both see it.
(2) routes/inference.py: the chat-completions route previously
dropped chat_template_kwargs on the floor because the whitelist
body builder only forwarded known fields.
- At the top of openai_chat_completions, lift
chat_template_kwargs.enable_thinking from payload.model_extra
onto the typed payload.enable_thinking field when the caller
did not set the latter, so the non-passthrough GGUF path's
generate_chat_completion(...) call honors the override.
- Teach _build_passthrough_payload to forward a
chat_template_kwargs dict, and have _build_openai_passthrough_body
derive that dict from payload.enable_thinking so
response_format requests (structured columns) also land at
llama-server with the reasoning preamble suppressed.
Net effect on a 10-row support-bot run with gemma-4-E2B-it-GGUF:
responses arrive without <think> tags, wall-clock per call drops
roughly in half, and structured columns stop leaking reasoning
tokens through the GBNF-constrained output.
* Studio: update GitHub Support Bot learning recipe with maintainer layout
Replace the template with the hand-laid-out export from the maintainer
so note nodes ship with real x/y positions (scattered around the
graph instead of all stacked at x=480) and the edges / canvas pan look
correct on first load. Also picks up the maintainer's prompt tweaks and
output schema names (coauthor_response / user_request / followups / task /
cites / confidence).
Diff is mostly ui.nodes positions and prompt bodies; runtime shape is
unchanged (seed_config / columns still target model_1 against the Local
Model provider).
* Studio: auto-size dataset sample columns; wide text gets a wide column
Drop the per-column click-to-expand toggle and the 180-char truncation.
Every column now renders its full value. Columns with long text get a
min-w of 48rem so the text is readable without wrapping into a tall
block; narrow-content columns get a 12rem min-w. The table wrapper
already has overflow-x-auto, so wide-column totals cause a horizontal
scrollbar instead of cramming everything into the viewport.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix GitHub scrape progress
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add resetApiBase export for test setup
* Studio: rename github-support-bot output columns to User / Assistant
Previously emitted user_request and coauthor_response, which did not
match the canonical User / Assistant chat-pair shape that downstream
SFT consumers expect. Renamed the columns in the recipe JSON (columns,
UI node ids, edges, notes, prompt Jinja refs) and the matching copy in
the learning-recipes index, data-recipes-page, and easy view.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: make stop button actually stop generation
The UI stop button routes through assistant-ui's cancelRun, which aborts
the frontend fetch. Four issues combined to let llama-server keep decoding
long after the user clicked stop:
1. request.is_disconnected() does not fire reliably behind proxies
(e.g. Colab) that don't propagate fetch aborts.
2. llama-server defaults n_predict to n_ctx when max_tokens is not sent,
so a cancelled request keeps producing tokens up to 262144.
3. The httpx.Client pool keeps TCP keep-alive, so even a cleanly closed
stream reuses the same connection and llama-server's liveness poll
never sees a disconnect.
4. No explicit backend route to cancel - every cancel path relied on
is_disconnected.
Changes:
- Add POST /api/inference/cancel keyed by session_id/completion_id, with
a registry populated for the lifetime of each streaming response.
- Have the frontend (chat-adapter.ts) POST /inference/cancel on
AbortController abort, alongside the existing fetch teardown.
- Send max_tokens=4096 + t_max_predict_ms=120000 as defaults on every
outbound chat completion to llama-server; honoured by user overrides.
- Disable httpx keep-alive on the streaming client so connection close
reaches llama-server and its 1s liveness check fires.
No behaviour changes for non-streaming paths or for existing callers
that already pass max_tokens/session_id.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: harden stop-button cancel path and scope cancel route
- Require at least one identifier for /api/inference/cancel so a missing
thread id cannot silently cancel every in-flight generation.
- Scope /cancel to a dedicated studio_router so it is not exposed under
the /v1 OpenAI-compat prefix as a surprise endpoint.
- Store a set of cancel events per key in _CANCEL_REGISTRY so concurrent
requests on the same session_id do not overwrite each other, and
deduplicate in _cancel_by_keys so the cancelled count reflects unique
requests.
- Always send session_id with chat completions (not only when tools are
enabled) so non-tool GGUF streams register under it and are reachable
from /cancel.
- Register the non-GGUF stream_chunks path in the cancel registry too,
so transformers-based stop-button works behind proxies that swallow
fetch aborts.
- Only apply the 2-minute t_max_predict_ms wall-clock cap when the
caller did not pass max_tokens, so legitimate long generations on
slow CPU/macOS/Windows supported installs are not silently truncated.
- Remove the abort listener on normal stream completion so reused
AbortSignals cannot fire a spurious cancel POST after the fact.
* studio: close cancel-race and stale-cancel gaps in stop path
- Register the cancel tracker before returning StreamingResponse so a
stop POST that arrives during prefill / warmup / proxy buffering
finds an entry in _CANCEL_REGISTRY. Cleanup now runs via a Starlette
BackgroundTask instead of a finally inside the async generator body.
- Add a per-run cancel_id on the frontend (crypto.randomUUID) and in
ChatCompletionRequest so /api/inference/cancel matches one specific
generation. Removes the stale-cancel bug where pressing stop then
starting a new run in the same thread would cancel the retry.
- Apply t_max_predict_ms unconditionally in all three llama-server
payload builders (previously gated on max_tokens=None, which made it
dead code for UI callers that always send params.maxTokens). Raise
the default to 10 minutes so slow CPU / macOS / Windows installs are
not cut off mid-generation.
- Make _cancel_by_keys refuse empty input (return 0) so a future
internal caller can not accidentally mass-cancel every in-flight
request.
- Accept cancel_id (primary), session_id, and completion_id on the
/api/inference/cancel route. Unify the three streaming sites on the
same _cancel_keys / _tracker variable names.
- Annotate _CANCEL_REGISTRY as dict[str, set[threading.Event]].
* Add review tests for PR #5069
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: harden stop-button cancel semantics and wall-clock cap
- Make /inference/cancel match cancel_id EXCLUSIVELY when supplied.
Previously the handler iterated ('cancel_id','session_id','completion_id')
and unioned matches, so a stale cancel POST carrying {cancel_id:old,
session_id:thr} would still cancel a later run on the same thread via
the shared session_id. cancel_id is now a per-run exclusive key;
session_id / completion_id are only used as fallbacks when cancel_id
is absent.
- Close the early-cancel race. If /inference/cancel lands before the
streaming handler reaches _TrackedCancel.__enter__() (stop clicked
during prefill / warmup / proxy buffering), the cancel was silently
dropped. Stash unmatched cancel_ids in _PENDING_CANCELS with a 30 s
TTL; _TrackedCancel.__enter__() now replays any matching pending
cancel by set()-ing the event immediately after registration.
- Make t_max_predict_ms = _DEFAULT_T_MAX_PREDICT_MS conditional on
max_tokens is None at all three llama-server payload sites. The cap
is a safety net for callers who leave max_tokens unset (otherwise
llama-server defaults n_predict to n_ctx, up to 262144). Callers who
set an explicit max_tokens are already self-limiting and must not be
silently truncated at 10 minutes on slow CPU / macOS / Windows
legitimate long generations.
- Guard each StreamingResponse return with try/except BaseException so
_tracker.__exit__ runs even if StreamingResponse construction or any
preceding statement raises between _tracker.__enter__() and the
BackgroundTask attachment. Prevents a registry leak on that narrow
window.
* studio: close TOCTOU race and restore wall-clock backstop on UI path
- Close TOCTOU race in the pending-cancel mechanism. The previous fix
split cancel_inference's (cancel_by_keys + remember_pending_cancel)
and _TrackedCancel.__enter__'s (register + consume_pending) into
four separate lock acquisitions. Under contention a cancel POST
could acquire-then-release the lock, find the registry empty, and
stash ONLY AFTER __enter__ had already registered and consumed an
empty pending map -- silently dropping the cancel. Both call sites
now do their work inside a single _CANCEL_LOCK critical section, via
the new atomic helper _cancel_by_cancel_id_or_stash() and an
inlined consume-pending step in __enter__. Reproduced the race under
forced interleaving pre-fix; 0/2000 drops post-fix under parallel
stress.
- Apply t_max_predict_ms UNCONDITIONALLY at all three llama-server
payload sites. The previous iteration gated the cap on
`max_tokens is None`, which turned out to be dead code on the
primary Studio UI path: chat-adapter.ts sets
maxTokens=loadResp.context_length after every model load, so every
chat request carries an explicit max_tokens and the wall-clock
safety net never fired. The cap's original purpose is to bound
stuck decodes regardless of the token budget; it must always apply.
- Raise _DEFAULT_T_MAX_PREDICT_MS from 10 minutes to 1 hour. 10
minutes was too aggressive for legitimate slow-CPU chat responses
(a 4096-token reply at 2 tok/s takes ~34 min); 1 hour accommodates
that and still catches genuine zombie decodes.
- Prune _PENDING_CANCELS inside _cancel_by_keys as well, so stashed
entries expire proportionally to overall cancel traffic rather than
only to cancel_id-specific POSTs.
* studio: trim verbose comments and docstrings in cancel path
* studio/llama_cpp: drop upstream PR hashes from benchmark comment
* Add review tests for Studio stop button
* Consolidate review tests for Studio stop button
* Align cancel-route test with exclusive cancel_id semantics
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: move cancel cleanup to generator finally; drop dead helper
- Move _tracker.__exit__ from Starlette BackgroundTask into each
streaming generator's finally block. Starlette skips the background
callback when stream_response raises (OSError / ClientDisconnect),
which leaked _CANCEL_REGISTRY entries on abrupt disconnect.
- Check cancel_event.is_set() at the top of each GGUF while loop so a
pending-replay cancel falls through to final_chunk + [DONE] instead
of propagating GeneratorExit out of _stream_with_retry.
- Remove unused _remember_pending_cancel; _cancel_by_cancel_id_or_stash
superseded it.
* Add review tests for Studio stop-button
* studio: wire audio-input stream into cancel registry
- Register cancel_event with _TrackedCancel on the audio-input streaming
path so POST /api/inference/cancel can stop whisper / audio-input GGUF
runs. Previously the registry stayed empty on this branch, so the stop
button returned {"cancelled":0} and the decode ran to completion.
- Apply the same finally-based cleanup and pre-iteration cancel-event
check used on the other three streaming paths.
- Update the _CANCEL_REGISTRY block comment to list cancel_id as the
primary key (was stale "session_id preferred").
* Consolidate review tests for Studio stop-button cancel flow
- Merge the 6 behavioral tests from test_stream_cleanup_on_disconnect.py
(finally cleanup on normal/exception/aclose, pre-set cancel_event
pattern, and its regressions) into test_stream_cancel_registration_timing.py,
which is the PR's existing file covering the same area.
- Extend structural invariants to include audio_input_stream alongside the
three GGUF / Unsloth streaming generators: no _tracker.__enter__ inside
the async gen body, cleanup via try/finally, no background= on
StreamingResponse.
- Delete test_stream_cleanup_on_disconnect.py (now empty).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: make cancel-via-POST interrupt Unsloth and audio-input streams
Close two remaining gaps in the stop-button cancellation wiring:
- stream_chunks (Unsloth path): add a top-of-loop cancel_event check and
call backend.reset_generation_state() so cancel POSTs flush GPU state
and close the SSE cleanly instead of relying on request.is_disconnected
(which does not fire through proxies like Colab's).
- audio_input_stream: run the synchronous audio_input_generate() via
asyncio.to_thread so blocking whisper chunks do not freeze the event
loop, matching the pattern already used by the GGUF streaming paths.
* Add review tests for Studio stop-button cancel flow
* Consolidate review tests for Studio stop-button cancel flow
- Delete standalone test_cancel_registry.py at repo root: tests duplicated
test_cancel_atomicity.py / test_cancel_id_wiring.py and re-implemented
registry primitives inline (scaffolding).
- Extend tests/studio/test_stream_cancel_registration_timing.py with
regression guards for the iter-1 cancel-loop fixes:
structural: each streaming generator checks cancel_event in its loop;
audio_input_stream offloads next() via asyncio.to_thread;
stream_chunks cancel branch calls reset_generation_state().
runtime: Unsloth loop breaks on external cancel and resets state;
audio loop stays responsive under blocking next();
both loops emit zero tokens on pre-set cancel (replay path).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: extend stop-path to passthrough streams; tighten wall-clock cap
- Lower _DEFAULT_T_MAX_PREDICT_MS from 1 hour to 10 minutes so the
wall-clock backstop actually bounds runaway decodes when cancel
signaling fails.
- Wire _TrackedCancel and cancel_event.is_set() into
_openai_passthrough_stream and _anthropic_passthrough_stream and
disable httpx keepalive so stop requests from /v1 and /v1/messages
tool-calling clients reach llama-server.
- Apply t_max_predict_ms to the tool-passthrough request body so the
backstop covers passthrough paths as well.
- Symmetric pre-registration stash for session_id/completion_id
cancels (_cancel_by_keys_or_stash) so early cancels by those keys
replay on later registration like cancel_id.
- Drop dead except BaseException guards around StreamingResponse()
at four streaming sites; cleanup lives in the generator's finally.
* studio: harden cancel registry against ghost-cancel and leak paths
- Revert the session_id/completion_id stash in the fallback cancel
helper. session_id is thread-scoped and reused across runs, so
stashing it on an unmatched POST would fire cancel_event for the
user's next unrelated request via _TrackedCancel.__enter__.
cancel_id remains the only per-run unique key that gets stashed.
- Default max_tokens to _DEFAULT_MAX_TOKENS in the tool-passthrough
body. Mirror the direct GGUF path so OpenAI/Anthropic passthrough
callers who omit max_tokens get the same zombie-decode cap instead
of relying on the wall-clock backstop alone.
- Wrap _openai_passthrough_stream setup with an outer try/except
BaseException. The inner except httpx.RequestError does not catch
asyncio.CancelledError at await client.send, which would otherwise
leave _tracker registered in _CANCEL_REGISTRY indefinitely.
- Frontend stop POST uses plain fetch + manual Authorization header
instead of authFetch. A 401 on the cancel POST no longer refreshes
tokens or redirects the user to the login page mid-stop.
* Add review tests for Studio stop-button cancel flow
* studio: trim comments on stop-button review changes
Collapse multi-paragraph rationale blocks on the cancel registry,
_openai_passthrough_stream, and the frontend onAbortCancel handler
into one-line explanations of why the non-obvious behaviour exists.
Drop authFetch import that became unused when the cancel POST
switched to plain fetch.
* Consolidate review tests for Studio stop-button cancel flow
Move review-added tests out of test_cancel_dispatch_edges.py into the
existing PR test files that already cover the same areas:
- backend registry fan-out / exclusivity / idempotency / falsy-keys
edge cases moved into tests/studio/test_cancel_atomicity.py
- frontend plain-fetch (not authFetch) + manual Authorization header
moved into tests/studio/test_cancel_id_wiring.py
Delete the now-empty test_cancel_dispatch_edges.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop default-capping responses at 4096 tokens (follow-up to #5069) (#5174)
* Studio: stop default-capping responses at 4096 tokens
Follow-up to #5069. The 4096 default introduced for runaway-decode
defense silently truncates any caller that omits max_tokens. The
Studio chat UI sets params.maxTokens = loadResp.context_length after
a GGUF load, so it's fine, but every other consumer is not:
- OpenAI-API direct callers (/v1/chat/completions, /v1/responses,
/v1/messages, /v1/completions) where the OpenAI default is
effectively unlimited per response. langchain, llama-index, raw
curl, and the openai SDK all rely on that.
- Reasoning models. Qwen3 / gpt-oss reasoning traces routinely exceed
4096 tokens before the model emits a single visible content token.
The user sees the trace cut off mid-thought.
- Long-form generation ("write a chapter", "produce a full SVG").
Reproduced on this branch: gemma-4-E2B-it-GGUF Q8_0, prompt asking
for a 10000-word story, no max_tokens in the request:
finish_reason: stop (misleading -- should be 'length')
content_chars: 19772
content_tail: ...'a comforting, yet immense, pressure.\n\n*"'
Body ended mid-sentence on a stray opening quote, right at the 4096
token mark.
After this patch the same request returns 38357 chars ending with
'...held in a perfect, dynamic equilibrium.' -- a natural stop, not
a truncation.
Implementation: rename the constant to _DEFAULT_MAX_TOKENS_FLOOR and
set it to 32768. Each call site now uses the model's effective
context length when known, falling back to the floor:
default_cap = self._effective_context_length or _DEFAULT_MAX_TOKENS_FLOOR
The 10-minute t_max_predict_ms wall-clock backstop from #5069 is
preserved as the second line of defense.
Plumbed _build_passthrough_payload + _build_openai_passthrough_body
through the routes layer so the Anthropic and OpenAI passthrough
paths also respect the model's context length.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: cancel passthrough streams during llama-server prefill + route through apiUrl for Tauri
Three reviewer-flagged correctness gaps in the stop-button mechanism.
1) `_openai_passthrough_stream` could not honor cancel during prefill.
The cancel check ran inside the `async for raw_line in lines_iter`
body, so a cancel POST that arrived before llama-server emitted the
first SSE line was unobservable until prefill completed. With a long
prompt under proxy/Colab conditions -- the exact target scenario for
this PR -- that left the model decoding for a long time after the
user clicked Stop. Add an asyncio watcher task that closes `resp` as
soon as `cancel_event` is set, raising in `aiter_lines` so the
generator can exit. The watcher polls a threading.Event because the
cancel registry is keyed by threading.Event for the synchronous
/cancel handler.
2) `_anthropic_passthrough_stream` had the same blocking-prefill pattern.
Same fix.
3) The frontend's stop-button cancel POST used a bare relative
`fetch("/api/inference/cancel", ...)`, which targets the webview
origin in Tauri production builds (where the backend is at
`http://127.0.0.1:8888`). Route through the existing `apiUrl()`
helper from `lib/api-base.ts` to match every other Studio call.
Browser/dev builds get the empty base, so behavior is unchanged
there.
Verified via temp/pr_simulation/sim_5069_prefill_cancel.py: cancel
during prefill terminates within ~250ms on both passthrough paths
(was 145s+ on the Anthropic path before this change), and the standard
non-passthrough chat path still cancels with no regression.
* Studio: log cancel-body parse errors instead of silently swallowing
Reviewer-flagged defensive logging gap. The bare `except Exception: pass`
in `cancel_inference` would mask malformed payloads that hint at a buggy
client or a transport issue. Log at debug so future investigation isn't
left guessing whether `body={}` came from a missing body or a parse
failure. Behavior is unchanged: an unparseable body still falls through
to the empty-dict path and the cancel call returns `{"cancelled": 0}`.
* Studio: Anthropic passthrough cancel parity with OpenAI passthrough
Two reviewer-flagged consistency gaps in the cancel surface for
/v1/messages.
1) Anthropic passthrough did not register cancel_id, so a per-run cancel
POST (the cleanest Studio-style cancel path) silently missed when
the route hit `_anthropic_passthrough_stream`. The OpenAI passthrough
has registered (cancel_id, session_id, completion_id) since this PR
was first opened; mirror that here. Also add `cancel_id` to
`AnthropicMessagesRequest` so the route handler can plumb it through.
2) The cancel handler's fallback key list checked only completion_id
and session_id, never message_id. Anthropic clients that send their
native `id` (returned in the SSE message_start event) for cancel had
no way to hit the registry. Add message_id to the fallback list.
Verified via temp/pr_simulation/sim_5069_prefill_cancel.py: P2 now
cancels by cancel_id in 137ms (was hanging pre-fix), and the new P2b
case cancels by message_id in 77ms. P1 (OpenAI) and P3 (standard chat)
still pass with no regression.
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: probe AMD GPUs in llama-server VRAM detection
_get_gpu_free_memory in studio/backend/core/inference/llama_cpp.py
only queried nvidia-smi. On AMD ROCm hosts that returns nothing, so
the GPU list is empty, the auto-fit logic falls into the no-gpus
branch, and llama-server gets --fit on with no -ngl to anchor it.
The model loads on CPU even though the GPU is detected elsewhere in
Studio. Addresses #5106.
Add a torch-based fallback that runs after nvidia-smi fails or returns
empty:
import torch
if torch.cuda.is_available() and hasattr(torch.cuda, "mem_get_info"):
for ordinal in range(torch.cuda.device_count()):
free, _total = torch.cuda.mem_get_info(ordinal)
gpus.append((ordinal, free // (1024 * 1024)))
Works on AMD because the ROCm torch wheels Studio installs reuse the
entire torch.cuda.* namespace via HIP. Also rescues NVIDIA hosts
where nvidia-smi is missing from PATH (a secondary cause of the bug
on Windows). Matches the convention
studio/backend/utils/hardware/hardware.py:412 already uses for the
same fallback purpose.
Verified locally: nvidia-smi path returns the expected GPU and free
MiB; torch fallback returns valid VRAM when nvidia-smi is forced to
fail. Note: PR #4874 is a draft taking a different approach
(parsing vulkaninfo); the two are complementary.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review feedback on PR #5172
torch.cuda.device_count() enumerates GPUs RELATIVE to the current
CUDA_VISIBLE_DEVICES (or HIP_VISIBLE_DEVICES on ROCm). Returning
those visible ordinals directly lets _select_gpus rewrite
CUDA_VISIBLE_DEVICES with the wrong physical IDs: a process started
with CUDA_VISIBLE_DEVICES=2,3 would get its child llama-server
relaunched with CUDA_VISIBLE_DEVICES=0,1, targeting the wrong GPUs
and violating any scheduler pinning.
Translate visible ordinals back through the active CVD/HIP/ROCR
mask before returning. Falls through to bare ordinal when no mask
is set. Also drop the redundant int() cast on // -- bytes // 2**20
already returns int.
Verified: with CUDA_VISIBLE_DEVICES=6 and nvidia-smi forced to fail,
the torch fallback now returns (6, free_mib) instead of (0, free_mib).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix ROCm visibility precedence + narrow ROCm child env
Two reviewer-flagged correctness bugs in the AMD GPU probe path.
1) ROCm visibility precedence was reversed. torch.cuda enumerates GPUs
relative to HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES on ROCm builds,
but the probe's env-var lookup checked CUDA_VISIBLE_DEVICES first. With
CUDA_VISIBLE_DEVICES=0,1 and HIP_VISIBLE_DEVICES=6,7 the probe returned
[(0, ...), (1, ...)] when torch's view was actually [(6, ...), (7, ...)].
The wrong physical IDs flowed downstream into CUDA_VISIBLE_DEVICES for
the llama-server subprocess, pinning it to GPUs 0,1 instead of 6,7.
Fix: branch on torch.version.hip. On ROCm, prefer HIP > ROCR > CUDA
(matches torch's own ordering). On NVIDIA, use CUDA only -- ignoring
any HIP/ROCR vars the parent happens to have set.
2) Child env narrowing only set CUDA_VISIBLE_DEVICES. On ROCm, llama-server
honors HIP/ROCR; if the parent shell exported HIP_VISIBLE_DEVICES=4,5
and the selector picked just GPU 4, the child still saw both because
we never narrowed HIP/ROCR. Now we set all three on ROCm so the AMD
subprocess actually sees the planned subset.
Both branches verified via temp/pr_simulation/sim_5172_rocm_precedence.py
(7/7 cases pass), including the reviewer's verbatim R5 case
(CVD=0,1 + HIP/ROCR=6,7).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: sort GPU probe result + honor explicitly empty ROCm masks
Two reviewer-flagged correctness nits on top of eff55fb8.
1) Gemini medium: the torch fallback returned an unsorted list when the
visibility mask was non-sequential (e.g. CUDA_VISIBLE_DEVICES=5,2,9),
diverging from the docstring guarantee and the nvidia-smi path. Now
sorted by physical id.
2) Codex P2: an explicitly empty HIP_VISIBLE_DEVICES="" should mean
"no GPUs" per the codebase convention in
utils/hardware/hardware.py::_get_parent_visible_gpu_spec. The previous
`or` chain treated empty string as falsy and silently fell through to
ROCR / CUDA, producing wrong physical IDs. Switch to `is not None`
checks to match.
Verified via sim_5172_rocm_precedence.py (9/9 cases pass) including the
two new R8 (sort) and R9 (empty-HIP honored) cases.
* Studio: align nvidia-smi probe with torch fallback (sort + robust CVD)
Two follow-up Gemini-medium nits on PR #5172.
1) Fragile CVD parsing on the nvidia-smi path: `cvd.split(",")` would
raise ValueError on a trailing comma like "0,1," because the empty
trailing token is not skipped. The torch fallback already filters
empty tokens via `if x.strip()`; mirror that here.
2) Missing sort guarantee on the nvidia-smi path: the docstring promises
sort-by-id, the torch fallback now sorts, but the nvidia-smi path
relied on driver enumeration order. Add an explicit sort.
Both changes match what shipped in 6b1cccd6 for the torch fallback, so
the two probe paths now have identical CVD parsing + ordering semantics.
* Studio: drop cvd.strip() truthiness so empty CVD filters all GPUs
Reviewer-flagged correctness bug. The previous `if cvd is not None and
cvd.strip():` guard treated `CUDA_VISIBLE_DEVICES=""` as if the variable
were unset, leaving `allowed=None` (and `physical_ids=None` on the torch
path). On the nvidia-smi path that mattered: nvidia-smi ignores CVD
entirely, so the probe's `allowed` filter is the only thing that
respects the parent's "no GPUs" intent. Pre-fix the probe returned every
physical GPU when the parent had explicitly hidden them.
Drop the `.strip()` truthiness check on both paths. The downstream
`if x.strip()` token filter still keeps trailing-comma masks like
"0,1," safe, and an empty mask now produces an empty allowed/physical
set as expected (matching utils/hardware/hardware.py convention).
Verified via sim_5172_rocm_precedence.py R10 + R11 (now 11/11 cases
pass): nvidia-smi path with `CUDA_VISIBLE_DEVICES=""` now returns []
instead of leaking the hidden GPUs.
* Studio: log ROCm env-var failures instead of silently swallowing
Reviewer-flagged defensive logging gap. The bare `except Exception: pass`
around the HIP/ROCR env-var assignment would mask anything from a
missing torch import to an unexpected version object shape. Log at
debug so a failed AMD child-env narrowing is at least traceable.
Behavior is unchanged: torch missing or version probe failing still
leaves the child with only CUDA_VISIBLE_DEVICES set.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: stop currency escape from breaking inline LaTeX
The currency-escape preprocessor in studio/frontend/src/lib/latex.ts
matched the opening dollar of any $<digits>...$ span and inserted a
backslash. The result was that text like "$30^\circ$" or
"**$90 - x$**" rendered as raw characters with stray dollar signs.
Fixes#5164.
Add two helpers in front of the escape:
- hasInlineMathCloser looks for an unescaped, non-doubled closing
dollar within the same line. Bold-wrapped spans (**$X$**) are always
treated as math since LLMs use that form for bold math.
- looksLikeMathBody filters multi-token bodies that look like prose
between two currency tokens ($5 to $10, $5, $10).
Verified against 111 inputs: the issue body, common LaTeX patterns
(Greek vars, fractions, integrals, vectors, exponents), prose currency
in lists and sentences, code blocks, and headings. All pass.
* Address review feedback on PR #5170
- Drop ^ and _ from MATH_OP_RE since LATEX_CHAR_RE already short-
circuits on those before MATH_OP_RE is consulted (Gemini comment).
- Treat compact currency ranges like $5-$10 and $5/$10 as currency
rather than math. The body between the first two dollars in those
forms is "5-" or "5/", a single non-whitespace token that previously
hit the math shortcut. Extend TRAIL_PUNCT_RE to strip - and / so the
trimmed body comes back as pure currency. (Codex comment.)
- Honour __underscore-bold__ around math the same way as **-bold**.
Markdown allows both delimiters and LLMs do reach for the underscore
form. (Gemini comment.)
Verified against the existing 18 cases plus 5 new ones for the range,
slash, and underscore-bold scenarios. All pass.
* Studio: fix numeric inline math + currency-as-closer in LaTeX preprocess
Two reviewer-flagged real-world misses in the inline-math heuristic.
1) Numeric-only operator forms like $2 + 2$, $100 < 200$, $1,000 - 500$
were getting their leading $ escaped, so the renderer never saw them
as math. The body has a math op but no lone-letter variable, so the
old looksLikeMathBody required the lone-letter clause and rejected
purely numeric expressions. Add SIMPLE_MATH_RE to recognise number-
or-letter operands joined by math operators.
2) Prose like "Starts at $5 + a $10 add-on" was being treated as one
math span "5 + a " with the second currency token mistaken for the
closer. The body satisfied the math-op + lone-letter check, so the
span got accepted and the renderer ate "10 add-on". In hasInlineMathCloser,
reject any candidate $ whose next character is a digit -- that's almost
always another currency token starting, not the closer of a real math
span (math doesn't follow $ with a bare digit).
Verified via temp/pr_simulation/sim_5170_latex.mjs: 25/25 cases pass,
including the 6 reviewer numeric-math cases, 3 currency-as-closer cases,
and 16 regression checks against the originally shipped behavior.
Daniel Han