* Studio: restore tensor parallelism for vision/mmproj GGUFs
#6416 disabled --split-mode tensor for any GGUF that ships an mmproj projector to
dodge a GGML_ASSERT crash (#6415) seen on an older llama.cpp build with consumer
Blackwell (sm_120). The blanket skip silently dropped tensor_parallel=true for
every multimodal/MTP GGUF (e.g. Qwen3.6-35B-A3B-MTP); on hardware where the model
fits on one GPU the load then collapsed to a single GPU. mmproj + --split-mode
tensor works on current builds (verified end to end on B200/sm_100), so the skip
was disabling a working configuration.
Make the vision skip self-healing per binary:
- attempt tensor for vision models by default
- skip upfront only on a binary already seen to abort on tensor + mmproj this
session (_vision_tensor_split_aborts), recorded when such a launch crashes at
startup (_record_vision_tensor_split_abort). Process scoped, so a studio update
re-probes the new build. The route-level layer-split fallback stays the net.
- add _select_gpus(min_gpus=...) so a downgraded tensor request can keep multiple
GPUs instead of collapsing to one (default 1, no behavior change).
Add tests/test_tp_vision_regression.py: an AST allowlist guard over the
tensor_parallel drop sites (which would have flagged #6416), plus cache and
_select_gpus coverage. No GPU required.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review on vision tensor-parallel self-healing
Three fixes from the PR review:
- Record a vision-tensor abort only after every startup retry fails. The first
version cached the binary on the first spawn crash, which on every build
(including capable ones) is the benign --fit step abort that the existing
--fit off retry resolves. That poisoned the cache so the next vision load in
the same process skipped tensor. Recording now happens at the post-retry
failure block (after fit-off, flash-attn-off and MTP-drop), so a binary that
actually works is never cached.
- Gate the record on the tensor/mmproj crash signature: a hard signal fault
(_is_signal_crash) with no non-tensor cause (_output_has_nonprojector_diagnostic
excludes OOM and unknown-arch), so an OOM, bad extra args, or MTP/flash-attn
crash no longer marks an otherwise capable binary incompatible.
- Preserve the multi-GPU request on the cached downgrade. The vision gate now
raises _layer_min_gpus to the visible GPU count and threads it through the
layer-split GPU selection (_select_gpus min_gpus and the subset loops), so a
downgraded tensor request still spreads across GPUs instead of collapsing to a
single card the model happens to fit.
Verified two vision+tensor loads in one backend process both tensor-split across
4 GPUs (the benign fit abort no longer poisons the cache). Tests updated.
* Studio: harden vision tensor-parallel self-healing (review round 2)
Address the second review round on the vision/mmproj tensor-parallel fix:
- Preserve vision on the first load: a --split-mode tensor + --mmproj
GGML_ASSERT now raises so the route-level tensor->layer fallback retries
layer split with the projector intact, instead of stripping --mmproj and
silently loading text-only (which returned success and skipped the fallback,
losing vision on the first load until the next cached load).
- Symmetric multi-GPU preservation: the pooled-VRAM tensor downgrade now raises
_layer_min_gpus from the usable tensor GPUs like the vision downgrade, so it
no longer collapses a multi-GPU request to a single card.
- Base the layer fallback minimum on usable GPUs: _select_gpus caps min_gpus to
the count of cards with usable VRAM, so a downgrade never forces a nearly-full
card in (or trips --fit) just to hit the count.
- Re-probe after in-app updates: key the per-binary abort cache on (path, mtime)
like _capability_cache, so POST /api/llama/update swapping the binary in place
(no backend restart) re-probes the new build instead of inheriting the old
build's abort.
- Bump _layer_min_gpus for a known-bad vision binary independent of the tensor
drop, so the route fallback's layer retry (tensor already off) still spreads
across GPUs.
Adds deterministic non-GPU regression tests for each.
* Studio: gate cached-vision layer minimum on the current tensor request
The cached-vision _layer_min_gpus bump fired for every later vision load on a
binary recorded as tensor+mmproj-incompatible, including loads that did not
request tensor parallelism. A plain non-tensor vision load that fits on one card
would then grab every GPU just because an earlier TP attempt aborted in the same
backend process.
Re-tie the bump to the current tensor request (back inside the tensor-drop
guard), so only a downgraded tensor request preserves the multi-GPU spread; a
non-tensor vision load minimizes device count as before.
* Studio: preserve GPU count + confirm assert on vision tensor fallback
Third review round on the vision/mmproj tensor-parallel fix:
- Preserve multi-GPU on the first tensor->layer fallback. The route-level retry
runs tensor-off, so the in-function downgrades can't see the original tensor
request and a fits-on-one-card model loaded the first successful fallback on a
single GPU. The GGUF load closure now passes preserve_multi_gpu_on_layer (the
toggle asked for tensor, this attempt is layer) and load_model raises
_layer_min_gpus for it, so the downgrade still spreads across GPUs.
- Cap the auto-context layer loops to usable GPUs. They bypass _select_gpus, so a
raised _layer_min_gpus could force a nearly-full card into the subset (or trip
--fit). They now start from _auto_min_gpus, capped to the GPUs with usable VRAM.
- Confirm the tensor/mmproj assert before caching. Recording (and the layer-retry
raise) now require the ggml assert marker via _is_tensor_split_assert, not the
bare-signal predicate shared with the projector-incompat branch, so a corrupt
or too-new projector that SIGSEGVs independent of split mode is no longer cached
as tensor/mmproj-incompatible.
Adds deterministic non-GPU regression tests for each.
* Studio: extend multi-GPU fallback to extra/env tensor + overhead-aware cap
Fourth review round on the vision/mmproj tensor-parallel fix:
- Preserve multi-GPU fallback for all tensor requests, not just the UI toggle.
Tensor can also be requested via --split-mode tensor in extra args or an
inherited LLAMA_ARG_SPLIT_MODE=tensor env; the fallback retries those too, so
the preserve_multi_gpu_on_layer hint now keys off _effective_tensor_parallel
(the same check the fallback uses), comparing the overall request against the
current attempt instead of only request.tensor_parallel.
- Cap the auto-context layer fallback to GPUs that can pay the per-device layer
overhead. The cap counted any card with positive usable VRAM, so a nearly-full
GPU with a few MiB free stayed eligible and could be exposed to llama.cpp and
OOM. It now mirrors _select_gpus: a card counts only if usable VRAM exceeds the
per-device pipeline overhead.
Adds deterministic non-GPU regression tests for both.
* Studio: match the #6415 split-axis assert + replay layer-preserve hint
Fifth review round on the vision/mmproj tensor-parallel fix:
- Narrow the tensor/mmproj crash signature. _is_tensor_split_assert matched any
GGML_ASSERT/GGML_ABORT, so an unrelated invariant a corrupt GGUF or projector
trips with --mmproj present could be cached as tensor/mmproj-incompatible. It
now matches the specific #6415 warmup assertion
(GGML_ASSERT(src_ss[0].axis != GGML_BACKEND_SPLIT_AXIS_0) in ggml-backend-meta),
whose split-axis signature is inherent to tensor splitting. A reworded future
assert just re-crashes-then-falls-back (vision preserved via layer split)
instead of poisoning the cache for other models.
- Persist the layer-preserve hint for respawns. A successful tensor->layer
fallback committed _last_load_kwargs without preserve_multi_gpu_on_layer, so
_respawn_if_dead replayed only --split-mode layer + tensor_parallel=False and a
mid-session respawn of a fits-on-one-card model came back single-GPU. The hint
is now in the replay snapshot, so recovery keeps the multi-GPU placement.
Adds deterministic non-GPU regression tests for both.
* Studio: tighten comments on the vision tensor-parallel fix
Make the comments and docstrings added by this PR succinct: collapse the
multi-line block comments in llama_cpp.py / inference.py to one or two lines,
trim the verbose test docstrings (the names and assert messages already carry the
intent), and shorten the module docstring. No code changes; verified comment-only
with scripts/comment_tools.py check --strip-docstrings.
* Studio: cache vision tensor abort only on the split-axis token
_is_tensor_split_assert also accepted any GGML_ASSERT/GGML_ABORT from
ggml-backend-meta, but that file holds many asserts, so an unrelated
scheduler/projector/model invariant on an --mmproj launch could cache the binary
as tensor/mmproj-incompatible and make later compatible vision models skip tensor
parallelism. Match the GGML_BACKEND_SPLIT_AXIS_* token itself (unique to the
#6415 warmup assert), not the source file name.
* Studio: don't leak the httpx test stub into later tests
The regression module stubbed httpx via sys.modules.setdefault, which installs
the lightweight stub even when real httpx is present but not yet imported. The
stub then persists for the whole pytest process, so provider/HF tests collected
later (importing httpx or huggingface_hub.errors) got a module missing
HTTPError/Response. Mirror the neighboring llama_cpp helper tests: import real
httpx first and only fall back to a stub on ImportError.
* Studio: latch the #6415 tensor-split abort on the first spawn, key it per model
The self-heal recorded the --split-mode tensor abort only in the post-retry
failure block, after the flash-attn-off retry. But SPLIT_MODE_TENSOR requires
flash_attn, so the flash-off retry can't run tensor and its output no longer
carries the warmup split-axis assert (ggml-backend-meta :541). The record
therefore never fired on the real reproducer and the crash loop repeated on
every load (reported by oobabooga on #6659).
Latch instead on the first spawn that shows the signal crash + split-axis
marker: record it, kill the process, and raise straight to the route's layer
fallback, skipping the futile flash-attn/MTP retry ladder for this crash.
The crash is a tensor-split geometry limit (e.g. MQA n_head_kv=1 splitting to
GGML_BACKEND_SPLIT_AXIS_0), not a vision/mmproj property: it reproduces without
--mmproj and even single-GPU tensor. So drop the vision/mmproj scoping, rename
_vision_tensor_* -> _tensor_split_*, and key the session cache on
(binary, mtime, model) rather than (binary, mtime) so one model's abort no
longer skips tensor for every other model on the same build.
Regression tests updated to pin the early-spawn record, the per-model cache,
and that an unrelated ggml-backend-meta assert is not treated as the marker.
* Studio: reload on explicit tensor-off after a multi-GPU layer fallback
When a tensor load is downgraded to layer but kept multi-GPU to honor the
tensor request (preserve_multi_gpu_on_layer, the geometry-cache gate, or the
budget downgrade), the server reports tensor_parallel=False with --split-mode
layer stored. A later Apply that explicitly turns the tensor toggle off then
matched the loaded state and deduped to already_loaded, so Studio kept the
fallback's all-GPU CUDA_VISIBLE_DEVICES placement instead of re-selecting
normal placement (a single GPU for a model that fits on one card).
Latch a _layer_preserves_tensor_intent flag in load_model whenever a tensor
request is downgraded to layer with the multi-GPU floor raised
(_layer_min_gpus > 1), clear it when tensor stays on or on unload, and force a
reload in _request_matches_loaded_settings when the user explicitly turns the
tensor toggle off while that flag is set. An Apply that does not touch the
toggle still dedupes, so a working multi-GPU layer server is not churned.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address reviewer.py findings on the tensor-split self-heal
P1 (dedup): tensor intent can be dropped via extras, not only the toggle. An
explicit llama_extra_args=["--split-mode", "layer"] matches the stored fallback
extras, so _request_matches_loaded_settings deduped to the preserved all-GPU
placement instead of reloading. Now reload when layer_preserves_tensor_intent
and the user explicitly drops tensor via the toggle OR via extras
(_effective_tensor_parallel of the explicit extras is false).
P1 (downgrade symmetry): the len(tp_gpus) < 2 compute-buffer downgrade cleared
tensor_parallel without raising _layer_min_gpus, unlike the budget and geometry
downgrades. GPUs below tensor's replicated compute-buffer reserve can still take
layer split's lower overhead, so keep the multi-GPU request (len(gpus) >= 2) and
let _select_gpus cap unusable cards.
P2 (cache key): key the tensor-split abort cache on st_mtime_ns, so a binary
replaced in place within the same second after an abort is re-probed instead of
inheriting the stale entry.
P2 (test hygiene): load routes/inference.py via importlib in the regression
tests instead of importing the routes package, which runs routes/__init__.py and
pulls in every router (e.g. python-multipart). Added regression coverage for the
extras-off reload, the compute-buffer multi-GPU preservation, and the same-second
nanosecond cache invalidation.
* Studio: record the tensor-split abort on the Windows CRT abort exit too
The first-spawn split-axis latch only recorded when _is_signal_crash matched
(POSIX signal or 0xC0000000+ NTSTATUS). On MSVC builds GGML_ASSERT terminates
through the CRT abort() path with exit code 3, which is neither, so the cache
never filled on Windows and every later load of the same bad binary/model
repeated the tensor crash before falling back to layer.
The split-axis marker is definitive, so accept either a signal crash or the
Windows abort() exit (3) when the marker is present. Add _is_abort_exit and a
unit test, and assert the early latch honors it.
* Studio: fix UnboundLocalError on --fit-on fallback, reload backend fast path
Two follow-ups from review on the tensor-split self-heal:
UnboundLocalError: _layer_min_gpus was initialized inside the GPU-selection try.
If NVML probing or GGUF/mmproj sizing raised, the except path logged "using
--fit on" and fell through to the command builder, where the new
self._layer_preserves_tensor_intent = _layer_min_gpus > 1 then raised, turning a
safe --fit-on layer fallback into a hard load failure. Bind _layer_min_gpus
before the try so the except path always has it.
Backend fast path: _request_matches_loaded_settings forces a reload when a
preserved tensor->layer fallback gets an explicit tensor-off request, but
load_model's own _already_in_target_state still matched the tensor-off/layer
settings and short-circuited, so the placement re-selection never ran. Mirror
the guard there: reload when layer_preserves_tensor_intent and the request drops
tensor intent. The flag clears on that reload, so there's no loop.
Added regression coverage for both.
* Studio: testable tensor-split record decision; skip futile fit-off retry
Follow-ups from a deeper review of the tensor-split self-heal:
Extract the record decision into _should_record_tensor_split_abort(rc, output)
(marker AND (signal crash OR Windows abort)) and call it from the early latch.
The combined boolean was only covered by source-inspection substring checks, so
an or->and typo would silently stop recording on Windows (CRT abort exit 3 is
not a signal) with every test still green. Add a behavioral test over the
POSIX / Windows / NTSTATUS / clean-exit / SIGKILL / no-marker matrix.
Skip the --fit off retry inside _spawn_and_wait when the crash already shows the
split-axis marker: that abort is fit-independent, so the retry just warms up and
crashes a second time before the latch records it. Skipping it lets the caller
latch immediately and corrects the latch comment.
Also clarify the dedup-guard comments (toggle read from model_fields_set vs
extras via _effective_tensor_parallel without env; the backend fast path is
intentionally broader and only ever forces a reload).
* Studio: don't reload-loop tensor-off requests under env tensor
The preserved-fallback reload guard fired on the raw tensor toggle, ignoring
LLAMA_ARG_SPLIT_MODE=tensor. For an env-driven tensor user, an explicit
tensor_parallel=false request then forced a reload that re-engaged tensor via
the env and re-created the same preserved layer fallback, so every /load
reloaded -- bypassing the env-downgrade matching that exists to avoid exactly
this loop.
Gate the guard on the env-aware effective tensor state: reload only when an
explicit toggle/extras change leaves _effective_tensor_parallel (which consults
the env) off. If the env still forces tensor, fall through to the existing
env-downgrade match, which dedupes instead of looping. Added a regression test
with LLAMA_ARG_SPLIT_MODE=tensor set.
* Studio: tighten comments and test docstrings on the TP self-heal
Condense the verbose comments and test docstrings added across the review rounds
into fewer, succinct lines without changing their intent: the early-latch and
downgrade-site rationale, the cache/key and helper docstrings, the dedup-guard
comments, and the per-test docstrings. No code changes (AST-verified comments
and docstrings only); tests and lint unchanged.
* Studio: clear preserved tensor flag on diffusion; carry it across non-drop reloads
Two follow-ups on the preserved-fallback machinery:
Diffusion: the DiffusionGemma path early-returns from load_model before the
command builder that sets/clears _layer_preserves_tensor_intent, so the flag
from a prior tensor->layer fallback leaked onto a later diffusion load and
forced needless reloads of the diffusion server on tensor-off/extra Applies.
Clear it when starting diffusion.
Settings reload: the preserve hint was recomputed only from the new request, so
a reload for an unrelated setting (e.g. max_seq_length) with the tensor toggle
omitted dropped a preserved multi-GPU layer placement back to one GPU. Carry
llama_backend.layer_preserves_tensor_intent into the hint when the request is
not an explicit tensor-off/extras-off drop, so a fitting model stays multi-GPU.
Added regression tests for the diffusion clear, the carry-forward, and the
updated tensor-intent computation.
* Studio: gate the preserve carry-forward on the same model being loaded
The tensor-intent carry-forward read llama_backend.layer_preserves_tensor_intent
without checking it belonged to the model being loaded. On a direct model switch
(load B without an explicit /unload of A), the flag is still set from A's
downgrade (it isn't reset until B's load_model reaches the command builder, after
the route reads it), so a plain load of B got preserve_multi_gpu_on_layer=True
and was spread across all GPUs even though it fits on one and the user never
requested tensor for it. The backend dedup doesn't have this leak (it checks
model_identifier first); the leak was only in the route hint.
Extract the decision into _carry_preserved_tensor_intent(preserved, same_model,
explicit_drop) and gate it on the backend still holding the same model. Add a
behavioral truth-table test (catches a `not` inversion and a missing same-model
guard) and tighten the compute-buffer downgrade test to bound its source window.
* Studio: match the HF quant too when carrying preserved tensor intent
The same-model guard on the preserve carry-forward compared only model_identifier,
which is variant-agnostic for HF repos. A later load of the same repo with a
different gguf_variant (which already bypassed dedupe on the variant mismatch)
was treated as the same model, so a request that omits tensor settings inherited
the prior variant's preserved intent and forced multi-GPU layer placement for a
quant that never requested tensor. Also require the loaded hf_variant to match for
HF repos (local direct-file loads already differ by model_identifier path). Added
a regression test for the variant guard.
* Studio: match the loaded GGUF by path too when carrying preserved tensor intent
A local directory holding multiple GGUF variants keeps one variant-agnostic
model_identifier (the directory) while config.gguf_file selects the file, so the
same-model guard let variant B inherit variant A's preserved tensor->layer
fallback and forced B onto multi-GPU. Mirror _already_in_target_state's identity
logic: match by resolved path when both sides have a local file, else by HF
variant. #6659
* Studio: let implicit same-settings reloads dedupe after a preserved fallback
The backend _already_in_target_state mirror forced a reload on ANY effective
tensor-off request once a tensor->layer fallback was preserved. In the HF
auto-pick / local-directory flows the route-level dedup is skipped, so an
identical /load with tensor omitted reached this guard and reloaded every time
even without an explicit drop. Thread the route's preserve_multi_gpu_on_layer
decision in so only an explicit drop reloads; implicit carry-forward dedupes. #6659
* Studio: only an explicit tensor/split-mode change drops preserved intent
The explicit-drop test treated request.llama_extra_args is not None as a drop,
so a same-model reload that merely added an unrelated pass-through arg (e.g.
--top-k 20) without touching the tensor field or --split-mode disabled the
carry-forward and collapsed a fitting model back to one GPU. A drop now requires
an explicit tensor_parallel field change or a non-tensor --split-mode override,
via a shared _is_explicit_tensor_drop helper used by both the already-loaded
dedup and the load carry-forward so the two readers agree. #6659
* Studio: treat an explicit clear of extras as a tensor drop
When tensor intent was extras-driven (--split-mode tensor) and fell back to a
preserved layer split, a later request that explicitly clears extras
(llama_extra_args=[]) but omits tensor_parallel left the empty list with no
split-mode override, so the carry-forward kept the model pinned multi-GPU instead
of returning to normal layer selection. _is_explicit_tensor_drop now also counts
an explicit empty-list clear as a drop, while an unrelated extra (--top-k) or
inherit (None) still carries the preserved intent. #6659
* Studio: don't treat the UI's tensor_parallel echo as a tensor drop
The Studio frontend always sends tensor_parallel and copies the /load response's
resolved value back into its state, so after a tensor->layer fallback every
ctx/settings reload carries tensor_parallel=false even though the user never
changed it. Keying the drop on the field (or on an empty extras clear) collapsed
the preserved multi-GPU placement on the next reload. A fallback also always
stores --split-mode layer, never a tensor split mode, so a clear never wipes
tensor intent. _is_explicit_tensor_drop now drops only on an explicit non-tensor
--split-mode override; the bare field echo, an empty clear, an unrelated extra,
and inherit all keep the preserved placement, and --split-mode tensor /
tensor_parallel=true re-engage tensor. #6659
* Studio: match the resolved config.identifier when carrying tensor intent
The same-model guard for the carry-forward compared the raw request id, but
ModelConfig.from_identifier normalizes it (adds the unsloth/ prefix for a
shorthand, fixes repo-id case) before load_model stores config.identifier. So a
ctx/settings reload using the shorthand id missed the match, dropped
_carry_preserved_tensor_intent, and could collapse a preserved multi-GPU layer
placement to one GPU. Compare against config.identifier (what the backend stores),
keeping it symmetric with _already_in_target_state. #6659
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Daniel Han