* report a complete load once llama-server is healthy
load_progress() derived its fraction purely from the llama-server's VmRSS over the GGUF shard total. With layers offloaded to VRAM (-ngl) the process releases the mmap'd weight pages after upload, so VmRSS sinks back well below the shard total: the fraction climbs toward ~1.0 during mmap, then collapses to a small value (~8%) once the weights are on the GPU. A fraction-driven progress bar therefore restarts and sticks there indefinitely even though the model is loaded and serving, which reads as a hang at "Starting model...".
Once the server is healthy the load is complete by definition, so report
fraction 1.0 (and bytes_loaded == bytes_total) in the ready phase regardless of resident set size. The VmRSS read is factored into _read_rss_bytes() with its original semantics preserved (0 on a missing VmRSS line, None when /proc is unavailable) so it can be unit-tested off Linux.
Fixes#5740
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* stub heavy deps in the load-progress test and guard a valueless VmRSS
Two review fixes:
1. The new test imported core.inference.llama_cpp at module top, which pulls in
loggers/structlog/httpx and fails collection with ModuleNotFoundError in the
lightweight backend test env when the file is run on its own. Stub loggers,
structlog and httpx via sys.modules.setdefault before the import, mirroring
test_llama_cpp_load_progress_matrix.py; setdefault keeps the real modules when installed. Verified the file now collects and passes with only pytest present.
2. Catch IndexError in _read_rss_bytes: a "VmRSS:" line with no value column
would make line.split()[1] raise and crash a load-progress poll. Return None
instead, with a test for the valueless line.
* Hold load-progress high-water mark and explain a never-healthy load (#5740)
load_progress() now holds a per-process VmRSS high-water mark, so the bar
no longer regresses to ~8% when -ngl offloads the weights and frees the
mmap pages mid-load.
A live server that never returns 200 on /health now gets a specific error
(context/VRAM too large, or a local proxy/VPN intercepting the loopback
probe) instead of the generic invalid-GGUF/out-of-memory message.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Hakan Baysal <hakan.baysal@trmix.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: fall back to text-only when a vision projector hard-crashes llama-server
The text-only mmproj fallback (#6075) only fired when llama-server printed a
recognizable projector-format error ("Unknown projector type", exit -6). An
installed llama.cpp that predates a model's projector can instead SIGSEGV
(exit -11) with no parseable output, e.g. unsloth/Qwen3.5-4B-MTP-GGUF +
mmproj-F16 on an older gfx1151 prebuilt: llama-server crashes on load, the
--fit off retry crashes the same way, and load_model gives up with a hard 500
instead of dropping vision.
Generalize the decision: a vision (--mmproj) launch killed by a signal (POSIX
returncode < 0, e.g. -11 SIGSEGV / -6 SIGABRT; Windows 0xC0000000+ access
violation) is treated like a projector incompatibility, so the load retries
once text-only. The retry is skipped if a cancel/unload is pending, mirroring
the MTP guard. Clean non-zero exits (bad GGUF, port bind) and hung processes
keep their own handling; non-vision launches are unaffected.
Reproduced and verified on gfx1151 (Radeon 8060S, ROCm 7.2.1): a current
prebuilt (llama.cpp b9596) loads the exact model + args fine, confirming the
crash is a stale prebuilt. With a wrapper that SIGSEGVs on --mmproj, Studio now
recovers: the load returns 200 (is_vision=false) and serves at ~31 tok/s
text-only instead of failing. New _is_signal_crash helper plus tests pin the
decision.
Also normalize a few em-dashes to ASCII punctuation in existing comments.
* Studio: refine mmproj hard-crash fallback (signal scope + last argv)
- Limit _is_signal_crash to genuine program faults (SIGSEGV, SIGABRT,
SIGILL, SIGFPE, SIGBUS) and Windows 0xC0000000+ statuses. SIGKILL,
SIGTERM and SIGINT no longer count, so an OOM-killer, unload or
supervisor kill is not masked as a projector incompatibility.
- Strip --mmproj from the last attempted argv so the text-only retry
keeps --fit off / --spec-default instead of resurrecting the original
spec flags (matters for MTP vision models on an older llama.cpp).
- Drop stray temp files committed by mistake and gitignore the "~" dir
so they cannot be re-added.
* Studio: tighten comments in mmproj hard-crash fallback
* Studio: retry --flash-attn off before dropping vision on a startup crash
When llama-server hard-crashes at startup, the recovery chain now tries the
least-destructive mitigation first. Flash-attention kernels SIGSEGV at load on
some ROCm/GPU builds (often inside the vision tower's attention); disabling
flash attention keeps BOTH vision and MTP, so a hard program fault with
--flash-attn on now retries once with --flash-attn off before the MTP-drop or
the text-only (mmproj-strip) fallbacks. _is_signal_crash already gates this to
genuine faults (SIGSEGV/SIGABRT/SIGILL/SIGFPE/SIGBUS), so an OOM-kill or unload
(SIGKILL/SIGTERM/SIGINT) does not trigger a retry.
Field context: a gfx1151 user crashes loading a vision GGUF even on the latest
prebuilt, so an update cannot help, and the same model and args load fine on
another gfx1151 box, pointing at a runtime/flash-attn fault. New
_with_flash_attn_off helper plus tests. Verified on hardware with a wrapper
that SIGSEGVs on --flash-attn on: Studio recovers with is_vision=true (vision
and MTP intact) instead of failing or losing vision.
* Studio: name the OOM kill on a too-large model load
When the OS kills llama-server with no diagnostic output (SIGKILL/SIGTERM,
almost always the OOM killer, e.g. a BF16 model too large for the WSL VM's
RAM cap), the recovery ladder correctly does not retry an external kill, so
this is the message the user sees. It fell through to the generic "is the
GGUF valid / out of memory" text. Make it actionable: name the signal and
point at a smaller or more quantized GGUF, a lower context length, or raising
the WSL memory limit. Output-based diagnoses still win and a hard fault keeps
the generic fallback.
* Studio: refuse a model too large for system RAM on a unified-memory APU
On gfx1150/gfx1151 APUs the weights load into shared system RAM (GGML
unified memory). _get_gpu_free_memory reports the full ROCm/APU budget as
free (often ~100 GB), but under WSL the VM's RAM cap is the real ceiling.
Studio trusted the budget, spawned a load larger than RAM, and the OS killed
it mid-flight, taking the Studio process with it (a silent "Terminated" with
no error, the model resident in RAM not VRAM).
Add a pre-flight guard on the APU path: if the weights exceed available
system RAM (psutil, then /proc/meminfo), refuse before spawning with a clear
message (smaller/more-quantized GGUF, lower context, or raise the WSL memory
limit). Weights only so KV/context auto-reduction is not double-counted;
unknown RAM never refuses; non-APU and discrete-GPU paths are untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: refine recovery ladder (keep diagnosed errors, flip all flash-attn)
Two review points on the hard-crash recovery ladder:
1. The signal-only text-only fallback stripped --mmproj on any hard fault,
even when llama-server had already printed a non-projector cause (an OOM
such as "cudaMalloc failed: out of memory", an unsupported architecture, or
a tensor-parallel limit). That masked the real error and told the user to
update llama.cpp for vision. New _output_has_nonprojector_diagnostic gates
the signal path: it fires only when no such marker is present, so a bare
SIGSEGV with no output still retries text-only, but a diagnosed OOM surfaces
the real error instead of silently dropping vision.
2. _with_flash_attn_off only flipped the first --flash-attn. llama.cpp is
last-wins, so a leftover enable from extra_args (--flash-attn on, -fa on, or
the = form) could keep flash attention on and re-crash the retry. It now
flips every occurrence and returns None only when nothing is flippable.
test_llama_cpp_mmproj_fallback.py and the classification/APU suites: 103 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: pass the text-only retry's exit code to the failure classifier
When the text-only fallback retry itself fails, read its exit code before
_kill_process() clears it and forward it to _classify_llama_start_failure, so an
OS-killed retry surfaces the actionable out-of-memory message instead of the
generic one (matching the primary failure path).
* Studio: scope APU RAM guard to selected GPUs, count MTP drafter, neutral SIGTERM
Three refinements to the startup recovery work in this PR:
- The unified-memory APU RAM guard fired whenever any visible GPU was a
gfx1150/gfx1151 APU, so on a mixed APU+dGPU host it could refuse a valid
load placed on the discrete GPU. Scope _amd_apu_wants_unified_memory to the
selected gpu_indices (physical ids, mapped via CUDA_VISIBLE_DEVICES like
_is_datacenter_gpu); None still means every visible GPU. Applied to both the
RAM guard and the GGML_CUDA_ENABLE_UNIFIED_MEMORY env set.
- The RAM guard counted only the main GGUF plus mmproj, so a separate MTP
drafter (also resident in unified system RAM, even when offloaded to CPU)
could push the load past the RAM cap and still get OS-killed mid-load. Add
the drafter weights to the APU RAM total.
- The startup classifier reported SIGTERM (-15) as 'most likely out of memory',
but SIGTERM is also how an unload/cancel or a supervisor stops the server.
Keep the OOM wording for SIGKILL (-9, the OOM killer) and report -15
neutrally.
Tests updated/added accordingly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review on the APU guard and decode-probe ladder
- Map APU physical ids via the active ROCm mask (HIP, then ROCR, then CUDA),
mirroring _get_gpu_memory, so a HIP_VISIBLE_DEVICES-selected APU is matched.
- Only add the MTP drafter to the APU RAM total when MTP will actually engage,
so a stale LLAMA_ARG_SPEC_DRAFT_MODEL cannot refuse a non-MTP load.
- After an MTP first-decode hard fault, retry --flash-attn off (keeps MTP)
before dropping speculative decoding, matching the startup rung.
- Fold the --flash-attn= / -fa= rewrite into one branch.
Tests: tensor-parallel decode-probe assertion updated for the FA-off rung.
* Studio: tighten two comments in the APU guard and RAM preflight
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: refine flash-attn retry and APU RAM guard per review
- _with_flash_attn_off now decides on the effective last-wins value: it returns
None when FA is already off (no wasted retry), and neutralizes a bare
--flash-attn / -fa (which llama.cpp reads as on) so the retry cannot re-enable
it. Length is preserved so downstream index slices stay valid.
- _amd_apu_wants_unified_memory uses 'gpu_indices is not None' so an empty
selection is respected (not treated as all-visible).
- The APU RAM refusal now checks the base model only (main + mmproj); an
optional MTP drafter is dropped by the existing MTP-drop fallback rather than
causing a hard pre-spawn refusal of an otherwise loadable model.
Tests: bare-flag / effective-off / empty-selection / HIP-mask cases added.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Trim and tighten code comments and docstrings across studio/ Python. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.
Classify llama-server startup failures so diffusion/image GGUFs (FLUX, Qwen-Image, LTX, ERNIE-Image, Z-Image, ...) point users to the Images page instead of a misleading out-of-memory error. Other unknown architectures get a precise unsupported message; Ollama and OOM fallbacks are preserved.
Architecture is matched exactly against general.architecture, covering the arches Unsloth ships as GGUF: flux, qwen_image, ltxv, wan, lumina2.
Fixes#5842.
Hakan Baysal