is_cuda_server() treats a co-located libggml-cuda.so* as proof the server is
CUDA-ready. That's normally true (llama.cpp dlopens the backend from beside the
binary), but an *interrupted* build (thermal/power shutdown -- common on the
NVIDIA-ARM laptops this path targets) can leave a half-linked libggml-cuda.so
next to the server: present, so is_cuda_server() matches, yet the backend fails
to load at runtime. The post-build path already wipes+rebuilds such a partial
.so, but the step-0 early-skip trusted it and never rebuilt -- so Studio could
report GGUF CUDA inference ready while running a broken/non-CUDA backend.
Gate the early-skip with cuda_server_probe(): 'llama-server --list-devices'
enumerates backends and exits (cheap, no server spin-up). Only a definitive
'flag supported, ran, but no CUDA device' triggers a clean rebuild; a timeout or
an old pin without --list-devices stays inconclusive and keeps trusting the .so,
so we never force a needless, thermally-expensive rebuild. Probe logic verified
against healthy/broken/unsupported/timeout stubs (0/1/2/2).
Addresses Codex review P2 (provision_llama_cuda.sh).
The WoA+NVIDIA WSL2 fallback bridges UNSLOTH_NO_LLAMA_CUDA / UNSLOTH_PYTHON /
UNSLOTH_SKIP_WSL_WINDOWS_SHORTCUT into the distro, but not UNSLOTH_PYTORCH_MIRROR.
install.sh's get_torch_index_url() reads it (as does install.ps1's own native
Get-TorchIndexUrl), yet Windows env vars don't cross into WSL -- so a
mirror-required / restricted-network install silently fell back to
download.pytorch.org inside the distro even though the outer installer honored
the mirror.
Forward it alongside the other vars, guarded by a strict http(s)-URL allow-list
(no shell metacharacters) and single-quoted so the value can't break out of the
bash -lc string. Verified: legit mirror URLs (incl. host:port and query strings)
forward; space/';'/$()/quote-injection and non-http schemes are rejected.
Addresses Codex review P2 (install.ps1).
The foreground source build in setup.sh used -j(nproc), which on the
lightly-cooled NVIDIA-ARM boxes this WoA/WSL path targets (DGX Spark /
GB10, N1X RTX Spark laptops) draws enough sustained power during the
nvcc compile to trip a thermal shutdown -- the exact reason
provision_llama_cuda.sh already caps its background build.
Mirror that cap for the foreground build (only reached when no prebuilt
llama.cpp was available and a CUDA toolkit is present): gate on
aarch64/arm64 + GPU_BACKEND=cuda, then use ~half the cores, also bounded
by ~1.5 GB/nvcc job. Other platforms and CPU builds keep full -j(nproc).
Override anywhere with UNSLOTH_LLAMA_BUILD_JOBS=N.
Verified: nproc=20/29GB box -> -j10; override=6 -> -j6; CPU build and
x86_64 stay uncapped.
Recognise the Gemma 4 separate-drafter MTP family, auto-download the drafter with retry, fall back to n-gram with a clear reason when it cannot be resolved, and retry the download on reload. Gemma 3n (ships no drafter) and embedded-MTP models (Qwen) are unaffected.
Fixes#6406
* Added Windows RTX 50-series troubleshooting guide
* Update README.md to link Blackwell Windows Troubleshooting Guide
Added a troubleshooting guide for Windows installation issues with RTX 50-Series.
* Fix install path, troubleshooting accuracy and markdown for PR #6286
* Condense Windows Blackwell tips into README and drop standalone guide for PR #6286
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: label Apple Silicon as Metal/unified memory instead of CPU-only in installers
* Studio: drop redundant aarch64 check from the macOS GPU label detection
* studio: run /generate/stream's sync generator off the event loop to avoid blocking it
* fix: close generator in finally on client disconnect in generate_stream
* Fix/adjust generate stream test for PR #6466
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust generate stream cancellation for PR #6466
* Fix/adjust generate stream cleanup for PR #6466
* fix: cancel incomplete generate stream cleanup
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
_estimate_mtp_overhead_bytes is also reached for the separate-drafter spec modes
(draft-simple / draft-eagle3) through _user_draft_via_extras. Those modes load a
small distinct drafter with its own KV -- already counted in the draft KV +
weights -- and keep no duplicated full target context; only MTP runs a second
context over the target model's own KV geometry (llama.cpp ctx_tgt). Charging the
~main-KV-sized f16 copy there over-reserved by tens of GiB on an MLA model and
needlessly shrank the advertised context, the same under-advertising #6312 set
out to fix.
Thread mtp_keeps_target_ctx through _estimate_mtp_overhead_bytes (True for MTP,
False for separate-drafter modes) and derive _engaged_is_mtp at the fit call site
so the target copy is added only when the engaged mode is actually MTP. MLA + MTP
(GLM-5.2 / DeepSeek / Kimi) is unchanged, so the GLM-5.2 OOM fix is preserved;
non-MLA and the draft-simple / draft-eagle3 paths no longer pay the copy.
test_mtp_mla_target_ctx.py adds a case asserting the separate-drafter reserve
collapses to the draft KV (no target copy) while the default MTP path keeps it.
* Studio: Auto disables MTP for MLA models (GLM-5.2 et al.); UNSLOTH_MLA_MTP_ENABLED to re-enable
Studio's Auto speculative mode promotes any embedded-MTP model >=3B to
--spec-type draft-mtp. For MLA models (GLM-5.2/DeepSeek/Kimi) that is a
regression: llama.cpp's MLA/DSA MTP path keeps a duplicated full target-KV
context and recomputes the sparse-attention indexer every draft step, so it
runs ~2x slower than no speculation (GLM-5.2 UD-IQ1_S bench: 27 vs 45 tok/s,
flat across draft depth 1..6 and 96-100% acceptance, on both prose and code).
vLLM/SGLang get a speedup from the same model, so this is a llama.cpp
implementation gap, not a model property.
Auto now drops embedded MTP for MLA models and falls back to ngram-mod (or
spec-off when the binary lacks ngram-mod), mirroring the existing sub-3B
fallback. The metadata separator is kv_lora_rank: it is present on MLA models
and absent on non-MLA embedded-MTP models (Qwen3.x-MTP), whose MTP module is
structurally identical but fast, so a "full layer" heuristic cannot tell them
apart. Qwen MTP, separate drafters (Gemma, --model-draft), and non-MTP models
are unchanged.
Explicit overrides still engage the slower MTP route: choosing MTP / MTP+Ngram
in Settings, or passing --spec-type in extra args. UNSLOTH_MLA_MTP_ENABLED=1
re-enables Auto promotion for MLA once the upstream path is optimized.
A new spec_fallback_reason value "mla_mtp_disabled" surfaces this as an
Auto-mode policy downgrade (not a binary/update problem), with a settings
banner that points users at the MTP override. It is deliberately kept out of
the "Update llama.cpp" affordance since updating does not help.
Tests: resolver-matrix rows for MLA->ngram-mod / MLA-no-ngram->off /
non-MLA-Qwen->draft-mtp / MLA-separate-drafter->draft-mtp /
non-MTP-MLA->default / forced mtp|mtp+ngram on MLA->draft-mtp / env flag;
kv_lora_rank metadata fixtures; and reload-skip coverage (Auto ngram-mod is
idempotent, forced mtp bounces a reload).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add an Open button to reveal the models folder in the file manager
* Studio: report models folder creation failures
---------
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
test_safetensors_capability_advertise: detect_reasoning_flags now returns a
reasoning_effort_levels key, so the none-template expectation must include it.
test_tensor_parallel::test_runtime_recovery_reloads_without_mtp: the assertion
raced the recovery thread, which sets _spec_fallback_reason just before its
finally clears _mtp_runtime_fallback_in_progress. Wait for the flag to clear
before asserting.
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Cosmetic follow-up to a4ad50e (review nit): the retry loop slept 2s even
after the last attempt, adding ~2s only when nvidia-smi is permanently hung.
Sleep only between attempts.
On Windows-on-ARM + NVIDIA the Studio install runs inside WSL2, where
nvidia-smi is served over GPU-PV and can take far longer than its usual
sub-second response when the host is under heavy CPU load (the concurrent
pip / frontend / cmake work during install). detect_host probed nvidia-smi
with a single 20s timeout; under that load it raised TimeoutExpired, the GPU
was treated as ABSENT, and the host was misrouted to the ggml-org CPU prebuilt
-> rejected on an NVIDIA host -> slow (and on thermal-limited laptops, risky)
CUDA source build, even though a usable arm64 CUDA prebuilt was published.
Add _nvidia_smi_capture(): retry the three detect_host nvidia-smi probes with
a generous 60s per-attempt timeout. It is only reachable when nvidia-smi exists
on PATH, so CPU-only hosts incur no extra wait. Measured: nvidia-smi took
42-59s under a -j20 build on an N1X; with the fix the probe rides it out and
detect_host correctly reports has_usable_nvidia + compute_cap, so the CUDA
prebuilt is selected (no source build).
* Fix clipped card shadows in Hub trending carousel
The carousel scroller only had vertical padding, so with overflow-x set
the first and last cards had their drop shadow clipped on the horizontal
edges. Add px-2 with a matching -mx-2 so the shadow has room while the
cards stay aligned with the section heading, and scroll-px-2 so snap-start
does not scroll the padding away on load.
* Align carousel edge fades with the scroll clip edge
The shadow fix gave the scroller an -mx-2 bleed, but the left/right fade
overlays stayed pinned to the wrapper edges, 8px inside the clip edge. That
left a thin strip where a card showed beside the fade, so the fade read as a
separate block instead of blending into the background. Offset both fades by
the same 8px so their opaque edge sits on the clip edge.
* Add click-and-drag panning to the Hub card carousel
The rows only scrolled by wheel or trackpad, and grabbing a card started
a native drag of its avatar image, so the cards could not be dragged to
move the row. Add mouse drag-to-scroll (touch and pen keep native
scrolling), swallow the click a drag would otherwise fire on a card, keep
plain clicks working, and block the avatar's native drag.
* Trim carousel edge fade width from 56px to 44px
* Smooth out carousel drag panning
Scroll snap was correcting the position on every drag frame, which made
the pan feel sticky. Disable snap while a drag is active and restore it on
release so the row follows the pointer and then settles on a card.
* Drop stale carousel drag when the button is released off-element
If a press ended outside the scroller before the drag threshold was
crossed, no pointerup reached us and the drag stayed armed, so a later
buttonless mousemove would scroll the row. Bail out and clear the drag
whenever the primary button is no longer held.
- find_nvcc now prefers the highest /usr/local/cuda-<ver> toolkit so a stale
unversioned `cuda` symlink or an older nvcc earlier on PATH can't win and
rebuild with CUDA 12.x (re-hitting the glibc>=2.41 / Blackwell clash this
script avoids); falls back to a PATH nvcc only when no versioned toolkit.
- Validate the GPU compute_cap is purely numeric before using it as
CMAKE_CUDA_ARCHITECTURES: some WSL GPU-PV / driver combos report "N/A",
which CMake rejects (aborting an otherwise-usable build) instead of letting
"native" autodetect.
- Gate the native-Linux aarch64 provisioner on _SKIP_GGUF_BUILD: when a non-root
user declines the sudo prompt (or lacks sudo) for GGUF deps, don't then run a
provisioner that does its own sudo apt-get installs.
_run_update imports routes.inference.get_llama_cpp_backend, which on a
fully-installed host pulls a real Studio singleton and blocks on its load
lock. Default the autouse fixture to a no-backend stub (the fail-open
path); the load-coordination tests still inject their own backend over it.
* Studio: reserve the duplicated MTP target KV context for MLA models
GLM-5.2 UD-IQ1_S advertised its native 1,048,576-token context, loaded, then
crashed cublasCreate on the first generation with "CUDA error: the resource
allocation failed" on a 2x B200 box. The model loaded fine; the decode OOMed.
Cause: when MTP speculative decoding is engaged, llama.cpp keeps a second full
copy of the target model's KV context for draft verification (ctx_tgt=yes in the
spec log), at f16. On an MLA model that copy is ~the main KV again -- for GLM-5.2
at 1M ctx llama.cpp sized it at ~97.5 GiB -- but the auto-fit reserve only
counted the tiny embedded draft head (~2 GiB), 46x too low. So weights (~202 GiB)
+ main KV (~83 GiB) + a 2 GiB reserve looked like it fit in 2x182 GiB, when the
real footprint with the ~97 GiB MTP copy is ~382 GiB and overruns the cards.
Disabling speculative decoding removed the copy and the same context ran fine.
_estimate_mtp_overhead_bytes now adds the duplicated target context (the main KV
re-estimated at f16) for MLA models, so auto-fit backs the context off (or selects
more GPUs) instead of advertising one that OOMs. It is gated strictly on MLA
(kv_lora_rank present), which is exactly the family that keeps the extra copy
(GLM-5.x, DeepSeek, Kimi-K2); non-MLA MTP (Qwen, Gemma) is byte-for-byte
unchanged. The reserve stays deterministic from GGUF dims, matching #6312.
test_mtp_mla_target_ctx.py covers it: the MLA reserve includes the f16 target
copy and dominates the draft head, the copy is f16 regardless of the main cache
type and scales with context, non-MLA embedded heads keep overhead == draft KV,
and _fit_context_to_vram on the GLM-5.2 / 2x B200 budget now returns a context
below the requested 1M where the old draft-only reserve kept the full 1M.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stub structlog/loggers in MTP-MLA test so it is import-order-independent
The new test imports core.inference.llama_cpp, which pulls in orchestrator ->
structlog. In the lightweight test env structlog is absent, so when this file is
collected before test_mtp_vram_budget.py (it sorts first) or run directly,
collection aborted with ModuleNotFoundError. Install the same loggers/structlog
(+ conditional httpx) stubs the sibling MTP tests use before the import, matching
the established per-file convention.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: fix Bypass Permissions menu freeze and show decimal GB for model sizes
Bypass Permissions freeze: the warning dialog lived inside the composer
"+"/More dropdown and kept the menu mounted via onSelect preventDefault,
so confirming or cancelling the dialog left both popovers frozen open.
Lift the dialog out of the menu into a store-driven
BypassPermissionsConfirmDialog mounted at a stable spot in the composer.
The menu item now closes normally on select and just toggles a new
bypassConfirmOpen store flag, so the popovers dismiss as expected.
Model search sizes: formatBytes divided bytes by 1024 but labelled the
result "GB", so unsloth/GLM-5.2-GGUF:UD-IQ1_S showed 201.8 GB where
Hugging Face reports 217 GB. Switch the search display to decimal
(base-1000) units to match what Hugging Face reports. The GPU-fit math
stays base-1024 since VRAM capacity is binary.
* Studio: address review feedback and add GLM-5.2 high/max/disabled thinking
Review feedback on the Bypass Permissions and size-format changes:
- Mount the Bypass Permissions warning dialog once at the chat-page root
instead of inside each Composer. It is driven by global store state, so
the per-composer mount meant Compare mode (multiple composers) rendered
duplicate dialogs and the shared-composer menu had none. A single root
mount fixes both.
- Defer opening the dialog past Radix's menu-close focus restoration with
setTimeout(0), so the dropdown does not steal focus back and break the
dialog's focus trap.
- Clamp the unit index in formatBytes so units[i] cannot go out of bounds
past TB (and to absorb log() float error at exact powers of 1000).
GLM-5.2 reasoning levels:
GLM-5.2's template gates thinking with enable_thinking and also reads a
reasoning_effort level ('high' or 'max'), so it needs high / max /
disabled rather than the binary toggle it got before (its style was
detected as enable_thinking, which made 'high' unreachable). Add a new
reasoning style 'enable_thinking_effort' that reuses the effort dropdown
but, unlike gpt-oss, can be fully disabled:
- detect_reasoning_flags classifies a template that has both
enable_thinking and reasoning_effort, extracting the discrete levels
from the quoted effort literals it branches on. Templates with only one
of the two (gpt-oss, Qwen3, DeepSeek, GLM-4.6) are unchanged.
- _request_reasoning_kwargs maps the new style to enable_thinking plus an
in-range reasoning_effort; disabling sends enable_thinking=false. The
gpt-oss reasoning_effort path is left untouched.
- The backend reports reasoning_effort_levels on the load/status response;
the frontend carries them through to the effort dropdown and sends
enable_thinking + reasoning_effort for this style.
Verified: backend reasoning kwargs render the real GLM-5.2 template to
"Reasoning Effort: High/Max" (thinking) and an empty <think></think>
(disabled); tsc, eslint, i18n parity and the production build all pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review feedback on reasoning effort and formatBytes
- chat-adapter localReasoningEffort: accept 'minimal' so a template that
branches on it (extracted into reasoning_effort_levels) is sent through
instead of being coerced to 'low' and then dropped by the backend.
- formatBytes: return '0 B' for non-finite / non-positive sizes (missing
metadata -> NaN, Infinity, negatives) and clamp the unit index lower
bound to 0, so sub-1-byte values can't produce a negative index.
* Studio: hybrid reasoning none gate and decimal GB in load progress
- _request_reasoning_kwargs: for enable_thinking_effort models, treat a
raw reasoning_effort='none' (OpenAI 'no reasoning' sentinel) as the
enable_thinking=false off gate, so a direct API caller can disable
thinking even without passing enable_thinking. The frontend already
sends enable_thinking=false; this only affects raw API callers.
- use-chat-model-runtime: the download / 'X of Y GB in memory' load
progress divided bytes by 1024**3 but labelled GB, so it disagreed with
the model picker and Hugging Face. Use decimal GB (1e9) to match.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry hybrid reasoning levels on all load paths and harden formatBytes
Review follow-ups on the enable_thinking_effort work:
- Every model-load path now copies reasoning_effort_levels and derives
supportsReasoningOff, via a shared reasoningCapsFromLoad() helper. The
shared/Compare composer load and the three chat-adapter auto-load paths
previously set only reasoningStyle, so a GLM-style hybrid model loaded
through Compare or first-chat auto-load fell back to the default
low|medium|high and lost its Max / Off controls.
- The local send path clamps the effort to the loaded model's advertised
levels (clampReasoningEffortToLevels) instead of a hard-coded list. A
stale "max" carried over from an external provider no longer reaches a
pure reasoning_effort (gpt-oss) model that only accepts none|low|medium|
high, where the backend would have dropped it.
- formatBytes divides iteratively instead of via Math.log, which has float
error at exact powers of 1000 (log(1e12)/log(1000) = 3.9999... would
label 1 TB as "1000 GB"). Keeps the non-finite/non-positive guard.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: restore sidebar roundness to its pre-#6349 state
* Studio: keep train recents as a rounded rectangle, not a pill
---------
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
* Studio: fall back to text-only when a vision projector hard-crashes llama-server
The text-only mmproj fallback (#6075) only fired when llama-server printed a
recognizable projector-format error ("Unknown projector type", exit -6). An
installed llama.cpp that predates a model's projector can instead SIGSEGV
(exit -11) with no parseable output, e.g. unsloth/Qwen3.5-4B-MTP-GGUF +
mmproj-F16 on an older gfx1151 prebuilt: llama-server crashes on load, the
--fit off retry crashes the same way, and load_model gives up with a hard 500
instead of dropping vision.
Generalize the decision: a vision (--mmproj) launch killed by a signal (POSIX
returncode < 0, e.g. -11 SIGSEGV / -6 SIGABRT; Windows 0xC0000000+ access
violation) is treated like a projector incompatibility, so the load retries
once text-only. The retry is skipped if a cancel/unload is pending, mirroring
the MTP guard. Clean non-zero exits (bad GGUF, port bind) and hung processes
keep their own handling; non-vision launches are unaffected.
Reproduced and verified on gfx1151 (Radeon 8060S, ROCm 7.2.1): a current
prebuilt (llama.cpp b9596) loads the exact model + args fine, confirming the
crash is a stale prebuilt. With a wrapper that SIGSEGVs on --mmproj, Studio now
recovers: the load returns 200 (is_vision=false) and serves at ~31 tok/s
text-only instead of failing. New _is_signal_crash helper plus tests pin the
decision.
Also normalize a few em-dashes to ASCII punctuation in existing comments.
* Studio: refine mmproj hard-crash fallback (signal scope + last argv)
- Limit _is_signal_crash to genuine program faults (SIGSEGV, SIGABRT,
SIGILL, SIGFPE, SIGBUS) and Windows 0xC0000000+ statuses. SIGKILL,
SIGTERM and SIGINT no longer count, so an OOM-killer, unload or
supervisor kill is not masked as a projector incompatibility.
- Strip --mmproj from the last attempted argv so the text-only retry
keeps --fit off / --spec-default instead of resurrecting the original
spec flags (matters for MTP vision models on an older llama.cpp).
- Drop stray temp files committed by mistake and gitignore the "~" dir
so they cannot be re-added.
* Studio: tighten comments in mmproj hard-crash fallback
* Studio: retry --flash-attn off before dropping vision on a startup crash
When llama-server hard-crashes at startup, the recovery chain now tries the
least-destructive mitigation first. Flash-attention kernels SIGSEGV at load on
some ROCm/GPU builds (often inside the vision tower's attention); disabling
flash attention keeps BOTH vision and MTP, so a hard program fault with
--flash-attn on now retries once with --flash-attn off before the MTP-drop or
the text-only (mmproj-strip) fallbacks. _is_signal_crash already gates this to
genuine faults (SIGSEGV/SIGABRT/SIGILL/SIGFPE/SIGBUS), so an OOM-kill or unload
(SIGKILL/SIGTERM/SIGINT) does not trigger a retry.
Field context: a gfx1151 user crashes loading a vision GGUF even on the latest
prebuilt, so an update cannot help, and the same model and args load fine on
another gfx1151 box, pointing at a runtime/flash-attn fault. New
_with_flash_attn_off helper plus tests. Verified on hardware with a wrapper
that SIGSEGVs on --flash-attn on: Studio recovers with is_vision=true (vision
and MTP intact) instead of failing or losing vision.
* Studio: name the OOM kill on a too-large model load
When the OS kills llama-server with no diagnostic output (SIGKILL/SIGTERM,
almost always the OOM killer, e.g. a BF16 model too large for the WSL VM's
RAM cap), the recovery ladder correctly does not retry an external kill, so
this is the message the user sees. It fell through to the generic "is the
GGUF valid / out of memory" text. Make it actionable: name the signal and
point at a smaller or more quantized GGUF, a lower context length, or raising
the WSL memory limit. Output-based diagnoses still win and a hard fault keeps
the generic fallback.
* Studio: refuse a model too large for system RAM on a unified-memory APU
On gfx1150/gfx1151 APUs the weights load into shared system RAM (GGML
unified memory). _get_gpu_free_memory reports the full ROCm/APU budget as
free (often ~100 GB), but under WSL the VM's RAM cap is the real ceiling.
Studio trusted the budget, spawned a load larger than RAM, and the OS killed
it mid-flight, taking the Studio process with it (a silent "Terminated" with
no error, the model resident in RAM not VRAM).
Add a pre-flight guard on the APU path: if the weights exceed available
system RAM (psutil, then /proc/meminfo), refuse before spawning with a clear
message (smaller/more-quantized GGUF, lower context, or raise the WSL memory
limit). Weights only so KV/context auto-reduction is not double-counted;
unknown RAM never refuses; non-APU and discrete-GPU paths are untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: refine recovery ladder (keep diagnosed errors, flip all flash-attn)
Two review points on the hard-crash recovery ladder:
1. The signal-only text-only fallback stripped --mmproj on any hard fault,
even when llama-server had already printed a non-projector cause (an OOM
such as "cudaMalloc failed: out of memory", an unsupported architecture, or
a tensor-parallel limit). That masked the real error and told the user to
update llama.cpp for vision. New _output_has_nonprojector_diagnostic gates
the signal path: it fires only when no such marker is present, so a bare
SIGSEGV with no output still retries text-only, but a diagnosed OOM surfaces
the real error instead of silently dropping vision.
2. _with_flash_attn_off only flipped the first --flash-attn. llama.cpp is
last-wins, so a leftover enable from extra_args (--flash-attn on, -fa on, or
the = form) could keep flash attention on and re-crash the retry. It now
flips every occurrence and returns None only when nothing is flippable.
test_llama_cpp_mmproj_fallback.py and the classification/APU suites: 103 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: pass the text-only retry's exit code to the failure classifier
When the text-only fallback retry itself fails, read its exit code before
_kill_process() clears it and forward it to _classify_llama_start_failure, so an
OS-killed retry surfaces the actionable out-of-memory message instead of the
generic one (matching the primary failure path).
* Studio: scope APU RAM guard to selected GPUs, count MTP drafter, neutral SIGTERM
Three refinements to the startup recovery work in this PR:
- The unified-memory APU RAM guard fired whenever any visible GPU was a
gfx1150/gfx1151 APU, so on a mixed APU+dGPU host it could refuse a valid
load placed on the discrete GPU. Scope _amd_apu_wants_unified_memory to the
selected gpu_indices (physical ids, mapped via CUDA_VISIBLE_DEVICES like
_is_datacenter_gpu); None still means every visible GPU. Applied to both the
RAM guard and the GGML_CUDA_ENABLE_UNIFIED_MEMORY env set.
- The RAM guard counted only the main GGUF plus mmproj, so a separate MTP
drafter (also resident in unified system RAM, even when offloaded to CPU)
could push the load past the RAM cap and still get OS-killed mid-load. Add
the drafter weights to the APU RAM total.
- The startup classifier reported SIGTERM (-15) as 'most likely out of memory',
but SIGTERM is also how an unload/cancel or a supervisor stops the server.
Keep the OOM wording for SIGKILL (-9, the OOM killer) and report -15
neutrally.
Tests updated/added accordingly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review on the APU guard and decode-probe ladder
- Map APU physical ids via the active ROCm mask (HIP, then ROCR, then CUDA),
mirroring _get_gpu_memory, so a HIP_VISIBLE_DEVICES-selected APU is matched.
- Only add the MTP drafter to the APU RAM total when MTP will actually engage,
so a stale LLAMA_ARG_SPEC_DRAFT_MODEL cannot refuse a non-MTP load.
- After an MTP first-decode hard fault, retry --flash-attn off (keeps MTP)
before dropping speculative decoding, matching the startup rung.
- Fold the --flash-attn= / -fa= rewrite into one branch.
Tests: tensor-parallel decode-probe assertion updated for the FA-off rung.
* Studio: tighten two comments in the APU guard and RAM preflight
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: refine flash-attn retry and APU RAM guard per review
- _with_flash_attn_off now decides on the effective last-wins value: it returns
None when FA is already off (no wasted retry), and neutralizes a bare
--flash-attn / -fa (which llama.cpp reads as on) so the retry cannot re-enable
it. Length is preserved so downstream index slices stay valid.
- _amd_apu_wants_unified_memory uses 'gpu_indices is not None' so an empty
selection is respected (not treated as all-visible).
- The APU RAM refusal now checks the base model only (main + mmproj); an
optional MTP drafter is dropped by the existing MTP-drop fallback rather than
causing a hard pre-spawn refusal of an otherwise loadable model.
Tests: bare-flag / effective-off / empty-selection / HIP-mask cases added.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: free chat model VRAM at training start only when the GPU is tight
The training start route unconditionally tore down the transformers/MLX
inference subprocess before training, and never stopped the llama.cpp GGUF
server at all, so a loaded GGUF chat model kept holding VRAM for the whole
run. Conversely the HF model was always unloaded even when there was plenty
of room to keep it.
Make the unload VRAM aware and cover every inference backend:
- Add routes/training_vram.py with summarize_resident_chat(),
can_keep_chat_during_training() and free_chat_models_for_training(). The
keep/unload decision reuses the same estimator and live per device free
VRAM reader the training GPU selection already uses (auto_select_gpu_ids,
estimate_required_model_memory_gb, get_visible_gpu_utilization), so the
probe agrees with the placement computed later in start_training.
- When a chat model is resident and training fits alongside it with a
conservative margin (required_gb * 1.15 + 4 GB), keep it loaded so the
user can train and chat at the same time; on a multi GPU box training
lands on a different GPU and both coexist. Otherwise unload the HF/MLX
orchestrator and the llama.cpp GGUF server before training starts.
- The export subprocess shutdown stays unconditional and now runs first so
its freed VRAM is reflected in the decision.
Default deny: non CUDA backends, unestimable models, or any probe error
fall back to the previous always unload behavior.
Adds tests/test_training_vram_coexistence.py and updates two existing route
tests in test_gpu_selection.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: per-GPU floor for explicit GPU lists + don't unload chat on invalid gpu_ids
Address review feedback on the chat coexistence probe:
- Explicit gpu_ids mode now enforces a per-GPU floor in addition to the
aggregate free-VRAM check, mirroring auto_select_gpu_ids' min_per_gpu_N.
Without it, an uneven split such as free [45, 10] for a 40 GB job passed
the aggregate threshold and kept chat loaded even though the 10 GB GPU
could not hold its training shard, risking an OOM.
- Invalid explicit gpu_ids (ids outside the visible set, or a UUID/MIG
mask) make resolve_requested_gpu_ids raise. That request is rejected with
a 400 before training starts, so leave the resident chat model untouched
instead of unloading it.
- Tighten the target_modules / gpu_ids type hints to List[str] / List[int].
Adds tests for the per-GPU floor (uneven split unloads, even split keeps)
and for invalid gpu_ids keeping the chat model loaded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: only free chat VRAM once training will start; handle in-flight and CPU-only chat
Address the second review pass on the chat-coexistence path:
- Run the chat/export VRAM teardown as a before_spawn hook inside
TrainingBackend.start_training, fired only after the start guards pass.
Previously the route freed chat VRAM before calling start_training, so a
refused start (e.g. a lingering pump thread) would tear down the resident
chat model even though no training job began.
- Treat an in-flight HF chat load (loading_models set, no active model yet)
as not safely sizeable: free it rather than risk both OOMing as the load
keeps allocating after training starts.
- Do not count or tear down a GGUF llama-server confirmed to run entirely on
CPU (_gpu_offload_active is False): it holds no VRAM, so killing it cannot
help training fit.
Adds tests for the before_spawn hook (runs on start, skipped when a
subprocess is alive or a pump thread will not die, survives a hook error),
the in-flight load flag, and the CPU-only GGUF exclusion in both the resident
summary and the unload path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: treat any in-flight chat load (HF swap / mid-start GGUF) as unsafe to keep
Tighten the in-flight detection in summarize_resident_chat so the keep check
never sizes a load that is still allocating:
- Flag loading on ANY non-empty loading_models, not only when active_model_name
is empty. load_model adds the new model to loading_models before clearing the
old active_model_name, so a replacement load during a swap was previously
sized as a normal resident and could OOM as the new model finishes loading.
- Flag a GGUF server that is active but not yet healthy (is_loaded False) as
in-flight: it is still mmaping/offloading layers, so its final VRAM footprint
is unknown.
Consolidates the signal into a single resident["loading"] flag; the route frees
the chat model whenever it is set. Adds tests for the replacement HF load and
the mid-start GGUF cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten comments in chat/training VRAM coexistence (comments only)
* Studio: run before_spawn VRAM hook only after GPU-selection validation
Reviewers found the before_spawn hook fired before prepare_gpu_selection
validated gpu_ids (and before config build), so a refused start (invalid
gpu_ids -> 400, or a bad grad-clip value) could still tear down chat/export
VRAM. Move the hook to immediately before proc.start(), once all synchronous
validation and process construction have passed. This also fixes the route's
in-flight-chat loading branch, since that teardown runs inside the same hook.
Add test_hook_skipped_when_gpu_selection_rejects.
* Studio: recompute GPU auto-selection after the before_spawn VRAM hook
Codex P2: with before_spawn moved after prepare_gpu_selection, placement was
frozen against the pre-teardown VRAM state while the hook freed export/chat
afterward. Auto-selection could pin training onto a GPU the hook then cleared
(or onto a kept chat model). Split validation from placement: explicit gpu_ids
are still validated before the hook (raise -> 400, no teardown; explicit
placement is VRAM-independent), but VRAM-dependent auto-selection now runs
after the hook so it sees the freed memory.
Add test_auto_placement_runs_after_hook and test_explicit_placement_validated_before_hook.
* Studio: allow chatting during training (lift sidebar gate + VRAM-aware load guard) (#6335)
* Studio: allow chatting during training (lift sidebar gate + VRAM-aware load guard)
The sidebar disabled New Chat, project, and home navigation while a training
run was active, so users could not chat during training even though the backend
serves inference fine alongside a run. This removes that gate and adds a backend
guard so the one genuinely risky operation, loading a new local chat model
mid-training, is refused with a clear 409 when it would not fit beside the run.
Frontend (app-sidebar.tsx): drop the chatDisabled = isTrainingRunning gate and
its consumers. Navigation triggers no model load on its own, so chat stays
usable during training.
Backend (routes/training_vram.py, routes/inference.py): add
can_load_chat_during_training plus a load/validate guard that sizes the same
effective load the loader performs (LoRA 4-bit to 16-bit resolved first, HF auto
placement via auto_select_gpu_ids, explicit multi-GPU per-GPU floor, GGUF sized
from on-disk shards and companions or the selected remote variant). It is a
no-op when training is inactive, never blocks external providers or
already-resident models, and default-denies only on a CUDA sizing failure so a
load can never OOM the run. Validate refuses early with the real settings so the
frontend does not unload the resident chat model for a load that would be
rejected.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review feedback for chat-during-training load guard
- Run the load/validate VRAM guard via asyncio.to_thread so the sync
nvidia-smi + HF metadata work never blocks the event loop.
- Size the GGUF KV cache at the requested context (_estimate_gguf_kv_gb)
and add it to the local GGUF estimate so large-context picks are not
under-counted.
- Keep the requested quantization when adapter_config.json is malformed
(not a JSON object) instead of raising in _effective_load_in_4bit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: size the training load guard at the launcher's effective GGUF context
The GGUF KV-cache estimate used max_seq_length only, but the llama.cpp
launcher honors a user --ctx-size/-c in llama_extra_args. A load such as
max_seq_length=4096 with --ctx-size 131072 was sized against a 4k cache
while the server allocates 131k, so the guard could approve a long-context
GGUF load that then OOMs training. Size the guard's KV at the larger of
max_seq_length and the parsed --ctx-size (reusing the launcher's own
parse_ctx_override), keeping the conservative f16 cache so the estimate is
never smaller than what the server allocates.
The chat model picker also validated with the raw max_seq_length while
/load sizes with resolveLoadMaxSeqLength, so validate could pass, unload
the current model, then have /load reject the native-context load. Validate
now uses the same effective context; the load path is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: size the GGUF training guard at the server parallel-slot count
The KV-cache estimate assumed a single slot, but llama-server allocates the
cache across --parallel slots (app.state.llama_parallel_slots). On a Studio
launched with --parallel N>1 the guard under-sized the cache N-fold and could
approve a GGUF chat load that then OOMs training. Thread the same slot count
the loader uses into the guard's KV estimate; default 1 leaves single-slot
setups unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments for chat-during-training guard
* Studio: keep chat generation alive across navigation; Train spinner + Return to Chat
Hoist the base chat runtime above the routed outlet so navigating to Train (or any tab) no longer aborts an in-flight generation; only an explicit Stop cancels. Add a Train sidebar spinner and swap New Chat to Return to Chat while a run is active, with a lightweight completion watch so the spinner clears from any tab. Also respawn a chat llama-server killed mid-session and guard unreadable HF cache dirs that 500'd the hub model list.
* Studio: show Return to Chat on the Train tab whenever a chat is live
Previously the top sidebar item only swapped to Return to Chat while training was running; on the Train tab with an idle/just-finished run it stayed New Chat, which started a fresh thread and cancelled an in-flight generation. Show Return to Chat (and navigate back, preserving the run) whenever a generation is running or its thread is still active, or training is in progress.
* Studio: keep a running chat alive when starting a New Chat
Starting a New Chat (or switching threads) while a generation was in flight
remounted the single-chat runtime provider, which detached the in-flight run
and cut the previous chat off (it showed up frozen / empty when reopened).
Key the single-chat view by project instead of by thread or new-chat nonce so
the provider stays mounted and assistant-ui switches to a fresh thread in place.
The previous generation keeps streaming in the background and autosaves on
completion, and returning to that thread reattaches the live run instead of
reloading a half-saved one.
Also:
- "Return to Chat" now lands on the thread that is still generating rather than
the empty new chat that became active after New Chat.
- Skip the explicit /inference/cancel POST when an abort comes from a runtime
detach (navigation / background switch) rather than an explicit Stop, so a
backgrounded generation is never cancelled behind the scenes.
* Studio: make model export non-blocking and inline
The Export tab opened a full-screen modal that trapped focus, could not be
closed or cancelled while running, and showed no progress. It also stopped
training and unloaded the chat model before loading, so export could not run
alongside them.
Export now mirrors the training runtime pattern:
- Inline panel embedded where the Export Model button was, with no modal or
backdrop, so the rest of the UI stays usable during an export.
- Global export runtime store plus an app-root lifecycle hook, so a run keeps
going and streaming across navigation and is reflected on the Export nav item
from any tab.
- The worker log stream now stays connected across the load to export phase
boundary instead of stranding on "Waiting for worker output".
- Progress bar driven by phase and quant index (quant N of M for GGUF), with
elapsed time and a working Cancel.
- load-checkpoint no longer stops training or unloads inference; export loads in
its own subprocess in parallel and surfaces out-of-memory as a clear error.
- Add POST /api/export/cancel and is_export_active on /api/export/status.
* Studio: show Return to Chat on the Export tab too
Extend the New Chat to Return to Chat swap to the Export route so leaving a
running chat for Export offers a way back to the live generation, matching the
Train tab.
* Studio: smooth out Export animations and polish the panel
- Drop the height-based reveal animations (source switch, run panel, quant
picker, hub fields) that caused flashing and reflow; use instant swaps and
quick opacity fades instead.
- Method and quant cards now transition colors only, with no transition-all or
hover lift, so selecting a method or quant is crisp instead of jumpy.
- Auto-scroll the export panel into view when it opens and add a scroll-to-bottom
button when its output is below the fold, like Chat.
- Show Return to Chat on the Export tab while an export is running, matching how
training drives it on the Train tab.
- Surface the current phase or stage in the live output before the first worker
line arrives so the panel never looks stuck while progress is advancing.
* Studio: show Return to Chat on every non-chat tab
Generalize the Return to Chat swap from just Train/Export to any non-chat route
(Recipes, Projects, Hub, ...) so a running or active chat is always one click
away, instead of showing New Chat there.
* Studio: stream export logs over the Cloudflare tunnel; drop janky export animations
Exporting over a --secure Cloudflare quick tunnel showed "connecting..." with no
logs while the progress bar advanced. Cloudflare buffers text/event-stream and
only flushes when the stream closes, so the SSE log stream never reached the
browser during the run (direct localhost is unaffected, which is why this only
showed up over the tunnel).
Add a tunnel-safe JSON poll fallback (GET /api/export/logs?since=) that the
runtime lifecycle hook polls while a run is active. Short JSON responses are not
buffered by the proxy, so logs show up in near real time over the tunnel. It
shares the orchestrator's monotonic seq cursor with the SSE stream and the store
de-dupes by seq, so the two transports run together (SSE on localhost, poll over
the tunnel) without double-printing. A successful poll marks the panel
"streaming" instead of leaving it stuck on "connecting...".
Also remove the framer-motion AnimatePresence reveals from the export config and
run panel (quant picker, hub fields, the inline run panel, and the live log
section). The expand/slide animations flashed and felt clunky; the sections now
render in place.
* Studio: recover export over the Cloudflare tunnel when the blocking POST times out (524)
A model export over a --secure Cloudflare quick tunnel showed "Request failed
(524)" even though the export succeeded on the backend (the GGUF was written).
Cloudflare returns 524 when a single request takes longer than ~100s to respond,
and a GGUF conversion routinely runs for minutes, so the blocking per-method
export POST is cut off while the backend keeps going.
Confirm completion via short status polls instead of relying on the long POST
response (the same approach that fixed log streaming):
- The orchestrator records each finished op's outcome (status / output_path /
error) with a monotonic seq, exposed on GET /api/export/status.
- parseJson now preserves the HTTP status; a 524/520/522/523/502/503 or a
status-less network drop is classified as a recoverable transport error.
- runExport wraps each phase (load, every export method, each GGUF quant): on a
recoverable failure it keeps the run alive (logs keep streaming, the panel
shows "reconnecting...") and polls status until the still-running op finishes,
then settles from the recorded result, recovering the output path for the
success banner. A real 4xx still fails immediately; localhost still uses the
fast POST response. applyBackendStatus also settles a reloaded run from the
last-op record.
Verified over the tunnel: a 3m14s gemma-4-E4B-it GGUF export now ends on the
success banner with the output path instead of 524.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep the export method + logs visible after navigating away mid-export
While an export was running, navigating to another tab and back to Export
remounted the page and reset the local form state (exportMethod, quant levels),
so the method card showed unselected and the run panel's log area was hidden
until the card was re-clicked. The run itself lives in the global store and was
unaffected.
Seed exportMethod / quantLevels from the active run's summary via lazy useState
initializers on (re)mount, and gate the panel's log area on the live run
(isExporting / logLines / the run's method) rather than only the local form
selection. The card stays selected and the logs/progress stay visible across
navigation; nothing changes when no run is active.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: address export/training review findings
- Export: guard Start against an empty GGUF quant selection so an inline-panel
run with no quant can't settle as success with no file produced.
- Export: thread the source HF token into the background load so gated/private
HF source exports (and gated bases) authenticate, matching the consent path.
- Export: only settle a recovered (non-owned) run as a finished export when the
last backend op was an export, not a standalone load_checkpoint.
- Training: free the export subprocess whenever an export is active, not only
once a checkpoint is loaded, so an in-flight export load can't race training
for VRAM (current_checkpoint is unset during the load phase).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Windows installer: repair a stale CPU PyTorch instead of looping forever
A Windows machine with an NVIDIA CUDA 13 driver (e.g. RTX 6000 Pro on enterprise
drivers) could get permanently stuck at:
Stale venv detected (torch cpu != required cu130).
[ERROR] The existing Studio environment needs repair.
Re-run install.ps1 so it can replace the environment safely with rollback.
Re-running install.ps1 did not help. install.ps1 installs torch with
"torch>=2.4,<2.11.0" --index-url .../cu130 but no --force-reinstall, so when a
torch==X+cpu is already present uv treats it as satisfying the range (PEP 440
ignores the +cpu/+cuXXX local label) and makes no change -- the CPU wheel is
never replaced. setup.ps1 then rejects the venv as cpu != cu130 and exits, but it
cannot create a venv or install torch, so the loop never resolves. The migrated-
venv branch also preserves existing torch and never reinstalls it.
After the install step, detect the installed torch flavor (cuXXX/cpu/rocm) and,
when it does not match the tag implied by the selected index, force-reinstall the
torch/torchvision/torchaudio triplet from the correct index via three
--reinstall-package flags. No-op on a healthy matching venv; skipped for
--no-torch, ROCm (already --force-reinstalls), and CPU-only machines.
Adds two pure helpers (ConvertTo-TorchFlavorTag, Get-ExpectedTorchFlavorTag), a
PowerShell unit test (tests/studio/test_torch_flavor.ps1), and a CI parse gate for
install.ps1 (previously unparsed).
* install.sh: repair a stale CPU PyTorch on Linux too (parity with install.ps1)
install.sh has the same latent bug as the Windows installer: the CUDA torch
install uses "torch>=2.4,<2.11.0" --index-url .../cuXXX with no
--force-reinstall, so an already-present torch==X+cpu satisfies the version
range (PEP 440 ignores the +cpu/+cuXXX local label) and uv leaves it in place.
The migrated-venv branch also preserves existing torch. Unlike Windows there is
no stale-venv check in setup.sh, so on Linux the symptom is silent CPU training
rather than a hard loop -- same root cause.
Mirror the install.ps1 fix: after the install block, detect the installed torch
flavor (_torch_flavor_tag) and, when it does not match the index tag
(_expected_torch_flavor_tag), force-reinstall the torch/torchvision/torchaudio
triplet from the selected index via --reinstall-package. No-op on a healthy
matching venv; skipped for --no-torch, ROCm (its own repair force-reinstalls),
and CPU-only / macOS hosts. Adds tests/sh/test_torch_flavor.sh (run in
studio-backend-ci and run_all.sh).
* Installer: catch CPU-fallback on AMD/WSL too (repair ROCm, warn when unfixable)
Extend the torch-flavor safety net beyond NVIDIA:
- install.sh now auto-repairs a stale CPU torch on standard pytorch.org ROCm
indexes too (the rocm-index install path lacked --force-reinstall, unlike the
Windows ROCm install). Reuses the rocm-adjusted $TORCH_CONSTRAINT + rocm index,
so it pulls the correct ROCm wheels.
- Both installers gain a universal post-install warning: when a GPU build was
expected (cuXXX / rocm, including the repo.amd.com gfx* arch indexes) but torch
is still CPU-only, warn loudly instead of silently training on CPU. This catches
the cases auto-repair cannot safely fix (AMD gfx arch indexes that need
--find-links, a migrated AMD venv on Windows where the ROCm install was skipped).
- Mac / Intel / CPU-only hosts resolve to the cpu index -> expected == installed
-> no-op, no false warning. WSL uses install.sh, so the NVIDIA repair + warning
apply there.
Adds Get-InstalledTorchTag (ps1) and _torch_index_repairable (sh) helpers and
extends both unit tests. gfx*/AMD indexes now map to the 'rocm' expected flavor.
* Installer: tighten torch-flavor comments (no logic change)
Condense the rationale comments added for the stale/CPU PyTorch repair in
install.ps1, install.sh and the two helper unit tests; same intent, fewer
lines. Comment-only: AST parse of install.ps1/setup.ps1 clean, helper unit
tests (15 ps1, 24 sh under bash and dash) and the integration sims
(24 ps1, 28 sh) still pass, banner markers the sims slice on are unchanged.
* Installer: bound torch probe, auto-repair gfx, fix ROCm gate parity
install.ps1: in Get-InstalledTorchTag, call WaitForExit(30000) and drain stdout
and stderr asynchronously instead of reading stdout synchronously first, so a
hung or noisy "import torch" (a wedged CUDA/driver, the exact failure this PR
targets) can no longer block the probe past the timeout.
install.sh and install.ps1: treat the repo.amd.com gfx* indexes as plain
--index-url reinstallable. They are PEP 503 simple indexes uv resolves in full
(torch plus every transitive dep) via --index-url, the same URLs the fresh
ROCm install paths already use, so a stale CPU torch on AMD Strix now auto-repairs
to the correct ROCm build instead of only warning.
install.sh: include */gfx* alongside */rocm* in the bitsandbytes install and
ROCm torch repair gates, so a custom UNSLOTH_AMD_ROCM_MIRROR whose path lacks
/rocm/ still installs the AMD bitsandbytes build and repairs ROCm torch.
tests/sh/test_torch_flavor.sh: gfx indexes now assert repairable, plus a
gfx1151 case and an unknown-mirror not-repairable case.
* install.ps1: guard Get-InstalledTorchTag against an empty python path
Make the early return explicit for an empty $PythonExe instead of relying on
Test-Path -LiteralPath '' returning false, so the probe stays safe under
Set-StrictMode or a future refactor that drops the [string] annotation.
* Studio: polish Bypass permissions toggle and add it to chat menu settings
- Recolor the pill, menu item and settings caption from alarm red to a
bright yellow accent via a new --bypass token (light and dark).
- Use the shield icon on the composer pill (X on hover, like other pills).
- Expand the composer to two rows the moment Bypass permissions is on.
- Keep Bypass leftmost among tool pills; Compare still sits ahead of it.
- Add a Bypass permissions row to Settings > Chat menu so it can be pinned.
- Widen the "More" submenu so the label fits on one line.
* Studio: use shield-ban icon and lighten the Bypass permissions pill
- Swap the lucide shield-off icon for the Hugeicons shield-ban across the
pill, menu item and settings row.
- Lighten the pill background and deepen the yellow text a touch so the
label stays readable on a near-white tint.
* Studio: drop the Bypass permissions resting fill, keep it on hover
Show only the yellow icon and label at rest; the rounded hover pill picks
up the yellow accent like the other toggles.
* Studio: lighten the Bypass permissions hover fill
Drop the hover tint so the yellow label stays readable on hover.
---------
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Switch the sidebar profile button from rounded-full to rounded-[5px] so its hover background is a rounded rectangle instead of a pill. The collapsed icon-only state stays circular. Styling only.
The base-install lines passed a bare unsloth-zoo spec and relied on the
co-installed unsloth>=2026.6.7 (whose pyproject pins unsloth_zoo>=2026.6.5)
to floor unsloth-zoo transitively. Make the floor explicit so the install
scripts stay in sync with pyproject and the bare spec can never resolve below
the version that ships unsloth_zoo.diffusion_studio (the DiffusionGemma Studio
runner), first released in unsloth-zoo 2026.6.4. Leaves the reinstall/upgrade
flags and the git-main overlay untouched.
* Load repo-code VLMs that register AutoModel in auto_map
FastModel.from_pretrained already falls back from the VLM auto class to
AutoModelForCausalLM for repo-code VL models that register only that class
in their auto_map (e.g. Nemotron-VL). Models like DeepSeek-OCR and
DeepSeek-OCR-2 instead register their architecture under AutoModel, so they
fell through to AutoModelForImageTextToText and raised "Unrecognized
configuration class ... for AutoModelForImageTextToText".
Generalize the guard: when neither vision auto class is registered, fall
back to whichever generic auto class the repo actually registered
(AutoModelForCausalLM, else AutoModel).
* Do not hard-error on a newly initialized position_ids buffer
RaiseUninitialized turns transformers' "some weights of ... were not
initialized" warning into a hard error. position_ids is a deterministic
arange buffer that transformers itself lists in
_keys_to_ignore_on_load_missing, so re-initializing it is correct rather
than a sign of a corrupt checkpoint. Some VLMs (e.g. DeepSeek-OCR) ship it
non-persistently, which tripped the guard. Allowlist position_ids alongside
the existing classifier/predictions head weights.
* Only ignore missing-weight records that are exclusively position_ids
The previous substring check skipped the whole "Some weights of ..." record
whenever position_ids appeared anywhere in it. Transformers reports every
missing key in one record, so a corrupt or incompatible checkpoint missing a
real parameter could load with randomly initialized weights as long as one
missing key contained position_ids. Parse the "newly initialized: [...]" list
and suppress only when every listed key is a position_ids buffer; otherwise
raise as before.
* Match the concrete VLM auto class name when checking auto_map
Transformers resolves remote code by the exact auto class name being called,
and AutoModelForVision2Seq aliases to AutoModelForImageTextToText on
transformers >= 5. Checking for both spellings treated a config that only
registers the legacy AutoModelForVision2Seq key as having a supported VLM
class, skipping the AutoModelForCausalLM fallback that used to load it and
failing as an unrecognized config under AutoModelForImageTextToText. Match
only the concrete class name we would actually pass, keeping the AutoModel
and AutoModelForCausalLM fallbacks.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Preserve VLM mode on the vLLM path when falling back to AutoModel
A repo-code VLM that registers only AutoModel or AutoModelForCausalLM (DeepSeek-OCR, Nemotron-VL) routes to that generic class, so is_vlm, derived from the resolved auto class, is False. That is correct for processor selection (these repos ship no AutoProcessor) but wrong for the vLLM path, where is_vision_model=is_vlm made vLLM treat a vision_config model as text-only and skip the VLM guard and conversion.
Add is_vlm_config, derived from the config vision_config (and gated on not text_only so a text-only resolve still wins), and use it for the fast_inference VLM guard and the is_vision_model flags passed to load_vllm, get_vllm_state_dict and convert_vllm_to_huggingface. Processor selection still uses is_vlm, so DeepSeek-OCR keeps loading via its tokenizer. DeepSeek-OCR with fast_inference now raises the clear 'Fast inference is only supported for ...' error instead of being mishandled as text-only.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Package scanners: close fail-open gaps in the sdist fallback and hidden-payload paths
Follow-up hardening on the now-blocking scanners so the enforcing gate cannot
report clean while a malicious artifact goes unscanned.
scan_packages.py
- Hidden payload: also flag a network call AND an os/subprocess exec that live
only in a blanked docstring/string of an exec/eval file (the fetch-then-run
shape of an exec(__doc__) dropper). Either alone in real code was already
covered; hidden together they are the payload.
- Pinned releases fail closed: _release_files no longer falls back to the latest
artifact when a pinned version is missing or empty, so a yanked/bad pin is an
error instead of a different file being scanned in its place.
- requires_dist is read from the pinned release's metadata, not the project-level
(latest) document, so a sdist-only pin follows its own dependency tree.
- Environment markers are evaluated (PEP 508) instead of dropping any marker that
merely contains the word extra, so default-true markers like extra != 'dev' are
kept; conservative fallback keeps a dep on any uncertainty.
- Transitive recovery is a depth-bounded worklist: a wheel dependency whose own
child is sdist-only is fetched (--no-deps) and scanned, then its children are
recovered in turn, rather than being silently skipped.
scan_npm_packages.py
- Baseline keys use the package-relative path instead of the basename, so the
same basename in a different directory is not over-suppressed.
Tests cover each case; full scripts pass AST and ruff checks.
* Address review: tighten marker scope, decoy-proof the dropper check, fail closed on missing pin metadata
- Markers: keep any dep whose marker can hold on another install target
(sys_platform == 'win32', python_version == '3.13'); only drop a marker that
depends solely on extra and is false with no extra. A scanner runs on one
target but must cover code installed on others. Pure-extra markers are
evaluated against default_environment() with extra unset.
- Hidden dropper: the network+exec docstring check now inspects the removed
(blanked) span directly, so a benign visible network or subprocess call cannot
mask a payload that still lives in a docstring. Carrier checks stay
blanked-only (an in-code carrier is already caught by the normal check), so
corpus findings are unchanged.
- requires_dist: a pinned version whose own metadata cannot be fetched recovers
nothing rather than substituting the latest release's dependency tree.
- Transitive recovery: the last-ditch direct-sdist branch also chases the
recovered package's declared deps, matching the other branches.
- npm baseline: schema bumped to v2 (package-relative keys); a pre-v2 baseline
with entries is ignored (fail closed) instead of mis-applying basename keys.
Tests cover each case; scripts pass AST, ruff, and the import-hoist verifier.
* Scanner: exclude comments from hidden-payload check, flag missing pin metadata as incomplete
Hidden network+exec detection now inspects only docstring/string spans (what exec(__doc__)/exec(<str>) can actually run), so a real exec() beside comments that mention a network and a subprocess call no longer false-positives. Missing pinned-release metadata in transitive recovery records a download_error so the --with-deps path fails closed instead of treating it as no dependencies. Adds regression tests for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: fail fast on an invalid first training batch
Training a base vision-language model (e.g. Qwen/Qwen2-VL-7B or
unsloth/Qwen2-VL-7B) on a conversational image dataset crashed on the first
step with 'Expected ... Long, Int; but got torch.cuda.FloatTensor (embedding)'.
Root cause: the base model's chat template is a flat, media-only template that
renders to an empty string for role-based messages, so UnslothVisionDataCollator
hands the processor empty text, the processor returns empty input_ids, torch
defaults the empty tensor to float32, and the embedding lookup rejects it.
Add a preflight that runs one real batch through the trainer's own tokenization
and collation right before train(), and stops the run with an actionable message
when input_ids is empty or non-integer (pointing to the instruction-tuned variant
for the base-model case). Faithful across text, vision and audio-VLM paths, and
never blocks a run whose first batch is valid.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments in the training preflight
* Stub unsloth/trl in preflight test so backend CI collection passes
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The colocate GRPO setup set args.vllm_enable_sleep_mode from the raw
UNSLOTH_VLLM_STANDBY env var (== '1'). When unsloth-zoo's vision standby gate
disables sleep mode for a multimodal run, the engine is built with
enable_sleep_mode=False but TRL was still told sleep mode is on, so it could
drive sleep/wake against an engine that did not enable it.
Read enable_sleep_mode from the colocated engine
(model.vllm_engine.llm_engine.vllm_config.model_config), the same path
check_sleep_mode uses, and fall back to the standby env var (!= '0', matching
load_vllm/patch_vllm) only when the engine cannot be introspected.
Pairs with unslothai/unsloth-zoo#768.
* Fix scan_packages.py --fix crash on download_packages() tuple return
`download_packages()` returns `(results, download_errors)`, but the two
`--fix`-path call sites still treated the return value as the bare results
list. `find_safe_version` did `downloaded = download_packages(...)` followed
by `if not downloaded:` (always false: a 2-tuple is truthy) and
`for _, archive_path in downloaded:`, which unpacked the results list into
two variables -> ValueError in the normal single-archive `--no-deps` case.
`_run_fix` indexed `downloaded[0][1]`, i.e. the second archive of the results
list instead of the first archive's path -> IndexError. So `--fix` crashed
exactly when a CRITICAL finding needed remediation. The main scan path already
unpacks the tuple; this aligns the two `--fix` sites with it.
Adds CPU-only regression tests for both sites.
Closes#6412
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Update scripts/scan_packages.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update scripts/scan_packages.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Studio: show an actionable message when the GGUF runtime is missing
Selecting a GGUF model with no llama-server installed surfaced a generic
"Invalid model" in the UI, because validate_model's catch-all discarded the
real cause. Add LlamaServerNotFoundError (a RuntimeError subclass) raised by the
GGUF preflight in ModelConfig.from_identifier, and catch it in the validate
route so users get an actionable message: run `unsloth studio setup` to
download the prebuilt llama.cpp runtime. Other validation failures keep the safe
generic message. Adds a regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: also map missing GGUF runtime to a 400 in load_model
validate_model already surfaces the actionable 'install the runtime'
message for LlamaServerNotFoundError; load_model fell through to the
generic 500 'Failed to load model'. Catch it there too so a GGUF load
without llama-server gives the same install hint instead of a 500.
* Trim comments for PR #6327
* Studio: fix stale validate test after #6398 and surface missing GGUF runtime on /load
- test_other_runtime_errors_do_not_get_gguf_message: after merging #6398,
validate_model surfaces a RuntimeError's own message, so a plain RuntimeError
no longer returns "Invalid model". Assert it does not receive the GGUF
install message instead (the prior assertion was stale after the main merge).
- Raise LlamaServerNotFoundError (not a plain RuntimeError) at the backend
load-time missing-binary branch, after diffusion routing, so /load returns the
actionable 400 like remote validation, instead of a generic 500.
- Share LLAMA_SERVER_NOT_FOUND_DETAIL between the from_identifier preflight and
the load-time raise so the message stays in sync.
- Add a propagation regression test for the non-tensor load path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: reach the published source asset when a mix build's commit 404s
A llama.cpp "mix" prebuilt records a merge commit that is never pushed to
the fork, so the codeload/archive URLs for that commit 404. The merged
source tree is instead published as a release asset alongside the prebuilt
(llama.cpp-source-commit-<sha>.tar.gz). The installer resolves that asset
URL from the approved-checksums manifest, but when the manifest omits the
top-level repo/release_tag the URL resolves empty, hydration falls through
to the 404-ing commit archive, and the prebuilt install drops to a slow
source build (or fails outright).
Extract exact_source_asset_url() and resolve the asset's host and tag
defensively: the artifact's own repo, then the manifest repo, then the
source repo; and the manifest release tag, then the tag we actually
installed the prebuilt from (the source asset is its sibling on the same
release). Normal installs build the identical URL as before, so this only
adds a working fallback for the degenerate manifest.
Add unit coverage for the resolver, including the empty repo/release_tag
regressions.
* Studio: cover exact_source_asset_url through the real parser chain
Add TestExactSourceAssetUrl.test_resolves_through_real_parser_chain, which runs
parse_approved_release_checksums -> preferred_source_archive -> exact_source_asset_url
so a regression in the parser or source-selection wiring cannot pass while only the
hand-built helper unit tests stay green.
* Keep server-side tools enabled under --secure and on every bind
--secure binds loopback and exposes Studio only through an authenticated
Cloudflare HTTPS tunnel, but it was grouped with a raw 0.0.0.0 bind and
force-disabled all server-side tools (web search, Python, terminal). The
process tool policy overrode the client's enable_tools request, so the
model was never told the tools existed and answered in plain text. The
plain 'unsloth studio' command had no way to re-enable and printed nothing.
Tools now default on for every bind. The bind host and --secure no longer
change the tool policy; only an explicit --enable-tools/--disable-tools
forces it on or off. Both 'unsloth studio' and 'unsloth studio run' accept
the flags and the startup banner states the resolved policy.
- run.py: replace _apply_default_tool_policy(host, secure) with
_apply_cli_tool_policy(enable_tools); add an enable_tools kwarg to
run_server and --enable-tools/--disable-tools to the argparse.
- _tool_policy.py: resolve_tool_policy defaults to on for every host and
no longer prompts on a network bind.
- studio.py: drop the secure-as-public tool gating, add the flags to the
plain command, and reword the startup banner.
- Update and extend the secure-flag and tool-policy tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add tool-policy notice to plain server banner and refresh run --help
Follow-up to PR review:
- run.py: the plain 'unsloth studio' / --secure / direct run.py path went
through _emit_startup_output without any tool-policy line, so a
network-reachable launch was silent about code execution now that tools
default on. Thread enable_tools through _emit_startup_output /
_emit_secure_startup_output and print a one-line policy notice, followed by
a single stop hint.
- studio.py: the 'unsloth studio run' --enable-tools/--disable-tools and --yes
help still described the removed loopback-on/network-off default and the
confirmation prompt; reword to match the new policy.
- Add tests for the banner notice and the refreshed help text.
* Update CI tool-policy resolver tests for default-on behavior
tests/python/test_unsloth_run_tool_policy_resolver.py still asserted the
removed network-bind policy (0.0.0.0 and LAN IP default off, explicit enable
prompts and aborts on a declined prompt), so it failed the Python CI jobs.
Rewrite the truth table: every bind defaults on, explicit on/off always wins,
and the resolver never prompts (yes/silent/prompt kept for compatibility).
* Trim comments for the tool-policy change
Shorten the verbose docstrings and block comments added for --secure tool
handling; keep the security-relevant intent. Verified comment-only via an AST
diff (code unchanged).
* Add deterministic test that server-side tools execute under --secure
Drive the GGUF agentic tool loop with a fake llama-server stream and let the
real execute_tool run: python counts 1..100, terminal returns a UTC datetime,
and web_search runs through real _web_search with only the ddgs network
boundary mocked. A policy assertion pins that the post-fix --secure path
(policy None + per-request enable_tools) is what keeps these executions
reachable. No model, GPU, or live network; runs in the existing backend CI.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align _emit_startup_output banner test with the moved stop hint
The tool-policy notice now prints between the access banner and the stop
hint, so the stop hint is emitted once at the end instead of inline in the
banner (include_stop_hint is False and print_studio_stop_hint runs once).
Update the plain-localhost case to match; the mismatch and wildcard cases
already asserted this wiring.
---------
Co-authored-by: Michael Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Reap Studio child processes when the parent dies abnormally
Standalone `unsloth studio` launches orphaned cloudflared and llama-server when
the parent exited without running the cooperative shutdown path (terminal-window
close, Task Manager End Task, SIGKILL): the children reparented to init and kept
running, leaving an authenticated Cloudflare tunnel up for days.
Add utils/process_lifetime.py: a parent-owned Windows Job Object
(JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, children auto-inherit) plus Linux
PR_SET_PDEATHSIG, behind a best-effort helper that mirrors the desktop app's
windows_job.rs. initialize_parent_lifetime() runs at the top of run_server;
long-lived spawns (cloudflared, llama-server, RAG embedder, llama.cpp updater)
get the PDEATHSIG preexec, multiprocessing workers are adopted into the job, and
_graceful_shutdown plus atexit gain a terminate_all() backstop sweep. The
cooperative shutdown path is otherwise unchanged.
Verified on Linux: killing the parent now reaps cloudflared and llama-server
within ~2s instead of orphaning them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: add real Windows kill-on-job-close integration test
Spawn a parent that installs the job and a child that inherits it, terminate
the parent, and assert the child is reaped. Skipped off Windows. Also make the
liveness probe Windows-safe (os.kill(pid, 0) terminates on Windows).
* Fix Win64 handle truncation in the Job Object calls
Set explicit argtypes so the 64-bit job/process handles are not marshaled as
c_int (which truncated them on Win64, failing AssignProcessToJobObject). Assert
install success in the Windows integration test.
* Bind multiprocessing workers to parent death; harden the sweep
Review follow-ups:
- Multiprocessing workers (inference/export/training/data-recipe/Xet) cannot be
given a preexec_fn by the parent, so adopt_pid alone left them orphanable on a
Linux SIGKILL. They now bind themselves with PR_SET_PDEATHSIG at startup via
bind_current_process_to_parent_lifetime(), wired into the shared
run_without_native_path_secret entrypoint and the Xet child entry.
- Wire the previously-missed data-recipe worker through adopt_pid.
- terminate_all now honors its timeout: SIGTERM, wait, then SIGKILL the
survivors, so cooperative children can exit cleanly.
- Track adopted pids with a /proc starttime identity and add forget_pid, so the
shutdown sweep never signals a recycled pid.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: cross-session backstop to reap a leftover llama-server on startup
Builds on the parent-lifetime reaper: the Windows Job Object / PR_SET_PDEATHSIG
path kills children when the parent dies, and terminate_all() sweeps the living
parent's children. Neither covers an orphan left by an already-dead Studio:
terminate_all()'s registry is in-memory, PR_SET_PDEATHSIG has no macOS
equivalent, and both are best-effort.
This records the spawned llama-server PID to a pidfile under the active studio
root (removed on _kill_process). The startup reaper kills that exact PID first,
verifying it is still a llama-server to guard against PID reuse, then clears the
pidfile. It is path-independent, so it also catches an orphan the install-root
match would miss; the pidfile only ever names a Studio-spawned server, so
unrelated user processes (vllm, games) are never candidates. The existing
root-gated enumeration stays as a further fallback.
Adds tests: kills a recorded live server (real subprocess, verifies the actual
SIGKILL), skips a reused non-llama PID, cleans a stale/missing pidfile, and
clears the pidfile on kill.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: only reap a recorded llama-server when it is a true orphan
Harden the pidfile cross-session reaper so it cannot kill a live server
and does not crash on Windows.
- Reap the recorded PID only when its parent is gone (a genuine orphan),
so constructing a second LlamaCppBackend in-process (the helper and
advisor paths each build one) can never kill the active chat server.
The check is topology independent: it holds whether the sweep runs in
the main process or a worker.
- Record pid:starttime and verify the start-time identity before killing,
so a PID recycled to another process is never reaped.
- Fall back to SIGTERM when signal.SIGKILL is undefined (Windows), where
os.kill maps it to TerminateProcess, instead of raising and leaving the
orphan alive while clearing the record.
Update and extend the pidfile tests: a live server with a running parent
is spared and its record kept, an identity mismatch is skipped, the
record-to-reap round trip kills a matching orphan, and the Windows
SIGKILL fallback uses SIGTERM.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Michael Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Reap Studio child processes when the parent dies abnormally
Standalone `unsloth studio` launches orphaned cloudflared and llama-server when
the parent exited without running the cooperative shutdown path (terminal-window
close, Task Manager End Task, SIGKILL): the children reparented to init and kept
running, leaving an authenticated Cloudflare tunnel up for days.
Add utils/process_lifetime.py: a parent-owned Windows Job Object
(JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, children auto-inherit) plus Linux
PR_SET_PDEATHSIG, behind a best-effort helper that mirrors the desktop app's
windows_job.rs. initialize_parent_lifetime() runs at the top of run_server;
long-lived spawns (cloudflared, llama-server, RAG embedder, llama.cpp updater)
get the PDEATHSIG preexec, multiprocessing workers are adopted into the job, and
_graceful_shutdown plus atexit gain a terminate_all() backstop sweep. The
cooperative shutdown path is otherwise unchanged.
Verified on Linux: killing the parent now reaps cloudflared and llama-server
within ~2s instead of orphaning them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: add real Windows kill-on-job-close integration test
Spawn a parent that installs the job and a child that inherits it, terminate
the parent, and assert the child is reaped. Skipped off Windows. Also make the
liveness probe Windows-safe (os.kill(pid, 0) terminates on Windows).
* Fix Win64 handle truncation in the Job Object calls
Set explicit argtypes so the 64-bit job/process handles are not marshaled as
c_int (which truncated them on Win64, failing AssignProcessToJobObject). Assert
install success in the Windows integration test.
* Bind multiprocessing workers to parent death; harden the sweep
Review follow-ups:
- Multiprocessing workers (inference/export/training/data-recipe/Xet) cannot be
given a preexec_fn by the parent, so adopt_pid alone left them orphanable on a
Linux SIGKILL. They now bind themselves with PR_SET_PDEATHSIG at startup via
bind_current_process_to_parent_lifetime(), wired into the shared
run_without_native_path_secret entrypoint and the Xet child entry.
- Wire the previously-missed data-recipe worker through adopt_pid.
- terminate_all now honors its timeout: SIGTERM, wait, then SIGKILL the
survivors, so cooperative children can exit cleanly.
- Track adopted pids with a /proc starttime identity and add forget_pid, so the
shutdown sweep never signals a recycled pid.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Michael Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Daniel Han