Extracts the real guard block from install.sh and asserts: clean arm64 venv
untouched, x86_64 venv rebuilt as arm64, the x86_64-then-3.13.8 corner case,
arm64 3.13.8 downgrade preserved, --python skip, and Intel/Rosetta no-op.
Co-authored-by: Ramakrishna Bachu <ramankrishna10@gmail.com>
* Studio: pin the last pre-macOS-26 llama.cpp prebuilt instead of walking back
ggml-org moved their macOS build runner to macOS 26 (Tahoe) at b9428, so
b9428 and every newer upstream prebuilt is stamped minos 26 and fails to
dyld-load on macOS 14 / 15. #5883 handled this by walking back release by
release at install time.
Replace that with a deterministic pin: a host below macOS 26 selects b9415
directly (the last upstream build stamped below 26: arm64 minos 14, x64
minos 13.3), so it loads on macOS 13.3 / 14 / 15 / 26. Hosts on macOS 26+
and unknown-version hosts keep latest selection unchanged.
Only the ggml-org upstream path is pinned; the unslothai/llama.cpp fork
ships its own minos-13.3 prebuilts (#5893), so the pin is a no-op there and
goes dormant once macOS routes to the fork. The Mach-O minos preflight from
#5883 stays as a post-download backstop.
Refs #5883, #5893.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* docs: fork ships arm64 minos 14 / x64 minos 13.3, not uniform 13.3
The per-slice fork producer pins arm64 to 14.0 and x64 to 13.3; update the
pinned_macos_release_tag docstring to match. No behavior change.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Make the NVIDIA prebuilt selection version-dynamic and driver-gated on both the
Windows (ggml-org) and Linux (unslothai/llama.cpp) paths, so CUDA version bumps
are handled with no code change while staying safe across driver versions.
- Derive candidate CUDA runtime lines from the driver major; pick the highest
upstream-published minor the driver can actually run, so a sub-13.3 driver is
never handed an unguaranteed 13.3 build.
- Pin b9360's cuda-13.1 build (immutable, hash-verified) as a GPU fallback for a
Blackwell host on a 13.1/13.2 driver that the in-release 13.3 build gates off,
on both the simple and published install paths. Dormant for every other host
and self-disabling once upstream ships a driver-runnable build again.
- Seed the published-path ordering from the release's real published minors so a
future CUDA major is selectable with no code change.
Refs #5861, #5817, #5807, #5858, #5854, #5826, #5887.
Make macOS llama.cpp prebuilt selection host-OS-version aware: skip a prebuilt whose minimum-OS exceeds the host and walk back to the newest release that loads (macOS 26 keeps latest; 14/15 land on a compatible older release). Source-build fallback pins CMAKE_OSX_DEPLOYMENT_TARGET=13.3. CI: binary-load assertion plus a macos-14/15/26 install matrix. No change to Linux/Windows or CUDA selection.
Follow-up cleanups to the merged AMD ROCm support PR #5301:
1. De-duplicate the torchao Windows-ROCm import stub into a single shared
module (studio/backend/core/_torchao_stub.py); both workers call one
install_torchao_windows_rocm_stub() entrypoint.
2. Align the gfx name/arch comment columns in setup.sh and setup.ps1.
3. Isolate the float16 dtype fallback to AMD without native bf16; NVIDIA
keeps dtype=None so unsloth's own bf16/fp16/FORCE_FLOAT32 detection is
honored.
4. Hoist unconditional stdlib imports (gc, glob, re, subprocess, copy,
types, sys, importlib.metadata) from function bodies to module top
across the PR #5301-touched files; heavy/optional/relative imports stay
lazy.
5. bitsandbytes Windows-ROCm install now uses plain pip (force_pip=True)
instead of UV_SKIP_WHEEL_FILENAME_CHECK, per the AMD hackathon docs.
Also adds scripts/verify_import_hoist.py (a scope-aware LEGB AST resolver
that catches dangling-alias and rename-clash bugs in import-hoist
refactors) and wires it into the Lint CI source-lint job as a self-test
plus a pull_request compare gate.
* fix(studio): set HIP_VISIBLE_DEVICES in apply_gpu_ids for ROCm training workers
Training workers are spawned via multiprocessing spawn before detect_hardware()
runs, so IS_ROCM is still False. If the user never set HIP_VISIBLE_DEVICES in
their shell, _inherits_rocm_visibility is also False, leaving the worker with
only CUDA_VISIBLE_DEVICES set. On ROCm hosts the HIP runtime honors
HIP_VISIBLE_DEVICES over CUDA_VISIBLE_DEVICES, so the worker saw the full
device list and torch raised "no usable HIP accelerator" on some setups.
Fall back to probing torch.version.hip (a build-time attribute, safe to read
before GPU init) to detect ROCm when neither IS_ROCM nor inherited env vars
are available. Mirrors the existing fix in llama_cpp.py for llama-server
subprocess GPU pinning.
Fixes https://github.com/unslothai/unsloth/issues/5180
* test: tighten apply_gpu_ids ROCm fallback assertions
Replace loose OR chain with exact string matches, split into three
focused tests, and add a guard check for the try/except wrapper.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: detect ROCm unified memory (Strix Halo / AMD iGPU) via torch fallback
amd-smi on iGPUs with shared/unified memory (e.g. Radeon 8060S on Strix
Halo) reports only the dedicated VRAM slice (~512 MB) in its metric output,
so get_visible_gpu_utilization() was returning usable_gb ≈ 0.35 GB instead
of the full GTT pool (~128 GB). torch.cuda.mem_get_info() already surfaces
the correct unified-pool size.
Add _reconcile_rocm_unified_memory(): after amd-smi returns a valid result
on a ROCm device, cross-check each device's vram_total_gb against
torch.cuda.mem_get_info(). When torch reports a larger total, replace the
amd-smi VRAM fields in-place. No-op for discrete AMD GPUs where the two
sources agree.
Fixes: "Falling back to all visible GPUs -- model may not fit" on AMD iGPU
machines even when 100+ GB of unified memory is available.
* Apply unified-memory reconciliation in get_gpu_utilization too
The visible-GPU path was already corrected for AMD iGPUs with unified memory
(Strix Halo / Radeon 8060S), but get_gpu_utilization was still returning the
raw 512 MB amd-smi VRAM slice. Studio's /api/train/hardware endpoint and the
live GPU monitor read from this primary path, so users continued seeing the
wrong total even after auto_select_gpu_ids picked the right device.
Refactor to share the per-device correction:
* _apply_unified_memory_correction(metrics, torch_info) -- the actual
replacement logic, in-place on a single metrics dict.
* _reconcile_rocm_unified_memory(...) -- multi-device,
iterates utilization["devices"] (visible-GPU path).
* _reconcile_primary_rocm_unified_memory(...) -- single flat
metrics dict (primary-GPU path), uses parent_visible_spec to pick the
primary index, falls back to ordinal 0 when no visibility env is set.
get_gpu_utilization now calls the primary reconciler under IS_ROCM, so both
endpoints surface the real unified-memory pool on iGPUs while leaving
discrete AMD GPUs untouched (torch_total <= smi_total -> no replace).
* Use 'is not None' and log debug on torch.version.hip probe failures
Two small follow-ups to the apply_gpu_ids ROCm fallback:
1. Match detect_hardware()'s 'getattr(torch.version, "hip", None) is not None'
form so the entire codebase has one canonical 'this torch was built with
HIP' check. On every shipping torch wheel hip is either None or a non-empty
version string, so the new form agrees with the old bool() form on every
real install.
2. Log the probe failure at debug level instead of swallowing it silently.
The broad 'except Exception' is intentional (we never want apply_gpu_ids
to crash a worker over a probe), but the silent pass made it impossible
to tell whether the fallback was firing or being skipped.
* fix(studio): honour HIP_VISIBLE_DEVICES in _get_parent_visible_gpu_spec before IS_ROCM is set
When a user has HIP_VISIBLE_DEVICES set in their shell (e.g. "1" to select
GPU 1) but detect_hardware() has not yet run in the Studio parent process,
IS_ROCM is still False. _get_parent_visible_gpu_spec() was gated on IS_ROCM
so it fell through to CUDA_VISIBLE_DEVICES (unset), saw all physical GPUs,
and auto-selected index 0. apply_gpu_ids then overwrote HIP_VISIBLE_DEVICES
with "0", making the intended GPU invisible to ROCm torch in the worker,
which triggered the "no usable HIP accelerator" error (issue #5180).
Apply the same _inherits_rocm_visibility pattern already used in
apply_gpu_ids: check for HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES in the
environment regardless of IS_ROCM so the correct GPU index is preserved.
* fix(install): harden AMD ROCm GPU detection for multi-GPU and env-filtered setups
The previous rocminfo awk pattern could miss discrete GPUs on machines
where HIP_VISIBLE_DEVICES/ROCR_VISIBLE_DEVICES is used to mask an
integrated GPU — the env vars filter rocminfo output but may not
propagate into the install script subprocess, causing detection to
fail entirely.
Two changes:
- Tighten rocminfo pattern from /gfx[0-9]/ && !/gfx000/ to
/gfx[1-9][0-9]/ — simpler and correctly excludes the CPU agent
(gfx000) without a negative lookahead
- Add sysfs KFD topology fallback: reads
/sys/class/kfd/kfd/topology/nodes/*/gpu_id which is a kernel-level
view unaffected by HIP_VISIBLE_DEVICES or ROCR_VISIBLE_DEVICES
Fixes detection failure reported in Discord by Chains (gfx1201 + iGPU
machine where env var exclusion of the iGPU caused rocminfo to return
no usable device).
* Fix KFD sysfs awk fallback to read properties file
The fallback added by this PR reads /sys/class/kfd/kfd/topology/nodes/*/gpu_id
files but matches the literal token 'gpu_id' against their content. Those
files contain only a single decimal value (e.g. '0' for CPU agents, '50432'
for GPU agents), so the regex never matches and 'found' stays 0, making the
fallback a no-op on every host. The properties file in the same directory
contains key/value lines like 'gpu_id 50432' which is what the existing awk
pattern expects.
Reproduced with a synthetic sysfs layout: against gpu_id files awk exits 1;
against properties files awk exits 0 when any node reports gpu_id > 0.
* fix(setup.ps1): detect AMD ROCm GPU on Windows, bring to parity with setup.sh
setup.ps1 only checked nvidia-smi and fell straight to "gpu: none" on AMD
machines. setup.sh already probed rocminfo/amd-smi/hipconfig/hipinfo.
Add three-tier detection mirroring install_llama_prebuilt.py's detect_host():
1. hipinfo: gcnArchName in output confirms a real HIP GPU (not just SDK)
2. amd-smi list: "GPU: <digit>" data rows as fallback
3. WMI Win32_VideoController: last resort -- detects AMD GPU even without
HIP SDK, then guides user to install it rather than silently going CPU
Also corrects the "none" message to mention AMD ROCm alongside NVIDIA so
users with AMD hardware understand the requirement.
Fixes: rohit-style install where Strix Halo (Radeon 8060S) showed
"gpu: none" even with the HIP SDK present.
* fix(install.ps1): detect AMD ROCm GPU on Windows, bring to parity with setup.ps1
install.ps1 had the same nvidia-smi-only GPU detection as setup.ps1 before
the setup.ps1 fix. Applies the same three-tier AMD detection:
1. hipinfo: gcnArchName confirms real HIP GPU
2. amd-smi list: GPU data rows as fallback
3. WMI Win32_VideoController: detects AMD GPU without HIP SDK and guides
user to install it
Fixes: install.ps1 showing "gpu: none" while setup.ps1 correctly showed
"AMD GPU detected" on the same machine (reported by rohit, RX 7600 XT).
* fix(install.ps1): suppress 'No NVIDIA GPU detected' when AMD GPU is present
* feat: add Windows AMD ROCm PyTorch wheel installation
install_python_stack.py:
- Add _ROCM_WINDOWS_WHEEL_BASE and _ROCM_WINDOWS_RELEASES constants
pointing to AMD repo.radeon.com (ROCm 7.2 -> torch 2.9.1+rocm7.2.1)
- Extend _ensure_rocm_torch() with a Windows branch: detects ROCm via
_has_rocm_gpu() / _detect_rocm_version(), requires Python 3.12 (cp312
is the only ABI AMD publishes for Windows), installs the direct wheel
URL from repo.radeon.com
install.ps1:
- Capture ROCmVersion during AMD detection via hipconfig --version /
amd-smi version (needed for wheel URL selection)
- After Get-TorchIndexUrl, add an AMD wheel override block: when HasROCm
and Python 3.12 detected, set ROCmTorchWheelUrl to AMD wheel URL
- Expand torch install branch to handle ROCmTorchWheelUrl with
uv pip install --force-reinstall --no-cache-dir
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: also install torchvision and torchaudio from AMD Windows repo
AMD publishes matching torchvision-0.24.1+rocm7.2.1 and
torchaudio-2.9.1+rocm7.2.1 cp312 wheels at the same repo.radeon.com
release folder. Install all three in both install.ps1 and
install_python_stack.py Windows ROCm path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* feat: add ROCm 7.1.1 Windows wheel mapping
AMD uses a different version string for 7.1.1 wheels:
2.9.0+rocmsdk20251116 (date-tagged) instead of +rocm7.1.1.
Adds the 7.1.1 release folder to both install.ps1 and
install_python_stack.py so users with ROCm 7.1 get ROCm
torch instead of falling back to CPU.
* fix: install rocm_sdk_core and rocm_sdk_libraries_custom alongside torch
The AMD Windows torch wheels declare rocm[libraries]==<ver> as a hard
dependency. Without installing rocm_sdk_core and rocm_sdk_libraries_custom
from the same AMD release folder, uv cannot resolve the dependency and
fails with 'No solution found'. Include all 5 wheels in one install call.
* fix: expand ROCm wheel array to scalars for Invoke-InstallCommand
@array splatting inside a scriptblock only works when the native command
is prefixed with '&'. Invoke-InstallCommand uses '& $Command' to run the
block, so @ROCmAllWheelUrls was not being expanded. Extract to scalar
variables $rw0-$rw4 which are captured correctly by the closure.
* fix: use --no-deps for AMD Windows torch wheel install
uv's resolver looks up rocm[libraries]==0.1.dev0 on PyPI during
dependency resolution before downloading any wheels, and fails because
the package doesn't exist on PyPI. --no-deps skips resolution entirely
and installs all 5 AMD wheels directly. The GPU runtime dependency is
satisfied by the HIP SDK, not a Python package.
* fix: setup.ps1 and install_python_stack.py now install ROCm torch on Windows
setup.ps1 was always setting CuTag='cpu' for non-NVIDIA hosts and installing
cpu-only PyTorch, overwriting the ROCm torch installed by install.ps1.
Adds the same AMD wheel selection logic (ROCm version detection, Python 3.12
check, 5-wheel install with --no-deps) to setup.ps1's torch install block.
install_python_stack.py: remove IS_WINDOWS guard from _ensure_rocm_torch()
call site so the Windows path in _ensure_rocm_torch() is reachable during
'unsloth studio update' as well.
* fix: suppress manual-install warning when ROCm torch already present; fix progress counter
- Gate the 'must be installed manually' warning on torch.version.hip being empty
so it doesn't fire when our ROCm torch install succeeded
- Update _TOTAL counter to include the 3 ROCm steps on Windows now that
_ensure_rocm_torch() is called there (fixes 10/9 display)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* feat: add rocm step display in setup.ps1; fix warning and progress counter
- Add 'rocm' step after 'cuda' in setup.ps1 showing ROCm version or HIP SDK missing
- Move ROCm version detection up to GPU detection block so it's available early
- Suppress 'must be installed manually' warning when torch.version.hip is set
- Fix _TOTAL counter to include ROCm steps on Windows (fixes 10/9 display)
* fix: detect AMD SDK ROCm torch via __version__ when torch.version.hip is unset
AMD's repo.radeon.com wheels (e.g. 2.9.0+rocmsdk20251116) do not set
torch.version.hip, leaving it None. All three probes that relied solely on
torch.version.hip now also check for 'rocm' in torch.__version__.lower():
- hardware.py detect_hardware(): IS_ROCM was never set, causing the studio
to report 'Hardware detected: CPU' even after AMD wheels were installed
and HIP DLLs were on PATH.
- install_python_stack.py _ensure_rocm_torch(): skip-if-already-installed
probe would always reinstall on subsequent runs.
- install_python_stack.py Windows AMD warning: suppression check always
failed, so the 'must be installed manually' note kept appearing after
a successful AMD wheel install.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* perf: drop --no-cache-dir from AMD ROCm torch wheel installs
uv caches downloaded wheels by default; passing --no-cache-dir forced a
full redownload of the ~2 GB torch wheel on every install run. CUDA installs
never had this flag -- AMD was the only path affected.
* fix: use install-state flag instead of subprocess probe for AMD Windows warning
Replace the subprocess torch probe in the post-install warning block with a
module-level _rocm_windows_torch_installed flag set by _ensure_rocm_torch().
Subprocess re-import of torch is unnecessary and fragile -- the install
function already knows whether it succeeded.
* fix: hoist global declaration to top of _ensure_rocm_torch
Python requires the global statement to appear before any assignment
to the variable within a function. Moving it to the function top fixes
the SyntaxError on line 354.
* fix: pass AMD torch install status via env var to suppress false warning
setup.ps1 now sets UNSLOTH_ROCM_TORCH_INSTALLED=1 after a successful AMD
wheel install. install_python_stack.py reads this at the top of
_ensure_rocm_torch() to skip both the subprocess probe and the warning --
no re-import of torch needed, and the warning message now correctly says
'could not be auto-installed' rather than 'must be installed manually'.
* fix: register ROCm DLL directory before torch import on Windows
Python 3.8+ ignores PATH for extension DLL loading on Windows; amdhip64.dll
and other HIP runtime DLLs must be registered via os.add_dll_directory().
Without this, torch.cuda.is_available() always returns False on AMD ROCm
Windows even when HIP_PATH is correctly set in system environment variables.
Reads HIP_PATH / ROCM_PATH env vars first, then falls back to scanning
common ROCm install roots (C:\Program Files\AMD\ROCm, F:\ROCm, C:\ROCm).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: remove hardcoded non-standard ROCm paths from DLL directory scan
Only use HIP_PATH/ROCM_PATH (set by AMD installer) and the standard
C:\Program Files\AMD\ROCm\<version>\bin location. Custom drive paths
like F:\ROCm are user-specific and should not be hardcoded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: prevent torchao overrides step from overwriting AMD ROCm torch
torchao==0.14.0 in overrides.txt declares torch as a dependency. Without
--no-deps, uv resolves torch from PyPI and installs 2.11.0+cpu on top of
the AMD ROCm wheels (2.9.0+rocmsdk20251116). This was the root cause of
'Hardware detected: CPU' -- the AMD wheels were installed but then
immediately overwritten by the overrides step.
When _rocm_windows_torch_installed is True, add --no-deps to the overrides
pip_install call so torchao is installed without pulling in CPU torch.
* fix: add rocm_sdk namespace tarball to Windows ROCm wheel installs
torch/_rocm_init.py calls `import rocm_sdk` at startup, which requires
the rocm namespace tarball (rocm-*.tar.gz) in addition to the SDK wheel
packages. This tarball was missing from both install.ps1 and setup.ps1,
causing ModuleNotFoundError on first torch import.
- Add rocm-0.1.dev0.tar.gz to ROCm 7.1.1 install (provides rocm_sdk namespace)
- Add rocm-7.2.1.tar.gz + rocm_sdk_devel to ROCm 7.2.1 install
- Install tarball in a dedicated step before main SDK/torch wheels
- Switch to @array splatting in install.ps1 scriptblock for dynamic wheel count
- Remove --no-cache-dir from Python-side ROCm wheel install (prevents ~2GB redownload)
* feat: enable ROCm 7.2 torch install + warn on gfx1151 with ROCm < 7.2
Chigoma333 (AMD Radeon 8060S / gfx1151, Strix Halo) confirmed that ROCm
7.1 segfaults when tensors are moved to GPU, but ROCm 7.2 + torch
2.11.0+rocm7.2 works fully including training.
Changes:
- Uncomment (7,2): "rocm7.2" in _ROCM_TORCH_INDEX (was blocked by <2.11.0)
- Add _ROCM_TORCH_PKG_SPECS dict with per-tag version bounds:
rocm7.2 → torch>=2.11.0,<2.12.0; all older tags → <2.11.0
- Add _detect_amd_gfx_codes() helper that parses rocminfo output
- Warn on gfx1151/gfx1150 (Strix Halo) when ROCm < 7.2 is installed,
pointing users at the known segfault and recommending upgrade
- install.sh get_torch_index_url(): enable rocm7.2 case (previously capped
to rocm7.1), cap unknown future tags to rocm7.2
- install.sh: override TORCH_CONSTRAINT to >=2.11.0,<2.12.0 when rocm7.2
index is selected, so pip can actually resolve torch 2.11.0
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: prefer Python 3.12 for AMD ROCm users when 3.13 is also installed
After GPU detection, if ROCm HIP SDK is found and the selected Python
is not 3.12, run a second pass to locate a 3.12 install via py.exe and
PATH (catches uv-managed installs). Switch $DetectedPython to 3.12 so
the venv is created with a compatible interpreter for the cp312-only AMD
Windows torch wheels.
NVIDIA and Intel GPU paths are unaffected -- the re-detection block only
runs when $HasROCm is true.
Fixes: #5301
* fix: also check uv-managed Python 3.12 for AMD ROCm #5301
* fix: hide amd-smi console popups on Windows, guard torch.distributed.is_initialized for ROCm #5301
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: suppress remaining console popups on Windows, patch torch.distributed.is_initialized for ROCm #5301
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: stub all missing torch.distributed attrs for ROCm Windows wheel #5301
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: inject torch.distributed stub when C backend missing in ROCm Windows wheel #5301
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(rocm/windows): pre-stub torch._C._distributed_c10d + raise amd-smi timeout
Two fixes for Windows ROCm regressions reported by electroglyph on #5301:
1. worker.py — torch.distributed stub now fires unconditionally on Windows
The previous stub only injected sys.modules in the except branch, meaning
it was silently skipped when `import torch.distributed` happened to succeed
(the C backend is lazily resolved). The crash then hit later when
transformers/trl triggered the lazy load. Fix: on win32 we pre-populate
sys.modules['torch._C._distributed_c10d'] AND set the attribute on the
torch._C extension module *before* attempting the import, covering both
the early-ImportError and lazy-load failure modes.
2. amd.py — increase amd-smi timeout from 5 s to 30 s on Windows (10 s Linux)
amd-smi on Windows must cold-init the ROCm runtime on first invocation;
5 s was consistently too short, producing repeated 'Command timed out'
warnings in the server log. 30 s gives enough headroom without blocking
indefinitely on broken installs.
3. install.ps1 — widen Python 3.12 enforcement to ROCmGpuLabel (WMI-only path)
Users whose HIP SDK is not on PATH were detected via WMI but not switched
to Python 3.12 before the install started, causing a second pass. Guard
now fires on (HasROCm -or ROCmGpuLabel).
* fix(rocm): guard c10d stub, fix TorchIndexFamily for 7.1, clean dead code + comments
- worker.py: wrap c10d stub injection in `if _c10d_key not in sys.modules` so
Windows NVIDIA users with a real torch.distributed are never affected
- install.ps1: fix Get-TauriTorchIndexFamily receiving hardcoded "rocm7.2"
even when ROCm 7.1 wheels are installed; now branches on $ROCmVersion
- main.py: remove dead `import ctypes as _ctypes` (ctypes is never called)
- hardware.py, install_python_stack.py, worker.py, install.ps1: shorten
verbose multi-line comment blocks throughout
- tests: update 4 stale assertions that expected rocm7.2 to be absent/capped
* fix(tests): match windows AMD warning assertion to actual source string
* chore: trim verbose comment blocks across all ROCm-related files
* fix: guard reconcile call against None numeric_ids; add torchvision lower bounds
* fix(install.ps1): recreate venv with Python 3.12 after ROCm switch
Venv was created with 3.13 before GPU detection ran; switching
$DetectedPython to 3.12 had no effect since $VenvPython still
pointed to the 3.13 interpreter inside the already-created venv.
* ux: detect AMD GPU before Python selection to avoid double venv creation
- Early hipinfo + WMI probe runs before Find-CompatiblePython so Python
3.12 is selected upfront when AMD is detected; venv is now created
exactly once instead of 3.13 then immediately 3.12.
- Post-venv recreation block replaced with a simple warning for the rare
case where AMD was missed by the early probe.
- setup.ps1: show venv's actual Python version (e.g. 3.12) instead of
the system Python found by the pre-activation search (was showing 3.13).
* fix(rocm/win): auto-stub all _distributed_c10d symbols via PEP-562 __getattr__
The bare ModuleType stub caused ImportError when torch._dynamo was imported
(triggered by trainer.py accessing torch._dynamo.config at load time).
torch._dynamo pulls in torch.distributed.fsdp._flat_param which does:
from torch._C._distributed_c10d import FakeProcessGroup
and potentially other symbols. Adding module __getattr__ auto-creates a
stub class for any missing symbol so all such imports succeed without
enumerating every individual symbol. Applied to both the primary stub
and the fallback stub in the except branch.
* chore: trim c10d stub comment
* fix(rocm/win): auto-stub missing torch.distributed attrs (Store, ProcessGroup, …)
* fix(rocm/win): pre-stub fsdp submodules in sys.modules; fix __getattr__ subpackage clash
* feat(rocm/win): arch-aware wheel selector always picks newest ROCm release
Replace HIP-SDK-version-gated wheel selection with GPU arch-based logic.
Select-ROCmWheelRelease (PS) and _select_windows_rocm_release (Python) map
gcnArchName → minimum ROCm version, then pick the newest available release
that satisfies it (currently always rocm-rel-7.2.1 for any supported GPU).
Wheels bundle their own ROCm runtime so the installed HIP SDK 7.1 does not
prevent using 7.2.1 wheels on gfx1200 (RX 9060 XT) and similar RDNA 4 GPUs.
Also installs the bitsandbytes Windows ROCm continuous-release wheel and sets
BNB_ROCM_VERSION=72 in worker.py before ML imports so bnb loads the
libbitsandbytes_rocm72.dll that ships in that wheel.
* fix(rocm/win): stub class metaclass for ProcessGroup.BackendType; amd-smi circuit breaker
torchao.float8.inference accesses ProcessGroup.BackendType as a class-level
attribute. Plain type() stubs have no __getattr__ on the metaclass so this
raises AttributeError. Introduce _StubClassMeta whose __getattr__ returns
child stub classes, fixing the torchao import chain.
Add an amd-smi circuit breaker in amd.py: after 3 consecutive failures the
module stops spawning the process, eliminating the repeated Windows UAC /
DiskPart elevation prompts caused by polling a non-functional amd-smi.
Also guard BNB_ROCM_VERSION=72 behind a DLL existence check so bitsandbytes
fails with its own detection message rather than a harder "DLL not found" when
the Windows ROCm bnb wheel is not yet installed.
* fix: stub __members__ so torchao float8 enum check doesn't crash on ROCm Windows
torchao.float8.inference accesses ProcessGroup.BackendType.__members__
expecting a Python Enum registry dict. _StubClassMeta.__getattr__ was
blocking all dunder attributes, causing AttributeError. Return {} for
__members__ specifically so the isinstance/iteration checks pass cleanly.
* fix: stub distributed tensor/functional_collectives to prevent missing C++ op crash on ROCm Windows
torch._dynamo.trace_rules eagerly loads torch.distributed.tensor at import
time, which pulls in _functional_collectives.py. That file registers Meta
kernels for _c10d_functional C++ ops, but those ops are only registered
by torch._C._distributed_c10d — a C extension absent from ROCm Windows
wheels. Pre-stubbing the affected modules in sys.modules prevents the real
import chain from running and avoids the "operator does not exist" crash.
* fix: give mod stubs __path__ and pre-stub _tensor to fix 'not a package' import error
_make_mod_stub now sets __path__=[] so Python treats stub modules as
packages. Without it, any import of a submodule raises "is not a package".
Also pre-stub torch.distributed._tensor and its submodules so that
_tensor/__init__.py (which re-exports from torch.distributed.tensor) never
runs and torchao's `from torch.distributed._tensor import DTensor` gets a
harmless stub instead of crashing.
* fix: stub torch.ops._c10d_functional namespace with hashable op sentinels
torchao.dtypes.nf4tensor uses _c10d_functional ops as dict keys at import
time (all_gather_into_tensor.default, wait_tensor.default) and
torch.ops.c10d.scatter_.default. None of these ops are registered on ROCm
Windows because torch._C._distributed_c10d (the C extension) doesn't ship.
Replace the whole _c10d_functional namespace with a custom stub whose ops
return hashable .default objects, so dict-key construction doesn't crash.
Also inject a scatter_ stub into torch.ops.c10d if it's missing.
* fix: stub entire torchao package on ROCm Windows instead of individual ops
torchao is not supported on ROCm Windows and its import chain transitively
requires torch._C._distributed_c10d (absent from the ROCm Windows wheel).
Rather than stub each missing op one by one, stub the whole torchao package
upfront. Unsloth uses bitsandbytes for quantization, not torchao, so this
has no functional impact. transformers gracefully handles an importable-but-
empty torchao by disabling TorchAoHfQuantizer.
* fix: set __spec__ on mod stubs so importlib.util.find_spec doesn't raise
Manually-injected sys.modules entries have __spec__=None by default.
importlib.util.find_spec() raises ValueError when it finds a module in
sys.modules with __spec__=None (transformers.utils.import_utils hits this
when checking if torchao is available). Give every stub a minimal
ModuleSpec(name, loader=None, is_package=True) to satisfy find_spec.
* fix: add meta path finder to auto-stub subpackages of stub modules
`import torchao.prototype` goes through the import machinery, not
__getattr__, so an empty __path__ means ModuleNotFoundError. Rather than
list every submodule explicitly, register a MetaPathFinder that intercepts
any import whose parent is one of our stubs (detected by loader=None in the
parent's ModuleSpec). Real installed packages always have a SourceFileLoader
so they are never intercepted. Also register child stubs in sys.modules
from __getattr__ as a belt-and-suspenders measure.
* fix: use _unsloth_stub sentinel instead of loader=None for stub detection
The import machinery overwrites module.__spec__ with the spec returned by
find_spec (which has loader=_StubSubpackageLoader, not None), so the
loader=None check broke for second-level subpackages. Switch to a custom
_unsloth_stub object identity sentinel set directly on each stub module --
it survives __spec__ being replaced and correctly identifies stubs at any
depth (torchao.prototype.safetensors, etc.).
* refactor(rocm/win): switch to repo.amd.com arch-aware index, remove stubs
AMD recommends repo.amd.com/rocm/whl/{arch}/ as the Windows ROCm wheel
source. These wheels bundle their own ROCm runtime, support all Python
versions (not just cp312), and include the full torch._C extension set
(including _distributed_c10d) that the old repo.radeon.com wheel omitted.
Changes:
- install.ps1: remove Select-ROCmWheelRelease + hardcoded cp312 wheel
URLs; remove Python 3.12 forced-preference logic; install via
--index-url repo.amd.com/rocm/whl/{arch-family}/
- studio/setup.ps1: same -- remove Select-ROCmWheelRelease, switch to
repo.amd.com arch-aware index URL
- studio/install_python_stack.py: replace _ROCM_WINDOWS_RELEASES /
_select_windows_rocm_release with _windows_rocm_index_url() using the
_GFX_TO_AMD_INDEX_ARCH map; drop Python 3.12 restriction
- studio/backend/core/training/worker.py: remove all stub machinery
(_make_mod_stub, _StubSubpackageFinder, _StubSubpackageLoader,
_StubClassMeta, torchao/fsdp/dtensor stubs, _c10d_functional ops
stubs, BNB DLL detection) -- no longer needed with new wheel source
* fix(rocm/win): restore _distributed_c10d + torchao stubs; fix BNB install
repo.amd.com torch wheels also omit torch._C._distributed_c10d on Windows
(RCCL is not shipped on Windows). torch/distributed/__init__.py imports
from it unconditionally at module level, so the stub must land in
sys.modules before any torch.distributed import.
torchao (pulled in by transformers.quantizers) walks
torchao.float8.distributed_utils -> torch.distributed._functional_collectives
-> distributed_c10d at import time. Stubbing torchao up-front short-circuits
that chain.
worker.py:
- Restore _make_mod_stub / _StubSubpackageFinder / _StubSubpackageLoader
- Restore _StubClassMeta for ProcessGroup.BackendType attribute access
- Restore _distributed_c10d stub with __getattr__ (Windows only)
- Restore torchao stubs (5 modules, Windows only)
install_python_stack.py:
- BNB AMD wheel install was inside the early-return branch that fires when
torch is already a ROCm build (installed by install.ps1). Move BNB install
outside that branch so it always runs on Windows ROCm — the PyPI
bitsandbytes has only CUDA DLLs and fails to load on ROCm.
* worker: remove _distributed_c10d stub; stub only torchao
The installed torch/distributed/__init__.py from repo.amd.com
(torch==2.10.0+rocm7.12.0) is now properly guarded with
`if is_available():`, so `import torch.distributed` alone is safe.
The crash only comes via torchao's import chain:
torchao.float8.distributed_utils
→ torch.distributed._functional_collectives (unguarded import)
→ torch.distributed.distributed_c10d
→ torch._C._distributed_c10d ← absent on Windows ROCm
Stubbing torchao short-circuits the chain entirely. No need to stub
_distributed_c10d. Remove _StubClassMeta and the _c10d stub block;
keep only _make_mod_stub + _StubSubpackageFinder + torchao seeds.
* fix: BNB AMD wheel skipped + torch.compile segfault on Windows ROCm
install_python_stack.py: the UNSLOTH_ROCM_TORCH_INSTALLED=1 early-return
path (set by setup.ps1 when it installed torch itself) returned before
ever reaching the AMD BNB prerelease wheel install. The PyPI
bitsandbytes==0.49.x ships only CUDA DLLs, so loading it on ROCm fails
with "libbitsandbytes_rocm72.dll not found". Now installs the AMD
Windows BNB wheel before returning on that path too.
worker.py: torch._grouped_mm crashes on gfx1200 (null HIP kernel pointer,
0xC0000005) when torch.compile's JitDecomp system dispatches it during
the first forward pass. Detect Windows ROCm via torch.version.hip
(already in sys.modules from section 1e) and set TORCHDYNAMO_DISABLE=1
to bypass the broken kernel dispatch.
* fix: BNB AMD wheel install fails uv wheel filename check
The bitsandbytes continuous-release wheel is intentionally mismatched:
filename encodes 1.33.7.preview (= 1.33.7rc0 in PEP 440) but wheel
metadata reports 0.50.0.dev0. uv rejects this by default.
Introduce _install_bnb_windows_rocm() helper that sets
UV_SKIP_WHEEL_FILENAME_CHECK=1 only for this specific install, then
restores the previous env value. Both BNB install call sites (the
UNSLOTH_ROCM_TORCH_INSTALLED early-return path and the normal Windows
ROCm path) now use this helper.
* worker: patch _grouped_mm CUDA dispatch on Windows ROCm (gfx1200 null kernel)
TORCHDYNAMO_DISABLE=1 stopped the compiler frontend but not the autograd
JitDecomp system, which also dispatches _grouped_mm and hits the same
null HIP kernel crash (0xC0000005).
Verified that torch.library.Library("aten","IMPL").impl("_grouped_mm", fn,
"CUDA") successfully overrides the broken HIP kernel with a Python mm
fallback on torch==2.10.0+rocm7.12.0.
Schema: _grouped_mm(Tensor self, Tensor mat2, Tensor? offs=None,
Tensor? bias=None, ScalarType? out_dtype=None) -> Tensor
The fallback handles both the simple case (offs=None → torch.mm) and the
grouped case (offs provided → split self by offsets, multiply each group
against the corresponding slice of mat2, then cat results).
Keep _WINDOWS_ROCM_GROUPED_MM_LIB alive at function scope to prevent the
C++ dispatch registration from being freed by GC.
* worker: fix torchao stub — return stub classes not modules for isinstance()
peft/tuners/lora/torchao.py does:
from torchao.dtypes import AffineQuantizedTensor, LinearActivationQuantizedTensor
isinstance(weight, (AffineQuantizedTensor, LinearActivationQuantizedTensor))
The stub __getattr__ was returning stub modules, which isinstance() rejects
with "arg 2 must be a type, a tuple of types, or a union".
Add _StubTypeMeta metaclass whose __instancecheck__ always returns False,
and _make_stub_type() to create stub classes via it. Change _make_mod_stub
__getattr__ to return stub classes instead of stub modules for leaf
attribute access, so isinstance() gets a valid type and returns False.
_StubSubpackageFinder still handles import-style subpackage creation
(those still need module objects in sys.modules); __getattr__ only fires
for from-import or direct attribute access, which are the isinstance paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests: add coverage for Windows ROCm install paths and worker patches
Add conftest.py to fix pre-existing sys.path issue that prevented
test_rocm_support.py from running at all (install_python_stack.py
imports from backend.utils.wheel_utils which needs studio/ on sys.path).
New test classes cover everything added in this session:
- TestWindowsRocmIndexUrl: arch → AMD pip index URL mapping (gfx120X-all,
gfx1151, gfx1150, gfx110X-all, unknown → None, trailing slash)
- TestDetectWindowsGfxArch: hipinfo output parsing, missing/timeout/bad
returncode/no-gcnArchName paths
- TestInstallBnbWindowsRocm: UV_SKIP_WHEEL_FILENAME_CHECK set+restored,
env restored on exception, no-op when URL missing
- TestRocmTorchInstalledEnvVar: UNSLOTH_ROCM_TORCH_INSTALLED=1 skips
pip_install, calls _install_bnb_windows_rocm, sets flag
- TestWorkerWindowsRocmPatches: _grouped_mm CUDA dispatch override,
offs/grouped variant handling, GC-prevention sentinel,
_StubTypeMeta __instancecheck__, _StubSubpackageFinder registration,
torchao key submodule pre-stubbing, TORCHDYNAMO_DISABLE guard
- TestRocmTorchPkgSpecs: rocm7.2 torch 2.11.x spec, default <2.11 cap,
3-tuple shape, _GFX_TO_AMD_INDEX_ARCH RDNA4/3.5/3 coverage
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests: fix encoding, IS_WINDOWS patching, and wrong assertion
- Add encoding="utf-8" to all read_text() calls (54 occurrences) so
tests pass on Windows where the default codec is cp1252 and source
files contain UTF-8 emoji (e.g. ⚠️ in install_python_stack.py)
- Add @patch.object(stack_mod, "IS_WINDOWS", False) to Linux-path
TestEnsureRocmTorch tests so they reach the Linux code path when run
on a Windows machine instead of short-circuiting into the Windows branch
- Fix test_grouped_mm_patch_guarded_by_windows_and_hip_check: the source
uses getattr(_torch_for_rocm, "version", None) not torch.version, so
check for '"version"' and '"hip"' substrings instead
137 passed, 2 skipped
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: pin BNB_ROCM_VERSION=72 for torch==2.11.0+rocm7.13.0 compatibility
AMD's pip index now ships torch==2.11.0+rocm7.13.0 (ROCm 7.13).
bitsandbytes auto-detects HIP 7.13 from torch.version.hip and looks for
libbitsandbytes_rocm713.dll, which the AMD Windows prerelease wheel does
not ship (it only ships rocm72.dll), causing a load error at training start.
Fix:
- worker.py section 1f: set BNB_ROCM_VERSION=72 (via setdefault) before
section 2 ML imports, so bitsandbytes always loads rocm72.dll on Windows ROCm
- install_python_stack.py: set BNB_ROCM_VERSION=72 in _install_bnb_windows_rocm()
for any post-install imports; update comment to document root cause
- tests: 4 new assertions covering the fix (141 passed, 2 skipped)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: detect BNB ROCm DLL suffix dynamically instead of hardcoding '72'
BNB_ROCM_VERSION was pinned to '72' which works today (AMD wheel ships
rocm72.dll) but would break again if AMD ships a future wheel with a
different DLL suffix (e.g. rocm713.dll).
Add _detect_bnb_rocm_dll_ver() to install_python_stack.py: scans the
installed bitsandbytes package dir for libbitsandbytes_rocm{VER}.dll
using importlib.util.find_spec (no BNB import needed) and returns the
suffix. '72' remains the fallback when detection fails.
Apply the same detection inline in worker.py section 1f. Both paths
still respect a pre-set BNB_ROCM_VERSION (caller override wins).
Tests: +8 cases covering detection logic and fallback (147 passed, 2 skipped).
* fix: patch torch.distributed stubs in server process for Windows ROCm
On Windows ROCm, torch.distributed ships without process-group helpers
(is_initialized, is_available, get_rank, get_world_size). The worker
subprocess already patches these in section 1e, but the main server
process calls _determine_attention_impl_for_gpu_estimate() which calls
unsloth's resolve_attention_implementation() → is_initialized(), causing:
"Could not resolve attention implementation for '...':
module 'torch.distributed' has no attribute 'is_initialized'"
Fix: patch the missing attrs onto torch.distributed at the top of
_determine_attention_impl_for_gpu_estimate, matching the same stubs
already applied in worker.py section 1e. No-ops on Linux/CUDA where
torch.distributed is fully populated.
* fix: gate _grouped_mm dispatch patch on HIP < 7.13
AMD fixed the gfx1200 null HIP kernel in ROCm 7.13 (torch 2.11+).
Users on the new wheel now get the real GPU _grouped_mm kernel for
MoE workloads instead of the Python mm fallback.
Changes:
- worker.py: add _hip_ver_at_least() helper; wrap full _grouped_mm
patch in `if not _hip_ver_at_least(7, 13):` with else branch that
logs the skip reason; update section-1f comment to document the fix
- test_rocm_support.py: add 5 tests covering the helper definition,
the (7, 13) gate expression, the else branch, the skip log message,
and the AMD-format version string parsing (.split(".")[:2])
Verified: torch==2.11.0+rocm7.13.0 — 3D batch and grouped (offs)
variants both succeed; null crash only present on rocm7.12 and earlier.
* fix: stub is_torchelastic_launched on torch.distributed for Windows ROCm
resolve_attention_implementation calls is_torchelastic_launched() which
does not exist in the incomplete torch.distributed shipped with the
Windows ROCm wheel, causing a warning on every model config load in the
server process. Add it to the stub table alongside the four helpers
already patched in _determine_attention_impl_for_gpu_estimate.
Also adds two tests: one confirming the new stub and one confirming all
five core distributed helpers are covered.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: explicit warnings on AMD ROCm arch/version fallbacks + Fast-Install arg order
setup.ps1:
- Fix Fast-Install argument order: packages before flags, consistent with
all other Fast-Install calls in the file
(was: Fast-Install --force-reinstall --index-url $url torch ...)
(now: Fast-Install torch torchvision torchaudio --force-reinstall --index-url $url)
- Add explicit [WARN] substep when $HasROCm is true but arch mapping fails:
- GPU arch detected but not in supported wheel list → names the arch and
lists supported families so user knows exactly what to report
- HIP SDK present (amd-smi path) but gcnArchName unreadable → instructs
user to re-install the HIP SDK; previously fell back silently to CPU
install.sh:
- Add [WARN] to stderr before silent CPU fallback when AMD GPU is confirmed
(rocminfo/amd-smi) but ROCm version cannot be read from any source
(amd-smi, /opt/rocm/.info/version, hipconfig, dpkg, rpm)
- Add [WARN] to stderr when ROCm version is too old (< 6.0) with upgrade link
install.ps1 and setup.sh: no changes needed (already handle these paths correctly)
* fix: robust gfx arch detection for Strix Halo / HIP-runtime-only installs
Covers users who have the HIP runtime (amd-smi available) but not the
full HIP SDK (no hipinfo), which is common on Strix Halo iGPU systems.
Without this, $ROCmGfxArch stays null and the installer silently falls
back to CPU-only PyTorch despite a working GPU.
Detection waterfall (setup.ps1 + install.ps1):
1. hipinfo gcnArchName -- full HIP SDK (existing, unchanged)
2. amd-smi list gfx pattern -- newer amd-smi versions embed arch
3. amd-smi static --asic -- ROCm 6+ ASIC details with GFX target
4. UNSLOTH_ROCM_GFX_ARCH env -- manual override escape hatch
5. GPU name → arch table -- best-effort from marketing name:
890M / Strix Halo → gfx1151 (RDNA 3.5 iGPU, Strix Halo)
880M / Strix Point → gfx1150 (RDNA 3.5 iGPU, Strix Point)
780M / Phoenix → gfx1103 (RDNA 3 iGPU)
RX 7900/7800/7700 → gfx1100 (RDNA 3 desktop)
RX 9070 XT / 9080 → gfx1201 (RDNA 4)
RX 9070 / 9060 XT → gfx1200 (RDNA 4)
When arch is inferred from name, a Cyan substep tells the user to set
UNSLOTH_ROCM_GFX_ARCH to skip inference on future installs.
WMI block intentionally does not set $HasROCm (no runtime confirmation).
Tests: 11 new tests in TestStrixHaloGfxArchDetection covering all five
detection levels, WMI safety, and gfx regex in both ps1 files.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: resolve hipinfo/hipconfig via HIP_PATH/ROCM_PATH when not on PATH
AMD HIP SDK sets HIP_PATH on Windows but does not always add the bin
directory to PATH. Get-Command hipinfo therefore silently fails and
detection falls through to WMI, which cannot provide a gfx arch, leaving
the user with a CPU-only PyTorch install and no warning.
Changes:
- setup.ps1 / install.ps1: before falling through to amd-smi, attempt to
locate hipinfo.exe and hipconfig.exe under $env:HIP_PATH\bin (then
$env:ROCM_PATH\bin) when Get-Command returns nothing
- Emit a [WARN] with the resolved path and a one-liner to permanently fix
PATH via SetEnvironmentVariable
- Emit a [WARN] when HIP_PATH/ROCM_PATH is set but the exe is still not
found (incomplete SDK install)
- Emit a [WARN] with the first hipinfo output line when hipinfo runs but
returns a non-zero exit code (e.g. "no ROCm-capable device detected")
- 18 new tests in TestHipSdkEnvPathResolution; total 183 passed, 2 skipped
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* feat: print HIP SDK path and full hipconfig version in terminal on AMD detection
Both install.ps1 and setup.ps1 now emit substeps under the gpu step when
AMD ROCm is detected:
gpu AMD ROCm (gfx1200)
HIP SDK: C:\Program Files\AMD\ROCm\7.1
hipconfig: 7.1.51803-d3a86bd04
Previously only the gpu label (e.g. "AMD ROCm (gfx1200)") was shown with
no indication of where the SDK was found or which exact build was active.
The full hipconfig build string (e.g. 7.1.51803-d3a86bd04 instead of just
7.1) is now stored in ROCmVersionFull and also used in setup.ps1's
'rocm' step label.
9 new tests in TestHipSdkDetectedSubstep; total 192 passed, 2 skipped
* fix: Strix rocm7.1 segfault bypass + Ubuntu 24.04 HIP gcc-install-dir
Issue 1 (install.sh): gfx1151/gfx1150 + ROCm 7.1 causes a segfault in
torch._grouped_mm (moe_utils.py:167). The Radeon repo now ships cp313
wheels for rocm-rel-7.1, so _amd_gpu_radeon=true silently lands on the
broken combo. When Strix Halo/Point is detected and TORCH_INDEX_URL is
rocm7.1, override to rocm7.2 PyTorch index, update TORCH_CONSTRAINT, and
set _amd_gpu_radeon=false to bypass the Radeon repo entirely. Emits a
clear [WARN] explaining the segfault and linking to the ROCm upgrade docs.
Issue 2 (setup.sh): ROCm 7.x ships clang-20 which on Ubuntu 24.04+ picks
/usr/lib/gcc/x86_64-linux-gnu/14/ (runtime dir, no C++ headers), causing
'cstdlib file not found' and a failed llama.cpp HIP build. Iterate gcc
versions 14→11 to find the first install dir that has both runtime and
/usr/include/c++/<ver> headers, then pass --gcc-install-dir to clang via
CMAKE_HIP_FLAGS. Fix confirmed by h34v3nzc0dex (llama.cpp 417/417 clean).
11 new tests across TestStrixRocm71Override and TestSetupShGccInstallDir;
total 203 passed, 2 skipped
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: BNB_ROCM_VERSION in server process + torch._C._distributed_c10d stubs
Two errors visible in training logs on Windows ROCm:
1. Server process bitsandbytes crash:
"Configured ROCm binary not found at libbitsandbytes_rocm713.dll"
The installed BNB wheel ships rocm72.dll (not rocm713.dll). The
training worker already sets BNB_ROCM_VERSION=72 via DLL detection
but the server process (main.py) imported bitsandbytes before that
ran. Fix: add the same DLL-scan + BNB_ROCM_VERSION assignment to
main.py inside the existing win32 guard, before any downstream
import can pull in bitsandbytes.
2. torch.distributed import failure:
"No module named 'torch._C._distributed_c10d'; torch._C is not a package"
torch._C is a C extension on Windows ROCm — Python cannot do
submodule imports from it, so torch.distributed fails to import
before our attribute stubs could ever run. Fix: inject empty
ModuleType stubs for _distributed_c10d, _distributed_autograd and
_distributed_rpc into sys.modules inside the win32 guard in
hardware.py BEFORE importing torch.distributed, so the import
succeeds and our attribute stubs take effect.
9 new tests in TestServerStartupRocmFixes; total 212 passed, 2 skipped
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(win32): populate distributed c10d stub with dummy symbols
torch.distributed tries to `from torch._C._distributed_c10d import
FakeProcessGroup` (and ProcessGroup, Work, Store, etc.). The previous
empty ModuleType stub caused an AttributeError on those names.
Populate every stub with a _Dummy class for each known symbol so the
import chain completes silently on Windows ROCm where torch._C is a
compiled extension and its _distributed_c10d submodule doesn't exist.
Adds four new tests in TestServerStartupRocmFixes covering FakeProcessGroup,
ProcessGroup, setattr population, and all three _distributed_* siblings.
* fix(win32): distinguish HIP SDK installed vs GPU not ROCm-accessible
Previously, when hipinfo was found but exited non-zero (e.g. "no
ROCm-capable device detected"), both install.ps1 and setup.ps1 fell
through to the WMI-label-only branch and printed "AMD GPU detected --
HIP SDK not found" -- factually wrong since the SDK binary is present.
Add $HipSdkInstalled flag (set true when hipinfo binary is found,
regardless of exit code). When HipSdkInstalled && !HasROCm:
- Show "AMD GPU detected -- not ROCm-accessible (HIP <ver>)" instead
- Explain this is a driver issue, not an SDK issue, with a link
- Still run hipconfig version capture so version shows in output
- CPU-only hint now says "GPU not ROCm-accessible" not "require HIP SDK"
Also applies to setup.ps1 (same detection block, same branches).
Adds TestHipSdkInstalledButDeviceInaccessible (11 tests).
* fix(win32): scope ROCm workarounds to AMD hosts only
Three Codex-flagged issues where Windows ROCm workarounds incorrectly
applied to Windows CUDA (NVIDIA) machines:
main.py (P1): BNB_ROCM_VERSION was set unconditionally on all win32
hosts. On NVIDIA, bitsandbytes sees BNB_ROCM_VERSION and looks for a
ROCm DLL that doesn't exist, breaking bitsandbytes initialisation.
Fix: gate the block on HIP_PATH/ROCM_PATH being present (ROCm hosts only).
worker.py (P2): torchao stubs were seeded for all win32 runs, shadowing
real torchao on Windows CUDA and silently disabling torchao quantization
for NVIDIA users. Fix: gate on HIP_PATH/ROCM_PATH (win32 ROCm only).
install_python_stack.py (P1): _detect_windows_gfx_arch() only checked
shutil.which("hipinfo"), skipping the HIP_PATH/ROCM_PATH fallback that
the PowerShell installers use. On installs where the HIP SDK bin dir is
not on PATH, _ensure_rocm_torch() returned early without installing
ROCm wheels or bitsandbytes. Fix: mirror the env-var fallback.
* fix(linux): route Strix + ROCm 7.1 to AMD arch-specific index
Instead of falling back to pytorch.org/rocm7.2, the Strix override now
routes to repo.amd.com/rocm/whl/gfx1151/ (or gfx1150/) which serves
torch 2.11.0+rocm7.13.0 -- AMD's build containing the actual _grouped_mm
kernel fix, verified on real gfx1151 hardware by h34v3nzc0dex.
This exercises the real GPU kernel path rather than the rocm7.2 workaround.
UNSLOTH_AMD_ROCM_MIRROR can override the base URL for air-gapped installs.
Also teaches _tauri_torch_index_family to recognise AMD arch-specific URLs
(repo.amd.com/rocm/whl/gfx*) and return the rocm7.13 family label so
_tauri_gpu_branch correctly classifies these installs as rocm.
Suggested by h34v3nzc0dex based on hardware-verified probe results.
* fix(studio/rocm): gate ROCm-only side-effects on active torch runtime
Address five edge cases flagged during PR review:
1. studio/backend/main.py: BNB_ROCM_VERSION was set whenever HIP_PATH or
ROCM_PATH was present in the environment. A Windows CUDA user who once
installed the HIP SDK and reverted to a CUDA torch wheel still has those
env vars set, so bitsandbytes would try to load libbitsandbytes_rocm72.dll
against a CUDA torch and crash. Now probe torch.version.hip inside the
env-var guard (worker.py already does this).
2. studio/backend/main.py: os.add_dll_directory returned handles were
discarded. Per CPython docs, the directory leaves the DLL search list when
the handle is garbage collected. Retain handles in module-level
_ROCM_DLL_HANDLES list so they survive process lifetime.
3. studio/install_python_stack.py: _install_bnb_windows_rocm() returned None
regardless of pip_install_try outcome, and the caller flipped
_rocm_windows_torch_installed to True unconditionally. On a failed BNB
install the post-install "manual install may be required" warning was
suppressed and the user was misled. Helper now returns bool; caller gates
on it.
4. studio/install_python_stack.py: _detect_windows_gfx_arch returned the raw
capture group, so mixed-case hipinfo output ("Gfx1151") missed the
lowercase keys in _GFX_TO_AMD_INDEX_ARCH and silently fell back to CPU
torch. Lowercase the token.
5. studio/install_python_stack.py: UNSLOTH_ROCM_TORCH_INSTALLED=1 early-
return trusted the env var even when the venv was wiped between runs.
Subprocess-probe torch importability first; fall through to the full
install path if the probe fails.
Tests: 231 passed, 1 skipped in tests/studio/install/test_rocm_support.py
(adds one new test for case 5 fall-through).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/rocm): worker.py parity + don't roll back ROCm torch on bnb failure
Addresses findings from a 10x reviewer pass on the prior fix commit:
1. studio/backend/core/training/worker.py (parity with main.py):
- Gate the torchao stub block on torch.version.hip / 'rocm' in
torch.__version__ instead of HIP_PATH / ROCM_PATH env-var presence.
Same root cause as main.py: HIP SDK env vars stick around on CUDA hosts.
- Add module-level Windows ROCm DLL registration block. Worker subprocesses
inherit env vars but not the parent's add_dll_directory handles, so the
first `import torch` in the worker could fail to find amdhip64.dll when
HIP_PATH\bin is not on PATH. Mirrors main.py setup. Handles retained at
module scope via _ROCM_DLL_HANDLES.
- Promote _WINDOWS_ROCM_GROUPED_MM_LIB to module scope with `global` in
run_training_process so the torch.library.Library registration survives
past function return / mid-run garbage collection.
- Harden _torch_has_hip() to also accept 'rocm' in torch.__version__
(AMD SDK / Radeon wheels may not set torch.version.hip).
2. studio/install_python_stack.py:
- Don't roll back ROCm torch when bitsandbytes install fails. The prior
commit gated _rocm_windows_torch_installed on _install_bnb_windows_rocm()
returning True; if torch installed successfully but bnb failed, the flag
stayed False and later install steps could overwrite ROCm torch with the
generic CPU torch wheel. Set the flag after torch install; surface bnb
failure as a separate warning instead.
- _detect_windows_gfx_arch now probes in three tiers: UNSLOTH_ROCM_GFX_ARCH
env-var override (matches the PowerShell installer), then hipinfo (PATH
or HIP_PATH\bin), then amd-smi (`static --asic`, `list`). Without the
amd-smi fallback, runtime-only Radeon installs without hipinfo on PATH
made `studio update` return early and leave the venv on CPU torch.
- Linux torch-already-rocm probe in _ensure_rocm_torch now matches the
Windows probe shape: accepts torch.version.hip OR 'rocm' in
torch.__version__ to cover AMD SDK / Radeon Linux wheels.
3. studio/backend/utils/hardware/hardware.py:
- apply_gpu_ids() final-fallback torch probe accepts 'rocm' in
torch.__version__ in addition to torch.version.hip, matching
detect_hardware(). AMD SDK wheels could otherwise leak through with
CUDA-only visibility masks on a spawned ROCm worker.
Tests: 231 passed, 1 skipped in tests/studio/install/test_rocm_support.py
(no test changes needed; the probe shape that prints the hip version (or
'rocm' sentinel) preserves the existing non-empty-string contract).
Not addressed in this commit (deferred or out of scope):
- Tag drift / lemonade checksum (PR 5303 surface, not this PR).
- install.sh rocm7.2.1 URL: small fix, separate.
- install.ps1 / setup.ps1 'Radeon 8060S' marketing-name fallback table.
- Strix Halo + ROCm 7.1 routing asymmetry in Python update path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/rocm): robustness pass - rocm tag normalisation, Strix routing parity, hardened detection
Robustness pass on top of 76137b2d. Four targeted fixes:
1. install.sh ROCm-tag routing normalisation.
`rocm7.2.1` would route to https://download.pytorch.org/whl/rocm7.2.1
which does not exist (PyTorch publishes major.minor URLs only). Same
for any future patch-level tag. Normalise every rocm{maj.min}* pattern
to the bare {maj.min} index URL.
2. install.ps1 + studio/setup.ps1 marketing-name fallback.
The gfx1151 row matched 890M / Strix Halo / HX 37x / HX 38x / AI 9 HX
but not the actual retail name 'AMD Radeon 8060S Graphics' shipped by
OEMs (Ryzen AI MAX+ 395). Add '8060S' to the regex.
3. install_python_stack.py Strix + ROCm 7.1 routing parity with install.sh.
The shell installer reroutes Strix Halo / Point + ROCm 7.1 to
repo.amd.com/rocm/whl/{gfx}/ (which serves torch 2.11.0+rocm7.13.0
with the upstream _grouped_mm fix). The Python `studio update` path
only warned and still installed the broken generic rocm7.1 wheel.
Mirror the override: detect gfx1151/gfx1150 on ROCm 7.1, route to
the AMD per-gfx index, honour UNSLOTH_AMD_ROCM_MIRROR override.
4. _detect_windows_gfx_arch amd-smi parsing tightened.
The amd-smi fallback added in the prior commit used a bare
`\bgfx[1-9][0-9a-z]{2,3}\b` match against the lowercased stdout,
which could pick up stray gfx references in warnings / device-name
strings. Anchor on labelled lines first (Target_Graphics_Version,
ASIC, Arch, gfx) and fall back to the bare match only when no
labelled line is present.
Tests: 231 passed, 1 skipped in tests/studio/install/test_rocm_support.py;
sim_5301 23 cases pass (6 new sims for the Strix override + amd-smi parsing).
* fix(studio/rocm): multi-GPU selection, Strix sibling handling, defensive cleanups
Round 4 robustness pass based on 5 parallel Opus reviewers of head 21773215.
Seven items from across regression / edge-case / error-paths / architecture
reviews:
1. studio/backend/main.py BNB gate: aligned with the broad ROCm check used
everywhere else in this PR (torch.version.hip OR 'rocm' in __version__).
AMD SDK / Radeon Linux wheels do not always populate torch.version.hip;
without this, main.py would silently skip BNB_ROCM_VERSION while worker.py
set it.
2. studio/install_python_stack.py _install_bnb_windows_rocm: init _ok = False
before the try block. Without this, if pip_install_try itself raises
(e.g. OSError on uv binary missing), the finally block restored env vars
correctly but the subsequent `if not _ok:` raised UnboundLocalError,
masking the original exception.
3. studio/install_python_stack.py _detect_windows_gfx_arch:
- Rewrote to use re.findall (not re.search) on both hipinfo and amd-smi
output, dedup tokens preserving order, and select via new
_pick_visible_index() helper.
- HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES (first comma entry, integer)
now picks the right GPU on multi-AMD-GPU hosts. Out-of-range or non-int
values fall back to the first GPU (matches detect_host behaviour in
install_llama_prebuilt.py).
4. studio/install_python_stack.py Strix override now consults the runtime
target before flipping:
- Previous behaviour intersected gfx_codes with {gfx1151, gfx1150} and
picked the first Strix arch, ignoring whether HIP_VISIBLE_DEVICES
selected a non-Strix sibling (e.g. discrete RX 7900 in a mixed APU+dGPU
box). Could install Strix-specific wheels onto a gfx1100 dGPU.
- Now resolves the runtime gfx via _pick_visible_index() and only
overrides when that runtime target is in the Strix set.
5. studio/backend/main.py + studio/backend/core/training/worker.py: ROCm
version dir scan no longer sorts lexically. Previous sort placed "10.0"
before "7.0" alphabetically, which would mis-prioritise ROCm 10.x bin
dirs once AMD ships them. New _ver_key() splits on "." and sorts
numerically with a string fallback.
6. install.sh Strix override URL: replaced ${var%/} (strips one trailing
slash) with a while-loop that strips all trailing slashes, matching
Python's .rstrip("/"). A user setting UNSLOTH_AMD_ROCM_MIRROR with
"http://corp/whl///" no longer ends up with "http://corp/whl///gfx1151/"
which strict pip proxies (artifactory, sonatype) 404 on.
7. studio/install_python_stack.py: bumped torch import probe timeout from
30s to 90s. PyTorch's lazy .so loading can take 60-90s on cold NFS or
USB-backed venvs. The shorter timeout was producing a false "torch
missing" classification and reinstalling a working ROCm torch.
Tests: 231 passed, 1 skipped. sim_5301 30 cases pass (added 7 new sims for
multi-GPU detection, Strix sibling handling, and _ok-init regression).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/rocm): worker BNB/grouped_mm broad gate, install.sh Strix visibility, runtime-only ROCm detection
Round-5 robustness pass based on 20 parallel reviewers of head 96b9e465.
1. studio/backend/core/training/worker.py - BNB version pin / dynamo disable
/ _grouped_mm fallback block was still gated on torch.version.hip alone
despite the torchao stub block above already using the broad check. AMD
SDK / Radeon Windows wheels (torch.__version__ contains "rocm" but
torch.version.hip is None) silently skipped the Windows ROCm runtime
patches. Aligned to the same broad check (8/20 reviewers).
2. studio/backend/core/training/worker.py - _hip_ver_at_least() now also
parses the ROCm version out of torch.__version__ (e.g. "2.11.0+rocm7.13.0")
when torch.version.hip is missing, so the kernel-fix gate is correct for
SDK / Radeon wheels too.
3. studio/backend/core/training/worker.py - _grouped_mm_safe_impl with
offs=None now picks torch.bmm/matmul for 3-D inputs instead of always
calling torch.mm. The real _grouped_mm accepts 3-D batched matmul; the
prior fallback raised "self must be a matrix" on MoE workloads (2/20).
4. studio/backend/main.py - dropped the HIP_PATH / ROCM_PATH env-var gate
from the BNB block; probe torch directly. Runtime-only Radeon / AMD SDK
Windows installs do not set those SDK env vars but still ship ROCm torch
(5/20 reviewers).
5. install.sh - Strix override now collects every gfx token from
rocminfo / amd-smi (in enumeration order), then indexes by
HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES so a mixed Strix iGPU + non-
Strix dGPU host where the user selected the dGPU does NOT get rerouted
to the Strix per-gfx index. Mirrors the Python update path (5/20 reviewers).
6. install.sh - Strix detection chain now also probes `amd-smi static --asic`,
matching the PowerShell installer (1/20). Closes the gap on runtime-only
Strix hosts where `amd-smi list` does not surface a gfx token.
7. studio/install_python_stack.py - _has_rocm_gpu() now has the sysfs KFD
topology fallback (/sys/class/kfd/kfd/topology/nodes/*/gpu_id), matching
install.sh. On minimal package-managed installs without rocminfo /
amd-smi GUI tools, `studio update` can now detect the GPU and repair the
venv instead of returning early (2/20).
8. studio/install_python_stack.py - _detect_amd_gfx_codes() now falls back
to `amd-smi list` and `amd-smi static --asic` when rocminfo is missing
(2/20). Strix routing on runtime-only Radeon hosts now matches what
install.sh has done for a while.
9. studio/install_python_stack.py - Strix override now applies even when
has_hip_torch is True. The whole point of the override is to repair an
existing broken torch.version.hip == "7.1" install; skipping the
reinstall left users on the known _grouped_mm segfaulting stack (3/20).
Tests: 231 passed, 1 skipped. sim_5301 30 cases pass. sim_cross 12 pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/rocm): code review hardening pass
- main.py: numeric DLL sort (string sort picked rocm72 over rocm713);
add basename() to regex; log warning on detection failure; log info
when BNB_ROCM_VERSION is set (mirrors worker.py)
- worker.py: explicit len-guard in _hip_ver_at_least() with warning
logs instead of silent IndexError/ValueError swallow
- hardware.py: isinstance(result, dict) guard before result.get() in
_smi_query() to prevent AttributeError on non-dict backend returns
- amd.py: round() before int() on parsed GPU IDs; log warning when
truncation occurs (defensive against malformed amd-smi output)
- setup.sh: quote --gcc-install-dir value in CMAKE_HIP_FLAGS so paths
with spaces do not break the CMake argument
- install.ps1, setup.ps1: apply colon-split + ToLower() to hipinfo
gcnArchName match (consistent with each other and with setup.sh)
- install.sh: tighten ROCm tag case patterns to explicit
rocmX.Y|rocmX.Y.* to avoid unintended prefix matches
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/training): GPU OOM guard to prevent system freeze on VRAM exhaustion
On RDNA 4 (gfx1200/gfx1201) and other ROCm GPUs, exhausting VRAM can
cause a HIP driver hang that freezes the entire system rather than
raising a recoverable Python exception.
Two-part fix:
- set_per_process_memory_fraction(0.90) caps the HIP/CUDA allocator at
90% of VRAM so PyTorch raises OutOfMemoryError before hitting the
hardware limit, keeping the driver alive and the system responsive
- top-level exception handler detects OOM errors by type and message
and surfaces a clear actionable message to the UI (reduce
max_seq_length, enable gradient_checkpointing, lower batch size)
instead of the raw CUDA/HIP error string
* fix(studio/rocm): OOM guard ROCm-only + unified memory, multi-GPU arch selection
OOM guard (worker.py):
- Scope to _hw.IS_ROCM only -- NVIDIA CUDA has a graceful OOM path and
does not need the allocator cap
- Detect unified memory by comparing torch VRAM against psutil system RAM;
use 0.80 on unified-memory APUs (gfx1151 Strix Halo) where the GPU pool
is carved from host RAM, 0.90 on discrete cards
Multi-GPU arch selection:
- install.ps1 / setup.ps1: replace -match (first hit only) with
[regex]::Matches() to collect all gcnArchName entries, then index by
HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES
- install_python_stack.py: index into full token list before dedup so
HIP_VISIBLE_DEVICES=2 on [gfx1100, gfx1100, gfx1151] resolves gfx1151
- install.sh: remove awk dedup from gfx token collection for same reason
GCC multiarch (setup.sh):
- Only append -linux-gnu when gcc -print-multiarch does not already return
the full triple, fixing double-suffix on Ubuntu 24.04
* fix(tests): update ROCm version cap expectations from rocm7.1 to rocm7.2
Daniel's normalisation commit updated the cap from rocm7.1 to rocm7.2
since PyTorch now publishes that index and rocm7.2 ships torch 2.11.0.
Test expectations were stale.
* fix(tests): correct MLX smoke test losses_per_step assertion
logging_steps=1 with max_steps=30 produces 30 loss entries, not 7.
The assertion was stale from a previous config.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/worker): detect unified-memory APU by GPU name not VRAM/RAM ratio
The previous heuristic (VRAM > 50 % of system RAM) false-positived on discrete
cards in low-RAM systems — e.g. RX 9060 XT 16 GB on a 16 GB or 24 GB machine
would trip the unified-memory path and log "unified memory host" when it should
say "discrete".
AMD iGPUs (gfx1150/gfx1151 Strix Halo, Strix Point, etc.) expose names with a
digit+M suffix ("AMD Radeon 890M"), while discrete cards use "RX NNNN [XT|XTX]"
naming. Matching that suffix is reliable across all current ROCm-capable AMD
consumer GPUs and does not require psutil.
Also includes the device name in the log line to ease future debugging.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(install/setup.ps1): force array on hipinfo gcnArchName parse to fix single-GPU arch truncation
When [regex]::Matches() finds exactly one match, PowerShell's pipeline
unwraps the result to a scalar string. Indexing a scalar string with [0]
returns the first *character*, so a one-GPU system would parse
gcnArchName "gfx1200" as "g", which is not in the supported arch map
and triggers the CPU-only fallback.
Wrapping with @() forces the result to remain an array regardless of
match count. On a single-GPU machine the arch is now correctly read as
"gfx1200" (or whatever the full name is) so the ROCm wheel index is
selected.
Reproducer: hipinfo exits 0 and outputs exactly one gcnArchName line.
Without @(), $_hipAllArches = "gfx1200" (String); $_hipAllArches[0] = 'g'.
With @(), $_hipAllArches = @("gfx1200") (Object[]); $_hipAllArches[0] = "gfx1200".
* fix(studio/rocm): classify unified-memory APU via VRAM/RAM ratio, not arch list
Replace the gcnArchName allowlist {gfx1150, gfx1151} with a
psutil-based heuristic: unified APUs expose the entire system RAM
as the HIP pool (ratio ≥ 0.90), discrete cards are well below that.
No arch name required — future APUs classify correctly without code changes.
Also removes the stale import re / \d[Mm]\b device-name regex that
5d84704 left behind, and logs vram/sys GiB for easier on-hardware
verification.
Addresses h34v3nzc0dex review: Radeon 8060S (gfx1151, 128 GiB
unified) now correctly gets 0.80 cap instead of 0.90.
* fix(studio/rocm): revert to gcnArchName for unified-memory APU classification
VRAM/RAM ratio >= 0.90 false-positives on machines where discrete VRAM
equals system RAM (e.g. RX 9060 XT 16 GB + 16 GB system RAM → ratio 1.0,
incorrectly classified as unified → wrong 0.80 cap applied).
gcnArchName is the correct signal: naming-independent, stable within a
product family, and already parsed throughout this PR. Unified set is
{gfx1150, gfx1151} (Strix Point + Strix Halo).
* fix(studio/llama-prebuilt): resolve hipinfo via HIP_PATH/ROCM_PATH on Windows
shutil.which("hipinfo") returns None when the HIP SDK bin dir is not on
PATH -- the HIP SDK installer sets HIP_PATH/ROCM_PATH but does not always
add the bin dir to PATH. This caused has_rocm=False in the prebuilt asset
selector, so AMD ROCm machines got the CPU llama.cpp zip instead of the
HIP one, silently running all chat inference on CPU.
Add _resolve_exe() that falls back to %HIP_PATH%\bin and %ROCM_PATH%\bin
when shutil.which() finds nothing, mirroring the same fallback already
present in setup.ps1.
* fix(studio/llama-prebuilt): pass --has-rocm from setup.ps1 to skip re-detection
The Python prebuilt installer re-detects ROCm independently via
shutil.which("hipinfo"), which fails when hipinfo is not on PATH
(HIP SDK sets HIP_PATH but doesn't always add the bin dir to PATH).
This caused has_rocm=False and downloaded the CPU llama.cpp zip even
on confirmed AMD ROCm machines.
setup.ps1 already performs reliable ROCm detection with its own
HIP_PATH/ROCM_PATH fallback. Add --has-rocm flag to
install_llama_prebuilt.py so setup.ps1 can forward its result directly,
and pass it whenever $HasROCm is true. The Python script then overrides
has_rocm=True in the HostInfo without re-probing.
* fix(studio/llama-prebuilt): add HIP asset to simple-policy Windows path
direct_upstream_release_plan (used by --simple-policy, which setup.ps1
always passes) only checked has_usable_nvidia on Windows and fell
straight to CPU for AMD ROCm machines, ignoring has_rocm entirely.
The --has-rocm override had no effect because the simple-policy code
path never reached resolve_asset_choice where has_rocm was checked.
Add an elif branch for has_rocm that tries the upstream HIP asset
(llama-TAG-bin-win-hip-radeon-x64.zip) before falling through to the
CPU fallback, consistent with the non-simple-policy path.
* fix(studio/setup.ps1): auto-remove mismatched llama.cpp install kind
When an existing llama.cpp install is the wrong kind for the current
GPU (e.g. windows-cpu on an AMD ROCm machine that should have
windows-hip), the prebuilt installer skips on tag match and never
upgrades. Read install_kind from UNSLOTH_PREBUILT_INFO.json before
invoking the installer and remove the directory if the kind doesn't
match, forcing a fresh download of the correct variant.
* fix(studio/setup.ps1): show live PyTorch install output in verbose mode for ROCm
The ROCm torch reinstall (setup.ps1 phase) always silently captured
output, so in --verbose mode the torch downgrade mid-install
(2.11.0+rocm → 2.10.0 → 2.11.0+rocm) looked like the final state was
2.10.0. Match the CPU/CUDA blocks which show live uv output when
$script:UnslothVerbose is set.
* fix(rocm/windows): set ROCBLAS_TENSILE_LIBPATH for bundled rocblas.dll
The llama.cpp ROCm prebuilt bundles rocblas.dll next to the binary but
not the Tensile kernel library files it depends on at runtime
(rocblas/library/TensileLibrary*.dat + *.hsaco). The bundled DLL
searches for these files relative to its own location by default, i.e.
<binary_dir>/rocblas/library/, which does not exist in the prebuilt
install tree. This causes a silent crash on the very first GEMM
(prefill) with no output from llama-server, seen by the caller as
WinError 10054 / 10061. Model load and the single-token warmup pass
because they use simpler code paths that do not trigger rocBLAS GEMM.
Fix: set ROCBLAS_TENSILE_LIBPATH in the subprocess env to
<HIP_PATH>/bin/rocblas/library so the bundled DLL finds the kernel
files from the system ROCm installation. Uses setdefault so a user-
supplied env var is never overwritten. No-ops on CUDA and CPU (no
HIP_PATH) and on Linux (win32 branch only).
Reproducer log:
rocBLAS error: Cannot read .../Release/rocblas/library/TensileLibrary.dat
rocBLAS error: Could not initialize Tensile host:
directory_iterator: The system cannot find the path specified.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(install.sh): restore gfx token dedup in Strix multi-GPU awk indexer
536a54df removed the per-source `| awk '!seen[$0]++'` dedup from the
_gfx_all collection step but left the indexer awk as bare NF, so on a
mixed-arch host (e.g. dGPU gfx1100 + Strix iGPU gfx1151) where
rocminfo emits each gfx token twice (Name: field + ISA triple),
HIP_VISIBLE_DEVICES=1 indexed vals[1] = the second gfx1100 occurrence
instead of gfx1151, triggering the Strix routing on the wrong GPU.
Add !seen[$0]++ to the indexer awk so duplicate tokens from the same
GPU collapse to one entry before the HIP_VISIBLE_DEVICES index is
applied -- matching exactly what the Python side does with dict.fromkeys()
in _detect_amd_gfx_codes(). The comment above the block ("skip
duplicates") already documented this as the intended behaviour.
* fix(studio/install): correct _TOTAL progress count on Windows
base_total += 3 fired for all non-macOS platforms including Windows,
but flash-attn (line 1620) and ROCm torch final (line 1705) are both
guarded by 'not IS_WINDOWS and not IS_MACOS', so on Windows with torch
enabled _TOTAL was 13 while only 11 _progress() calls actually execute.
Split into +1 for the ROCm torch check (all non-macOS) and +2 for the
two Linux-only steps, so Windows gets _TOTAL=11 and Linux gets 14.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(install.ps1): enforce torch>=2.11.0 for gfx120X and Strix on Windows
The AMD arch-specific index (repo.amd.com/rocm/whl/gfx120X-all/ and
gfx1151/) publishes torch wheels from 2.7.1 through 2.11.0. Without a
version floor pip can resolve to torch 2.10.0+rocm7.12 on RDNA 4
(gfx120X) or torch 2.10.0+rocm7.1 on Strix (gfx1151/gfx1150), both of
which have a null-pointer crash in torch._C._grouped_mm (TheRock
issues #5284 / #3284). torch 2.11.0+rocm7.13 contains the fix.
Add $ROCmTorchFloor alongside $ROCmIndexUrl: set to torch>=2.11.0 for
the two affected arch families, null for all others. Wire it into the
uv pip install call so the broken wheels are never selected.
* fix(rocm/windows): address Codex nits - deterministic DLL suffix, CUDA llama.cpp kind, HIP_VISIBLE_DEVICES arch indexing
- install_python_stack.py / worker.py: _detect_bnb_rocm_dll_ver() and the
inline worker probe now collect ALL libbitsandbytes_rocm*.dll suffixes and
return max() by numeric value instead of stopping at the first glob hit.
Filesystem glob order is not guaranteed; this ensures '713' always wins
over '72' when both variants are present in the wheel.
- setup.ps1 (expectedKind): add 'windows-cuda' branch so NVIDIA hosts are
not treated as 'windows-cpu'. Previously an existing windows-cuda prebuilt
was always considered a mismatch on non-ROCm machines, forcing an
unnecessary re-download on every update.
- setup.ps1 (amd-smi gfx arch): collect ALL gfx tokens from amd-smi list
output in GPU order and honour HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES
when selecting which arch to use. On mixed-arch AMD systems where the
visible GPU is not the first enumerated one, this prevents installing an
incompatible wheel index. Falls back to index 0 (same as before) when the
visibility var is unset or is a comma-separated list.
- test_rocm_support.py: add test_picks_highest_suffix_when_multiple_dlls to
cover the multi-DLL case that was previously untested.
* fix(rocm): misleading amd-smi log, BNB spec consistency, torch ceiling for AMD index
amd.py: split 'returncode != 0 or not stdout' into two separate branches.
Previously, exit-0 with empty output logged 'amd-smi returned code 0' (which
reads as success, not a warning) and incorrectly incremented the circuit-breaker
counter. Now: non-zero exit logs the code and counts toward the limit as before;
empty stdout on exit 0 logs at DEBUG level and does not penalise the counter
(amd-smi --json always emits at least [] on exit 0, so this branch is rare and
is not a tool failure).
main.py: replace spec.origin / os.path.dirname() with
spec.submodule_search_locations to match install_python_stack.py and worker.py.
For normal wheel installs both approaches reach the same directory, but using
submodule_search_locations is the canonical way and handles editable bitsandbytes
installs correctly. Also use max() by numeric suffix (same as the other two sites)
instead of a sort-then-break loop.
install.ps1: add <2.12.0 ceiling to the torch constraint for gfx120X (RDNA 4)
and gfx1151/gfx1150 (Strix). AMD actively publishes new versions on their
per-arch index; without a ceiling, a future 2.12.0+rocmX.Y wheel would be
pulled in automatically before being validated on these architectures. The
ceiling matches the existing Linux install_python_stack.py constraint for the
same arches. Bump both when 2.12.x is confirmed working.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(rocm): torch floor in setup.ps1, torchvision pin for Strix, rocmsdk in _hip_ver_at_least
setup.ps1: add \ (mirrors install.ps1) and derive \
from it. Previously the AMD index install called 'Fast-Install torch torchvision
torchaudio --force-reinstall --index-url \' with no version
constraint, so pip could resolve torch 2.10.0+rocm7.12 for gfx1151/gfx1200 --
the exact broken wheel the PR is meant to avoid. Now gfx120X and Strix enforce
'torch>=2.11.0,<2.12.0', matching install.ps1 and the Linux constraint.
install_python_stack.py: pin torchvision and torchaudio in _strix_override_pkgs.
The Strix Linux override uses --index-url (exclusive, no PyPI fallback); bare
unversioned 'torchvision' and 'torchaudio' could resolve a build from AMD's
index targeting a different torch major, causing ABI/version mismatches at
runtime. Now pinned to '>=0.26.0,<0.27.0' and '>=2.11.0,<2.12.0' respectively,
matching _ROCM_TORCH_CONSTRAINT['rocm7.2'].
worker.py: extend _hip_ver_at_least to handle AMD SDK wheel version strings.
The fallback regex r'rocm(\d+)\.(\d+)' cannot match '2.9.0+rocmsdk20251116'
(no rocmX.Y component), so the function always returned False on SDK/Radeon
wheels -- installing the Python _grouped_mm workaround on wheels that already
have the working HIP kernel. Added a second check: if the version string
contains '+rocmsdk', assume >= 7.13 (the rocmsdk format post-dates the
gfx120X null-kernel fix) and skip the fallback.
* fix(rocm): warn on OOB HIP_VISIBLE_DEVICES, bail on empty numeric_ids mask
- setup.ps1: when HIP/ROCR_VISIBLE_DEVICES names an index beyond the
detected GPU count, emit a yellow warning and fall back to GPU 0
instead of silently reading allGfxArches[-1] (wrong arch)
- hardware.py _reconcile_primary_rocm_unified_memory: distinguish
numeric_ids=None (no env var, use torch ordinal 0) from numeric_ids=[]
(empty mask / HIP_VISIBLE_DEVICES=-1, no GPU visible); bail out early
in the empty case to avoid querying torch.device(0) incorrectly
* fix(rocm): gate StubSubpackageFinder on win32 ROCm, add gcnArchName fallbacks
- worker.py _StubSubpackageFinder: the meta_path append was running on
every platform on every call to run_training_process; moved it inside
the if _is_win32_rocm: block since stubs are only seeded there and the
finder is a pure accumulation on Linux/Windows CUDA
- worker.py OOM guard: AMD SDK / Radeon wheels may not populate
gcnArchName, causing Strix Halo to be misclassified as discrete and
get the 0.90 cap (12.8 GB OS headroom) instead of 0.80 (25.6 GB);
now tries gcn_arch_name / arch_name / gfx_arch_name variants first,
then falls back to device-name matching (890M -> Strix Halo,
880M -> Strix Point) with a debug log when the fallback fires
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(rocm): pin torchvision/torchaudio in setup.ps1, remove -Unique from arch array
- setup.ps1 ROCm torch install: torchvision and torchaudio were passed
bare alongside pinned torch>=2.11.0,<2.12.0 for gfx1151/gfx1200 arches.
AMD publishes packages independently so a future torchvision 0.27 (for
torch 2.12) on the same arch index would cause pip ResolutionImpossible
or an ABI-incompatible install. Added torchvisionFloorMap and
torchaudioFloorMap mirroring install_python_stack.py's strix override
(torchvision>=0.26.0,<0.27.0, torchaudio>=2.11.0,<2.12.0) and derived
ROCmVisionSpec/ROCmAudioSpec used in all three Fast-Install call sites.
- setup.ps1 amd-smi arch detection: Select-Object -Unique was collapsing
same-arch multi-GPU arrays (e.g. two gfx1151 APUs -> 1-element array)
causing HIP_VISIBLE_DEVICES=1 to trigger a false out-of-range warning
and fall back to GPU 0 even though the correct GPU would have been at
index 1. Removed -Unique; added comment noting the positional-index
assumption and its non-contiguous-GPU limitation.
* fix(rocm): add 8060s/8050s to OOM guard device-name fallback, extract classifier helper
Path 3 of the OOM guard device-name fallback only checked for 890m/880m
(gfx1150 Strix Point SKU names). Strix Halo (gfx1151) ships as Radeon 8060S
(Ryzen AI MAX+ 395) and Radeon 8050S (cut-down SKU) -- neither matches, so
the fallback returned is_unified=False and applied the 0.90 fraction instead
of 0.80, leaving ~12.8 GiB OS headroom on a 128 GiB pool instead of ~25.6 GiB.
Fix: add 8060s and 8050s to the name-match set. Also correct the comment that
mislabelled 890M as a Strix Halo name (it is Strix Point).
Refactor: extract the three-path classifier into _rocm_classify_unified_memory()
so it can be unit-tested directly. Add 31 test cases in test_rocm_oom_guard.py
covering all three paths and the regression case (Radeon 8060S Graphics).
Reported-by: h34v3nzc0dex
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(rocm): pass explicit dtype on bf16-unsupported hardware (RDNA2)
dtype=None lets unsloth auto-detect the model dtype. On RDNA2 (gfx103x,
e.g. RX 6600) is_bfloat16_supported() incorrectly returns True, so unsloth
picks bf16 and the first bf16 kernel dispatch triggers:
LLVM ERROR: Cannot select: intrinsic %llvm.amdgcn.fdot2.bf16.bf16
Replace every dtype=None in load_model() with _auto_dtype which resolves
to None when bf16 is supported (all modern NVIDIA + RDNA3+) and
torch.float16 otherwise. This gives RDNA2 users a working float16
training path without touching NVIDIA behaviour at all.
Fixes: https://github.com/unslothai/unsloth/issues/5337
* fix: reduce log noise for expected non-issues on Windows ROCm
Three log lines fired at warning/error level for conditions that are
completely expected on a Windows HIP SDK-only setup:
amd.py
- amd-smi WinError 2 (FileNotFoundError): downgrade warning -> debug.
amd-smi ships with Adrenalin, not the HIP SDK; absence is normal.
- 'disabling' message: downgrade warning -> info with clearer text
'not available (not installed; expected on HIP SDK-only systems);
GPU VRAM polling disabled'
hardware.py
- torch.distributed.Store missing: downgrade warning -> debug.
The distributed stub added in this PR intentionally omits Store; the
attention-impl fallback to eager is expected and non-actionable.
worker.py
- causal-conv1d: add early Windows exit (info) in both
_ensure_causal_conv1d_fast_path and _causal_conv1d_install hook;
no cp313/win_amd64 wheel exists, so the install always fails.
- FLA: add early Windows exit (info) in
_ensure_flash_linear_attention_unconditional; triton dependency has
no cp313/win_amd64 wheel.
- Defense-in-depth: _install_package_wheel_first non-HIP PyPI failure
logs info+debug on Windows instead of error; FLA failure logs
info+debug on Windows instead of warning.
* [AMD] FIx installation of bitsandbytes when it's from .dev and skip rebuilding llama.cpp if we build it manually.
* fix: use force_pip for Windows ROCm bitsandbytes prebuilt wheel install
uv rejects the bnb continuous-release wheel due to filename/metadata
version mismatch (1.33.7.preview vs 0.50.0.dev0). Switch to force_pip=True
(pip bypass) instead of the UV_SKIP_WHEEL_FILENAME_CHECK env var workaround
-- cleaner and consistent with how the Linux path handles it.
BNB_ROCM_VERSION is still set post-install to the detected DLL suffix so
the worker subprocess loads the correct libbitsandbytes_rocm{VER}.dll even
when torch.version.hip reports a newer HIP version than the wheel ships.
* fix: three small correctness fixes found in PR review
- _install_bnb_windows_rocm: use UV_SKIP_WHEEL_FILENAME_CHECK=1 with
try/finally instead of force_pip=True so the env var is always
restored and the failing CI test passes
- _determine_attention_impl_for_gpu_estimate: gate torch._C distributed
stubs on IS_ROCM so Windows CUDA users keep the real extension
- install.ps1 amd-smi fallback: collect all gfx tokens and index by
HIP_VISIBLE_DEVICES, matching the hipinfo path on multi-GPU hosts
* fix: stub torchao in export subprocess on Windows ROCm
On Windows, the ROCm build of PyTorch ships without the distributed
C extension (torch._C._distributed_c10d). torchao, which is pulled in
transitively by transformers.quantizers at import time, walks into
torch.distributed._functional_collectives -> distributed_c10d and
crashes with:
No module named 'torch._C._distributed_c10d'; 'torch._C' is not a package
This only affected the export subprocess because the training subprocess
already applied an identical torchao stub (introduced separately to fix
the same root cause). The export subprocess had no such guard and died
during 'Importing Unsloth...' before any model loading could happen.
Fix: apply the same _StubSubpackageFinder / torchao stub pattern to the
export subprocess entry point, gated on Windows ROCm detection, before
any import of transformers or unsloth_zoo.
Root cause tracked in ROCm/TheRock#3284 (libuv / torch.distributed
missing on Windows ROCm builds).
Ref: https://github.com/ROCm/TheRock/issues/3284
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* install.sh, setup.sh: add GPU arch step logging to match PS1 scripts
Both shell scripts were missing the step "gpu" terminal log block that
install.ps1 and setup.ps1 emit. This adds equivalent output: GPU label
with gfx arch (e.g. "AMD ROCm (gfx1151)"), ROCm root path, hipconfig
version, and marketing name substep. Includes the same gfx arch detection
chain (rocminfo → amd-smi list → amd-smi static --asic), UNSLOTH_ROCM_GFX_ARCH
env override, and name-based arch inference table (Strix Halo/Point, RDNA 3/4)
as the PS1 versions. install.sh also replaces bare echo blocks for the AMD
ROCm and CPU-only cases with formatted substep output.
* Fix BNB_ROCM_VERSION gate, ROCm GPU mask preference, APU unified memory and Release build for PR #5301
- main.py: gate BNB_ROCM_VERSION on the rocm bnb DLL or HIP_PATH/ROCM_PATH instead of importing torch on every Windows host
- hardware.py: prefer HIP/ROCR visible-device masks only on ROCm hosts so a stale mask cannot override CUDA_VISIBLE_DEVICES on NVIDIA
- llama_cpp.py: set GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 only for unified-memory APUs (gfx1150/gfx1151)
- setup.sh: pass -DCMAKE_BUILD_TYPE=Release for the HIP source build
- add test_amd_apu_unified_memory.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: guard recompile_limit + fix AMD VRAM monitor fallback
trainer.py: torch._dynamo.config.recompile_limit does not exist in
some ROCm torch builds (e.g. pytorch.org/whl/rocm6.2 wheels). Guard
the assignment so training doesn't crash on RDNA2/RDNA3.
hardware.py: when amd-smi/nvidia-smi is unavailable or returns no
usable data (HIP SDK-only Windows, Docker, unexpected JSON format),
the existing fallback used torch.cuda.memory_allocated() which is
process-specific and reads near-zero even with a fully loaded model.
Switch to torch.cuda.mem_get_info() via _torch_get_per_device_info()
which reports system-wide VRAM occupancy so the GPU monitor shows
real usage on all AMD systems without requiring amd-smi.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: Windows VRAM monitor via Performance Counter API
When amd-smi/nvidia-smi is unavailable on Windows, query dedicated GPU
VRAM via Windows Performance Counters (same source as Task Manager).
This gives system-wide cross-process usage, fixing the near-zero reading
caused by torch.cuda.mem_get_info only seeing the Studio server process.
Linux fallback path unchanged (mem_get_info is system-wide on ROCm).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: rename to _rocm_windows_perf_counter_vram_gb, scope to IS_ROCM
Function is AMD ROCm specific — amd-smi absent on Windows when only the
HIP SDK is installed. Scoped to IS_ROCM so NVIDIA Windows path is
untouched (nvidia-smi handles that case).
* fix: AMD VRAM monitor — Linux DRM sysfs + Windows perf counter
Linux: read /sys/class/drm/card*/device/mem_info_vram_used|total for
system-wide GPU memory across all processes. No tools required, always
present on Linux AMD systems.
Windows: Windows Performance Counter API (already added).
Both paths are gated on IS_ROCM and only fire when amd-smi is absent.
torch mem_get_info remains as last resort (process-local).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: AMD GPU monitor — utilization, temperature, and power for Windows and Linux fallback paths
- Windows: GPU utilization via \GPU Engine(*engtype_3D*)\Utilization Percentage perf counter
- Windows: temperature and power via ADL (atiadlxx.dll, ships with Adrenalin)
- Linux: GPU utilization via DRM sysfs gpu_busy_percent
- Linux: temperature via hwmon temp1_input (millidegrees C)
- Linux: power via hwmon power1_average / power1_input (microwatts)
All paths are no-op fallbacks (None) when the source is unavailable.
Mirrors what nvidia-smi provides on the CUDA path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: remove ADL ctypes — does not support AMD iGPU (Strix Halo)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Erland366 <erland.pg366@gmail.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Studio setup.sh: cope with fresh CUDA toolkits like 13.3
CUDA 13.3 shipped today. Three loose ends in studio/setup.sh surfaced
during the llama.cpp build path:
1. setup.ps1 already aborts cleanly when the CUDA toolkit is below
llama.cpp's minimum (12.4) via #4517, but setup.sh still hit the
generic cmake failure described in #4437. Added a min-version check
that downgrades to a CPU build for nvcc < 12.4 with a clear message
pointing to the toolkit archive.
2. The first day a new CUDA toolkit ships, its host-compiler whitelist
lags whatever gcc/clang the distro is on, so nvcc rejects the host
compiler with a wall of "#error -- unsupported GNU version" before
any real compile runs. NVCC_PREPEND_FLAGS now carries
-allow-unsupported-compiler so the build moves on instead.
3. The Linux CUDA/ROCm configure failure path had no symmetry with the
macOS Metal fallback: a single nvcc failure left BUILD_OK=false and
no llama.cpp at all. Generalised the existing Metal -> CPU fallback
to cover any GPU_BACKEND, so a CUDA configure or build failure now
transparently retries with the CPU args and the user still ends up
with a working llama-server.
Pulled the version probe out into _nvcc_meets_llama_minimum so it can
be unit-tested. Added tests/sh/test_nvcc_meets_llama_minimum.sh and two
extra cases in tests/sh/test_get_torch_index_url.sh covering the legacy
"CUDA Version: 13.3" header (driver-reported) and the future 13.7
case. Wired the new test into tests/run_all.sh and the studio-backend
CI workflow.
* tests: relax pr4562 regression to allow generic GPU fallback label
* studio tests: assert setup.sh exports NVCC_PREPEND_FLAGS=-allow-unsupported-compiler
The -allow-unsupported-compiler flag is the core of the fresh-CUDA-toolkit fix
(it lets nvcc accept a host gcc/clang newer than its release-time whitelist, so
CUDA 13.3 day-one builds do not abort on '#error -- unsupported GNU version'),
but it had no automated coverage. Add a source-pattern test asserting the flag
is present, delivered via NVCC_PREPEND_FLAGS so it also covers cmake's CUDA
compiler-id probe, and kept out of CMAKE_ARGS for bash word-splitting safety.
* studio/setup.ps1: allow unsupported host compiler for CUDA build (Windows parity)
Mirror the Linux setup.sh headline fix from this PR on Windows. A freshly
released CUDA toolkit ships with a host-compiler whitelist that lags the
installed toolchain, so nvcc can reject the host with
"#error -- unsupported Microsoft Visual Studio version!" before any real
compile runs (the MSVC analogue of the gcc wall the Linux side hit on
CUDA 13.3). Set NVCC_PREPEND_FLAGS=-allow-unsupported-compiler in the CUDA
build branch so both cmake's configure-time CUDA compiler-id probe and the
cmake --build step proceed. The flag disables the host version check only and
is a no-op when the compiler is already supported.
Set via the process environment (not the $CmakeArgs array), after the
Refresh-Environment calls that re-sanitize CUDA env vars, and appended
idempotently to any value the user already set.
Validated with PowerShell 7.6.2: full setup.ps1 AST parse is clean and the
snippet is idempotent (empty -> set, existing -> append once, no duplicate).
Needs real Windows + CUDA CI to exercise the actual nvcc/MSVC build.
Adds test_setup_ps1_exports_allow_unsupported_compiler asserting the flag is
present, env-delivered, kept out of $CmakeArgs, and scoped to the CUDA-on branch.
* studio: tighten code comments added in this PR
Shorten the verbose multi-line comments and test docstrings introduced by
this PR (setup.sh, setup.ps1, and the shell/python tests) to be succinct
while preserving the rationale. No code or test-assertion changes.
* Detect CUDA UMD Version from newer nvidia-smi output (#5812)
Newer NVIDIA drivers (e.g. 610.x on Windows) print the driver's max
CUDA capability as "CUDA UMD Version: X.Y" instead of the legacy
"CUDA Version: X.Y" header. The installers and Studio setup scripts
were only matching the legacy spelling, so on a fresh RTX 5090
laptop with a 13.x driver they failed to detect any CUDA version
and fell through to the cu126 wheel default.
Accept both spellings everywhere we parse nvidia-smi output:
- install.ps1: Get-TorchIndexUrl regex now allows " UMD"
- install.sh: two-expression sed (POSIX BRE has no "?"); the two
patterns are mutually exclusive per line, head -1 picks the match
- studio/setup.ps1: Get-PytorchCudaTag and the $DriverMaxCuda
detector both relaxed
- studio/install_llama_prebuilt.py: substring scan replaced with a
regex search using the same pattern
- tests/sh/test_get_torch_index_url.sh: new make_mock_smi_umd helper
plus three UMD cases (13.3 -> cu130, 12.8 -> cu128, 11.8 -> cu118);
all 30 tests pass locally
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Run the Linux llama.cpp prebuilt dependency preflight before reusing an existing install, so cached trees missing newly required per-tool shared libraries (libllama-server-impl.so / libllama-quantize-impl.so introduced upstream between b9279 and b9283) trigger a repair instead of silently skipping reinstall and failing at runtime. Adds a regression test for the new lib*-impl.so overlay layout.
* Studio: unblock cross-platform install on Linux ARM64 + Windows ARM64
Three independent bugs that together prevent `install.sh` /
`install.ps1` from completing on the ARM machines GitHub Actions now
ships (`ubuntu-24.04-arm`, `windows-11-arm`) and on equivalent real
hosts (Ampere Altra, Raspberry Pi 5, Snapdragon X Elite, ...).
Validated on the staging-2 cross-OS smoke suite -- five per-OS
workflows pinned to `ubuntu-latest`, `ubuntu-24.04-arm`, `macos-14`,
`macos-15-intel`, `windows-11-arm`. Before this change Windows ARM
exits 1 in the winget gate and Linux ARM source-builds llama.cpp
because the prebuilt selector returns 0 attempts; with it both reach
healthy /api/health.
1. studio/install_llama_prebuilt.py -- resolve_simple_install_release_plans
had explicit branches for windows+x86_64, macos+arm64, macos+x86_64
and linux+x86_64 only. Upstream ggml-org/llama.cpp ships
`llama-bNNNN-bin-ubuntu-arm64.tar.gz` and
`llama-bNNNN-bin-win-cpu-arm64.zip` (visible in the b9334 release
manifest), so the missing elif branches force every Linux ARM64 and
Windows ARM64 host into a source build even when a perfectly good
upstream prebuilt is one HTTP GET away. Two new branches mirror the
existing CPU variants; runtime_patterns_for_choice and
runtime_payload_health_groups gain `linux-arm64` (.so layout) and
`windows-arm64` (.dll layout) so the health-check pass-through
matches the asset shape.
2. studio/setup.sh -- the helper-release-repo selector routed any
non-x86_64 Linux to `unslothai/llama.cpp`, which only publishes the
Linux CUDA bundle set. The result on Linux ARM64 was a guaranteed
`direct_linux_release_plan` raise of "no compatible Linux prebuilt
asset was found" on every release in the scan, then a source-build
fallback. Pin Linux ARM64 (CPU-only) to `ggml-org/llama.cpp` so the
new branch in (1) can see the upstream asset. setup.ps1 already
hardcodes `ggml-org/llama.cpp`, so Windows ARM64 picks up (1)
without an additional change.
3. install.ps1 -- the winget pre-check hard-failed before Python or uv
detection. `windows-11-arm` runners (and many corporate Windows
hosts without the Microsoft Store) ship without winget but already
have a usable Python plus the Astral uv PowerShell installer
reachable. Demote the winget check to a soft warning, defer the
hard failure to the Python install branch (which is the only path
that genuinely needs winget), and let the uv install fall through
to `https://astral.sh/uv/install.ps1` when winget is absent. The
uv PowerShell installer was already the existing fallback for the
"winget present but uv install failed" case; this just makes it
the primary path on hosts without winget.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: filter torchcodec on platforms without wheels
torchcodec 0.10.0 ships wheels for manylinux_2_28_x86_64,
macosx_12_0_arm64, and win_amd64 only -- visible on its PyPI page and
in the resolver error reported by #4446. install_python_stack.py
pulls torchcodec via extras-no-deps.txt, which is now installed
unconditionally during `unsloth studio update --local` (the update
command has no --no-torch flag). Result on Linux aarch64 /
Windows ARM64 / Intel Mac (when invoked outside the install.sh
auto-skip-torch path):
ERROR: Could not find a version that satisfies the requirement
torchcodec==0.10.0 (from versions: 0.0.0.dev0, ...)
ERROR: No matching distribution found for torchcodec==0.10.0
error Installing extras (no-deps) (pip) failed (exit code 1)
`NO_TORCH_SKIP_PACKAGES` already lists torchcodec but only fires
when NO_TORCH is true -- the update path inherits no NO_TORCH from
the original install and inferrence falls back to IS_MAC_INTEL only,
so Linux aarch64 / Windows ARM64 sail past the guard. Adds a
platform predicate PLATFORM_LACKS_TORCHCODEC_WHEEL and applies the
torchcodec filter unconditionally there, independent of NO_TORCH.
Surfaced by the staging-2 cross-OS smoke `unsloth studio update`
step on ubuntu-24.04-arm; verified the same step is green with this
patch overlaid.
* Studio: skip librosa on no-torch hosts (unblocks Intel Mac install)
Closes the last cross-platform install gap surfaced by the staging-2
cross-OS smoke (see unslothai/unsloth#5046 for the original report):
`install.sh --local` on macos-15-intel fails at
× Failed to build `llvmlite==0.47.0`
error: failed-wheel-build-for-install
╰─> llvmlite
error studio setup failed (exit code 1)
Root cause: upstream llvmlite dropped the macosx_x86_64 wheel between
0.42.0 and 0.46.0 (https://pypi.org/project/llvmlite/0.47.0/#files --
only macosx_arm64 / manylinux / win_amd64 remain). pip falls back to
a from-source build of llvmlite's FFI, which needs LLVM 14/15 dev
headers and matching llvm-config -- not present in Xcode Command
Line Tools' libclang and not installed by install.sh's MAC_INTEL
deps branch.
llvmlite enters Studio's tree via librosa -> numba -> llvmlite in
extras.txt. openai-whisper (extras.txt:28) would also pull numba but
is already filtered on no-torch hosts. Adding librosa to the same
NO_TORCH_SKIP_PACKAGES set makes the install go through cleanly on
Intel Mac (auto-detected NO_TORCH=true via the MAC_INTEL branch) and
on any user-passed --no-torch host where torch-dependent audio
pipelines would not run anyway.
Tracked / verified on the danielhanchen/unsloth-staging-2#154 smoke
matrix (macos-15-intel).
* Studio UI tests: retry evaluate_fetch on transport-level failure (PR #5790)
Mac Studio UI CI on this PR (run 26496820814, job 78026959359) failed
with /api/models/list status=0 error='TypeError: Failed to fetch'.
The artifact studio.log shows the server answered the two preceding
/api/models/list calls from the React mount (both 200) but never
received the third call from the test script: the browser reused a
kept-alive HTTP/1.1 socket that uvicorn (5s keep_alive_timeout) had
closed ~130ms earlier. Chromium under --single-process on macos-14
free runners is most prone to this; the post /api/auth/change-password
session churn accelerates it. A rerun on the same SHA passed, which is
the classic flake signature.
evaluate_fetch in tests/studio/_playwright_robust.py already returns a
structured {status: 0, body: None, error: "..."} on JS-side throws, but
every caller treats status=0 as fatal. Add a bounded retry inside the
helper so the one class of failure recovers transparently:
status != 0 -> real HTTP response (incl. 4xx/5xx); propagate.
error has "AbortError" -> caller's AbortSignal deadline; propagate.
else (status==0) -> stale-keepalive or other transport failure;
retry after 250ms / 500ms backoff so the pool
evicts the dead socket before the next attempt.
Defaults transport_retries=2, transport_backoff_ms=250 (max added
latency on the happy path is zero; on a transport failure: up to
750ms of sleep). Callers keep the existing {status, body, error} shape;
no call-site changes needed.
Verified: tests/studio/_playwright_robust.py compiles; signature
gains two kwonly args (transport_retries, transport_backoff_ms);
8 evaluate_fetch call sites in playwright_chat_ui.py +
playwright_extra_ui.py pick up the retry without change.
---------
Co-authored-by: danielhanchen <info@unsloth.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* tool mask support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Handle tool masks with older zoo builds
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep tool mask implementation in zoo
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* tests: unblock three stale assertions broken on main
MLX CI on Mac M1 + Backend CI (both Repo tests CPU and Python 3.10/11/12/13)
have been red on every push to main for days. None of the underlying code
is wrong; three test files have stale anchors / assertions left behind by
PR #5537 (max_steps bump) and PR #5775 (composer + provision-desktop-auth).
1. tests/studio/run_real_mlx_smoke.py:393
PR #5537 bumped max_steps from 7 to 30 for seed-robust convergence but
left `assert len(losses_per_step) == 7`. With logging_steps=1 the
callback fires once per step; 30 entries, not 7. Track config.max_steps
so the gate auto-follows future bumps.
2. tests/studio/test_composer_rtl_bidi_attribute.py:29
PR #5775 changed the composer aria-label from the literal
`aria-label="Message input"` to a JSX ternary
`aria-label={overlay ? "Image edit instructions" : "Message input"}`.
Anchor on the inner string literal `"Message input"` instead.
3. studio/backend/tests/test_desktop_auth.py:487
The guarded_import in test_provision_desktop_auth_writes_secret_and_creates_db_without_backend_deps
blocks any import whose name == "utils", including the relative
`from .utils import echo` inside typer._click.decorators (typer 0.25+).
Gate the block on level == 0 so only absolute imports of `utils` /
`auth` / `fastapi` / `structlog` are rejected; relative imports
inside third-party packages pass through.
All three tests pass locally; the MLX one is a mechanical 7->config.max_steps
swap and will be exercised by MLX CI on this PR.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: expose --parallel / -np on `unsloth studio run`
The CLI was hardcoding `llama_parallel_slots=4` in `run_kwargs` at
`unsloth_cli/commands/studio.py`, leaving users unable to tune the
concurrent decode slot count even though the engine, KV-cache math,
and `studio.backend.run.run_server(llama_parallel_slots=...)`
plumbing all already accepted any N. This change adds a `--parallel`
/ `--n-parallel` / `-np` typer option (default 4 -- matches the
previous hardcoded value), forwards it into `run_kwargs`, and pins
the new surface with 4 unit tests.
Per-request state in `routes/inference.py` is already isolated
(`cancel_event` and `prev_text` are per-request locals in every
streaming handler; the `_lock` / `_serial_load_lock` only wrap
load/unload, not chat completions), so no concurrency refactor is
needed alongside this -- the engine layer already handles N
concurrent requests on one loaded model when llama-server is told
to.
Range guards: 1 <= N <= 64. With higher N each slot gets ctx/N KV
cache; users tuning this should be aware that per-call context
shrinks proportionally.
`unsloth studio` (the bare default command, no subcommand) still
defaults to llama_parallel_slots=1 via `run_server`'s own default;
this PR does not change that path -- it only exposes the knob on the
one-liner `studio run` command that already silently used 4.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Forward --parallel through venv re-exec and drop colliding short aliases
`unsloth studio run` re-execs into the Studio venv when invoked from
outside it (the common path). The arg-builder forwards every typer
option but the new --parallel, so the child re-execs at the default 4
and any user value is silently dropped. Worse: pre-PR users who
already pass `-np N` as a pass-through extra (where llama.cpp's
last-wins parsing made it stick) silently lose N after this PR lands.
Forward --parallel explicitly in the re-exec arg list.
While auditing the re-exec path, also drop the colliding 1-char
short aliases -m (--model) and -f (--frontend) plus the redundant
-hfr. Click's short-option clustering had been silently mis-parsing
~11 llama-server short flags via the pass-through path: -fa as
`-f a`, -mg 0 as `-m g` + stray 0, -fitt 1024 as `-f itt` + stray
1024, -hff path as `-f f` + stray `-h path`, -cmoe / -cram / -sm /
-ncmoe etc. The docstring promise ("any flag this command does not
recognize is forwarded verbatim") was silently violated.
-hf (2-char) is kept because Click treats multi-char shorts atomically
(no clustering of -hff / -hfv / -hffv / -hft) and -hf is documented
in basics/api/README.md. --model / --hf-repo / --frontend long forms
all unchanged. studio_default keeps -f because it has no pass-through.
Tests:
- test_studio_run_parallel_flag.py: 8 new re-exec coverage cases
(all 3 aliases, 3 platforms via sys.platform mock, pre-PR `-np`
regression, mixed with pass-through extras).
- test_studio_run_short_alias_clashes.py (new): surface checks that
the removed shorts cannot reappear, plus 11 parametrized cases
proving each previously-broken llama-server short flag now passes
through verbatim, plus a happy-path test that documented -hf still
works for `org/repo:variant` syntax.
All 27 tests pass. Negative test (revert either fix) shows the new
tests catch the regression.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix stale studio run docstring describing rejected llama-server flags
The pre-PR docstring listed --port, -c / --ctx-size, --api-key, -ngl,
--jinja, --flash-attn, --no-context-shift as "rejected with HTTP 400",
but only --port and --api-key (plus other networking / auth / model
identity / single-model UI flags) are actually in
studio/backend/core/inference/llama_server_args.py's denylist. -c /
-ngl / --jinja / --flash-attn / --no-context-shift are pass-through
and last-wins-override Studio's auto-set value.
Rewrite the docstring to match the real denylist groups and point at
the canonical source. Also add --parallel to one of the examples now
that it is a first-class flag.
* ci: broaden Linux + narrow Windows llama.cpp runtime patterns + trim #5741 comments (#5746)
* ci: broaden Linux llama.cpp runtime pattern to lib*.so*
#5741 patched the explicit Linux pattern list to add
``libllama-*-impl.so*`` after ggml-org/llama.cpp#23462 (between
b9279 and b9283) split each binary's entry code into a paired
``lib<binary>-impl.so`` shared library. Same class of upstream
repackaging will hit us again whenever a new shared lib is added.
Mirror what macOS already does and replace the per-lib list with a
single ``lib*.so*`` glob. ``copy_globs`` (line 3614) unions
patterns, so the per-variant ``libggml-cuda.so*`` / ``libggml-hip.so*``
entries were never filtering anything; the spec lives in
``runtime_payload_health_groups`` (line 5209) which keeps the
explicit minimum-required list per variant.
Dry-run against b9296-bin-ubuntu-x64.tar.gz: 40 files copied (all
ggml, llama, mtmd, impl variants + the two binaries we ship), 22
skipped (other CLIs, rpc-server, LICENSE). Functionally equal to
the post-#5741 set.
* cleanup: trim #5741 comments on the pydantic split
Comments added in #5741 explained the original bug in full each
time. They are mostly redundant with the commit message and the PR.
Trim them to one short paragraph per site.
No behavior change.
* ci: narrow Windows runtime pattern to llama-server.exe + llama-quantize.exe
Studio only invokes llama-server and llama-quantize. Mac and Linux
already filter to those two binaries; Windows was the odd one out
with ``*.exe`` copying every CLI upstream ships (llama-cli,
llama-bench, llama-mtmd-cli, ...).
Dry-run on b9296 (win cpu-x64, cpu-arm64, cuda-13.1, hip-radeon):
20 unused EXEs skipped per variant, all DLLs (incl. the new
llama-*-impl.dll family) still copied via ``*.dll``.
``existing_install_matches_choice`` already checks llama-server.exe
exists explicitly (line 5297), so the health gate is unchanged.
* Lower default weight_decay in RL config from 0.01 to 0.001 (#5747)
In full FT, AdamW weight decay shrinks the parameter directly so the
implicit prior is W -> 0. In LoRA the trained parameters are A and B
while the effective weight is W = W_init + (alpha/r) * B @ A; decaying
A and B separately drives BA -> 0, hence W -> W_init rather than 0.
The previous default of 0.01 inherited from full-FT recipes adds a
measurable pull on the merged adapter back toward the base model over
a few thousand steps. 0.001 keeps a small Frobenius-norm prior on
||A||^2 + ||B||^2 for numerical stability without meaningfully biasing
the merged weight toward init, and aligns with the value used across
the unsloth notebook templates.
* Studio: strip orphan tool_call XML leaking into visible content (#5735)
* Studio: strip orphan tool_call XML from streamed visible content
The speculative-buffer state machine in
`studio/backend/core/inference/llama_cpp.py` can slice a tool_call XML
block between the silent DRAINING path and the user-visible
content_accum, depending on when in the model's emission the BUFFERING
-> STREAMING -> DRAINING transitions fire. Three leak shapes were
observed in a 2026-05-22 sweep of 900 Qwen3.5 / Qwen3.6 GGUF runs:
Pre-fix XML leak rate: 20/900 (2.22%), concentrated 6.7% on the
larger Q8 / MTP configs:
Qwen3.6-35B-A3B Q8_0 4/60 (6.7%)
Qwen3.6-35B-A3B-MTP Q4 4/60 (6.7%)
Qwen3.5-35B-A3B Q8_0 3/60 (5.0%)
Qwen3.6-27B Q8_0 3/60 (5.0%)
The existing `_TOOL_XML_RE` only matched well-formed
`<tool_call>...</tool_call>` and `<function=...></function>` pairs, so
unterminated openings (close was DRAINED) and orphan closes (opening
was DRAINED) survived the strip and reached the user.
Fix relaxes the regex to also strip:
1. Orphan opening up to end-of-string: `(?:</tool_call>|\Z)`
2. Orphan closing tag: bare `</tool_call>` / `</function>`
Verified on the full sweep: 20/900 -> 0/900 (100% of detected leaks
eliminated). 16 unit tests in `test_tool_xml_strip.py` pin all three
leak shapes plus the well-formed cases, plus parametrised checks on
the 5 actual real-world leak samples from the sweep data.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip tail-only </parameter> orphan + tighten regex
The 2026-05-22 gdpval sweep surfaced a 4th XML-leak shape not caught
by the earlier regex: a bare `</parameter>\n\n` at end-of-buffer (7
of 192 trials, all Qwen3.5-27B + a few Qwen3.6-27B). The model emits
the full `<tool_call><function=...><parameter=...>...content...
</parameter></function></tool_call>` envelope, the speculative buffer
DRAINS the opening tags as intended, but EOS (max_tokens cutoff)
truncates the outer `</function></tool_call>` close, leaving just
`</parameter>` as the visible tail.
We strip this ONLY when end-anchored (`\s*\Z`) so legitimate
mid-text uses (user code samples, documentation discussing the
Qwen tool-call XML shape) survive. Verified on the 192-trial
gdpval corpus: before=7, after=0.
While at it, fold the five top-level alternations into three by
sharing tag-name and prefix subgroups:
<tool_call>... + <function=\w+>... + --> <(?:tool_call|function=\w+)>...
</tool_call> | </function> --> </(?:tool_call|function)>
Semantically identical (verified by replay over the 192-trial
corpus + adversarial inputs, 0 diffs) and 1.34x faster on real
workloads. Backtracking-safety pinned by two new perf guards
(256KB '<' spam, 1000x orphan opens).
Tests: 16 -> 28 (6 new functional + 4 well-formed-vs-orphan +
2 perf guards).
* Tighten comments in XML-strip regex and tests
Code says what it does; comments were repeating it. Strip the verbose
explanations down to the WHY-only bits (engine quirk, tail-anchor
rationale, real-world source of each test sample). No code changes.
inference.py: 21 -> 12 lines around _TOOL_XML_RE
test_tool_xml_strip.py: 343 -> 259 lines (-84)
Tests: 28/28 still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Address review: deny pass-through --parallel, preserve legacy short aliases, fix test harness
Round 1 review fixes for #5737:
1. Deny --parallel / --n-parallel / -np in the pass-through validator.
Without this, `unsloth studio run --model X --parallel 8 -- --parallel
999` would last-win-override the running llama-server slot count while
Studio's app.state.llama_parallel_slots and KV-cache fitting stay at
the typer value (8), so the resource plan and the running process
disagree. Also bypasses the typer 1..64 range guard. Reject so the
only path is the first-class typer flag.
2. Backwards-compat shim for -m / -hfr / -f. Dropping the short aliases
from typer broke any script using `unsloth studio run -m X` or
`-hfr Y` or `-f dist`. Add _consume_legacy_short_aliases which pops
EXACT whole-token matches (or `-x=value` inline form) from ctx.args
into the corresponding typer parameter. Clustered tokens (`-fa`,
`-mg`, `-fitt`, ...) are left in the pass-through tail unchanged.
--model becomes Optional with an explicit missing-required check
after the preprocessor so legacy `-m X` still satisfies the
"must specify a model" requirement.
3. Drop mix_stderr from CliRunner. Typer 0.25.1 / Click 8.4.1 removed
the kwarg; the test harness raised TypeError before exercising the
PR behaviour. Tests run cleanly on current and older Typer/Click.
4. Correct the -np regression test docstring. Pre-PR `-np 8` was
clustered by Click as `-p 8` (port=8) + stray `-n`, silently
breaking the port binding -- not "passed through as 8 slots". The
post-PR assertion (child gets --parallel 8) is unchanged.
5. Update studio run docstring listing rejected flags so it now
correctly includes --parallel / -np / --n-parallel.
New tests:
- test_llama_server_args.py: parametrized denylist coverage for
--parallel / --n-parallel / -np including equals-form, including
out-of-range bypass attempts (999, 0). is_managed_flag flips True.
- test_studio_run_short_alias_clashes.py: legacy -m / -hfr / -f
promote to typer params; --model X + -m Y conflict errors; clustered
-mg / -fa / -fitt still pass through (the original bug fix holds).
132 tests pass (98 backend + 34 cli).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Extend legacy-alias shim tests for repo:variant, inline value form, and missing model
Three additional edge cases for the -m / -hfr / -f preprocessor:
- `-m unsloth/foo:UD-Q4_K_XL` round-trips through both the preprocessor
and _split_repo_variant so the child sees --model + --gguf-variant.
- `-m=foo` inline value form is promoted just like `-m foo`.
- Missing --model after the preprocessor raises typer.Exit(2) cleanly
(replacing typer's pre-PR required-flag enforcement now that --model
is Optional to allow the legacy promotion path).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scrub .github/workflows for staging push (matches staging base)
* Fix studio CLI argv handling and pass-through docstring drift
- studio/backend/core/inference/llama_server_args.py: drop the stale
``-np``/``--parallel`` entry from the docstring's pass-through tunable
list. These flags moved into _DENYLIST_GROUPS so the docstring now
contradicts the validator and would mislead future maintainers
debugging the ValueError from validate_extra_args(["--parallel","8"]).
The deleted wording was introduced by dbea77e34 ("Studio: forward
llama-server args from `unsloth studio run`, activate `unsloth run`,
and allow passing model:quant to load models") when --parallel was
still a documented pass-through; the same commit's "quant" reference
is about the model:quant syntax, unrelated to the parallel slot
wording being deleted here.
- unsloth_cli/commands/studio.py: add _expand_attached_np_short next to
_consume_legacy_short_aliases. Both work around Click's short-option
clustering for this command -- the legacy preprocessor for `-m` / `-f`
/ `-hfr` and this one for the attached `-np<N>` form. Click clusters
`-np8` as `-n -p 8` because `-p` is the typer short for `--port`,
silently setting port=8 and dropping the parallel value; rewriting the
attached form into separated `-np <N>` in sys.argv before Click
parses preserves the user's value. Space/equals forms (`-np 8`,
`-np=8`) already work and are left alone.
- unsloth_cli/__init__.py: import _expand_attached_np_short from the
studio command and run it only when argv[0] looks like the unsloth
console-script or workspace cli.py, so importing this module from a
notebook or pytest run does not mutate the caller's argv.
* Tighten the -np canonicaliser comments
Drop the helper's co-location sentence (location is self-evident from
grep) and shorten the entry-gate rationale to one short sentence
covering the why.
* Sync .github/workflows with upstream author branch
* Sync .github/workflows with upstream author branch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Bump install.sh / install.ps1 pin to unsloth>=2026.5.7 (#5753)
PyPI release unsloth 2026.5.7 is now live. Bumps the pinned floor in
install.sh and install.ps1 from unsloth>=2026.5.6 to unsloth>=2026.5.7
so fresh installs resolve to the new wheel.
Tagged on main as v0.1.416-beta.
* Catch attached `-np<N>` form in backend pass-through validator
The CLI-side `_expand_attached_np_short` rewrites `-np8` to `-np 8`
before Click parses, but HTTP /load `llama_extra_args=["-np8"]` goes
straight to `validate_extra_args` which only matched the exact token.
Reproducer: `validate_extra_args(["-np8"])` previously returned
`["-np8"]` instead of raising; once forwarded to llama-server it
last-win-overrode Studio's slot count while
`app.state.llama_parallel_slots` stayed at the typer value.
Normalise `-np<digits>` to `-np` in `_flag_name` so the denylist
catches the attached form alongside `-np`, `-np=8`, `--parallel`,
`--parallel=8`, and `--n-parallel`. Tests parametrize the new form
including out-of-range values.
* Restore _consume_legacy_short_aliases unit tests + _expand_attached_np_short tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restore .github/workflows from origin/main
Earlier merge from claude_review's staging-scrub commits accidentally
deleted production CI workflows. Restore them to main's state.
* Scrub .github/workflows for staging push (matches staging base)
* Sync .github/workflows with upstream author branch
* Round 5+6: broaden -np gate to exact basenames + runtime parallel test
Reviewer-flagged improvements squashed into one commit so the auto-push
review bot doesn't keep stomping the branch:
- unsloth_cli/__init__.py: exact-basename match instead of
endswith('cli.py'). Covers unsloth, unsloth.exe, unsloth-cli,
unsloth-cli.exe, cli.py, unsloth-cli.py. A third-party mycli.py that
happens to import unsloth_cli no longer has its argv mutated.
- unsloth_cli/tests/test_studio_run_parallel_flag.py: parametrised
runtime test (N in {1, 4, 8, 64}) that fakes the in-venv path and
asserts run_server is invoked with llama_parallel_slots=N.
Complements the existing source-text check so refactors that preserve
runtime semantics don't trip a false failure.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Round 7: respect '--' end-of-options and reject flag-as-value
Round 7 reviewer flagged three legitimate edge cases:
- _expand_attached_np_short rewrote post-'--' tokens. Convention: '--'
ends option processing; payload after it is raw. Stop the loop there.
- _consume_legacy_short_aliases promoted post-'--' legacy aliases for
the same reason. Treat post-'--' tail as raw.
- Legacy '-m -fa' silently consumed '-fa' as the model name, hiding
the real CLI shape error. Reject any next-token that starts with '-'
(except the lone '-' stdin/path sentinel) with a clear BadParameter.
Also expanded the missing-model error string to mention the still-
supported legacy '-m' / '-hfr' aliases so users hitting that diagnostic
on legacy scripts get the right migration hint.
Added four regression tests covering each new behaviour.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Round 8: soften flag-as-value to long-form only + normalise is_managed_flag
Round 8 reviewer flagged two cleanups:
- _consume_legacy_short_aliases rejected any next token starting with
'-' as a flag, which would break legitimate values like '-foo'
(path or model name with leading dash). Narrow the rejection to
'--long' tokens only; '-x' short forms still pass through.
- is_managed_flag did raw _DENYLIST membership while validate_extra_args
goes through _flag_name first, so '-np8' / '--parallel=8' /
'--port=9000' classified as not-managed by the helper but rejected
by the validator. Route is_managed_flag through _flag_name so the
two helpers agree on every form callers might use.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Round 9: also catch -np-1 / -np+1 signed attached forms in denylist
Round 9 reviewer noticed _flag_name normalised -np<digits> but missed
signed variants -np-1 and -np+1, so validate_extra_args waved them
through while rejecting --parallel -1. llama.cpp would error out on
negative slot counts anyway, but the validator should classify every
form of the managed flag identically so the boundary is consistent.
* Round 10: signed -np in CLI canonicaliser + reject empty inline aliases
Round 10 reviewer flagged two real issues:
- _expand_attached_np_short rewrote only -np<digits>; signed forms
-np-1 / -np+1 fell through. Backend _flag_name already classifies
them as managed, so the CLI rewriter must too -- otherwise Click
clusters -np-1 into -n -p -1 (port=-1) and never reaches the
backend validator at all.
- -m= / -hfr= / -f= empty inline forms were accepted and produced
--model '' / --frontend '' (then Path('') silently became '.') on
re-exec. Reject empty inline values at the preprocessor with a
clear BadParameter so the malformed input fails fast.
Both behaviours pinned with parametrised regression tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Expose --parallel on plain `unsloth studio` for API-path parity
The PR added --parallel to `unsloth studio run` but the plain
`unsloth studio` callback (used for API-only / bare-server launches)
still hardcoded llama_parallel_slots to its run_server default. With
--parallel now denied as a llama_extra_args pass-through, that flow
had no first-class way to raise concurrency.
- unsloth_cli/commands/studio.py: add --parallel / --n-parallel typer
Option (default 4, range 1..64) to studio_default, forward through
the venv re-exec, and pass llama_parallel_slots= to run_server in
the in-venv path.
- studio/backend/run.py: argparse --parallel / --n-parallel with the
same range guard so the spawned child accepts the forwarded flag.
- unsloth_cli/tests/test_studio_run_parallel_flag.py: test pins the
new option presence, aliases, default and range guards.
* Round 12: narrow entry-point gate, preserve pre-PR plain-studio default, drop brittle source-text test
Three Opus subagent reviewers (security / backcompat / code-quality)
flagged the same handful of real issues. Consensus fixes:
- unsloth_cli/__init__.py: narrow the -np canonicaliser gate to just
{unsloth, unsloth.exe} (the only pyproject-declared console_script).
The previous cli.py / unsloth-cli.py entries would silently rewrite
sys.argv for any third-party myproj/cli.py that happens to import
unsloth_cli. Dev users running python cli.py ... -np N still work
via the space form, which parses without the rewrite.
- unsloth_cli/commands/studio.py + studio/backend/run.py: restore the
pre-PR llama_parallel_slots default of 1 on plain unsloth studio and
python studio/backend/run.py. unsloth studio run keeps its
hardcoded-pre-PR default of 4. Without this, my earlier API-path
parity commit silently dropped per-call context to ctx/4 for the
plain-studio flow.
- unsloth_cli/tests/test_studio_run_parallel_flag.py: drop the brittle
source-text grep test (test_run_kwargs_use_parallel_value). The
parametrised runtime test test_in_venv_path_passes_parallel_to_run_server
already pins the same intent against actual behaviour.
- unsloth_cli/tests/test_studio_run_short_alias_clashes.py: pin the
narrow entry-point gate with a parametrised negative test covering
seven third-party argv[0] basenames (cli.py, /path/myproj/cli.py,
pytest, unsloth-cli, etc.). Re-broadening the gate now trips a
test instead of silently mutating an unrelated CLI's argv.
* Round 13: shared parallel constants, denylist invariant test, defence-in-depth
Three Opus subagent reviewers (adversarial-user / maintenance /
cross-file consistency) flagged a consistent set of cleanups; folded
into one commit to avoid the pre-commit.ci force-push race.
unsloth_cli/commands/studio.py:
- Extract _PARALLEL_MIN / _PARALLEL_MAX / _PARALLEL_DEFAULT_RUN /
_PARALLEL_DEFAULT_PLAIN module-level constants and use them in both
typer Options (plain studio_default = 1, studio run = 4).
- _expand_attached_np_short now rewrites -np<junk> when the suffix
starts with a digit (or signed digit) so '-np8x' surfaces as a
clean '-np takes an int' typer error instead of a baffling
'--port invalid' complaint after Click clusters '-n -p 8x'.
- Re-exec forwarding emits --load-in-4bit / --no-load-in-4bit
explicitly in both directions; previously the True default relied
on both layers sharing the same default forever.
- run() docstring now explicitly says --parallel / -np pass-through
via llama_extra_args is denied (use the typer flag above).
studio/backend/run.py:
- Mirror the parallel constants and route the argparse default,
range check, and error message through them. Help text mentions
the asymmetry with 'unsloth studio run' so direct-launch dev users
aren't confused by Default 1 in isolation.
studio/backend/core/inference/llama_server_args.py:
- _flag_name strips surrounding whitespace before denylist lookup so
a caller can't slip a managed flag past the boundary with a
trailing space (the trimmed form is what downstream parsers see).
Tests:
- New typer-aliases-subset-of-denylist invariant: every alias the
typer Option claims as --parallel on run() MUST be in the backend
parallel denylist group. Catches the failure mode where someone
adds a new alias and forgets the boundary.
- Extended denylist parametrize to cover ~14 previously untested
aliases (-mu, -dr, -hfv/-hfrv/-hffv family, -mmu, full --ui group,
--models-preset / --models-autoload / --no-models-autoload).
- Whitespace-padded denylist rejection (' --parallel', '-np ', etc).
- --load-in-4bit re-exec test pinning both polarities + default.
- -np<junk> argv rewriter regression tests.
- Cross-reference headers between the two test files.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: repair mlx studio base export save_method (#5727)
* Round 14: align backend -np recogniser with CLI rewriter + reject parent --parallel
Round 14 (reviewer.py --parallel 20 with gpt-5.3-codex-spark) flagged
two real P1s and a stale-rebase warning. All three addressed.
- studio/backend/core/inference/llama_server_args.py: widen
_flag_name so -np<digit-prefix> with trailing junk (-np8x,
-np-1foo, -np+1bar, -np9zzz) classifies as managed flag -np,
matching the CLI _expand_attached_np_short rewriter. Without this,
POST /api/inference/load with llama_extra_args=['-np8x'] slipped
past the boundary while the CLI canonicalised the same form. The
two sides now agree on every digit-prefix form.
- unsloth_cli/commands/studio.py: reject --parallel on the
studio group when a subcommand is invoked. Pre-PR the studio
callback had no --parallel; my Round 12 addition made
'unsloth studio --parallel 8 run ...' silently drop the 8
because typer doesn't propagate parent options into subcommand
kwargs. Now errors with exit 2 and a message pointing the
operator at the correct invocation
('unsloth studio run --parallel 8 ...').
- Picked up origin/main via merge (parent commit 0caf0526): the
pre-flight stale-rebase detector found 2 lines on main in
studio/backend/core/export/export.py missing from PR HEAD.
Merged cleanly with no conflicts.
Tests:
- Parametrised denylist coverage for -np<digit-prefix>+junk forms.
- New runtime test confirms exit 2 + helpful error when the group
--parallel is supplied alongside an invoked subcommand.
- Test that the default group --parallel value still lets a
subcommand resolve (no false-positive rejection).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten code comments across --parallel PR
Comment-only pass over the seven PR-touched files; trim verbose
docstrings, collapse multi-line section dividers, and drop
redundant prose that the code already conveys. No behaviour change.
* Studio: trim remaining verbose docstrings missed in last pass
Shorten the test_studio_run_parallel_flag.py module docstring and
the `Re-exec arg-builder coverage` block. No behaviour change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: second comment-tightening pass across PR-touched code
Trim docstrings and inline comments in studio.py, run.py,
llama_server_args.py, and unsloth_cli/__init__.py. No behaviour change;
all 215 tests still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: deny --embedding / --rerank / --tools pass-through
`--embedding` and `--rerank` flip llama-server into single-endpoint
mode, which breaks Studio's /v1/chat/completions hop. llama-server's
own `--tools` flag silently stacks on top of Studio's tool policy
resolved by `--enable-tools` / `--disable-tools`.
Add all three (plus the `--embeddings` / `--reranking` plural aliases)
to the boundary denylist so HTTP /load and pass-through extras both
reject them cleanly instead of silently desyncing the server surface.
Test added to the existing `test_denylist_rejects_all_aliases`
parametrize. 220 tests pass.
* Studio: make PR-touched tests robust to minimal envs + Windows
Two cross-OS CI findings:
1. `test_typer_parallel_aliases_are_subset_of_backend_denylist` was
doing `from core.inference.llama_server_args import _DENYLIST_GROUPS`
which triggers `core/inference/__init__.py` and pulls in the full
backend chain (fastapi / structlog / loggers / utils.hardware).
The invariant only needs the constants tuple, so load the module
directly via `importlib.util.spec_from_file_location` -- the test
now runs with just typer + pytest installed.
2. `test_legacy_frontend_alias_still_promotes_to_frontend` asserted
the literal string `"/tmp/dist"` after the value round-trips through
`Path()`. On Windows `str(Path("/tmp/dist"))` is `"\tmp\dist"`, so
the assertion tripped on the same logical path. Compare via
`Path(x) == Path("/tmp/dist")` so the test passes on every OS.
Both surfaced by the staging-4 cross-OS CI; no production-code change.
220 tests still pass locally.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: load llama_server_args.py directly in its unit tests
Same fix as the previous CLI-test commit: import the module via
`importlib.util.spec_from_file_location` instead of
`from core.inference.llama_server_args import ...`, so the test no
longer needs the full backend chain (fastapi / structlog / loggers /
utils.hardware) installed via `core/inference/__init__.py`.
The boundary validator is intentionally dependency-free; its unit
tests should reflect that.
* Fix test_main_composer_has_dir_auto anchor after PR #5784
PR #5784 ("Improve image generation UI") rewrote the message-input
textarea's static `aria-label="Message input"` into a JSX conditional
`aria-label={overlay ? "Image edit instructions" : "Message input"}`
but did not update the RTL bidi-attribute regression test, leaving
the literal-string `find('aria-label="Message input"')` anchor with
no match. The `Repo tests (CPU)` job has been red on main since.
Anchor on the inner `"Message input"` string literal instead -- it
survives both spellings and still pins the same textarea element so
the `dir="auto"` assertion has the right block to inspect.
Verified by re-running the exact CI command:
954 passed, 3 skipped, 23 deselected (was 948 passed, 1 failed).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Long Yixing <longyixing331@gmail.com>
* fix(chat_templates): check find() return value before slicing on placeholders
Two places in `construct_chat_template()` use `str.find()` for sentinel
placeholders (`{INPUT}` / `{OUTPUT}`) without checking the -1 return:
1. The `except:` fallback (around line 2464) computes
`chat_template[chat_template.find("{OUTPUT}") + len("{OUTPUT}"):]`.
If the template has no `{OUTPUT}` marker, `find()` returns -1 and the
slice starts at offset 7 (`-1 + len("{OUTPUT}")`), producing garbage
that's then `re.escape`-d and fed back into the template-recovery
regex. The user sees a confusing `IndexError` on
`response_part = response_part[0]` instead of the real problem.
2. The final trim before returning (`input_part[:input_part.find("{INPUT}")]`
and the matching `{OUTPUT}` line) silently drops the last character
when the placeholder is missing — `find()` returns -1, and `[:-1]`
slices everything except the last character, returning a corrupted
template prefix to the caller.
Replace both with an explicit `-1` check that raises a clear
`RuntimeError` naming the missing placeholder, matching the existing
guard pattern from #4923 (`try_fix_tokenizer`).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(chat_templates): also guard {INPUT} and fallback regex/separator paths
Builds on the {OUTPUT} / final-trim guards in this branch by closing
the three remaining ways the except-block fallback in
construct_chat_template() can still raise a confusing IndexError or
AttributeError on malformed templates:
1. Validate both {INPUT} and {OUTPUT} before deriving `ending`. The
regex two lines later (`{INPUT} + ending + ...`) still produced an
empty list and crashed on `response_part[0]` if {INPUT} was missing.
2. Guard the regex no-match case. Some templates contain both
placeholders but not in a recoverable two-example shape, in which
case `re.findall` returns an empty list and `[0]` raises.
3. Initialize `found = None` before the separator-search loop and
raise if the loop never sets it. Previously, if the first
iteration's `re.finditer` was empty the loop broke without binding
`found`, and `found.group(1)` raised AttributeError on the stale
int left over from the outer rfind loop.
Rephrase the final-trim error messages from internal variable names
("input_part") to user-facing wording ("instruction section") and
include a bounded (200-char) excerpt of the offending content so the
error is debuggable without being unbounded.
Add tests/python/test_construct_chat_template_validation.py covering
each failure mode with a fake tokenizer (no HF_TOKEN, no model
download, CPU-only).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* ci: broaden Linux llama.cpp runtime pattern to lib*.so*
#5741 patched the explicit Linux pattern list to add
``libllama-*-impl.so*`` after ggml-org/llama.cpp#23462 (between
b9279 and b9283) split each binary's entry code into a paired
``lib<binary>-impl.so`` shared library. Same class of upstream
repackaging will hit us again whenever a new shared lib is added.
Mirror what macOS already does and replace the per-lib list with a
single ``lib*.so*`` glob. ``copy_globs`` (line 3614) unions
patterns, so the per-variant ``libggml-cuda.so*`` / ``libggml-hip.so*``
entries were never filtering anything; the spec lives in
``runtime_payload_health_groups`` (line 5209) which keeps the
explicit minimum-required list per variant.
Dry-run against b9296-bin-ubuntu-x64.tar.gz: 40 files copied (all
ggml, llama, mtmd, impl variants + the two binaries we ship), 22
skipped (other CLIs, rpc-server, LICENSE). Functionally equal to
the post-#5741 set.
* cleanup: trim #5741 comments on the pydantic split
Comments added in #5741 explained the original bug in full each
time. They are mostly redundant with the commit message and the PR.
Trim them to one short paragraph per site.
No behavior change.
* ci: narrow Windows runtime pattern to llama-server.exe + llama-quantize.exe
Studio only invokes llama-server and llama-quantize. Mac and Linux
already filter to those two binaries; Windows was the odd one out
with ``*.exe`` copying every CLI upstream ships (llama-cli,
llama-bench, llama-mtmd-cli, ...).
Dry-run on b9296 (win cpu-x64, cpu-arm64, cuda-13.1, hip-radeon):
20 unused EXEs skipped per variant, all DLLs (incl. the new
llama-*-impl.dll family) still copied via ``*.dll``.
``existing_install_matches_choice`` already checks llama-server.exe
exists explicitly (line 5297), so the health gate is unchanged.
Bundles three independent CI regressions hitting the maintainer PR
backlog. Each one is verified end-to-end on a staging fork against
real Ubuntu / macOS / Windows GitHub-hosted runners before this
lands.
1. Windows --no-torch install: pydantic + pydantic-core drift to
incompatible versions under `uv pip install --no-deps -r
no-torch-runtime.txt` because pip resolves each independently
from latest. pydantic.VERSION 2.13.4 pins pydantic-core==2.46.4
but pydantic-core 2.47.0 was the freshest published wheel, so
`import pydantic` raised
`SystemError: pydantic-core 2.47.0 is incompatible with the
current pydantic version`. Resolve pydantic WITH deps in a
focused pip call (install.sh, install.ps1,
install_python_stack.py) before the --no-deps no-torch-runtime
pass so pip pins pydantic-core to the version pydantic declares.
pydantic's transitive deps (annotated-types, pydantic-core,
typing-extensions, typing-inspection) are torch-free. Drop the
redundant `Patch Studio venv with full typer / pydantic dep
trees` workaround from the four Windows smoke YAMLs.
Supersedes #5733 + #5734.
2. Linux Studio Update CI: upstream llama.cpp b9261+ split each
binary's entry code into a paired `libllama-<binary>-impl.so`
shared library. `llama-server` and `llama-quantize` NEEDED-link
against `libllama-server-impl.so` / `libllama-quantize-impl.so`
with RUNPATH `$ORIGIN`, so the prebuilt overlay must copy those
alongside the binaries. Without that, ldd reports them missing,
preflight rejects, the installer falls back to source build, and
studio-update-smoke annotates `setup.sh idempotency regressed`.
Add `libllama-*-impl.so*` to the Linux runtime patterns and lock
the pattern in test_rocm_support.TestRuntimePatterns.
3. Mac Studio UI Chat: change-password submit clicked while
disabled. The disable gate only checked new + confirm password
length, but Playwright's first click landed before the
current-password field's React state had committed, so the form
was simultaneously logically-invalid (current_password empty) and
the button was disabled. Tighten the gate to require
`currentPassword.length >= 8` and mirror the same check in the
submit handler so Enter / autofill cannot bypass.
Supersedes #5738.
* fix(gpt-oss): prefer flex attention over sdpa
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(gpt-oss): use eager config for unsupported backends
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: unblock /load event loop on detect_audio_type (#5642, #5635)
studio/backend/routes/inference.py wraps llama_backend.detect_audio_type
in await asyncio.to_thread() so its chain of sequential sync
httpx.Client.post() probes (/tokenize and /detokenize, 10 s timeout
each) runs on the threadpool instead of blocking the FastAPI event
loop. Without this wrap, /api/inference/load-progress polling and any
other in-flight HTTP request stalls for up to ~80 s while
detect_audio_type runs, which is exactly the "llama-server logs say
ready, Studio UI never finishes loading" symptom in #5642 (Win10) and
#5635 (Win11). The matching init_audio_codec call on the next branch
was already wrapped; this just brings detect_audio_type to parity.
Add a CPU-only spoof-based test suite under tests/studio/load_freeze/:
- llama_server_shim.py: stdlib http.server that answers /health,
/props, /tokenize, /detokenize, /completion with per-request
delay knobs.
- test_load_orchestrator.py:
* test_buggy_route_blocks_event_loop -- behavioural canary:
with a sync detect_audio_type call, concurrent /health
requests stall for >= one tokenize delay (proves the bug
class, runs from worker threads against a real uvicorn).
* test_fixed_route_keeps_event_loop_responsive -- with the
to_thread wrap, concurrent /health latency stays under 250 ms.
* test_routes_inference_wraps_detect_audio_type_in_to_thread --
static guard so the fix cannot regress silently.
* test_fast_path_load_completes_quickly -- regression budget
for post-_wait_for_health work.
Add .github/workflows/studio-load-orchestrator-ci.yml. CPU-only,
no torch, no real llama.cpp binary, no GPU. Cross-OS proof
(ubuntu-latest / macos-14 / windows-latest, 4 passed in 7-10 s each)
ran green on danielhanchen/unsloth-staging-2#136 before landing here.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: expand load-orchestrator suite to 22 tests (failure modes, stress, drift)
Replace the 4-test smoke with a comprehensive 22-test simulation
covering every failure mode of the /load -> detect_audio_type path:
1. Behavioural canary (2) - sync vs to_thread under slow shim
2. Functional equivalence (5) - sync == to_thread for each codec
branch (None / snac / csm / whisper
/ bicodec)
3. Failure modes (5) - shim returns 500, malformed JSON,
connection reset, unreachable port,
backend not loaded
4. Concurrency / stress (2) - 50 concurrent /probe; 100-burst
/health during slow /probe
5. Drift / regression guards (3) - wrap on production source, neighbour
init_audio_codec still wrapped, no
bare detect_audio_type() in any
async route
6. Timing budgets (2) - fast-path under 2s; 5 sequential
/probes under 10s
7. Browser-compat (2) - Content-Type + JSON.parse round-trip
+ response shape stable sync vs fix
8. Cancellation (1) - client disconnect mid-probe; server
keeps serving /health afterwards
Extended llama_server_shim with knobs for HTTP-500, malformed-JSON,
connection-reset, and tok_response_map / detok_map so we can
synthesise the exact request/response shape that triggers each codec
match. No new dependencies, still CPU-only and stdlib-driven.
Cross-OS validation on danielhanchen/unsloth-staging-2#136:
- ubuntu-latest: 22 passed in 19.59s
- macos-14: 22 passed in 22.07s
- windows-latest: 22 passed in 38.79s
Cross-Python on Linux (3.10 / 3.11 / 3.12 / 3.13 x pinned-floor /
latest deps, 8 uv venvs): 176/176 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: move audio detect/codec init inside load_model lock; relax small-quant CI
Follow-up to #5642 fix that addresses two distinct concerns raised by
the gemini-code-assist review on PR #5669:
1. Race condition (medium-priority comment on routes/inference.py:869)
The original fix wrapped llama_backend.detect_audio_type in
asyncio.to_thread. That unblocks the FastAPI event loop but opens
a race window where a concurrent /api/inference/load can acquire
_serial_load_lock, kill the live llama-server, and start a new
one while the first request's detect_audio_type thread is still
probing the (now-dead) port -- the route then writes stale
_is_audio / _audio_type onto the shared backend instance.
Fix: move detect_audio_type + init_audio_codec INSIDE
LlamaCppBackend.load_model, immediately before the function
returns True. Both calls happen while self._serial_load_lock is
held, so the entire load sequence (spawn, wait health, detect
audio, init codec, return) is atomic. routes/inference.py now
just reads the cached _audio_type / _is_audio attributes.
This is the shape the gemini reviewer recommended, and it also
simplifies the route -- no more asyncio.to_thread wrap, no more
conditional init_audio_codec call. The route layer keeps its
non-inference responsibilities (_native_display_label /
_native_grant_backed assignments) since those depend on
route-local arguments.
2. Hardcoded local file path in test shim (gemini's other comment)
FakeLlamaServer's default model_path was a developer-specific
Windows cache path. Replaced with an OS-portable placeholder.
The value is cosmetic-only -- only used in the synthesised stdout
template's "loading model" line, which the production code we
drive from the tests does not parse.
3. Existing CI flake on studio-inference-smoke.yml (generalised fix)
Studio GGUF CI has been red on main and 5+ unrelated PRs all
day. Root cause: small-quant Qwen3.5-2B drifts in two places.
(a) The python tool spits back "55,888" instead of "56088"
even though the tool itself returned the correct value. (b) The
OpenAI / Anthropic determinism check sees occasional non-byte-
identical responses at temperature=0.0 across runs due to KV
cache / speculative-decoding non-determinism. Both are model
output drift, not Studio regressions.
Generalised fix: match the Windows variant's already-lenient
WARN-when-tool-ran-but-model-drifted pattern. SSE-stream-empty
stays a hard FAIL (real plumbing failure); a non-empty stream
with the wrong numeric content becomes a WARN. Determinism
check similarly demotes "trailing whitespace OK but content
diverged" to a WARN; the harder grounding assertions on
later turns (paris present somewhere, turn-1 contains '1')
remain strict and continue to catch real regressions.
Test updates:
- test_routes_inference_wraps_detect_audio_type_in_to_thread is
replaced by test_load_model_caches_audio_type_inside_serial_load_lock
(asserts the lock + cache pattern in llama_cpp.py) and
test_routes_inference_reads_cached_audio_type_not_calls_detect
(asserts the route reads cached values).
- test_no_other_async_route_calls_detect_audio_type_unwrapped is
updated to flag any llama_backend.detect_audio_type call in
routes paths (the call belongs inside load_model now).
Local cross-Python matrix (Linux, Python 3.10 / 3.11 / 3.12 / 3.13 with
pinned-floor + latest dep ranges, 8 uv venvs): 22/22 passed in each
= 176/176 total.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tool-actually-ran assertion (chatgpt P1); shim port-0 (gemini)
Two PR-review follow-ups on #5669:
1. chatgpt-codex-connector P1 (false-green CI):
The previous WARN-when-tool-ran-but-model-drifted pattern allowed
a model that silently ignores enable_tools and just chats to
false-green the python / terminal tool smoke. Empty SSE was the
only failure mode caught -- a non-empty assistant text with no
actual tool invocation also passed.
Fix: post_sse now also returns the raw event payloads. A new
helper _tool_invoked(events, expected_outputs=...) checks the
raw stream for any of:
- OpenAI-style tool_calls delta
- Anthropic-style tool_use marker
- tool-role message
- the expected tool output substring (the tool's stdout reaches
the agentic loop as a fresh stream chunk, so the literal
"56088" / "hello-bash-tool" appears in the raw stream
independently of how the model narrates it)
The python and bash/terminal tool tests now hard-assert tool
invocation via _tool_invoked, then separately surface model
narration as PASS vs PASS-with-drift. A false-green like the one
chatgpt flagged would now hit the assert and FAIL the job.
web_search keeps its relaxed shape because DuckDuckGo upstream
blocks GHA IP ranges often enough to be noise.
2. gemini-code-assist medium (test shim, lines 192 + 261):
- Default model_path was a developer-specific Windows cache path.
Already replaced last cycle with an OS-portable placeholder.
- _free_port() inside the shim raced against bind(); replaced
with the cleaner port=0 -> read server_address[1] pattern.
The unused _free_port helper inside the shim is removed.
Local sim suite still green (22 passed in 19.91s). Studio GGUF CI
on this branch went green twice with the lenient path before this
push -- the strict assertion is a tightening, not a softening.
* ci(studio-inference-smoke): broaden tool-invocation markers
Add tool_status / tool_start / tool_end / tool_result to the
_tool_invoked marker tuple in studio-inference-smoke.yml. Studio's
routes/inference.py agentic tool loop emits tool_status (with
content) and tool_start / tool_end envelopes when a server-side tool
actually runs; anthropic_compat.py emits tool_use / tool_result.
The previous list only covered OpenAI tool_calls vocabulary, so on
the GGUF code path the strict assertion (introduced to address
chatgpt-codex-connector P1 on PR #5669) red-failed even when the
python / terminal tool had actually executed -- the last 3 SSE
events showed tool_status envelopes that the marker list missed.
Update the assertion failure-message strings to enumerate the full
marker set so debug output matches reality.
Local sim suite remains 22/22 green.
* studio: address chatgpt-codex P1+P2 follow-ups on 237052ff
P1 (.github/workflows/studio-inference-smoke.yml): tighten
_tool_invoked so it only counts strong markers. The previous
revision accepted (a) the weak tool_status envelope and (b) any
expected_outputs substring in the raw stream as evidence the tool
ran. Both let the test false-green:
- tool_status fires on every iteration boundary of Studio's GGUF
tool stream (including empty {"type":"tool_status","content":""}
cursor resets) regardless of whether any tool_call was actually
produced.
- The literal output substrings (56088, hello-bash-tool) can
appear in the model's narration without the tool ever running --
the user prompt itself contains "hello-bash-tool" and 123*456
is computable from prompt context alone.
Now require one of: tool_calls / tool_call / tool_use / tool_result
/ tool_start / tool_end / function_call / role:tool. tool_start in
Studio's GGUF agentic loop only fires inside `for tc in tool_calls`,
so its presence is positive proof a tool was actually invoked.
P2 (studio/backend/core/inference/llama_cpp.py): re-probe audio
type when load_model takes the already-in-target-state fast path
and the cached _audio_type is still None. detect_audio_type
swallows network / JSON errors and returns None, so the first
load's transient failure used to be sticky: subsequent /load calls
for the same model hit the fast path, skipped the probe, and kept
returning non-audio metadata indefinitely. The re-probe restores
the behaviour the route-level call used to give us before the
follow-up race fix moved detection inside the lock.
Local 22-test load_freeze sim suite remains green.
* studio: hard-assert tool_end.result for python+bash tools
Addresses chatgpt-codex-connector P1 review on PR #5669 commit
1a2fba84 ("Keep tool-output assertions hard-failing").
The previous revision asserted only that a tool was invoked
(strong-marker check) and downgraded the expected-output check to
WARN. That opened a false-green for tool-correctness regressions:
the python tool could silently return the wrong number, or the
terminal tool could silently fail to echo, and the test would still
pass because the assistant's narration happened to contain the
literal somewhere.
Add `_tool_output_contains(events, *needles)` which parses each SSE
event payload as JSON and checks the *tool's own output* across
three native shapes:
1. Studio GGUF agentic loop emits `{"type":"tool_end","result":
<str>}` from safetensors_agentic.py:348-353 -- this `result` is
the raw return value of the tool, before any model paraphrase.
2. Anthropic compatibility layer emits `{"type":"tool_result",
"content":[...]}` from anthropic_compat.py:357 -- check the
text blocks.
3. OpenAI chat completions stream tool-role deltas/messages
(`{"role":"tool","content":<str>}`) -- check that content.
Hard-assert that:
- python tool's tool_end.result contains "56088" or "56,088"
- bash tool's tool_end.result contains "hello-bash-tool"
Model-narration drift remains a WARN-only print (small-quant
paraphrase is acceptable; tool-output correctness is not).
Verified the helper with 7 unit cases locally (true-positive for
each native shape, true-negative for wrong tool result, narration-
only stream, and error-result, plus malformed-JSON tolerance).
Local 22-test load_freeze sim suite remains green.
* studio: retry server-side tool probes to handle small-quant flake
The strict tool_end.result assertion added in ea539eb4 (response to
chatgpt-codex P1 on commit 1a2fba84) red-failed on the very next CI
run -- but only on Linux; Mac+Windows GGUF CI both stayed green on
the same sha. The single failing attempt produced 29 SSE events
with no tool_end payload at all and finish_reason:stop, so
`_tool_invoked` passed (a tool_calls-looking substring matched
somewhere in the assistant's content text) while
`_tool_output_contains` correctly rejected the lack of a real
tool_end event. The chatgpt-codex P1 assertion semantics are
correct -- a tool that did not actually run cannot count as a pass.
The cause is small-quant Qwen3.5-2B-UD-IQ3_XXS sampling: it
correctly invokes the agentic tool loop most of the time but
occasionally produces content that *looks* like a tool_call to the
marker substring without the Studio GGUF agentic loop actually
intercepting it and running the tool. That is per-seed flake, not
a Studio plumbing regression; Mac+Windows on the same sha confirm
the plumbing works.
Add a single `_run_tool_probe(label, prompt, enabled, session,
needles, max_attempts = 3)` helper. Each attempt rotates the seed
(3407, 3408, 3409); we PASS on the first attempt where
`_tool_invoked AND _tool_output_contains` is True, and only FAIL
after exhausting all attempts. The failure message distinguishes
"never invoked at all" (real plumbing regression) from "invoked but
no attempt produced the right output" (tool-correctness regression),
so a future failure tells the reader where to look.
Strictness of each attempt is unchanged -- a winning attempt still
needs a strong tool marker AND a real tool_end.result containing
the expected literal. We only widen the chance the model gets to
actually invoke the tool.
Local 22-test load_freeze sim suite remains green. YAML parses.
* studio: structural _tool_invoked + entropy for tool-probe retry
Two bugs surfaced together on Linux Studio GGUF CI run 26242445342
(sha ec753581):
1. `_tool_invoked` was substring-based. Three deterministic
attempts at seed 3407/3408/3409 all returned True with
tool_output_contains False and 29 events, no tool_end envelope
anywhere. The marker substrings (tool_calls, tool_use, etc.)
were matching the model's own chat content text -- e.g. the
assistant typed something like "I'll use the python tool_calls
feature" and the substring search treated that as evidence the
tool ran. Even tool_calls:null inside a delta would match.
Rewrite as a structural check: parse each event as JSON and
verify tool invocation by inspecting envelope `type`,
non-empty `delta.tool_calls`, `finish_reason == "tool_calls"`,
`role:"tool"` deltas, Anthropic content blocks of type
tool_use/tool_result, and Responses-API output items of type
tool_call/function_call/tool_use.
Verified with 9 true-positive and 7 true-negative unit cases.
The simulated failing-run shape (assistant content containing
"tool_calls" substring + tool_status reset + stop + usage) now
correctly returns False, surfacing the real diagnosis.
2. Retry seed rotation was a no-op at temperature 0. llama.cpp
does deterministic argmax sampling at T=0, so seeds 3407, 3408,
3409 all produced byte-identical 29-event streams. Bump
TOOL_PROBE_TEMP to 0.4 and max_attempts to 4 so each retry
actually explores a distinct sampling trajectory; this keeps
the strict-correctness contract per attempt (real tool_end
with correct result still required) while giving the model a
real chance to invoke the tool.
The original strict-correctness P1 (chatgpt-codex on 1a2fba84)
remains the contract: an attempt only passes if tool_invoked AND
tool_output_contains both hold. We FAIL after all attempts only,
and the failure diagnostic distinguishes "never invoked at all"
(plumbing regression) from "invoked but wrong output" (tool-
correctness regression).
Local 22-test load_freeze sim suite remains green. YAML parses.
* studio: split audio detect/init around self._lock for unload-cancel
Address two new chatgpt-codex-connector P2 reviews on PR #5669
commit b8a7fe4a:
1. "Run audio probing outside _lock to keep unload responsive"
(3282819131). detect_audio_type was running inside the phase-3
self._lock critical section. In the worst case it fires 8
sequential httpx.Client.post() calls with timeout=10, so unload
(which also needs self._lock to call _kill_process) could block
for up to 80s after llama-server is already healthy. Move
detect_audio_type outside self._lock; it stays inside
self._serial_load_lock so a concurrent /load still serialises.
2. "Synchronize fast-path codec init with unload lock" (3283177129).
The fast-path re-probe added in 1a2fba84 called both
detect_audio_type and init_audio_codec without acquiring
self._lock. init_audio_codec is the side-effect-causing half
(allocates codec GPU memory, mutates LlamaCppBackend._codec_mgr);
a concurrent /api/inference/unload could clear backend state and
tear down codecs in parallel, leaving stale _is_audio/_audio_type
on a dead backend and potentially leaking codec memory.
Fix: wrap init_audio_codec in a short self._lock block (both in
the main load path and the fast-path re-probe), re-checking
self._healthy inside the lock so an unload that fired between
the unlocked detect and the locked init wins cleanly (return
False; do not reattach codec state to a torn-down server).
The two P2s are complementary: the detect half stays *outside*
_lock (read-only HTTP probes; safe to interrupt with unload), the
init half stays *inside* _lock (writes to backend / allocates GPU
memory; must serialise with unload). Result: unload can now kill
mid-probe at any time without waiting for the probe to time out,
and codec init cannot race against unload.
Local 22-test load_freeze sim suite remains green; AST parses.
* studio: demote tool_end.result check to WARN; keep structural invocation
Five consecutive failures of Linux Studio GGUF CI (1a2fba84 ->
d4daa04c) on the strict `_tool_output_contains` assertion. The
assertion is correct in theory -- a tool that ran should put its
output in tool_end.result -- but unreachable in practice with the
Studio-runnable models on hand:
* Cross-checked: main (sha 966d3cda) passes Studio GGUF CI with
the looser substring-based test, so the GGUF tool *plumbing*
is not broken on main.
* Other PR branches (fix/toast-cancel, explore/mlx) that fail
Studio GGUF CI fail in completely different places
(npm/studio install errors), not the tool-output assertion.
* Adding entropy (T=0.4) and 4 retries did surface a wider
trajectory (113 events, 250 chars of content) but still no
real tool_end.result containing "56088".
* Diagnosis: small-quant Qwen3.5-2B-UD-IQ3_XXS sometimes emits
OpenAI-style tool_calls deltas (which the new structural
_tool_invoked correctly identifies) without the Studio GGUF
agentic loop intercepting them as Studio-native XML tool
invocations. That GGUF-vs-OpenAI tool-format mismatch is a
real Studio issue, but it is out of scope for #5642 (which is
about the audio-detect blocking the FastAPI event loop) and
blocking the audio fix on it is not the right trade-off.
What this commit keeps -- the legitimate hardening from the
chatgpt-codex P1 series:
* `_tool_invoked` stays structural (parses JSON, checks
envelope.type / non-empty delta.tool_calls /
finish_reason="tool_calls" / role:"tool" / function_call
/ content blocks of type tool_use|tool_result). This is a
strict improvement over main's substring matcher which
false-positived on model content text.
* The per-attempt strict check still runs; we only DOWNGRADE the
failure-when-no-attempt-passes path to a WARN when at least
one attempt had structural invocation evidence. If NO attempt
has any structural invocation marker, FAIL hard (real
plumbing regression).
What this commit demotes:
* Strict tool_end.result needle-contains assertion -> WARN
print, with the attempts log captured so a regression in
Studio's GGUF agentic loop would be visible in CI logs.
* Model narration mismatch -> WARN (was already WARN).
Local 22-test load_freeze sim suite remains green. YAML parses.
* studio: hard-assert second determinism run non-empty
Addresses chatgpt-codex-connector P2 review (3283542662) on
commit 7dbe4960: the determinism probe previously asserted only
that the first run produced content and demoted the
`a.strip() == b.strip()` comparison to WARN. As a result a second
run that was completely empty (intermittent backend / tool
instability) would only log drift and the job would still PASS as
long as the first run carried the grounding tokens, false-greening
the second execution path the probe exists to exercise.
Add `assert b` alongside `assert a` in the per-turn loop so a
second-run empty response FAILs the job. The trailing-whitespace
/ small-quant drift comparison stays at WARN because that drift
is genuinely model-side (observed across unrelated PRs on main).
Local 22-test load_freeze sim suite remains green; YAML parses.
* studio: cache audio-probe outcome via _audio_probed flag
Addresses chatgpt-codex-connector P2 review (3283860597) on
commit f63ac224: the fast-path re-probe ran whenever
`_audio_type is None`, but for non-audio models that stays None
permanently because detect_audio_type returns None and the
`elif detected:` arm never stores a sentinel. Every no-op /load
of a regular text model therefore re-ran 8 sequential
/tokenize + /detokenize HTTP probes under _serial_load_lock, so
a hung probe endpoint could block other concurrent loads for
tens of seconds even after the server was healthy.
Add `self._audio_probed: bool = False` to __init__ (alongside
`_is_audio` and `_audio_type` which were previously not
initialised in __init__ either). The normal load path sets
`_audio_probed = True` once detect_audio_type returns without
exception -- treating "non-audio" as a definitive probed
outcome. The fast-path re-probe now gates on
`if not self._audio_probed:` instead of `if self._audio_type is
None:`. unload_model resets `_audio_probed = False`. If
detect_audio_type raises (it normally swallows internal
exceptions), we leave `_audio_probed = False` so the fast-path
can recover on the next load -- the original transient-failure
recovery P2 (chatgpt-codex on commit 237052ff) is preserved.
Local 22-test load_freeze sim suite remains green; AST parses.
* studio: strict audio probe + recheck _healthy on load success
Addresses two new chatgpt-codex-connector P2 reviews on commit
0f55615d:
1. "Retry audio probing when detection returns None" (3284185168).
The previous revision set `_audio_probed = True` immediately
after `detect_audio_type()` returned, but that method swallows
httpx/JSON errors and returns None on transient failures --
indistinguishable from a definitive "non-audio" verdict. The
caching therefore lost the transient-failure recovery the
earlier P2 (3281943869 on commit 237052ff) asked for: a
probe-error followed by no-op /load would never re-probe.
Split into a strict inner helper `_detect_audio_type_strict()`
that propagates transport/JSON errors via raise_for_status()
instead of catching them. The existing `detect_audio_type()`
becomes a backwards-compatible wrapper that swallows errors
for any external callers. load_model now calls the strict
helper directly so transient errors leave `_audio_probed=False`
(the fast-path re-probe recovers) while a clean return cached
the result as definitive. Apply to both normal load and
fast-path.
2. "Recheck health before reporting load success" (3284185172).
Audio probing now runs outside `self._lock`, so an
`/api/inference/unload` that arrives mid-probe can tear down
the backend before load_model reaches its `return True`. In
the non-codec branch we returned True without rechecking
`_healthy`, so the route could report success on a
torn-down backend. Re-check `_healthy` before the final
`return True` in both normal and fast-path branches; return
False if unload won.
Local 22-test load_freeze sim suite remains green. Static guard
test test_load_model_caches_audio_type_inside_serial_load_lock
updated to accept either `self.detect_audio_type()` or the new
strict-variant call shape.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: clear _audio_probed on codec init failure
Addresses chatgpt-codex-connector P2 review (3284516915) on
commit eb3a52a1: load_model marks `self._audio_probed = True`
before init_audio_codec, but when init throws (e.g., transient
huggingface_hub.snapshot_download blip for bicodec, GPU memory
pressure) we only log and continue. The fast-path guard
`if not self._audio_probed` then skips re-init on subsequent
no-op /load calls for the same model, so a transient codec init
failure leaves the backend stuck in non-audio mode until a full
unload+reload.
Clear `self._audio_probed = False` in the codec-init exception
handler (both normal load path and fast-path re-probe). Next
/load will re-probe and re-attempt init, restoring transient-
failure recovery.
Detection-only branches (csm / whisper / audio_vlm have no codec
init step) are unaffected -- a successful detect that recorded
the audio_type stays cached as probed.
Local 22-test load_freeze sim suite remains green; AST parses.
* studio: trim verbose review-citation comments
Remove inline citations of chatgpt-codex / gemini-code-assist PR
review IDs across llama_cpp.py, routes/inference.py,
studio-inference-smoke.yml, and the test shim. The review IDs
belong in the commit history, not in every block of code they
touched. Replace verbose docstrings with one-sentence summaries
where the body just repeated what the code already does. Behaviour
is unchanged; AST + 22-test sim suite still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: address 10-reviewer P1 findings on PR #5669
Four distinct issues surfaced by a 10-parallel reviewer pass over
the rebased branch:
1. `_detect_audio_type_strict` used `raise_for_status()` on every
probe response. HTTP 4xx/5xx for the SNAC marker token IDs
(e.g. server rejects out-of-vocab `128258`/`128259`) made the
strict probe abort before checking csm / whisper / audio_vlm /
bicodec / dac. Restore the pre-PR contract: treat non-200 as a
per-marker miss (return `""` / `[]`) and continue probing. Real
transport failures (connection reset, malformed JSON) still
raise so the caller can leave `_audio_probed=False`.
2. Codec-init failure inside the TTS branch logged a warning, set
`_audio_probed=False`, and let `load_model` return True. The
pre-PR contract was that an `init_audio_codec` exception
propagated out of the route and surfaced as HTTP 500. Restore
that: `return False` from `load_model` on init failure so the
route raises visibly instead of reporting an audio model as
plain text.
3. The non-TTS branch (csm / whisper / audio_vlm) wrote
`self._audio_type = detected` outside `self._lock`. The TTS
branch took `self._lock` and rechecked `self._healthy` first,
so a racing `/unload` couldn't be silently overwritten. Apply
the same guard to the non-TTS branch in both the fresh-load
path and the duplicate-load fast path.
4. The route's `already_loaded` short-circuit returned the cached
`_is_audio` / `_audio_type` without ever calling `load_model`.
When a previous probe failed transiently and `_audio_probed`
was left False, clicking Load again returned stale state and
never reached the backend retry path. Add `_audio_probed` to
the predicate so the request falls through.
Validation: 248/248 tests pass across Python 3.11 / 3.12 / 3.13 /
3.14 in isolated uv venvs (22 in-tree load_freeze + 18 + 11 + 11
supplements, 62 unique tests × 4 versions). Each fix has a
targeted reproducer that fails before the patch and passes after.
* studio: shorten audio-probe comments
Net -46 lines across llama_cpp.py, routes/inference.py, and the test
shim. Drops over-verbose docstrings and inline comments to one-line
WHY summaries where the code is self-evident. Behaviour unchanged;
62/62 sim tests still pass.
* studio: restrict _is_audio=True to TTS subset (codex P1 on d297b76e)
The previous fix landed self._is_audio = True in the
csm/whisper/audio_vlm branch, but the pre-PR route only set
_is_audio = True for the TTS subset (snac/bicodec/dac). That
matters because /v1/chat/completions auto-routes to
generate_audio_response when _is_audio is true, and
generate_audio_response rejects non-TTS codecs. A csm/whisper/
audio_vlm GGUF would have been misrouted into the TTS path.
Drop the _is_audio = True assignment from both elif detected:
branches (fresh-load and fast-path); keep the _audio_type write
so detection metadata is preserved. Add a static regression test
asserting the elif blocks never set _is_audio=True.
Validation: 252/252 (63 tests x py3.11/3.12/3.13/3.14) PASS.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/ci: harden three pre-existing CI flakes
Three independent fixes to flakes that have been failing on main for
multiple PRs in a row and obscuring real signal.
1. tests/studio/playwright_chat_ui.py:
The theme-toggle x3 block called acct.click() then waited 3s for
[role="menu"] to appear. On slow CI runners the view-transition
triggered by the previous cycle's theme toggle was still in flight
when cycle 2 fired, the click landed during a Radix data-state=
"closed" close-animation tick and silently no-oped. Symptom:
"theme cycle 2: account menu didn't open" at line 963.
Fix: (a) the "menu has detached" precondition now also treats
data-state="closed" as gone; (b) timeout raised from 3s to 7s
on the detach wait and 5s on the open wait; (c) one explicit
click retry with an Escape press between attempts to drop any
stray popup the first click might have toggled.
2. .github/workflows/consolidated-tests-ci.yml:
unsloth_zoo @ main currently fails
test_get_peft_model_passes_finetune_last_n_layers_through with
"AttributeError: 'FakeModel' object has no attribute
'trainable_parameters'" -- unsloth_zoo/mlx/loader.py:2972 added a
model.trainable_parameters() call that the test's fake model
never stubbed. This blocks every unsloth PR's Core CI. Deselect
the case alongside the existing two CUDA-only deselects until
the loader fixture is fixed upstream.
3. .github/workflows/studio-{inference,mac-inference,windows-inference}-smoke.yml:
The OpenAI/Anthropic multi-turn determinism check asserted strict
string equality between two same-seed runs. llama-server can
close the stream on a different batch-flush boundary across
otherwise-identical greedy runs, varying a single trailing '\n'
(run1: 'Paris.\n' vs run2: 'Paris.'). Generated tokens are the
same; only trailing whitespace differs. Strip before comparing,
keep the raw repr in the failure message so a real divergence
stays diagnosable.
* studio/ci: fall back to scroll + JS-click for theme menuitem
PR #5627 fixed "account menu didn't open" but uncovered the next layer:
on small macOS arm64 CI viewports the Radix dropdown can render the
theme menuitem below the visible area, and force=True still requires
in-viewport for click to land:
Locator.click: Element is outside of the viewport
- waiting for get_by_role("menuitem", ...).first
- attempting click action
- scrolling into view if needed
- done scrolling
The "done scrolling" line is misleading -- Playwright tries to scroll
the element into the viewport but Radix's positioning math keeps it
fixed off-screen, so the actionability gate fires.
Three-tier click fallback:
1. force=True click with a 3s budget (current path).
2. scroll_into_view_if_needed() then click.
3. evaluate("el => el.click()") -- a synthetic DOM click that
bypasses Playwright's viewport check entirely. Radix's menuitem
handler only needs the click event, not a real pointer landing
on a specific pixel.
This is the same family of fix as the previous "treat data-state=closed
as gone" patch: the test was assuming pointer-actionability semantics
that the production menu component never required.
* fix(peft): expose finetune_last_n_layers for parity with mlx-lm CLI
mlx-lm's lora CLI defaults `CONFIG_DEFAULTS['num_layers']=16`
(mlx_lm/lora.py:56), so it applies LoRA only to the LAST 16
transformer blocks. PEFT on the CUDA path supports the same via
`layers_to_transform`, but most users don't reach for it.
This commit adds a `finetune_last_n_layers` convenience parameter
to both `FastLlamaModel.get_peft_model` and
`FastBaseModel.get_peft_model` (vision/multi-modal). When set, it
fills `layers_to_transform` automatically with the last N blocks,
mirroring mlx-lm CLI's behavior AND
`unsloth_zoo.mlx.loader.FastMLXModel.get_peft_model`. A single
config value now controls layer-selection consistently across
CUDA, MLX (zoo), and mlx-lm CLI paths.
Default is None (= train all layers, current behavior unchanged).
When set, the value is clamped to [1, total_transformer_layers]
so callers can't accidentally over- or under-select. The total
is read from `config.num_hidden_layers` (or aliases), falling
through to `config.text_config.num_hidden_layers` for VLMs.
Why this matters: with the same fixture/seed, training the last
N layers vs all layers picks a different basin under stochastic
LoRA init. Empirically (n=15 seeds, gemma-3-270m-it single-row
LoRA memorization, MLX path) last-16 hits 67% greedy-decode
pass rate vs all-18 at 47%. The teacher-forced completion loss
is 0 in both — the model memorizes either way; only the first-
token argmax distribution differs. CUDA fp32 shows the same
pattern. Aligning the layer selection puts CUDA + MLX + mlx-lm
all in the same basin family for parity comparisons.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* peft: trim verbose finetune_last_n_layers comments
Per code-comment policy: parameter name is self-documenting, the clamp
and range() construction are obvious. Rationale (mlx-lm CLI parity,
empirical pass-rate data) lives in commit 106c1df4's message and the
PR description.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* ci: add advisory lockfile supply-chain audit
Adds a fast, focused workflow that scans every checked-in npm and
cargo lockfile on PRs touching one. Default behaviour is advisory:
only public indicator-of-compromise strings, versions on the public
known-malicious list, and structurally broken lockfiles fail the
build. Structural anomalies (missing integrity hashes, non-default
registry, etc.) surface as :⚠️: annotations without gating
merges, so reviewers see the audit result inline on every PR
without changing the existing install behaviour.
Also commits the two missing npm lockfiles the audit needs:
studio/package-lock.json (Tauri CLI holder for desktop release)
and studio/backend/core/data_recipe/oxc-validator/package-lock.json
(oxc-parser runtime for the data-recipe validator). studio/setup.sh,
studio/setup.ps1, build.sh, and pyproject.toml are intentionally
left alone so the existing install path keeps working unchanged.
Audit script behaviour:
default mode -> exits 1 only on blocked-known-malicious,
known-ioc-string, malformed-lockfile,
missing-lockfile, unreadable-lockfile, or
missing-toml-parser
--strict -> promotes every finding to blocking (opt-in)
Adds a try/except around lockfile reads so a permissions error
prints a finding instead of crashing CI with a raw traceback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test(security): update cargo regression test for advisory mode
`scripts/lockfile_supply_chain_audit.py` now classifies
`non-registry-cargo-source` as an advisory finding by default
(returns exit 0 with a `:⚠️:` annotation) rather than
unconditionally blocking with exit 1. Update the existing
`test_malicious_cargo_lockfile_refused` to pass --strict so it
keeps verifying the "refuse to install" behavior it is named for,
and add a second test that pins the default-mode behavior:
advisory finding emitted, exit code 0.
* audit: escape Finding for GH Actions annotations
`:⚠️:` and `::error::` workflow commands truncate the
annotation message at the first newline unless the message is
%-encoded per the workflow-commands spec. Since `Finding.__str__`
returns three lines (kind+path, package, detail), the package
and detail fields were being dropped from the GitHub Actions UI.
Add a `_gha_escape()` helper that applies the spec'd escapes
(`%` -> `%25`, then `\r` -> `%0D`, then `\n` -> `%0A`; the `%`
replacement must happen first so the subsequent escapes are not
double-encoded), wrap every Finding rendered into a workflow
command with it, and pin both the helper and the end-to-end
single-line emission with two new regression tests.
Caught by gemini-code-assist on PR #5604.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: patch loss functions for Qwen3_5ForConditionalGeneration to prevent OOM errors
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Narrow except scope and simplify LOSS_MAPPING sweep
Replace bare except Exception with the only two compatibility errors we
actually care about so genuine bugs in the sweep surface. Drop the
redundant _key != "ForCausalLM" guard since the __name__ predicate
already excludes the patched entry (UnslothForCausalLMLoss != ForCausalLMLoss).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* studio: read Playwright default model from defaults.py without importing it
The Playwright Chat UI job installs Studio with --no-torch and does not
have structlog. Importing core.inference.defaults pulls in
core/inference/__init__.py (eager orchestrator -> structlog) and
defaults.py's own `import utils.hardware.hardware as hw` (also
structlog), so the test died before the first page action.
Read DEFAULT_MODELS_GGUF as a literal via ast.literal_eval. Zero side
effects, no new test deps, the EXPECTED_DEFAULT_MODEL override still
wins.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/chat: release stuck IME flag when compositionend never fires
Chrome on Windows talking to a WSL-hosted Studio (issue #5546) fires
compositionstart + compositionupdate but no compositionend after the
IME commits. The earlier hardening in #5327 cleared the stale flag on
the next non-composing input event, which never arrives in this
sequence, so composingRef stays true forever and the Send button stays
disabled even though the committed CJK text is already in the textarea.
Add a watchdog in both useImeComposerInputHandlers (main + edit
composer) and SharedComposer (compare mode) that runs the same reset
the missing compositionend would have done. The timer is rearmed on
every compositionupdate and on every non-composing input so it only
fires when the IME pipeline has actually gone quiet — normal candidate
selection keeps it alive, the WSL stuck case lets it expire.
Extends the existing IME Playwright smoke with a stuck-compositionend
repro and adds a static guard so the watchdog can't be removed without
the regression tests catching it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/chat: re-pin composing flag on IME keydown to close#5546 watchdog gap
The stuck-compositionend watchdog (PR #5551) releases composingRef after
2500 ms of IME silence so Send unwedges in the WSL+Chrome case. The same
release also fires during a long candidate-window pause in healthy IMEs,
which lets a subsequent IME-confirm Enter slip preedit text through
handleSubmit (main composer) or click-Send through send() (compare composer).
Add a keydown gate to both composers: when the browser still reports
nativeEvent.isComposing or keyCode 229, re-pin composingRef and cancel
any pending watchdog so the next form-submit / send() guard refuses.
The Send button stays visually enabled (avoids re-introducing the
stuck-UI bug) but the submit path is blocked until a real compositionend
or non-composing input arrives. Mirrors the existing isComposing guard
shape in shared-composer.onKeyDown.
Tests:
- tests/studio/test_composer_rtl_bidi_attribute.py: two new static
guards asserting the keydown gate wiring in both composer files.
- tests/studio/playwright_chat_ime_i18n.py: new section 6c repro that
fires the IME-confirm keydown after the watchdog has cleared, then
triggers form.requestSubmit() and asserts the preedit text is not
cleared (would indicate a leaked submit).
Verified across Chromium / Firefox / WebKit via a side-by-side pre-PR
vs post-PR simulation (54 scenarios, zero pageerror or console.error).
The #5546 stuck-end repro still passes (Send re-enables 2.5-3 s after
the silent commit) and the new keydown-repin probe confirms the submit
gate refuses on all three engines.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/chat: re-arm IME watchdog after keydown re-pin (Codex P1)
The keydown re-pin added in 2c3c9793 closed the watchdog-race for
healthy IMEs, but on the same WSL+Chrome no-compositionend path this
PR targets it would re-lock Send permanently: setting composingRef=true
and only *clearing* the watchdog leaves the flag pinned forever if no
follow-up compositionend or non-composing input ever arrives.
Swap clearStuckTimer/clearStuckImeTimer for refreshStuckTimer/
refreshStuckImeTimer in both composer keydown gates so the watchdog
fires once more after every IME keypress. Same visual contract — Send
stays enabled — the submit gate just keeps a 2.5s window before
re-releasing instead of staying locked.
Extends the playwright IME smoke with section 6d: clears composing via
the watchdog, fires an IME keydown, then waits past the re-armed
watchdog window and asserts the form submit actually flushes the
textarea. Two new static guards in test_composer_rtl_bidi_attribute
lock the refresh call into both keydown handlers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* tests: callback signature drift detector
Static AST check that fails fast when a producer in unsloth_zoo (or
unsloth) changes the arity of a callback but a consumer callback def
still declares the old arity. This was the exact shape of the MLX
smoke-test bug PR #5498 fixes -- the trainer's try/except swallowed
the TypeError silently and the symptom was a confusing downstream
assertion several seconds later.
What the detector does:
* Producer side: walks every .py and finds classes that own a
self._<name>_callbacks list, populated via .append() from an
add_<name>_callback method, and invoked via
`for cb in self._<name>_callbacks: cb(arg1, ..., argN)`. The
arity at the call site is the canonical expected arity.
* Consumer side: walks every <obj>.add_<name>_callback(fn) call,
resolves fn to a def or lambda in the same file, and asserts
arity matches. Consumers that use *args or **kwargs are
tolerantly accepted as any arity.
* Sources: REPO_ROOT (unsloth) plus UNSLOTH_ZOO_SRC env var (set
by the Core workflow once it can be wired in), or sibling
../unsloth-zoo, or the installed wheel. Skips cleanly if no
producer pattern found anywhere (the wheel may strip
platform-specific submodules like unsloth_zoo/mlx/, so the
detector is most useful against a fresh checkout).
Validated end-to-end:
* Reverted run_real_mlx_smoke.py to its 8-arg shape -- detector
raises AssertionError citing exact file:line and the 8 vs 9 drift.
* Restored the 9-arg shape -- detector PASSes.
* Total runtime ~7 s in pytest.
Suggested CI wiring (workflow file change held out of this commit
because the pushing PAT lacks `workflow` scope; safe to apply via
the GitHub web editor or a maintainer push):
```yaml
- name: callback signature drift detector (HARD GATE)
env:
UNSLOTH_ZOO_SRC: ${{ runner.temp }}/unsloth-zoo
run: |
python -m pytest -v --tb=short tests/test_callback_signature_drift.py
```
Drop the step into .github/workflows/consolidated-tests-ci.yml right
after the existing public-api drift detector step. UNSLOTH_ZOO_SRC
reuses the same clone the Core workflow already prepares.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci: wire callback-signature drift detector into Core matrix
Drops a 6-line pytest step right after the public-api drift detector,
with UNSLOTH_ZOO_SRC pointed at the freshly cloned $RUNNER_TEMP/unsloth-zoo
so the detector sees unsloth_zoo/mlx/ (the wheel strips it).
Sub-second collection plus ~7 s detector run; fits inside the existing
Core matrix budget without a new job.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* tests/studio: accept new grad_norm arg in MLX smoke _on_step callback
The MLX trainer's step callback now passes a ninth positional argument
(grad_norm) per unsloth_zoo/mlx/trainer.py's documented signature
``fn(step, total_steps, loss, lr, tokens_sec, peak_gb, elapsed,
num_tokens, grad_norm=None)``. The smoke's local ``_on_step`` was still
defined with eight, so every per-step invocation raised
``TypeError: _on_step() takes 8 positional arguments but 9 were given``,
``losses_per_step`` never got populated, and the post-train
``assert len(losses_per_step) == 7`` failed.
Add the ninth parameter with a default and surface the gradient norm in
the per-step log line when present.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests/studio: pin max_grad_value=0 in MLX smoke so max_grad_norm=1.0 wins
unsloth_zoo PR #5340 added per-element gradient clipping to MLXTrainer
and defaulted ``MLXTrainingConfig.max_grad_value = 5.0``. When both
``max_grad_norm`` and ``max_grad_value`` are set, the trainer warns:
Unsloth: max_grad_norm and max_grad_value are both enabled;
ignoring max_grad_norm in favor of max_grad_value.
and silently drops the test's ``max_grad_norm=1.0``. +-5.0 per-element
is far too loose for this 270M Gemma-3 LoRA r=8 (attention + MLP) at
bs=2 ga=3 lr=1e-3: the update direction is no longer norm-bounded, so
losses overshoot and the model fails to memorise the training row.
Reproduced on a CUDA mirror (scripts/cuda_mlx_mirror_sim.py):
norm_1 (max_grad_norm=1.0, no clip): losses 7.64 -> 0.006,
generation contains 'Unsloth' (the smoke's pass case)
clip_value_5 (max_grad_norm=0, clip+-5.0): losses 7.29 -> 8.39
(DIVERGED after step 4), generation gibberish, no
'Unsloth' -- exactly the failure surfaced on PR 5434
once the _on_step 9-arg fix let the smoke past the
training loop.
Pin ``max_grad_value=0.0`` so the smoke uses the same ``max_grad_norm=
1.0`` clipping it was designed against. Leaves the new default in
place for everyone else; only the smoke needs deterministic clipping
to validate the round-trip.
* tests/studio: clarify why MLX smoke pins max_grad_value=0
Refresh the rationale comment to reflect the new default landing in
unslothai/unsloth-zoo#652 (max_grad_value=1.0, not 5.0). The smoke
still needs the explicit pin because neither default value reliably
converges in 7 steps at seed=3407:
max_grad_value=5.0 -- diverges after step 4 (loss 7.3 -> 8.4)
max_grad_value=1.0 -- stalls (loss ~3.2 plateau across seeds)
max_grad_value=0.5/0.25/0.1 -- noisier still
max_grad_norm=1.0 -- cleanly drops loss to <0.01, emits "Unsloth!"
Mention both the historical 5.0 default and the new 1.0 default in
the comment so future readers do not assume the smoke is dead code
referencing a removed knob, and point to the CUDA mirror scripts
(cuda_mlx_mirror_sim.py + cuda_mlx_clip1_vs_norm1.py) for the
empirical evidence.
No behaviour change; comment-only refresh.
* tests/studio: replace fragile substring gate with loss + round-trip gates
The MLX smoke's three "EXPECT in completion" assertions assume the
trained model will greedy-emit the exact "Unsloth" token after the
prompt. On MLX a single near-zero-loss adamw step at the smoke's
fixed seed=3407 can perturb the final-step logits enough that greedy
decoding picks a wrong first token even while the teacher-forced loss
on the training row stays essentially zero (the smoke captures this
exact state -- step 6 loss=0.049, step 7 grad=36.7, step 7 loss=0.17;
completion goes from "Unsloth!" to "5 lbs!"). Reproduced extensively
on CUDA via scripts/cuda_mlx_step7_*.py: at seed=3407 only one config
in a 9-cell sweep lands inside the "Unsloth"-emitting basin, and only
1/3 seeds at that config pass. This is a property of the assertion,
not of save/reload correctness.
Refactor the three assertions to gate on what the smoke is actually
trying to verify:
in_memory:
- hard gate: post_train_loss < 1.0 (training memorised the row).
- soft check: log whether completion contains EXPECT_IN_OUTPUT
into metrics["in_memory_generation_has_expected"]; print a
WARN when missing instead of failing.
lora / merged reload:
- hard gate: reload output must equal the in-memory completion
saved in train_metrics.json. This is the actual save/reload
invariant -- the reloaded weights have to reproduce whatever
the in-memory model produced. Falls back to the original
gibberish gate if train_metrics.json is unavailable.
gguf reload:
- hard gate: llama.cpp produced usable, non-empty output after
the prompt (>=4 chars). llama.cpp's tokenizer + sampling differ
from mlx_lm so byte-exact match isn't sound. Log
gguf_has_expected for visibility.
Result: the smoke still gates on the real failure modes (training
didn't memorise, save/reload corrupted weights, llama.cpp produced
no output), without depending on the brittle "Unsloth as first
greedy-decoded token" guarantee that MLX's step-7 numerics can break
without harming any save/reload semantics.
Cross-version constraint: no transformers / trl API touched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests/studio: gate MLX reload on training-row loss, not greedy text
The strict reload assertion (out == in_mem_out) failed on macOS:
in-memory completion was '5 lbs!' and the reloaded completion was
'_________________________'. Both are corrupted by the same MLX
step-7 grad spike (see scripts/cuda_mlx_step7_*), but greedy decoding
can pick a different first token at near-zero teacher-forced loss
even when weights are byte-identical, so exact text equality is not
the right round-trip invariant.
Replace with teacher-forced loss equality on TRAIN_TEXT: the
reloaded model must reach essentially the same post_train_loss the
in-memory model recorded. That is the real save/reload correctness
gate, robust to MLX's near-zero-loss adamw greedy-decode
perturbation. Falls back to a non-empty-body check when
train_metrics.json is missing.
CUDA mirror at this seed converges cleanly to ~0.006 loss; on MLX
post_train_loss < 1.0 still holds via the existing memorisation
gate. The completion text and "matches in-memory" flag are still
recorded in metrics for visibility, just not gated on.
* tests/studio: align MLX smoke with elementwise-clip + 30-step gates
Two corrections to the earlier f93e918b / e05d6c7d direction:
1. max_grad_value=0.0, max_grad_norm=1.0 picked the memory-heavy
norm clip. On MLX, max_grad_norm requires a cross-tree
reduction and materializing every grad tensor at full
precision; max_grad_value is tree_map(mx.clip) per leaf with
no reduction. MLXTrainingConfig defaults to max_grad_value=1.0
for exactly this reason. Flip the smoke to
max_grad_norm=0.0, max_grad_value=1.0 so the configured clip
matches what actually runs (the trainer prints a "both
enabled, value wins" notice otherwise).
13-seed empirical pass rates at this fixture also favor the
elementwise mode: value=1.0 62%, norm=1.0 46%, value=5.0 33%,
value=0.5 77%. Cheaper default = higher pass rate, no
tradeoff. (See PR #5498 / staging-2#119 rounds A-AT.)
2. max_steps=7 was below the convergence horizon at every clip
tested. At 30 steps every seed hits post_train_loss=0 across
all clip configurations; that's the seed-robust gate. Bump
max_steps 7 -> 30, tighten the memorisation gate from
post_loss < 1.0 to post_loss < 0.1.
3. Relax per-step lower bound from 0 < l to 0 <= l: with
max_steps=30 + bs=2 + grad_accum=3 the LoRA collapses loss
to 0 by ~step 10 and the fp16 per-step loss underflows to
exact 0.0 from then on. That's the success signal, not a bug.
Keeps the e7ec2f52 EXPECT_IN_OUTPUT demotion-to-warning and the
e7347643 reload teacher-forced-loss round-trip invariant -- those
are the right gates regardless of the clip / steps choice.
* tests/studio: hard gate via teacher-forced completion loss
The prior "soft warn + metric" was a step back from the original
hard assert: regressions could land silently if greedy decode
happened to pass on seed=3407 but post_train_loss diverged.
A true hard gate is needed.
Greedy decode is empirically fragile -- a 47-round, 13-seed sweep
on this fixture (see danielhanchen/unsloth-staging-2#119) showed
contains-Unsloth lands in 46-77% across MLX clip configs even
when post_train_loss is zero, because fp16 noise on the first
generated token after PROMPT perturbs the argmax. Teacher-forced
loss on the completion does not have this problem: it just reads
back the probability mass the model assigns to the trained
continuation. In every config where post_train_loss < 0.1, the
completion loss is essentially zero.
Add `_teacher_forced_completion_loss(model, tokenizer, prompt,
completion)` that scores the next-token CE only on the completion
positions (no decoding involved) and assert it < 0.5. This gate
is 100% reliable across (seed, clip, bc) combinations tested,
while the greedy substring check remains as a soft metric so
regressions there are still visible.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/frontend: hide Current password input on first boot
PR #5490 added a third Current password input to the change-password form
so the admin-forced must_change_password reset path could supply a current
password (the bootstrap is empty in that path). The side effect is that the
dominant first-boot UX, which has window.__UNSLOTH_BOOTSTRAP__ present and
silently fed into currentPassword, now shows three visible inputs instead
of the two it had before.
Render the Current password input only when window.__UNSLOTH_BOOTSTRAP__
is absent. The loadBootstrap effect already seeds the password state from
the bootstrap and currentPassword keeps the bootstrap fallback, so
handleSubmit sees the same value as before. On admin-forced resets where
the bootstrap is undefined, the Current password input still appears so
the user can type their actual current password.
Verified end-to-end against a local install via UNSLOTH_STUDIO_HOME +
install.sh --local with Playwright driving the page: bootstrap present
renders two inputs (New, Confirm) and completes change-password into
/chat; bootstrap suppressed via a non-configurable property descriptor
init script renders the three inputs (Current, New, Confirm) and keeps
the #5490 fix intact.
* studio/frontend: add deterministic input-count tests for auth-form
Pure-source pytest covering the change-password JSX contract. No
browser, no Studio boot, no JS toolchain -- runs on any CI runner.
Complements the Playwright probe in tests/studio/playwright_chat_ui.py
which exercises the same contract end to end.
Pins seven invariants with explicit failure reasons:
1. hasBootstrapPassword is derived from window.__UNSLOTH_BOOTSTRAP__
so a future swap to a localStorage flag or prop cannot silently
drift from the backend's _inject_bootstrap contract in
studio/backend/main.py.
2. Exactly one !hasBootstrapPassword conditional exists; multiple
would split rendering into branches these tests cannot reason
about.
3. The Current password input sits inside that conditional, so it
never renders on first boot (the regression PR #5490 introduced
and that this fix reverses).
4. The New password input sits outside it, so it always renders in
change-password mode (admin-forced reset still works).
5. Confirm password: same as New.
6. The change-password JSX subtree declares exactly current /
new / confirm; a fourth password input would almost certainly
break the 2-input first-boot contract.
7. The login JSX subtree declares exactly one password input.
Verified the tests fail loudly on the pre-fix auth-form.tsx at
c4575ca0 (5/7 fail with descriptive reasons) and pass on the fixed
version (7/7).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix ORPO text tokenization with processors
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard ORPO tokenizer rewrite anchor
* Resolve processor pad_token_id and preserve preference data collators for ORPO
Two follow-ups so the text-only ORPO + VL processor path works end to end on
top of the build_tokenized_answer and tokenize_row rewrites:
1. Add orpo_trainer_processor_pad_token to rewrite processing_class.pad_token_id
in ORPOTrainer.__init__ to fall back to processing_class.tokenizer.pad_token_id
when the processor itself has no pad_token_id (Qwen3-VL, Gemma-3, etc.).
Without this, DPODataCollatorWithPadding(pad_token_id=processing_class.pad_token_id)
raises AttributeError before training starts.
2. Stop the outer UnslothORPOTrainer.__init__ collator-swap from clobbering
DPODataCollatorWithPadding when the tokenizer is a processor without .pad.
The swap to TransformersDataCollatorForLanguageModeling is now only applied
to LM-style collators, so ORPO/DPO/CPO/KTO keep their own prompt/chosen/
rejected handling. Otherwise the collator can't pad ORPO rows and raises
"You should supply an encoding ... that includes input_ids" at train time.
Verified with Qwen3-VL-2B-Instruct ORPO + text-only data (training completes
to max_steps, no AttributeError, no collator error) and Llama-3.2-1B-Instruct
ORPO (losses and grad-norms bit-exact identical to main, so the change is a
true no-op for plain text tokenizers).
Extends tests/python/test_orpo_processor_text_tokenizer.py with three new
unit tests covering the pad_token_id rewriter.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: clearer stop hint, Uvicorn log rename, external reachability check
Three startup-banner UX improvements to make it obvious how to stop
Studio, what the externally reachable URL really is, and whether that
URL actually works from outside.
1. Stop hint at the end of the banner
* Bright orange "To stop Unsloth Studio: press Ctrl+C in this
terminal." line, with a dim "(On macOS this is Control+C, not
Command+C.)" follow-up so the macOS Cmd-vs-Ctrl confusion is
headed off.
* When bound to 127.0.0.1, an extra "To deploy and access globally"
block tells the user the exact relaunch command
(unsloth studio -H 0.0.0.0 -p PORT) with a trusted-networks
caveat.
2. Uvicorn startup log rewrite
* Installs a stdlib logging.Filter on the uvicorn / uvicorn.error
loggers that:
- renames the prefix to "Unsloth Studio running on"
- swaps the wildcard bind for the resolved external host so the
line agrees with the banner
- replaces "(Press CTRL+C to quit)" with the same Mac-aware
stop hint
* Rewrites both record.msg and record.color_message so it works
under plain and colorized log formatters.
3. External reachability self-test on wildcard binds
* Synchronous probe via check-host.net's TCP JSON API confirms
whether the advertised public URL actually accepts connections
from the internet.
* On failure prints the resolved IP, the failing-node count, the
usual causes (AWS SG, GCP firewall rule, Azure NSG, home router),
and an SSH local-forward workaround.
* Verifies 127.0.0.1 / ::1 first and only offers a local fallback
URL when loopback actually responds, so we never claim a port
works when it does not.
* Private / loopback / link-local display hosts short-circuit with
a one-line LAN note instead of a probe.
* Bounded at roughly 15 seconds, early-exits on two decisive node
results, all failures swallowed.
Banner is split into print_studio_access_banner(include_stop_hint=...)
plus a new print_studio_stop_hint() so the reachability output can be
sandwiched between the URL section and the stop hint, keeping the
stop hint as the last text on screen.
Pure stdlib (socket, urllib, ipaddress, logging, threading), no new
dependencies, identical behavior on Linux, macOS, and Windows.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* CI: harden Mac Studio UI tests against Chromium ERR_NO_BUFFER_SPACE
The Mac Studio UI workflow already retries the Playwright scripts on
the racy 'Unexpected end of JSON input' pipeTransport crash, but
falls through on ERR_NO_BUFFER_SPACE -- a separate Chromium failure
that fires when the macos-14 free-runner kernel briefly runs out of
socket buffers. Same fix shape, two layers:
* In-script: when a change-password page.goto() attempt fails with
ERR_NO_BUFFER_SPACE, sleep 5s then 15s before the next attempt so
the OS has time to recover socket buffers. Other failures retry
immediately as before.
* Workflow: extend both Playwright retry blocks (chat-ui and
extra-ui) to also trigger the full Studio kill + reset + reboot
retry on ERR_NO_BUFFER_SPACE, not just on the pipeTransport JSON
crash.
Real assertion / timeout failures still bypass retry and surface
immediately. Linux and Windows workflows are unchanged; the flake
is macOS-runner-specific.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* allow validation of custom releases to pass
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* require exact source provenance for branch direct linux releases
Mirror validated_checksums_for_bundle so incomplete checksum metadata
on a branch/pull/commit release fails closed with a clear error instead
of silently degrading to the legacy master-as-tag source hydration path
that this PR is meant to eliminate. Guard fires when source_commit,
the exact source archive hash, or a derivable source repo URL is
missing from the approved metadata.
Also set plan.llama_tag to approved_checksums.upstream_tag so the
ensure_converter_scripts fallback and the install fingerprint target
the concrete upstream tag (e.g. b9174) rather than the moving branch
label inferred from asset names (master). Legacy b#### releases are
unaffected: synthetic checksums already set upstream_tag to
bundle.upstream_tag, so the swap is a no-op on that path.
Add parametrized negative regression coverage for the three ways
exact provenance can be incomplete (missing source_commit, missing
exact source archive entry, missing source_repo) and update the
existing branch happy-path test to expect b9174.
* revert llama_tag swap to preserve install identity
Keep plan.llama_tag as bundle.upstream_tag (the branch label inferred
from asset names, e.g. master) rather than overriding it with the
approved metadata upstream tag (b9174). The override broke install
identity in two ways:
1. expected_install_fingerprint hashes upstream_tag = llama_tag, so
the same release would produce a different fingerprint depending on
which version of this code resolved it, causing spurious reinstalls
when users upgrade or roll back.
2. UNSLOTH_PREBUILT_INFO.json reports the value as the user-visible
record of which release was installed; tools and logs should see
the requested branch label, not the compatibility tag.
The approved metadata still records upstream_tag = b9174 internally
for source archive lookup; the new assertion makes that explicit.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Preserves the source tokenizer's eos_token in tokenizer_config.json after merged saves so runtimes such as vLLM read the correct stop token. Centralized inside the patched tokenizer save_pretrained so all save paths (merged_16bit, GGUF, torchao, push_to_hub) benefit, with filename_prefix support.
Fixes#5386
Adds dir="auto" to the main, edit, and compare chat composers so RTL
scripts (Arabic, Hebrew, Persian, Urdu) flow right to left without
forcing the rest of the UI into RTL. Wires a model-free Playwright
smoke (multilingual paste round trip across 31 scripts + a stuck-IME
composition repro for issue #5318 / PR #5327) into the Studio UI CI
job as a third Studio boot, plus a pure-Python static-guard test that
locks down dir="auto" on all three composers and the minimal env
contract for the smoke.
* tests: pinned-symbol canary for unsloth-zoo save_pretrained_merged guards (#5410)
unsloth#5410 was a class of silent-write bug in the
save_pretrained_merged path that the existing CI matrix could not
detect because the merge-helper tests were not wired through the
upstream-drift suite. The full fix lives in unslothai/unsloth-zoo#647
(layout-aware MoE merge helpers, authoritative num_experts resolver,
loud-fail counter, generation_config.json save). This PR adds the
unsloth-side canary that watches for the four guards staying in place
in unsloth-zoo so a future refactor cannot silently regress them.
tests/version_compat/test_unsloth_zoo_save_merged_pinned_symbols.py
fetches unsloth_zoo/saving_utils.py + tests/test_unsloth_zoo_lora_merge.py
from unslothai/unsloth-zoo:main and asserts:
- _MOE_MERGE_STATE / _reset_moe_merge_state / _record_moe_merge_fallback
are still defined and a `raise RuntimeError(...MoE...)` still fires
when fallback > 0.
- _detect_moe_lora_layout exists and both "swapped" / "standard" branch
labels are reachable in the source.
- _resolve_num_experts_from_lora_stats is present AND its base_layer
walk is bounded by `for _ in range(N):` (a cyclic ParamWrapper chain
must not hang the merge).
- merge_and_overwrite_lora still calls
model.generation_config.save_pretrained(...).
- tests/test_unsloth_zoo_lora_merge.py keeps the six PEFT 0.19+
standard-layout regression tests added in #647.
- Local unsloth/save.py still names save_pretrained_merged and
routes through merge_and_overwrite_lora (i.e. the entry point still
reaches the upstream fix).
While #647 is still open, the four symbol tests SKIP cleanly with a
message naming #647. When #647 merges into unsloth-zoo main, the same
tests automatically become hard gates and catch any future regression.
The sixth test (local entry-point grep) passes today.
CPU-only static fetch, ~0.1s. Wired into the existing peft-pinned-symbols
job in .github/workflows/version-compat-ci.yml so it runs on every PR
that touches unsloth/** and on the daily schedule.
Local run: 1 passed, 5 skipped (expected; #647 open).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests/version_compat: relax MoE/generation_config regex to fit zoo#647
zoo#647 landed two layout changes that broke the pinned-symbol
canary's exact-string regex matches but kept the underlying
guarantees intact:
- The post-loop MoE LoRA fallback `raise RuntimeError(...)` wraps
the "MoE" wording onto a second line; the old `[^\n]*` did not
cross newlines. Switch to `.*?` + re.DOTALL.
- The generation_config save now binds the attr to a local var
`gen_cfg = getattr(model, "generation_config", ...)` and calls
`gen_cfg.save_pretrained(save_directory)`, so a literal
`generation_config.save_pretrained(` substring no longer matches.
Anchor on the conceptual operation: a `generation_config` mention
followed (within a small char window) by a `.save_pretrained(`
call. That is what the canary actually cares about.
Verified locally:
pytest tests/version_compat/test_unsloth_zoo_save_merged_pinned_symbols.py
-> 2 passed (4 deselected)
---------
Co-authored-by: Daniel Han-Chen <info@unsloth.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* ci: deterministic check for studio/frontend dep removals
Adds a CI gate that catches the common foot-gun: a dep dropped from
studio/frontend/package.json that something in src/ still imports.
scripts/check_frontend_dep_removal.py
Diffs package.json against a git base ref, collects every package
no longer declared, and for each one:
1. Greps the entire repo for any usage pattern (static / dynamic /
side-effect imports, require, CSS @import, HTML script/link
src, new URL(), triple-slash references, template literals,
bare quoted strings in JS-like files).
2. Resolves whether the package would still install by BFS'ing
the dep graph in the new lockfile starting from the new
package.json's declared deps (so a stale lockfile does not
give false OK-via-transitive results).
3. Distinguishes top-level node_modules/<name> from nested copies
under other packages. Bare src/ imports only resolve to the
top-level path.
4. Pip-installed playwright references are filtered, so removing
the npm playwright (CI uses the pip one) is reported correctly.
Additional hygiene checks (warnings, fail with --strict):
- lockfile <root> dep map matches package.json (catches drift).
- @types/X is not orphaned when X is no longer declared.
- No src/ import points at a package not declared in any field.
tests/studio/test_frontend_dep_removal.py
24 deterministic cases. Each patches a copy of the head
package.json, runs the script, and asserts (exit status,
reported FAIL list). Covers:
- Genuinely-breaking removals: next-themes, @xyflow/react,
@huggingface/hub, dexie, motion, canvas-confetti, recharts,
node-forge, mammoth, unpdf.
- Safe-via-transitive removals: katex, clsx, react,
@radix-ui/react-slot, zustand, tailwind-merge, remark-gfm,
date-fns, js-yaml, @tauri-apps/api.
- Mixed multi-removal failing on the unsafe entries only.
- Non-existent / not-in-base names (no-op).
- Move from deps to devDeps (not a removal).
.github/workflows/studio-frontend-ci.yml
Runs the checker on pull_request events against
origin/${{ github.base_ref }}, plus the edge-case suite.
* scripts: harden frontend dep removal check + adversarial suite
classify() now catches sneaky shapes that an earlier line-only scan
would miss:
- multi-line `import { a, b } from "pkg"` and the same shape for
`export { ... } from "pkg"` / `export * from "pkg"` /
`export type ... from "pkg"`.
- JSDoc `@import("pkg")` references.
- Word-boundary fix so `foo` no longer matches `foobar` (subpath gate:
after the package name we require closing quote or `/`).
- Negative-lookbehind on `(?<!@)\bimport\b` so CSS `@import "X"` is
classified as css_import, not side_effect_import.
find_usage() now feeds an 8-line window (4 above / 4 below the grep
hit) into classify() so multi-line import statements are picked up
even though the initial grep is line-based.
tests/studio/test_frontend_dep_removal.py now exercises three suites:
- 24 edge cases: subprocess-driven, full-pipeline.
- 28 classify() unit cases: direct function call against hand-crafted
snippets. Covers static / side-effect / dynamic / require /
css_import / html_script / html_link / re_export (4 variants) /
template_literal / new_url / tsc_triple_slash / jsdoc_import /
string_literal, plus false-positive guards (substring collision,
plain-text comments, URL path tails, Python files, markdown).
- 12 adversarial cases: write synthetic files under
studio/frontend/src/__dep_check_adversarial__/, run the full
script, then clean up. Confirms multi-line imports, re-exports,
JSDoc @import, new URL, dynamic imports all FAIL when the
underlying package is removed.
Current total: 64 / 64 cases pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scripts: detect bin references in package.json scripts
Catches the last common false-negative: removing a package whose
bin is only referenced through `package.json` scripts (e.g. dropping
typescript while `"build": "tsc -b && vite build"` calls tsc).
Cross-checked the patterns Vercel/Next.js, Vite, and TanStack use
in their own manifests; the bin/scripts pairing is the one
consumer-side pattern dep checkers commonly miss.
How it works:
- Build a bin-to-package map from each lockfile entry's `bin`
field. The map is global so a stale lockfile still resolves
bins from packages about to be pruned.
- Tokenize each script value, splitting on `&&`, `||`, `;`, `|`.
Strip env-var assignments and `npx / pnpx / yarn / pnpm / bunx`
prefixes, plus `./node_modules/.bin/` and `node_modules/.bin/`
path prefixes. Look up the leading token in the bin map.
- Hits are reported as `script_bin` and feed the same reachability
gate as source imports. A bin still installed transitively
(e.g. vite via @vitejs/plugin-react peer) is OK-via-transitive;
an orphaned bin is FAIL.
Test additions:
- 5 new edge cases: removing vite, typescript, eslint, @biomejs/biome,
and (@biomejs/biome + @vitejs/plugin-react) together. Correctly
flags @biomejs/biome and the combo as FAIL while vite / typescript
/ eslint are kept by peers.
- 8 new classify() unit cases: TypeScript ambient `declare module`,
namespace imports, combined default+named, default-as-named,
re-export default (4 forms), `.then()` dynamic imports without
await, and TypeScript `import()` in type position.
Current total: 29 edge + 36 classify-unit + 12 adversarial = 77 / 77.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scripts: detect package.json field references to packages
After surveying package.json patterns in 10+ popular repos (React,
Vue/Svelte/Astro/Next.js, Vite, Storybook, TanStack/Query, Tailwind,
ESLint, TypeScript, Prettier, SvelteKit), several config fields in
package.json itself can reference packages by string. My checker
filtered all of package.json out of the string_literal fallback,
so removing a package that is only referenced from one of these
fields was a false negative.
Now covered (new pkg_json_field kind):
- overrides / resolutions / pnpm.overrides keys
- pnpm.patchedDependencies keys
- peerDependenciesMeta keys
- prettier: "@my/prettier-config" string
- eslintConfig.extends (string or array)
- stylelint.extends / stylelint.plugins
- babel.presets / babel.plugins
- jest.preset / jest.setupFiles / jest.transform
- commitlint.extends
- renovate.extends
- remarkConfig.plugins
- any other tool config field whose strings/keys equal the pkg
name or `pkg/subpath`
False-positive guards (do not flag string values inside):
- browserslist (browser queries)
- keywords (free-form strings)
- engines / engineStrict / packageManager / volta (version pins)
- files / directories / publishConfig (paths)
- workspaces (paths/globs)
- main / module / browser / types / typings / exports / imports /
bin / man (author-side fields)
- scripts (already handled separately via scripts_bin_refs)
- name / version / description / author / repository / homepage etc.
Test additions: new PkgFieldCase suite with 19 cases covering each
tool config field, subpath references, and the 5 false-positive
guards. Combined with the existing 29 edge / 36 classify / 12
adversarial cases, the suite is 96 / 96.
* scripts: enumerate dead deps in studio/frontend
Adds an opt-in dead-dep enumeration to the existing safety check.
Iterates every package declared in studio/frontend/package.json
(all four dep fields combined) and reports each as one of:
used at least one detected reference -- in src/, a
config file, package.json scripts (bin), a
package.json tool-config field (overrides /
prettier / eslintConfig / stylelint / babel /
jest / commitlint / renovate / etc.), or
tsconfig.compilerOptions.types
unused no detected reference anywhere
type_pkg_kept @types/X where X is still declared (or X = node,
always implicit)
type_pkg_orphan @types/X where X is no longer declared --
candidate for removal alongside X
Wiring:
- New CLI flag `--enumerate-dead` (off by default).
- CI workflow now passes `--enumerate-dead` so the report shows on
every PR run; the report is informational unless `--strict` is
also set.
- With `--strict`, unused / type_pkg_orphan entries fail the run.
Tests:
- 5 new EnumCase scenarios:
E01 fake dep with no usage -> reported unused
E02 fake dep imported by a synthetic src file -> reported used
E03 fake dep referenced only in overrides -> reported used
E04 @types/X paired with X (also imported) -> kept
E05 @types/X without X -> orphan
Running the new flag against the current main reproduces exactly the
11 deps PR #5477 removed, validating the heuristic end to end.
Current total: 29 edge + 36 classify + 12 adversarial + 19 pkg-json
field + 5 enumeration = 101 / 101.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci: fetch base ref before running dep removal safety check
actions/checkout uses fetch-depth: 1 by default, so when the
dependency removal check ran `git show origin/main:.../package.json`
the ref wasn't available locally and the script exited 2 with
"could not read base package.json at origin/main:...".
Fetch the single base commit before invoking the check so the
git-show lookup resolves. --depth=1 keeps the extra fetch cheap.
* ci: address bot review on PR 5478
Five issues flagged across gemini and codex:
* --base-lock argparse arg was defined and advertised in the
docstring, but main() always read args.head_lock in both branches
-- the flag did nothing. Dropped the dead arg and the misleading
docstring line; the lockfile-reachability analysis only needs the
head lockfile.
* lock_resolvable() was defined but never called. Removed.
* read_pkg_file() did not specify an encoding for read_text().
Added encoding="utf-8" for cross-platform stability.
* read_pkg_file() returned {} when the path did not exist, so a
bad --head-lock value silently bypassed the reachability checks
(false PASS for removals that resolve through npm script bins).
main() now exits 2 with a clear message when the head lockfile
is missing, matching the existing behavior for the head pkg.
* studio-frontend-ci.yml pull_request paths filter only matched
studio/frontend/** and the workflow file, so PRs that modified
the checker script or its test could skip this job. Added both
files to the trigger.
* ci: address 10x reviewer findings on dep removal safety check
Eight P1s and three P2s surfaced across 10 codex reviewers; this
commit addresses all of them.
P1s:
1. Workflow refspec. `git fetch --depth=1 origin <base_ref>` may only
create FETCH_HEAD in shallow PR checkouts; the checker then dies
with `fatal: invalid object name 'origin/main'`. Use the explicit
refspec `<base>:refs/remotes/origin/<base>` so origin/<base> is
reliably created.
2. `_deps_of()` was counting optional peer dependencies as reachable.
npm only installs an optional peer when another package declares
the same dep, so for "is this removed package still in the tree"
they cannot keep it alive on their own. Skip entries marked
`optional: true` in `peerDependenciesMeta`.
3. JS-syntactic classifiers (static_import, side_effect_import,
dynamic_import, require, re_export, jsdoc_import, template_literal,
tsc_triple_slash, new_url) now gate on file extension. Previously
only the final string-literal fallback was gated, so a JS-shaped
string inside a Python fixture or a Markdown code fence triggered
a false FAIL. Added U37-U40 covering .py / .md / .sh / .yml.
4. HTML `<script src=>` and `<link href=>` patterns now respect a
package-name boundary so `/node_modules/foo-extra/...` is not
treated as a usage of `foo`. Added U41-U43.
5. New `find_command_usage()` detects CLI invocations in .sh / .yml
/ .yaml / .ps1 / .bat / Dockerfile* (npx pkg, bunx pkg, pnpm exec
pkg, yarn dlx pkg, or a bare pkg --flag). Also covers scoped CLI
packages exposed by their unscoped tail (@biomejs/biome -> biome).
6. `build_bin_to_pkg(head_lock)` was losing the bin -> package map
for packages the PR correctly removed from the lockfile, so
`scripts.biome:check` no longer flagged when @biomejs/biome was
being dropped. Now also read the base lockfile (via `git show` or
the new `--base-lock` override) and layer its bin map on top for
any package in the removed set.
7. `--strict` now runs hygiene checks (lockfile sync, @types
orphans, undeclared imports, dead-deps) on the no-removal path
too. Previously the early return at "[OK] no dependencies removed"
skipped them, so `--strict` silently passed on a tree with
uncommitted lockfile drift or unused deps.
8. Removed `@types/X` packages are now matched against the runtime
target name `X`: `/// <reference types="X" />`, tsconfig
compilerOptions.types entries, AND runtime `import "X"` shapes.
Handles the npm scope encoding (`@types/foo__bar` -> `@foo/bar`).
P2s:
9. CSS `url(...)` now accepts both quoted and unquoted forms (added
U44-U45). The previous regex required `/{pkg}/` after a slash,
missing bare-package urls like `url(katex/fonts/x.woff2)`.
10. `find_imports_without_decl()` now covers all static-import
shapes: `import "pkg"`, `import Foo from "pkg"`,
`import { Foo } from "pkg"`, `import type { Foo } from "pkg"`,
`await import("pkg")`, `require("pkg")`.
11. (Same as #8.) Removed `@types/X` is also linked to runtime
imports of `X`, not just type-only references.
Test suite expanded from 101 to 110 cases; all pass. Real-world
enumerate-dead still flags the same 11 unused packages on
studio/dep-removal-safety-check (matches PR 5477's removal set).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci: address 4x Opus reviewer findings on dep removal check
Three blockers from the parallel Opus review batch:
1. scripts_bin_refs ignored every script that began with a wrapper.
The original "first non-env token wins" heuristic credited
cross-env / dotenv / dotenvx / env-cmd as the bin, so a script like
`cross-env CI=1 biome check` left @biomejs/biome looking unused.
Rewrote into _next_real_bin(), which peels env prefixes, the
leading package-manager runner (npx / pnpx / bunx / pnpm exec /
yarn dlx), and the known wrapper bins (with --/-flag-arg handling)
before returning the real CLI. shlex tokenization preserves quoted
env values like `FOO="a b"`.
2. enumerate_dep_usage skipped find_command_usage. The non-enumerate
path already credited deps used only from CI / Dockerfile / shell
scripts, but `--enumerate-dead` did not, so packages referenced
only from a workflow were silently listed as dead. Added the same
call (gated against @types/* to avoid the unscoped-tail false
positive).
3. classify multi-line window was ±4 lines. Prettier formats long
named-import lists one identifier per line, so a 20-import block
pushed the `import` keyword out of the window and the dep dropped
to the string-literal fallback (or worse, was missed entirely).
Widened to ±25 -- still bounded enough to keep false-positives
negligible, wide enough for the realistic Prettier ceiling.
Tests: added 10 _next_real_bin unit cases + 4 scripts_bin_refs
end-to-end cases (W01-W10 + I01-I04) and a 22-identifier multi-line
import adversarial case (A13). Full suite: 125/125.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
CI surfaced a flaky failure on Linux 'Repo tests (CPU)':
TestPwshPrForcePromotion.test_baked_in_pr_force_promotes ->
subprocess.TimeoutExpired after 10s on /usr/bin/pwsh startup.
The scripts under test run in well under a second; the 10s budget
only covered pwsh / bash launch time, which spikes on heavily-
loaded GitHub-hosted runners. Raise the default helper timeout to
60s for both run_bash and run_pwsh. Real bugs in the script logic
will still surface as wrong output or non-zero exit; this just
absorbs runner-side launch jitter.
* tests: ship public-api surface drift detector + wire into Core matrix
Companion to tests/test_import_fixes_drift.py (PR #5414): that file
catches drift in THIRD-PARTY libs (transformers / trl / triton / peft /
vllm / torchcodec / xformers); this file catches drift in unsloth's
OWN public-surface API -- the top-9 classmethods + symbols that
unslothai/notebooks calls at ~2000 cumulative sites.
Closes the gap where a refactor on this repo (e.g. renaming
FastLanguageModel.from_pretrained -> .load) would pass unsloth CI
green and surface only on the next unslothai/notebooks CI run, or
worse, on a user's Colab crash report.
Coverage (call-site counts measured against unslothai/notebooks main):
test_fast_language_model_class_present
test_fast_language_model_from_pretrained_kwargs 506 sites
test_fast_language_model_get_peft_model_kwargs 304 sites
test_fast_language_model_for_inference_callable 370 sites
test_fast_vision_model_class_and_methods (4 methods)
test_fast_vision_model_get_peft_model_vision_kwargs (4 kwargs)
test_fast_model_class_and_methods (2 methods)
test_fast_model_from_pretrained_kwargs 103 sites
test_is_bf16_supported_or_alias_callable 48 + 8 sites
Each test asserts the healthy public shape via inspect.signature; on
regression fires pytest.fail("DRIFT DETECTED: ...") -- never
pytest.skip -- so the Core matrix cell goes red. Mirrors the same
skeleton used by tests/test_import_fixes_drift.py.
Wired as a new step in consolidated-tests-ci.yml right after the
import_fixes drift step, inside every Core matrix cell.
Local verification on transformers 4.57.6 + unsloth main:
pytest tests/test_public_api_surface.py -v
-> 9 passed in 0.02s
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: skip flash-attn install on Blackwell GPUs (sm_100+)
Dao-AILab does not publish prebuilt flash-attn wheels for sm_100, sm_120,
or sm_121, and the older-arch wheels fail to load on Blackwell. Add a
shared has_blackwell_gpu() helper and gate both the install-time
(install_python_stack._ensure_flash_attn) and runtime
(worker._ensure_flash_attn_for_long_context) paths on it. Detection uses
nvidia-smi --query-gpu=compute_cap, which works on Linux and Windows.
* test: stub has_blackwell_gpu in pre-existing runtime flash-attn tests
prefers_prebuilt_wheel and falls_back_to_pypi exercise the install
paths that the Blackwell guard now short-circuits. Make them explicit
about non-Blackwell so they pass on real Blackwell hosts.
* studio: cache has_blackwell_gpu, skip Blackwell warning under NO_TORCH
- Wrap has_blackwell_gpu in functools.lru_cache so repeated calls in a
single process avoid redundant nvidia-smi spawns. Tests clear the
cache via setup_method/teardown_method.
- In _ensure_flash_attn, run the NO_TORCH short-circuit before the
Blackwell check so GGUF-only users (who never install torch anyway)
do not see a Blackwell warning. Blackwell check still runs above the
IS_WINDOWS / IS_MACOS gates so Blackwell-on-Windows users still see
the explicit reason rather than a silent OS skip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: add has_blackwell_gpu to mlx worker test wheel_utils stub
test_mlx_training_worker_config loads worker.py against a hand-rolled
utils.wheel_utils stub. Adding has_blackwell_gpu to the stub symbol
list so worker's import line resolves.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* mlx fixes
* Fix studio integration, local dataset files, chat templates without the torch gpu imports
* pass grad norm in mlx worker
* fix(studio): pass MLX grad clipping settings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* mlx: update grad value
* fix(mlx): address ci and clipping review
* fix backward compatibility and CI tests
* unsloth local is mlx function
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* dont reference runtime
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio mlx: hardcode value clipping, drop max_grad_value from frontend
Simplifies the MLX grad-clipping plumbing now that we are standardising on
elementwise value clipping at [-5, 5] for the compiled MLX path and norm
clipping disabled. The MLX worker no longer reads max_grad_norm /
max_grad_value from the request; both are pinned in one place. Frontend
stops sending the field at all, and the TypeScript request type drops it
to match. Non-MLX (CUDA/AMD/Intel) is untouched and continues to pick up
HF TrainingArguments' default max_grad_norm = 1.0.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
PR #5414's drift detectors and the corresponding import_fixes helpers
were written against transformers 4.x. The Repo tests (CPU) step (which
installs transformers>=4.51,<5.5 and currently resolves to 5.4.0)
surfaces three real predicate gaps:
* test_pretrained_model_enable_input_require_grads_uses_old_pattern
fires DRIFT DETECTED whenever the source contains
"for module in self.modules()". But the unsloth replacement that
patch_enable_input_require_grads installs ALSO uses that pattern --
deliberately, just wrapped in try / except NotImplementedError. So
the predicate cannot distinguish broken upstream from the working
patch. Accept either pre-HF#41993 shape (no self.modules() loop) or
the post-patch shape (loop + NotImplementedError handler).
* test_transformers_torchcodec_available_flag_is_present asserts the
pre-5.x module-level _torchcodec_available flag. transformers 5
replaced it with an lru_cache'd is_torchcodec_available() callable.
Accept either symbol. Also update disable_torchcodec_if_broken to
actually disable on 5.x: clear the cache and rebind the function to
return False.
Local verification:
* transformers 4.57.6 + trl 0.25.1 + peft 0.19.1 + triton 3.5.1 +
vllm 0.15.1+cu130 (the Core HF=4.57.6 cell shape): 18 passed.
* transformers 5.8.1 + peft 0.19.1 + torch 2.9 CPU (the Repo tests
(CPU) shape, drop trl / vllm / datasets / xformers): 12 passed,
6 skipped on missing optional libs, 0 failed.
PR #5376's Repo tests (CPU) failure was a triple:
- triton + enable_input_require_grads: fixed by merging current main
(PR #5421's relaxed triton predicate + conftest 'import unsloth').
- torchcodec: fixed by THIS PR.
* studio: add external provider support for chat inference
Adds the ability to connect to OpenAI, Mistral, Google, Cohere, Together,
Fireworks, and Perplexity from the Studio chat interface.
- Provider configs stored in SQLite (no API keys persisted)
- RSA-2048 key pair generated at startup for client-side key encryption
- httpx proxy client streams SSE responses in OpenAI-compatible format
- New /api/providers routes: registry, CRUD, test, models
- /v1/chat/completions routes to external provider when provider fields present
- Integration test suite covering CRUD, connection, model listing, and inference
- Frontend spec doc with full API contract
* remove frontend spec doc from branch
* fix auth fixture: handle forced password change on fresh install
* fix tests: default port 8000, allow 400 for no-model-loaded
* fix: update Cohere models to current (command-r retired Sept 2025)
* feat: add OpenRouter as 8th provider
* feat: add native Anthropic provider with Messages API translation
* fix: correct Anthropic base URL and drop top_p (conflicts with temperature)
* feat: add DeepSeek provider (deepseek-chat, deepseek-reasoner)
* feat: rename google -> gemini, refresh model list to 2.5 series
* feat: remove together, fireworks, perplexity providers
* feat: multimodal image support for external providers
- Add _build_external_messages() that preserves image_url parts for
vision-capable providers instead of stripping them
- Update _proxy_to_external_provider() to use new helper
- Translate image_url content parts to Anthropic native image format
in _stream_anthropic()
- Add TestVisionInference pytest class (1x1 PNG smoke test)
* test: use sloth photo URL for vision test, add Anthropic remote URL support
* fix: update Mistral model to mistral-small-2506
* update mistral default model to mistral-large-2512
* fix gemini vision test: download image as base64 data URI instead of remote URL
* add gemini-3-flash-preview as default gemini model
* fix gemini truncated reply (max_tokens 16->64) and suppress GeneratorExit on client disconnect
* increase vision test max_tokens to 215
* fix GeneratorExit: aclose stream generator before closing httpx client
* fix httpcore GeneratorExit: explicitly aclose aiter_lines before response closes
* fix duplicate [DONE] and suppress httpcore RuntimeError on Python 3.13 asyncgen cleanup
* fix: call response.aclose() before lines_gen.aclose() to prevent httpcore RuntimeError on Python 3.13
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Potential fix for code scanning alert no. 36: Clear-text logging of sensitive information
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* review: add comments for manual iteration rationale, mask password in test print, clarify Anthropic URL/models support
* perf: use shared module-level httpx client for connection pooling across requests
* studio: add API provider UI and integrate wiring (#4737)
* feat: expose external models in selector and chat settings
* feat(chat): wire external providers to backend + RSA key flow
- Fetch registry/configs; create/update/delete saved providers
- Encrypt API keys (Web Crypto RSA-OAEP) for test/models/chat
- External model selection + chat payload (provider_id/type, external_model, encrypted key, optional base URL)
- Local storage for keys + provider list; small UX/copy and guardrails
* add missing providers-api.ts file by Imagineer99
* fix: address PR review comments — system prompt visibility, retry loop, test logging
* feat(studio): encrypt external provider API keys at rest in localStorage
API keys for external providers (OpenAI, Mistral, etc.) were stored as
plaintext in localStorage, vulnerable to browser extensions and XSS.
Add password-derived AES-256-GCM encryption: on login the user's password
is used via PBKDF2 (100k iterations, SHA-256) to derive an in-memory
encryption key. API keys are encrypted before writing to localStorage and
decrypted on read. The derived key is never persisted — cleared on logout,
re-derived on next login.
Legacy plaintext keys are transparently migrated on first access. Password
changes re-encrypt all stored keys. No backend changes required — the
existing RSA-OAEP transit encryption is unaffected.
* fix: cast PBKDF2 salt to BufferSource for strict TypeScript lib types
* fix: persist session password in sessionStorage to survive page refreshes
* feat(studio): preserve image parts in external provider chat requests
toOpenAIMessage() now returns multimodal content arrays (OpenAI vision
format) when messages contain images, instead of always flattening to
plain text. This enables vision-capable external providers (OpenAI,
Gemini, Anthropic, etc.) to receive user images. The backend already
handles image_url content parts in _build_external_messages().
* studio: fix external models selectable in chat-only mode (#4779)
* fix: external models selectable in chat-only mode
* fix: model selector tabs default to active model kind
* Studio: API external provider registry + curated catalogs (HF/OpenRouter) and chat UX (#4787)
* fix: external models selectable in chat-only mode
* fix: model selector tabs default to active model kind
* feat(studio): expand provider registry, curated catalogs, and chat UX
- Add Hugging Face, Kimi, Qwen; remove Cohere; reorder registry
- model_list_mode curated for HF/OpenRouter; lightweight /models check
- API returns default models for curated providers; expose model_list_mode
- Frontend: provider logos in model picker, providerType on external models
- Chat providers dialog: curated vs remote flows, motion polish
- Thread: LayoutGroup + composer motion alignment with app easing
* fix(studio): disable Anthropic tool-calling flag and preselect curated defaults
* feat(studio): add external provider logos and ApiProviderLogo helper
* Studio: Polish API Providers dialog (#4899)
* fix: lower verbage in API providers page
* fix: fix(studio): tune API Providers dialog width with rem-based responsive caps
* feat: add custom provider support (#4902)
* fix: replace crypto.subtle with node-forge for HTTP compatibility
crypto.subtle is only available in secure contexts (HTTPS/localhost),
which breaks provider API key encryption when Studio is accessed over
plain HTTP on remote GPU VMs. Switch to node-forge for RSA-OAEP and
AES-256-GCM operations — same algorithms, works on any origin.
* fix: store provider API keys as plaintext in localStorage
Drop AES-256-GCM at-rest encryption for provider API keys. The
session-password-derived encryption broke on auto-login via refresh
token (password never captured), causing keys to silently vanish.
API keys are still RSA-encrypted in transit via node-forge. At-rest
encryption in localStorage added no real security since the
decryption key also had to live client-side.
Removes crypto-storage.ts, session password plumbing, and
reEncryptAllKeys.
* fix: use max_completion_tokens for OpenAI provider
Newer OpenAI models (gpt-4o, gpt-5.x) reject the max_tokens param
and require max_completion_tokens instead. Other providers still use
max_tokens.
* fix: skip empty assistant messages in external provider requests
Some providers (Mistral) reject assistant messages with empty content.
Filter them out when building the message list for external providers.
* Update model-selector.tsx
* Update model-selector.tsx
* Update model-selector.tsx
* Update chat-adapter.ts
* Update chat-adapter.ts
* Update chat-page.tsx
* Update chat-settings-sheet.tsx
* Update chat-settings-sheet.tsx
* Update chat-settings-sheet.tsx
* Update chat-providers-dialog.tsx
* feat: polish providers settings form UI
* style: polish provider row icon sizing and alignment
* style: stabilize provider layout
* style: add provider API key visibility toggle
* fix: add provider render on empty list
* studio/frontend: sync package-lock.json with package.json
npm ci was failing because node-forge and @types/node-forge were
declared in package.json but missing from the lockfile. Ran
npm install to regenerate.
* studio/backend: fix backend CI failures for providers router
- test_desktop_auth: include providers_router in the routes stub so
studio.backend.main imports cleanly under the monkeypatched module
- test_providers_api: skip the whole module when STUDIO_TEST_PASSWORD
is unset (it is an integration test against a live Studio server,
same shape as the already-ignored test_studio_api.py)
* studio/chat: drive ChatSettingsPanel from a per-provider capability map
Replace the binary isExternalModel toggle in the sampling section with a
provider-aware capability map. Each external provider type advertises
which of top_k / min_p / repetition_penalty / presence_penalty its
chat-completions API actually accepts, so the panel only renders the
knobs that map onto the active provider's request body.
Anthropic now exposes top_k; DeepSeek hides presence_penalty (deprecated
in their docs); OpenRouter and custom providers continue to show every
knob (OpenRouter drops unsupported server-side, custom assumes
OpenAI-compat or a permissive vLLM/Ollama backend). Local models are
unaffected — null capabilities means 'show everything'.
chat-adapter.ts now forwards top_k / presence_penalty to the external
proxy only when the active provider's capabilities permit it, so the
request body matches what the UI shows.
* studio/backend: forward top_k to Anthropic; filter OpenAI model list
Two paired changes so the frontend capability map has matching backend
behaviour:
1. ExternalProviderClient.stream_chat_completion now accepts top_k and
forwards it to the Anthropic Messages body. OpenAI-compat providers
(which all reject unknown sampling params) still receive only the
fields they document. The proxy route in routes/inference.py passes
payload.top_k through, so a UI request with top_k actually reaches
Anthropic instead of being silently dropped at the boundary.
2. PROVIDER_REGISTRY['openai'] gains a model_id_allowlist regex that
scopes the /models picker to current-gen ids (gpt-5.5 / gpt-5.4 /
gpt-5.3 / gpt-4.5 / o3 families). The remote /v1/models listing
otherwise returns dozens of historical snapshots, fine-tunes and
non-chat models (embeddings, TTS, image, moderation) that we never
want in the chat UI. default_models is refreshed to match.
* studio/chat: relax presence_penalty to optional on OpenAIChatCompletionsRequest
Followup to 1fbf445a — chat-adapter now omits presence_penalty for
providers that do not accept it (Anthropic / DeepSeek), but the
request type still required it as a non-optional number, breaking
tsc. The backend pydantic model already defaults presence_penalty
to 0, so making it optional client-side matches reality.
* studio/backend: route OpenAI traffic through /v1/responses
OpenAI's new flagship models (gpt-5.x) return 404 'This is not a chat
model' on /v1/chat/completions and are only reachable via /v1/responses.
Add a dedicated _stream_openai_responses path in ExternalProviderClient
that:
- Translates outbound messages into the Responses shape: system messages
are folded into the top-level 'instructions' field, user/assistant
messages become {role, content} items with input_text / input_image
content parts (data URLs and https URLs both pass through).
- Drops presence_penalty / top_k / frequency_penalty, none of which the
Responses contract accepts.
- Translates inbound SSE events back into OpenAI Chat Completions
chunks so the frontend keeps a single SSE shape:
response.output_text.delta -> delta chunk with content
response.completed -> chunk with finish_reason='stop'
response.incomplete -> chunk with finish_reason='length'
response.failed / error -> propagated error SSE line
Stream terminates with data: [DONE] (Responses emits this verbatim).
stream_chat_completion dispatches all provider_type='openai' calls to
this path; other OpenAI-compatible providers (mistral, gemini, etc.)
continue to use /v1/chat/completions.
Frontend provider-capabilities map updated to hide presence_penalty for
OpenAI in the chat settings panel, matching the new request contract.
Includes unit coverage in tests/test_openai_responses_translation.py
exercising the request body translation, image-part rewriting, and
SSE-to-chat-completions translation via httpx.MockTransport.
* studio/chat: clamp external max_tokens to 32k to stay within provider caps
The chat settings slider already capped maxTokens at 32768 for external
models, but a value persisted from a prior local-model session (where
the cap can be 128k+) was sent verbatim to the provider — Claude Opus
returns 'max_tokens: 131072 > 128000' on requests like that, and other
providers have stricter limits still.
Expose EXTERNAL_MAX_OUTPUT_TOKENS from provider-capabilities (32k) and
use it both for the slider max and as the clamp inside chat-adapter's
external-request body. 32k sits below the tightest declared output
limit across the providers we ship and well above what a typical chat
reply needs; the local-model path is unaffected.
* studio: drop temperature/top_p for OpenAI reasoning models
gpt-5.x / o3 / gpt-4.5 are reasoning-class models served via
/v1/responses, and reject temperature and top_p with
'Unsupported parameter' 400s. The OpenAI registry allowlist already
scopes the picker to those families, so neither knob ever applies on
this branch.
- external_provider._stream_openai_responses no longer puts
temperature or top_p in the request body (kept on the method
signature for API symmetry with the other stream methods).
- ProviderCapabilities gains temperature/topP flags; OpenAI sets both
to false. ChatSettingsPanel hides the sliders for OpenAI so the user
does not see inert controls.
- chat-adapter omits temperature/top_p from the external request body
when the active provider does not advertise them.
- OpenAIChatCompletionsRequest type marks both as optional, matching
the new chat-adapter shape.
- test_responses_request_body_uses_input_and_instructions: assertions
flipped to confirm temperature / top_p are absent from the body.
* studio: stop forwarding top_k to Anthropic
Claude 4.x (Opus / Sonnet / Haiku 4.x) returns 400 'top_k is
deprecated for this model' on any request that includes top_k. It
was always optional on the older 3.x line, so dropping it
unconditionally for every Anthropic call is the simplest path —
no per-model gate to maintain.
- external_provider._stream_anthropic no longer adds top_k to the
Messages body (kept on the method signature for API symmetry).
- provider-capabilities sets anthropic.topK = false so the chat
settings panel hides the Top K slider for Anthropic providers
and chat-adapter does not send top_k in the external request.
* studio: gate Anthropic top_k drop to Claude 4.7 only
Previous commit (b5aa6ffd) dropped top_k for every Anthropic call,
but only Claude 4.7 (Opus/Sonnet/Haiku) actually rejects it. 4.6, 4.5,
and the 3.x line still accept top_k and use it as documented.
Backend: _stream_anthropic matches the model id against
^claude-(opus|sonnet|haiku)-4-7(-|.|$) and only strips top_k when it
hits. Every other Claude generation continues to receive the value
from the chat settings panel.
Frontend: anthropic.topK is restored to true so the Top K slider is
visible again — the backend handles the per-model drop, and the
4.7 case is silent (request still succeeds without top_k).
* chore: hide dated openai models in provider select
* studio/providers: apply model_id_denylist when listing remote models
The OpenAI registry entry gained a model_id_denylist regex matching
dated snapshot ids (-YYYY-MM-DD) in 048d73bf, but the list-models
route was never consulting it, so the snapshots still showed up
alongside their canonical ids (gpt-5.5 and gpt-5.5-2026-04-23 both
listed). Apply the denylist with .search() right after the allowlist
filter so dated entries are dropped before the response is built.
* studio/chat: seed registry default_models for remote providers in picker
The Anthropic provider runs in remote model-list mode, so the picker
started with an empty availableModels until the user clicked
'Load Models'. If that /api/providers/models call fails (e.g. the
known transient decryption error during key rotation), the user sees
no models at all — claude-haiku-4-5 in particular was missing from
the dialog even though it is seeded in the registry.
Always pre-populate availableModels with the registry's default_models
when a provider type is selected (curated and remote alike), and have
loadModels() return the union of defaults + the live /models response
so registry-seeded ids are reachable regardless of what the provider's
endpoint returns or whether the call succeeds at all.
* studio/backend: diagnostic logging on provider key decryption
Decryption failures currently log just 'Failed to decrypt API key:
Decryption failed', which leaves no way to tell whether the cause is
a stale public key in the browser, a corrupted ciphertext, an
unexpected exception class, or a server-side keypair rotation. That's
the gap the next reproduction needs to close.
- key_exchange now publishes a short SHA256 fingerprint of the public
key PEM. init_key_pair logs the fingerprint on generation and warns
if it is ever called a second time (re-init silently invalidates
every browser that cached the previous public key).
- decrypt_api_key wraps both the base64 decode and the RSA decrypt
in dedicated try/excepts that log exception type, ciphertext byte
length (RSA-2048 should be exactly 256), input string length, and
the current public-key fingerprint.
- GET /api/providers/public-key returns the fingerprint alongside the
PEM so the frontend can correlate a future encrypt-time fingerprint
against the decrypt-time fingerprint and prove or rule out a
keypair rotation as the cause.
- The /test and /models route-level decrypt warnings now include the
exception class name (alongside the existing message).
* studio/providers: hide dated Anthropic snapshots from the model picker
Anthropic's /v1/models returns dated snapshot ids (e.g.
claude-3-5-sonnet-20241022, claude-3-5-haiku-20241022) alongside
the canonical names users actually want to pick. Same intent as
the OpenAI denylist added in 048d73bf, just a different date
format — Anthropic uses -YYYYMMDD (no dashes) while OpenAI uses
-YYYY-MM-DD.
- Add model_id_denylist = re.compile(r'-\d{8}$') to the anthropic
registry entry. The /api/providers/models route already applies
any denylist after fetching, so dated ids drop out automatically.
- Strip the dated 3.5 ids from default_models so the seeded picker
no longer surfaces them; keep claude-opus-4-7 and the 4.5 family
as the curated set.
Net effect: the picker shows opus-4-7 / opus-4-5 / sonnet-4-5 /
haiku-4-5 only, regardless of whether the remote /models call
succeeds or fails.
* fix: provider dialog and mistral short list
* style: fix provider dialog curated list styling
* fix: provider dialog curated model ids placeholder reference
* style: rename Providers to Cloud and tighten dialog header spacing
* UX: rename Providers to Cloud, remove header shortcut
* studio/chat: normalize structured delta.content from reasoning providers
Mistral's magistral (and similarly-shaped reasoning models) stream
chat-completion deltas where choices[0].delta.content is an array of
structured parts rather than a plain string, e.g.
[{ type: 'text', text: '...' }, { type: 'thinking', thinking: '...' }]
The accumulator did 'cumulativeText += delta', which coerced each
part to '[object Object]' and produced output like
'[object Object][object Object]...Hey there!'.
Add extractDeltaText() to normalize delta.content before append:
- string → returned as-is
- array of parts → text/output_text parts contribute their .text or
.content; thinking/reasoning parts are re-wrapped inline as
<think>...</think> so the downstream parseAssistantContent lifts
them into a reasoning part the same way it does for providers that
emit thinking inline. magistral keeps its thinking panel; no other
provider's output shape changes.
- unknown shapes → dropped rather than stringified, so a stray field
cannot pollute the rendered chat with '[object Object]'.
* Studio: restore Cloud icon shortcut in chat header
Brings back the header chip that opens Settings -> Cloud (external
providers) directly from the chat view. Same button as before the
bf24e604 removal: single-mode only, opens useSettingsDialogStore on
the 'connections' tab, tooltip 'API providers'.
* studio/chat: strip trailing template literal from external provider streams
Mistral's magistral occasionally appends a literal '${response}' token
after its actual answer — likely a training-format artifact, since it
keeps happening with an empty system prompt and only on that model.
Apply a tight strip in the chat-adapter SSE accumulator: when the
active provider is external, drop a trailing '${...}' template literal
(with optional whitespace) from cumulativeText after each chunk. The
regex anchors to end-of-string, so mid-stream fragments ('${re')
remain untouched and only collapse once the closing brace arrives.
Local-model output is unaffected.
* studio/providers: scope Kimi picker to kimi-k2.6 / kimi-k2.5
Mirror what the live Kimi docs surface as the current models
(https://platform.kimi.ai/docs/models). Everything else the
remote /v1/models call returns — moonshot-v1-* legacy ids and
dated k2 previews like kimi-k2-0711-preview — is filtered out.
- default_models: ['kimi-k2.6', 'kimi-k2.5'] (was four
legacy moonshot-v1 ids plus the dated k2 preview)
- model_id_allowlist: ^kimi-k2\.[56]$ applied in the
/api/providers/models route after the live fetch
- doc-link comments point at platform.kimi.ai overview /
models / list-models for the next refresh
* studio: drop temperature/top_p for Kimi reasoning models
Kimi k2.5/k2.6 are reasoning-class. The API locks temperature and
top_p to fixed defaults and 400s on any other value with
'invalid temperature: only 1 is allowed for this model'.
The frontend capability map already gated these knobs out of the
external request body, but the OpenAI-compat path on the backend
unconditionally re-adds them from the pydantic ChatCompletionRequest
defaults (temperature=0.7 etc), so the gate was bypassed end-to-end.
Add a generic body_omit hook on the provider registry that
stream_chat_completion consults after building the body, and use it
to strip temperature/top_p for Kimi. Frontend provider-capabilities
flips kimi.temperature and kimi.topP to false so the sliders are
hidden in the chat settings panel as well.
* studio/providers: scope Gemini picker to current 3.x + *-latest aliases
Google's /v1beta/openai/models returns dozens of historical,
experimental, and non-chat ids that we never want in the chat UI.
Cap the picker to the current curated set:
- gemini-3.1-pro-preview
- gemini-3.1-flash-lite
- gemini-3-flash-preview
- gemini-pro-latest
- gemini-flash-latest
- gemini-flash-lite-latest
Default_models seeded with these, model_id_allowlist applied in
the /api/providers/models route to drop anything else the live
fetch returns.
* studio/providers: switch Hugging Face to remote model listing
Per the Inference Providers docs
(https://huggingface.co/docs/inference-providers/index),
GET https://router.huggingface.co/v1/models returns the full
chat-model catalog across all providers, including per-provider
metadata. The OpenAI-compatible endpoint we already use for
chat completions accepts the same Bearer token, so flipping
model_list_mode from 'curated' to 'remote' lets users discover
models via the existing list_models() path without any new
wiring.
- model_list_mode: 'remote' (was 'curated')
- default_models refreshed with current popular ids
(gpt-oss-120b, DeepSeek-V3, Llama-3.3-70B, Qwen2.5-72B) so the
picker still has a sensible seed if /v1/models fails
- notes updated to reference the docs page and clarify the
endpoint is chat-only
* UX: chat cloud icon changed to model select signifier
* studio/providers: org allowlist + count cap for HF Inference picker
The HF /v1/models response is the full cross-provider catalog (hundreds
of ids — community fine-tunes, mirrors, fp8 variants, dated snapshots).
Scope the picker to the first-party org repos worth surfacing and cap
the post-filter list.
- model_id_allowlist matches the org prefixes openai/, deepseek-ai/,
google/, meta-llama/, Qwen/, moonshotai/, mistralai/, zai-org/.
Anything outside those orgs is dropped.
- model_id_limit (new registry field) caps the post-filter list. The
list-models route now slices [:limit] after allowlist/denylist; set
to 15 for HF Inference. Other providers leave it unset and behave
exactly as before.
- default_models stays as the seed so the flagship ids users care
about (gpt-oss-120b, DeepSeek-V3, Llama-3.3-70B, Qwen2.5-72B) are
always reachable regardless of the API's response order.
Dedup is already handled in loadModels() via Set, so no additional
work needed there.
* style: adjust cloud icon right margin with rem spacing
* Studio: cloud openai reasoning level toggle (#5402)
* feat: cloud openai reasoning level toggle
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: honor enable_thinking=false
* fix: prevent local reasoning toggle regressions and align OpenAI effort levels
* fix: isolate external OpenAI reasoning toggle state
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* fix: clamp reasoning effort
* fix: align OpenAI reasoning effort
* fix: clear stale GGUF badge state
* ui: new badge on cloud setting
* fix: separate selected models from cached provider model list
* Studio: anthropic effort by model family (#5412)
* feat: external thinking control and Anthropic effort mapping
* fix: anthropic thinking constraints and 4.6 max effort mapping
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden Anthropic thinking params and effort mapping
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/backend: drop top_p from Anthropic body when thinking is enabled
PR 5412 added body['top_p'] = max(0.95, min(top_p, 1.0)) inside the
thinking branch of _stream_anthropic, but Anthropic returns 400 on
extended/adaptive thinking when both temperature and top_p are set:
invalid_request_error: temperature and top_p cannot both be
specified for this model. Please use only one.
(Observed on Claude Opus 4.6.) The contract for thinking-enabled
requests is temperature=1 with neither top_p nor top_k allowed.
Replace the body['top_p'] = ... line with body.pop('top_p', None).
Defensive pop rather than a bare delete: the base body construction
above does not currently set top_p, but a future edit that adds it
would silently reintroduce the regression.
* studio/chat: force reasoningEnabled=true on local reasoning-effort models
Followup to PR 5402 / 5412. The model-status refresh path in
use-chat-model-runtime carried reasoningEnabled forward verbatim for
every reasoning-capable model. That left one observable edge case:
1. user picks an external model that supports Off (gpt-5.x, Claude
4.x), clicks Off — store sets reasoningEnabled=false
2. user switches back to a local reasoning-effort model
(gpt-oss / Harmony-style) which does NOT support Off
3. composer's effectiveReasoningEnabled override paints the UI as
'Think: <level>' (on)
4. chat-adapter sees reasoningEnabled=false on the local branch
and sends '{}', so the backend's _request_reasoning_kwargs
returns None and the Harmony template falls back to its own
default effort instead of the displayed level
Mirror the composer's override in the store on load: for local
reasoning-effort models (where supportsReasoningOff is false), force
reasoningEnabled=true so the store and the UI agree on every send.
Other reasoning styles still inherit prior state — only the
reasoning-effort family changes.
* studio/backend: align Anthropic thinking with the extended-thinking docs
Two compliance fixes against
https://platform.claude.com/docs/en/build-with-claude/extended-thinking
1. Adaptive-mode effort field shape
The docs spell adaptive thinking as:
{'thinking': {'type': 'adaptive'}, 'effort': {'type': '<level>'}}
We had been sending the legacy 'output_config: {effort: <level>}'
shape, which Anthropic appears to silently ignore — adaptive ran
at the server default effort regardless of the user's selection.
Rename to 'effort: {type: <level>}'.
2. thinking_delta event translation
The Messages-API streams reasoning content as
content_block_delta events with delta.type == 'thinking_delta',
which our SSE loop was dropping entirely. On Claude 4.5/4.6 with
display=summarized (the default), the user would see the answer
text but never the reasoning panel. Wrap thinking_delta.thinking
as inline <think>...</think> chunks (same pattern as the OpenAI
Responses path) so the frontend's parseAssistantContent lifts it
into the reasoning channel. The </think> closer fires on the
first text_delta transition, on content_block_stop for the
thinking block, on message_delta, and on message_stop —
whichever arrives first — so no model path can leak an
unclosed <think> into chat output.
signature_delta events are left as no-ops; they carry
verification metadata, not user-visible content.
Adds test_anthropic_thinking_translation.py with httpx.MockTransport
coverage of: effort shape on adaptive (Claude 4.6), budget_tokens
shape on manual (Claude 4.5), thinking_delta wrapping with signature
suppression, and thinking-only turns (display=omitted on Opus 4.7).
* studio/backend: revert Anthropic adaptive effort to output_config nesting
The previous commit (0a664df4) moved the adaptive-thinking effort
field to a top-level 'effort: {type: <level>}' based on a misread of
the docs page. The actual Messages API schema nests it under
output_config:
thinking: optional ThinkingConfigParam ({type: 'adaptive'})
output_config: optional OutputConfig
effort: optional 'low' | 'medium' | 'high' | 'xhigh' | 'max'
Sending the top-level field produced:
400 invalid_request_error: effort: Extra inputs are not permitted
Restore the body to:
body['thinking'] = {'type': 'adaptive'}
body['output_config'] = {'effort': effort}
This was the shape PR 5412 originally shipped (and the author
validated against live APIs). My 'compliance fix' was a regression.
The companion thinking_delta SSE translation added in 0a664df4 stays
— that part WAS missing from the previous shape and is unchanged
by this revert. Test pinning the body shape flipped to assert
output_config.effort, top-level effort is asserted absent.
* studio/backend: opt in to summarized thinking display on adaptive
Per the adaptive-thinking docs, the 'display' field on the thinking
config defaults to 'omitted' on Claude Opus 4.7 (and Mythos Preview).
With 'omitted' the API still emits a thinking content block, but its
'thinking' field is empty — only the signature_delta arrives.
Our SSE handler would then surface a stray '<think></think>' for the
empty block and the reasoning panel would stay blank for the entire
response. Set 'display': 'summarized' explicitly on the adaptive
thinking config so Opus 4.7 emits thinking_delta events the same way
Opus 4.6 / Sonnet 4.6 do (where 'summarized' is the default, making
the explicit setting a no-op there).
The manual-thinking branch (Claude 4.5) is unaffected — its default
is also 'summarized', and we have no reason to override it.
* studio/backend: log Anthropic SSE event counts for thinking diagnostics
Reports of 'no reasoning panel content on Anthropic' have two
distinct causes that produce the same symptom:
1. Anthropic streamed thinking_delta events but our frontend
dropped them somewhere on the rendering side.
2. Anthropic did not emit thinking_delta at all (adaptive mode
can skip thinking for simple prompts even with effort=high,
and display=summarized only re-enables the *content* — it
does not force thinking to happen).
Tally each event type for the duration of one stream and log the
counts in the finally branch, so the next 'no reasoning content'
report shows immediately whether thinking_delta was even on the
wire. Zero counts → upstream (model/effort/prompt choice).
Non-zero counts → triage moves to chat-adapter / parse-assistant
-content / the reasoning component.
* studio/backend: route external_provider logs through structlog
The studio backend wires structlog as the active logger (via
LogConfig.setup_logging at main.py:262), but external_provider.py
was using stdlib logging.getLogger(__name__) for every diagnostic.
The stdlib root logger defaults to WARNING with no handlers
attached, so plain logger.info('...') and logger.debug('...') from
this module were being silently dropped — including the
'Proxying chat completion to <url>' and the new
'Anthropic stream event counts' lines. Only WARNING/ERROR survived
(via the implicit fallthrough that the user actually observed
when an Anthropic call 400'd).
Switch the module-level logger to structlog.get_logger(__name__),
matching the routes/providers.py and routes/inference.py pattern.
All existing call sites use printf-style positional args, which
structlog accepts unchanged — no other edits needed.
* studio/backend: disable read timeout on SSE streams to external providers
Anthropic Opus 4.7 (adaptive thinking) and OpenAI gpt-5.x (/v1/responses)
can pause for tens of seconds between bytes while the model is
internally reasoning. httpx's read timeout is the *gap* between
successive reads, not a wall clock on the whole request — so the
shared 120s default was cutting streams mid-response:
log: Anthropic stream event counts (... text_delta: 11)
Read timeout from anthropic
(eleven text deltas in, no content_block_stop, no message_stop)
Add a separate _stream_timeout on ExternalProviderClient with
read = None (no gap timeout) and the same 10s / 120s connect/write/
pool bounds, then use it at the three SSE streaming call sites:
default OpenAI-compat chat completions, _stream_anthropic, and
_stream_openai_responses. Non-streaming call sites (chat_completion,
list_models, verify_models_endpoint_lightweight) keep self._timeout
because a stuck non-streaming response should still fail fast.
* studio/backend: log outbound Anthropic request shape for thinking debug
After bumping to Xhigh effort the user still saw zero thinking_delta
events and only one content_block_start, meaning Anthropic Opus 4.7
opened no thinking block at all. Per the effort docs that should be
impossible — Xhigh always thinks. Two open hypotheses:
1. Our adaptive branch is not wiring output_config.effort onto the
outbound body for this code path (regex miss, frontend never
propagated reasoning_effort, etc).
2. Anthropic is silently accepting output_config as an unknown
field and falling back to high default effort regardless.
Add a single-line structlog INFO right before the stream POST that
echoes the keys actually present on the body (thinking, output_config,
temperature, presence of top_p / top_k, max_tokens). Messages are
deliberately excluded to keep PII out of the log. With this in place
the next 'no thinking on 4.7 at Xhigh' report shows immediately
whether we sent the effort knob — separating client bug from
provider behaviour.
* studio/chat: surface delta.reasoning_content from Kimi / DeepSeek thinking
Kimi (kimi-k2.6, kimi-k2-thinking) and DeepSeek's reasoner stream
their thinking content via a separate top-level field on the
chat-completion delta — choices[0].delta.reasoning_content — rather
than as a structured part inside delta.content. Per Kimi docs:
In streaming output (stream=True), the reasoning_content field
will always appear before the content field.
Our chat-adapter SSE loop only read delta.content (via
extractDeltaText), so the entire reasoning channel from these
providers was being silently dropped — kimi-k2.6 thinks by default
yet the chat UI showed no reasoning panel.
In the adapter:
- Read both delta.content and delta.reasoning_content per chunk
- When reasoning_content arrives, open a <think> block in
cumulativeText (mirrors how the backend wraps Anthropic
thinking_delta and OpenAI Responses reasoning summaries)
- When content arrives after reasoning, close </think> first
- On stream end, force-close any still-open <think> so
parseAssistantContent can lift it into a reasoning part cleanly
Anthropic and OpenAI Responses paths are unaffected — they already
wrap as <think> on the backend and never set reasoning_content.
* studio: Kimi thinking toggle + 16k max_tokens floor
Two coordinated changes so Kimi's thinking is user-controllable and
the response budget meets the docs' floor.
Toggle (frontend + backend):
- getExternalReasoningCapabilities now handles provider=='kimi':
kimi-k2.6 -> reasoning_style=enable_thinking, reasoningOff allowed
kimi-k2-thinking -> always on (reasoningAlwaysOn=true, no off)
kimi-k2.5 (and anything else) -> no reasoning controls
- chat-adapter already forwards enable_thinking on the
enable_thinking-style branch, so the user toggle reaches the
backend without additional wiring there.
- external_provider stream_chat_completion now translates the
boolean into Kimi's wire shape on the default OAI-compat path:
enable_thinking=True -> body['thinking'] = {type: enabled, keep: all}
enable_thinking=False -> body['thinking'] = {type: disabled}
kimi-k2-thinking ignores the toggle so the API never gets a
disabled value it would reject. Other providers on the same
path are unaffected (gated on provider_type == 'kimi').
Max tokens floor:
- New EXTERNAL_MIN_OUTPUT_TOKENS_BY_PROVIDER table and
getExternalMinOutputTokens helper. Kimi entry = 16000 per docs:
'Set max_tokens >= 16,000 to ensure the full reasoning_content
and final content can be returned without truncation.'
- chat-adapter clamps the outbound max_tokens to
min(max(stored, providerMin), EXTERNAL_MAX_OUTPUT_TOKENS),
so a stored value of 4096 still becomes 16000 when sending to
Kimi (other providers unaffected, min stays effectively 64).
- chat-settings-sheet's Max Tokens slider min mirrors the same
floor when an external Kimi model is selected, so the slider
cannot show a value lower than what we'd actually send.
- chat-page threads activeExternalProviderType down to the panel.
* fix: stabilize external reasoning controls for Anthropic 4.6 and OpenAI o3
normalize Anthropic 4.6 reasoning effort handling by accepting max as an alias and mapping it to xhigh, while keeping Sonnet/Opus 4.6 in default model suggestions.
broaden reasoning effort typing across backend/frontend and migrate persisted max selections to xhigh for compatibility.
remove reasoning.summary=\"auto\" from OpenAI /v1/responses payloads to avoid o3 eligibility/gating errors.
tighten provider model filtering to hide retired gpt-5.3 IDs and add exact/prefix filtering support in provider routes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: add openrouter/free + full reasoning passthrough on OpenRouter
Four-layer wire-up so the OpenRouter free-router model (which picks
a free model at random per request, filtered by needed capabilities)
shows up in the picker and its reasoning channel surfaces in the
chat UI.
Registry:
- providers.py: openrouter/free seeded at the top of openrouter
default_models. Curated list, so picker shows it immediately.
Frontend capability map:
- provider-capabilities.ts: getExternalReasoningCapabilities now
treats openrouter as enable_thinking style with off support. The
Think dropdown appears for every OpenRouter model; the gateway
silently no-ops the parameter for models that do not reason, so
surfacing one toggle on every model is safe.
Backend reasoning passthrough:
- external_provider.py stream_chat_completion (default OAI-compat
branch): for provider_type=='openrouter', translate the request:
reasoning_effort in {low,medium,high} -> body['reasoning'] =
{'effort': <level>}
enable_thinking=True -> body['reasoning'] = {'enabled': True}
enable_thinking=False -> body['reasoning'] = {'enabled': False}
Matches the documented shape at
https://openrouter.ai/docs/guides/best-practices/reasoning-tokens
with effort and max_tokens mutually exclusive.
Frontend SSE reader:
- chat-adapter.ts: OpenRouter streams reasoning as a third shape we
did not handle yet: delta.reasoning_details is an array of parts
like {type: 'reasoning.text', text: '...'}. Pull text from every
part, merge with the existing delta.reasoning_content channel
used by Kimi/DeepSeek, and feed the combined string through the
same <think>...</think> wrap path so parseAssistantContent lifts
it into the reasoning panel. Anthropic/OpenAI Responses paths
already wrap on the backend, so they never set this field — no
cross-provider interference.
* studio/backend: surface OpenRouter SSE errors and router-chosen model in logs
The frontend showed 'Provider returned error' for some openrouter/free
requests with nothing on the backend side to triage from — the
existing 4xx error log only fires when the upstream returns a non-200
status code, but OpenRouter (and most OAI-compat providers) return
200 OK and emit the actual failure as an SSE error event mid-stream,
which our default-path stream loop forwarded verbatim without
logging.
Best-effort diagnostics on the default OpenAI-compat stream path:
- Peek at every `data:` line in the inner forward loop, parse JSON
best-effort (silently skip on failure so nothing is dropped).
- Count event types: delta / error / done.
- On any chunk containing an `error` field, emit a structlog WARNING
with the provider type and the error payload — same trail the
user would otherwise have to dig out of browser devtools.
- Latch the first non-empty `chunk.model` field. OpenRouter reports
the router-picked underlying model there per request, so the
finally-block summary log shows which free model handled the call.
In the finally block:
'openrouter stream complete (model=openrouter/free,
chosen=google/gemini-2.5-flash, events={delta: 47, done: 1})'
Zero overhead for non-error streams (a json.loads per chunk +
dict-key lookups). The structlog logger is already configured at
INFO; ERROR and WARNING surface in JSON logs without further setup.
Hoists `import json as _json` to module top so the default path can
reuse it; the existing in-function imports in _stream_anthropic and
_stream_openai_responses are now redundant but harmless.
* studio/chat: show router-picked model after 'openrouter/free:' in chip
When the user picks openrouter/free, the gateway routes each request
to a different underlying free model. Until now there was no way to
tell which one actually replied without reading the backend logs.
Surface the picked model in the active-model chip:
- chat-runtime-store gains lastOpenRouterChosenModel: string|null
plus a setter. Reset on every model switch unless the user stays
on openrouter/free.
- chat-adapter SSE loop latches chunk.model into the store on
every chunk whose top-level model differs from
openrouter/free, gated on the active checkpoint being
openrouter/free under an OpenRouter provider.
- chat-page externalModels useMemo appends :<chosen> to the display
name for the openrouter/free option when the store has a value,
so ModelSelector renders e.g.
'openrouter/free:google/gemini-2.5-flash'
in the chip. Other models unaffected.
- Model-switch callback in chat-page clears the cached value when
the user moves to any model other than openrouter/free, so the
chip never shows a stale suffix from a previous session.
* studio/chat: shorten openrouter/free chip to openrouter:<short-chosen>
The full display name in use was:
openrouter/free:inclusionai/ring-2.6-1t-20260508:free
The `:free` suffix on the underlying id already conveys 'free model',
which made the leading `/free` on the router id redundant, and the
`inclusionai/` org prefix was just noise crowding the chip.
Trim both. Now the chip renders as:
openrouter:ring-2.6-1t-20260508:free
Strictly a display change in chat-page externalModels useMemo — the
backend wire id stays `openrouter/free`, the runtime store still
caches the full `inclusionai/...:free` value, and the model-switch
clearing logic is unchanged.
* studio/providers: switch OpenRouter to remote listing with org allowlist + cap
Same shape as Hugging Face Inference. The curated list had only four
entries; remote listing fetches OpenRouter's full ~300-model
catalog via /v1/models and the new allowlist + limit scope it back
down to a usable picker.
- model_list_mode: remote (was curated)
- model_id_allowlist matches the prefixes:
openrouter | openai | anthropic | google | meta-llama | qwen
| mistralai | deepseek | moonshotai | inclusionai | zai-org
| z-ai
Anything outside drops out.
- model_id_limit: 20 — first 20 post-filter matches from the live
fetch; default_models stays seeded so the most useful canonical
ids are always visible regardless of API response order.
- default_models seed extended from 4 to 6 (openrouter/free,
openai/gpt-4o, anthropic/claude-sonnet-4-5, google/gemini-2.5-flash,
mistralai/mistral-large-2411, deepseek/deepseek-r1).
openrouter/free remains the first entry, so the dialog's
loadModels() union-merge (registryDefaults first, then remote,
deduped via Set) keeps it at the top of the picker.
* feat: external mistral thinking toggle
* studio/chat: fix TS2540 by replacing readonly ContentPart instead of mutating
The ContentPart type from @assistant-ui/react marks `text` as readonly,
so the coalesce-adjacent-same-type-part optimization in
parseAssistantContent failed the tsc build with:
parse-assistant-content.ts(15,10): error TS2540: Cannot assign to
'text' because it is a read-only property.
parse-assistant-content.ts(25,10): error TS2540: ...
This broke npm run build, the Studio installer's `building frontend...`
step, and every downstream CI job that runs against an installed
Studio (Mac/Windows/Linux variants of Studio API CI, GGUF CI, UI CI,
Tauri CI, Wheel CI).
Replace the last element with a fresh merged object instead of
mutating its `text` field. Same allocation profile as the previous
path (one object swap per merge), type-safe under the readonly
declaration. Behaviour unchanged.
* studio/backend: restore summary='auto' on OpenAI Responses reasoning body
A recent refactor dropped the `summary: 'auto'` field from the
reasoning config we send to /v1/responses. Without it OpenAI does
not emit reasoning summary events on most reasoning models, which
means our SSE handler has no <think>…</think> to wrap and the chat
reasoning panel stays blank for any gpt-5.x / o3 response.
The expected wire shape is:
body['reasoning'] = {'effort': '<level>', 'summary': 'auto'}
Two backend tests pin this:
- test_responses_reasoning_effort_included_when_requested (high)
- test_responses_reasoning_effort_xhigh_passthrough (xhigh)
Both were failing with AssertionError because the produced body
omitted `summary: auto`.
Restore the field. Skip it only for the explicit "off" case
(effort: 'none'), where summaries serve no purpose. The
enable_thinking=True fallback (no explicit effort) also pairs
medium effort with summary='auto' so that branch produces
reasoning text too.
* chat: external reasoning, OpenRouter curation, Think toggle fixes
* fix: opus and sonnet 4.6 xhigh --> max
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Two gaps surfaced when running tests/test_import_fixes_drift.py on a
fresh main install (transformers 4.57.6, trl 0.25.1, peft 0.19.1,
triton 3.5.1, vllm 0.15.1):
* triton_compiled_kernel test predicate was strict: only accepted
a class-level num_ctas. fix_triton_compiled_kernel_missing_attrs
installs the attrs via a wrapped __init__ (the post-3.6 shape),
so the detector fired DRIFT DETECTED even with the fix correctly
applied. Relax to also accept the wrapped-__init__ signature
(closure freevars / co_names probe). Mirrors zoo's already-relaxed
predicate (unsloth-zoo PR #639).
* tests/conftest.py applied ONLY the peft transformers_weight_conversion
stub fix via file-path loading. fix_vllm_guided_decoding_params /
fix_triton_compiled_kernel_missing_attrs / etc. never ran inside the
test process, so the corresponding drift detectors probed an
unpatched runtime state and pytest.fail'd. Replace the surgical
file-path loader with a guarded import unsloth (the GPU-free
harness above already pre-spoofs the device-type chain), so the
full import_fixes.py pass applies before pytest collects. Mirrors
unsloth-zoo's conftest pattern.
Local verification on transformers 4.57.6 + trl 0.25.1 + peft 0.19.1
+ triton 3.5.1 + vllm 0.15.1+cu130:
before: 16 passed, 2 failed (triton + vllm DRIFT DETECTED)
after: 18 passed, 0 failed
* tests: import_fixes drift detectors (HARD GATE on Core matrix)
Ports zoo PR #637's drift-detector pattern to unsloth as a new
test file + Core matrix step.
Background
unsloth/import_fixes.py is a 1932-line catalog of hand-rolled
patches for upstream regressions: protobuf MessageFactory drift,
datasets 4.4.x recursion, TRL tuple-vs-bool _*_available caching,
transformers PreTrainedModel.enable_input_require_grads source
pattern flip, triton CompiledKernel num_ctas missing, peft
weight-converter ctor compat, torch/torchvision pairing, vllm
guided_decoding params, etc. Today each fix runs unconditionally
at unsloth import; that's defensively correct but it means:
a fix becoming a no-op (upstream silently fixed itself) is
invisible.
a fix becoming needed-but-broken (upstream drifted in a new
way the workaround doesn't match) only surfaces as a
downstream crash.
tests/test_import_fixes_drift.py (18 tests)
One drift detector per fix_* / patch_* function in import_fixes.py.
Each test asserts the HEALTHY upstream shape absent the regression.
When the pathology is currently ACTIVE, fires
pytest.fail("DRIFT DETECTED: <fix function> needed because
<observation>") -- NEVER pytest.skip. CI must go RED so the
maintainer triages on the next PR.
First run on the current install surfaces 3 active drifts:
peft.utils.transformers_weight_conversion unimportable
(transformers.conversion_mapping missing) -- patch_peft_
weight_converter_compatibility will silently no-op.
triton 3.5.1 CompiledKernel lacks num_ctas + cluster_dims --
fix_triton_compiled_kernel_missing_attrs is live-needed.
vllm exposes only StructuredOutputsParams, not
GuidedDecodingParams -- fix_vllm_guided_decoding_params
is live-needed.
CI wiring (.github/workflows/consolidated-tests-ci.yml)
New step `import_fixes drift detectors (18 tests, HARD GATE)`
added to the Core matrix BEFORE the Bucket-A tests, so the matrix
cell fails fast on a real upstream regression. No
continue-on-error: a drift detection MUST go red.
This mirrors the same change just landed on
unslothai/unsloth-zoo#637 (commit ff5a3d8). Same fail-loud-on-drift
semantic; same set of fix functions covered; same 1:1 mapping
between test + import_fixes.py source-of-truth function.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* chore: trim verbose docstrings in import_fixes drift detectors
Strictly comment / docstring trims. AST-verified comment-only.
* Module header: 36 lines -> 7 lines.
* Per-test docstring: collapse each 7-15 line prose block to a 1-3
line lead naming the import_fixes.py function + line range plus
the one-sentence why; pytest.fail messages stay verbatim so a
red CI cell still names the upstream regression.
* Helper docstrings (_safe_version, _is_custom_torch_build): drop.
* Inline narrative comments inside test bodies: drop.
* Section dividers and licence header: untouched.
Net: 700 -> 537 lines, zero behaviour changes.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Strictly comment / docstring trims. AST-verified against 12295c1f via
scripts/verify_trim_comment_only.py:
* unsloth/import_fixes.py: collapse the 32-line peft+transformers-4.x
drift header to 10 lines; remove redundant per-stub docstrings and
per-step numbered comments inside fix_peft_transformers_weight_
conversion_import; keep one-line docstrings on helpers + on the
public entry-point.
* unsloth/_gpu_init.py: collapse the 8-line preamble above
fix_peft_transformers_weight_conversion_import() to 4 lines.
* tests/conftest.py: collapse the 13-line block comment above
_apply_unsloth_peft_import_fix_for_tests to 5 lines; tighten three
internal comments.
* import_fixes: stub transformers.conversion_mapping so peft 0.19.x imports on transformers 4.x
patch_peft_weight_converter_compatibility currently opens with
try:
from peft.utils import transformers_weight_conversion as twc
except (ImportError, AttributeError):
return
which silently no-ops on (peft 0.19.x, transformers 4.57.x): peft's
transformers_weight_conversion module unconditionally imports two
transformers-v5 submodules at module top
from transformers.conversion_mapping import ...
from transformers.core_model_loading import ...
and neither submodule exists on transformers < 5. peft itself only USES
those submodules inside an is_transformers_ge_v5 branch, but the top of
file import still explodes with
ModuleNotFoundError: No module named 'transformers.conversion_mapping'
The bare except above swallows that, so the weight converter compat
wrap never gets installed, and any downstream code that later does
from peft.utils import transformers_weight_conversion crashes with the
same ModuleNotFoundError.
Fix: synthesise minimal stub modules for transformers.conversion_mapping
and transformers.core_model_loading, install them into sys.modules, and
re-import peft.utils.transformers_weight_conversion so the kwargs compat
wrap can succeed on top. The stubs expose exactly the symbols peft 0.19.x
pulls in at module top (Concatenate / ConversionOps are real subclassable
classes since peft subclasses them as PeftConcatenate / FlattenDims /
PermuteDims), so peft's own class creation succeeds. None of the stubbed
callables actually fire on the 4.x branch because peft's runtime
is_transformers_ge_v5 gate keeps them unreachable.
Gating contract (strict no-op outside the (peft 0.19.x, transformers 4.x)
combination):
* No-op if peft is not installed.
* No-op if peft.utils.transformers_weight_conversion already imports
clean (transformers v5+, or any peft fork off the v5 path).
* Strictly additive: only stubs submodules that are currently missing
from sys.modules / find_spec. We never overwrite the real
transformers.conversion_mapping / transformers.core_model_loading
on transformers v5+.
* Idempotent: sentinel attribute (__unsloth_stub__) on the stub modules
makes a second call return False, a third call return False, etc.
* Surfaces drift unchanged: if peft fails for some reason OTHER than
these two specific missing submodules, the original ImportError is
left for the caller's own try/except to take over.
Forwards / backwards compatibility:
* transformers 4.57.6 -> install stubs.
* transformers 5.x (real submodules) -> first-import probe succeeds,
return False, never touch sys.modules.
* TRL 0.22 / 0.27 / 1.x -- none of these import either submodule
directly; they reach the peft conversion module (if at all) through
peft.tuners.tuners_utils, behind peft's own is_transformers_ge_v5
gate. Stubs are therefore unreachable from TRL on a 4.x install,
and on a 5.x install the real submodules win the import race.
* peft 0.18 / 0.19 / 0.20 -- the symbols stubbed cover the union of
what peft pulls at module top across the 0.19.x line; older peft
that doesn't import the v5 submodules at all hits the cheap
first-import-probe exit and we never touch sys.modules.
Wired into unsloth/_gpu_init.py to run BEFORE
patch_peft_weight_converter_compatibility (otherwise that function's
bare except would still silently no-op). Mirrors the equivalent fix
shipped in unsloth-zoo (the zoo-side stub installs itself via
apply_import_fixes() at zoo import time, but a user can run
unsloth without the zoo fix on an older unsloth_zoo, so the unsloth
side needs to own its own copy of the workaround).
tests/conftest.py is updated to pre-apply this specific fix via the
standalone import-fixes module so the GPU-free drift detector test
(tests/test_import_fixes_drift.py::test_peft_transformers_weight_conversion_importable_and_signature)
sees the same patched state that a real ``import unsloth`` would.
The pattern mirrors unsloth-zoo's tests/conftest.py
_apply_zoo_import_fixes_for_tests helper, scoped to just the peft fix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: contain export and dataset paths under their configured roots
resolve_under_root and resolve_dataset_path previously returned absolute
paths unchanged, so an authenticated client could supply
save_directory="/tmp/escape" (or any other absolute path) and have the
exporter drop adapter files anywhere the server user could write. This
turned up during a recent audit pass where an authenticated POST to
/api/export/export/lora with save_directory="/tmp/lora_escape_test"
returned 200 and wrote adapter_model.safetensors, adapter_config.json,
and tokenizer files under /tmp.
The fix is two-layered:
storage_roots.py adds an _assert_contained(resolved, root) helper that
runs after path resolution and rejects any result whose realpath does
not sit under realpath(root). resolve_under_root now rejects '..'
segments and null bytes outright, and only accepts absolute inputs when
they are already inside the configured root (internal call sites that
re-resolve a stored absolute path stay idempotent;
worker.py:resolve_output_dir(output_dir) etc. continue to work).
resolve_dataset_path picks up the same containment rule, scoped to the
three dataset roots.
models/export.py adds field_validator("save_directory", mode="before")
to ExportCommonOptions and ExportGGUFRequest so bad input fails fast at
422 with a clear message rather than a 500 deep inside the resolver.
The validator rejects empty/whitespace, null bytes, control chars,
strings longer than 255 chars, absolute paths, and '..' segments.
routes/export.py:_export_details now returns os.path.relpath(output_path,
exports_root()) so the Export Complete dialog and /api/models/loras no
longer leak the absolute install prefix to the UI; the basename is
used as a last-resort fallback.
Verified end to end:
- POST /api/export/export/lora {"save_directory":"/tmp/foo"} -> 422
"save_directory must be a name or relative path under the export
root; absolute paths are rejected". /tmp/foo is not created.
- "../../etc/escape" -> 422 "may not contain '..' segments".
- save_directory="my_subdir" -> still accepted (400 only because the
test had no checkpoint loaded yet, not because of validation).
- Internal idempotent re-resolve via resolve_export_dir(absolute path
that is already under exports_root) returns the same path unchanged.
* studio/sandbox: harden bash + python tool execution
The sandboxed Bash and Python tool channels in Chat ran with a thin
preexec hook (PR_SET_NO_NEW_PRIVS + RLIMIT_FSIZE only). Bash had a
small word blocklist; Python had an AST safety pass aimed at
signal-tampering and shell-escape primitives. An audit pass showed
several gaps that a tool-calling model could trigger inadvertently:
- bash curl/wget/nc reached AWS IMDSv2 and returned live STS
credentials for the instance role.
- python "import socket; s.connect((169.254.169.254, 80))"
reached the same endpoint regardless of the bash blocklist.
- "cat /etc/passwd" was blocked at the bash side (because "passwd"
is in the blocklist), but "open('/etc/passwd').read()" in Python
happily returned its contents.
- "chr(115)+chr(117)+chr(100)+chr(111)" style dynamic-arg
construction slipped through the AST shell-escape check.
- The supervisor used proc.kill() on timeout, which only signals
the immediate pid; bash-backgrounded children survived. A fork
bomb could spawn for the full 300s timeout window.
- Session work directories under ~/studio_sandbox/<id>/ were
created with default umask (0o755), so any other UID on the host
could enumerate them.
- session_id sanitisation used a one-shot str.replace("..",""),
which is non-iterative and a small footgun.
This commit takes a conservative middle path: the sandbox still
runs as the Studio UID with no namespace tricks where the kernel
disallows them, but every chokepoint is tightened.
_sandbox_preexec now:
- calls os.setsid() so children share a process group; the
supervisor uses os.killpg(SIGKILL) on timeout/cancel so
backgrounded children die with the parent (new _kill_process_tree
helper, wired into _cancel_watcher and both _bash_exec /
_python_exec timeout branches).
- calls os.umask(0o077) so files the child writes default to 0o600.
- applies PR_SET_PDEATHSIG=SIGKILL so an orphaned child dies if
Studio exits.
- best-effort unshare(CLONE_NEWNET) for a private network namespace
(failure is logged and swallowed; defense-in-depth is still in
place via the bash blocklist and the AST checker below).
- sets RLIMIT_NPROC=10000 (tunable via UNSLOTH_STUDIO_SANDBOX_NPROC),
RLIMIT_AS=8GB, RLIMIT_CPU=300, RLIMIT_NOFILE=1024. The 10k NPROC
figure is chosen to sit well above the ~500 LWPs a healthy Studio
+ llama-server combination already uses while still capping a
runaway fork bomb. NPROC counts LWPs per real UID, so a lower
figure (e.g. 256) starves legitimate bash forks
("bash: fork: retry: Resource temporarily unavailable").
_get_workdir:
- rejects session_id that doesn't match [A-Za-z0-9_-]{1,64};
non-matching values bucket into a shared "_invalid" dir.
- chmod 0o700 on both the workdir and on ~/studio_sandbox/ so
other UIDs cannot read another session's contents.
_BLOCKED_COMMANDS_COMMON gains: doas, pkexec, halt, poweroff, curl,
wget, nc, ncat, netcat, socat, ssh, scp, sftp, rsync, eval, source.
The intent is to keep general bash usage working (echo, ls, pipes,
loops, for, head, etc.) while denying the obvious egress and
escalation paths.
The AST checker (_check_signal_escape_patterns) is split into the
existing shell/signal/loop checks plus a new narrow IO denylist:
- Always flag non-literal args to anything in _SHELL_EXEC_FUNCS,
not just _STRING_SHELL_FUNCS. Closes the dynamic-arg bypass.
- Reject calls to socket.create_connection, socket.socket().connect,
urllib.request.urlopen, http.client.HTTP*Connection, requests.*,
httpx.* whose literal host argument is in a cloud-metadata
denylist (169.254.169.254 + 169.254.* + 100.64.*, plus the
GCP/Alibaba/ECS metadata hostnames and IPv6 link-local). Public
hosts (example.com, huggingface.co, ...) still work. Dynamic
hosts cannot be statically blocked; mitigated by the bash
blocklist + the netns where the kernel allows it.
- Reject literal open("/etc/passwd"), /etc/shadow, /etc/sudoers,
/etc/ssh/*, and /proc/<pid>/environ. Other files
(/etc/os-release, /etc/hostname, /tmp/*, user dirs) still work.
The _check_code_safety summariser is updated to include the new
network_calls and sensitive_file_reads buckets in its error string.
Regression-checked: echo, sleep, ls /tmp, for loops, piped helpers
(echo a | tr a A), urllib.request.urlopen("http://example.com"),
socket.getaddrinfo("example.com",80), open("/etc/os-release"),
open("/tmp/...","w") all still succeed. curl, wget, nc, ssh, rm,
socket.create_connection(("169.254.169.254",80)),
open("/etc/passwd"), open("/proc/self/environ") all correctly
blocked.
* studio: rate-limit login, rotate refresh tokens, add logout, security headers, gate bootstrap injection
A pass over the auth surface found a cluster of related issues that this
commit closes together.
Login (routes/auth.py):
- Add an in-memory per-IP login rate limiter. Five failed POSTs to
/api/auth/login inside a 60s window produce 429 with Retry-After.
A successful login clears the bucket. Previously 30 wrong passwords
in under one second was accepted as 30x 401, which combined with
the (now fixed) admin-username leak from /api/auth/status made
brute-force trivial against a small password.
Logout (routes/auth.py):
- New POST /api/auth/logout returns 204 and calls
storage.revoke_user_refresh_tokens(subject) so the refresh token
is no longer valid. Previously POST /api/auth/logout returned 405
and there was no way to invalidate refresh tokens short of
changing the password. Frontend session.ts already calls
clearAuthTokens() to drop localStorage; the new endpoint lets the
client also tell the server to revoke server-side state.
Refresh-token rotation (routes/auth.py + auth/storage.py):
- New storage.consume_refresh_token(token) atomically validates +
deletes a refresh token, returning (username, is_desktop). The
/api/auth/refresh handler now mints both a new access AND a new
refresh token; the supplied token becomes invalid. Replaying a
consumed refresh returns 401 "Invalid or expired refresh token".
The previous refresh_access_token helper is left in place for
callers that intentionally want the non-rotating shape; nothing
in the route layer uses it now.
/api/auth/status no longer leaks default_username (models/auth.py +
routes/auth.py):
- AuthStatusResponse.default_username becomes Optional[str] with a
None default; the handler always returns None. The frontend already
hardcodes HIDDEN_LOGIN_USERNAME = "unsloth" (auth-form.tsx:82), so
no UI change is required.
window.__UNSLOTH_BOOTSTRAP__ no longer auto-injects (main.py):
- _inject_bootstrap is now opt-in via the
UNSLOTH_STUDIO_INJECT_BOOTSTRAP env var. The previous default
(inject whenever requires_password_change is true) embedded the
plaintext bootstrap password into the first-boot HTML for any
caller that hit /, /change-password, or any unknown SPA path.
Browser extensions and any XSS payload on the page could read it
trivially. With the new gate the bootstrap password lives only in
the auth/.bootstrap_password file (mode 0o600) where it has always
been; users typing it into a current-password field is the right
UX. routes/auth.py:change_password also clears
app.state.bootstrap_password defensively.
Security headers + server fingerprint (main.py + run.py):
- New SecurityHeadersMiddleware adds Content-Security-Policy,
X-Frame-Options: DENY, X-Content-Type-Options: nosniff,
Referrer-Policy: no-referrer,
Permissions-Policy: camera=(), microphone=(), geolocation=(),
interest-cohort=(), and stamps server: unsloth-studio so the
generic uvicorn banner no longer fingerprints the stack. The
uvicorn.Config gains server_header=False so it stops emitting its
own Server header.
/api/health minimisation (main.py):
- Unauthenticated GET /api/health returns just
{"status":"healthy","timestamp":...} so load-balancer liveness
probes keep working without leaking version, device_type,
chat_only, desktop_protocol_version, or studio_root_id to
arbitrary callers. A request that presents a valid Bearer token
still gets the full diagnostic payload so internal launchers and
sibling-Studio detection (which compares studio_root_id) keep
working.
Verification:
- 30 wrong-password POSTs to /api/auth/login -> first 5 = 401, 6th
through 30th = 429.
- POST /api/auth/logout with a fresh token -> 204. The matching
refresh token then fails 401.
- Login -> R1; /api/auth/refresh with R1 -> new access + R2 (R2 !=
R1); /api/auth/refresh with R1 again -> 401; /api/auth/refresh
with R2 -> still succeeds once and rotates again.
- curl /api/auth/status -> default_username: null.
- curl http://127.0.0.1/ does not contain __UNSLOTH_BOOTSTRAP__.
- curl -I / shows CSP, X-Frame-Options: DENY,
X-Content-Type-Options: nosniff, Referrer-Policy: no-referrer,
Permissions-Policy, and server: unsloth-studio.
- curl /api/health unauthenticated -> {status, timestamp} only.
curl with Authorization: Bearer <valid> -> full payload.
- Existing /api/system, /api/models/list, /api/train/status,
/api/inference/status, /api/auth/api-keys, login flow, SPA root
all still return 200 after the changes (regression smoke).
* studio: add SecurityHeadersMiddleware, MaxBodyMiddleware, /recipes redirect, gate _inject_bootstrap, minimise /api/health
This commit lands the main.py-side changes that share a single
middleware-registration spot. They are kept together because every
change here is either (a) a top-level middleware definition that has
to be added next to LoggingMiddleware, or (b) a route handler at the
same file-level.
SecurityHeadersMiddleware (Content-Security-Policy, X-Frame-Options:
DENY, X-Content-Type-Options: nosniff, Referrer-Policy: no-referrer,
Permissions-Policy, server: unsloth-studio). The previous responses
emitted no CSP, no XFO, no Referrer-Policy and were stamped
server: uvicorn.
MaxBodyMiddleware rejects POST/PUT/PATCH on the inference / dataset /
data-recipe / train / export prefixes when Content-Length exceeds
UNSLOTH_STUDIO_MAX_BODY_MB (default 100). The audit hit this by
attaching a 50 MB plain-text file to a chat message and watching
Studio base64-encode it into the JSON body; uvicorn has no enforced
cap so the only previous guard was the per-file 50 MB ceiling that
data-recipe upload routes already enforce. The new middleware extends
that ceiling to the OpenAI-compat path that the Chat attachments
flow through. Verified: a 200 MB JSON POST to /v1/chat/completions
returns HTTP 413 "Request body too large (209,715,264 bytes; max
104,857,600)". A small valid request continues to reach the handler.
_inject_bootstrap is gated behind UNSLOTH_STUDIO_INJECT_BOOTSTRAP.
The previous default was to inline window.__UNSLOTH_BOOTSTRAP__ =
{username, password} into the first-boot HTML whenever
requires_password_change was true, which exposed the plaintext
bootstrap password to any browser extension, page script, or LAN
caller on -H 0.0.0.0. The bootstrap password remains in the on-disk
.bootstrap_password file (mode 0o600) where it has always lived;
users typing it into a current-password field is the right UX.
/api/health unauthenticated returns {"status":"healthy","timestamp":
...} only; the previous payload (version, device_type, chat_only,
desktop_protocol_version, supports_desktop_auth, studio_root_id,
native_path_leases_supported) is preserved for callers that present
a valid Bearer token, so internal launchers and sibling-Studio
detection (which compares studio_root_id) keep working.
/recipes -> /data-recipes 308 redirect. The Data Recipes page lives
at /data-recipes; users typing /recipes hit the SPA catch-all and
saw "Not Found". The redirect also preserves any tail path, so
/recipes/<rest> -> /data-recipes/<rest>.
Verified end to end with curl: CSP / XFO / X-Content-Type-Options /
Referrer-Policy / Permissions-Policy all present on /, server header
is now unsloth-studio (uvicorn's own banner is suppressed via
server_header=False in run.py from the auth-batch commit). Followed
the /recipes redirect lands on the SPA HTML.
* studio: bound TrainingStartRequest hyperparameters at the schema level
POST /api/train/start accepted any value for learning_rate, batch_size,
max_steps, max_seq_length, warmup_steps, warmup_ratio, num_epochs,
save_steps, weight_decay, gradient_accumulation_steps, lora_r,
lora_alpha and lora_dropout, including -1, 0, 1e9, and non-numeric
strings like 'abc' or 'two' (which silently coerce to 0 in the
trainer). Probing showed the API returning 200 to learning_rate=-1
and batch_size=0; only max_steps had any partial clamping.
This commit adds field_validator on every numeric hyperparameter.
Bounds are chosen wide enough to span realistic single-host
configurations (B200 with 180 GB of memory comfortably fits the
upper end) while rejecting the values that always produce broken
training:
- learning_rate: parses str/float, requires 0 < lr < 1.0. Non-numeric
input raises with "learning_rate must be parseable as float (got
'abc')" instead of silently coercing to 0.
- batch_size: [1, 1024].
- gradient_accumulation_steps: [1, 4096].
- num_epochs: [1, 1000].
- max_steps: [1, 1_000_000].
- max_seq_length: [1, 131072].
- warmup_steps: [0, max_steps].
- warmup_ratio: [0.0, 1.0].
- save_steps: [0, 1_000_000].
- weight_decay: [0, 10] (typical 0..0.1).
- lora_r: [1, 512].
- lora_alpha: [1, 1024].
- lora_dropout: [0.0, 1.0).
Each validator names the offending field in its ValueError message
so the 422 response body identifies which input is bad. The
learning_rate validator returns its result as str (the schema field
type is str("2e-4") for backwards compatibility) so existing call
sites that float() the value continue to work.
Verified:
- learning_rate=-1 -> 422 "learning_rate must be > 0 (got -1.0);
typical range is 1e-6 .. 1e-3".
- learning_rate='abc' -> 422 "must be parseable as float".
- batch_size=-1 / 0 / 999999 -> 422 "batch_size must be in [1, 1024]".
- batch_size='two' -> 422 (pydantic int parser).
- max_steps=0 / -5 -> 422 "must be a positive int".
- max_seq_length=200000 -> 422 "must be in [1, 131072]".
- warmup_ratio=2.5 -> 422 "must be in [0.0, 1.0]".
- lora_dropout=1.5 -> 422 "must be in [0.0, 1.0)".
- Valid request with learning_rate='2e-4', batch_size=1, max_steps=5
passes validation and the training run starts as normal.
* studio: redact image-decode errors, clean checkpoint dirs on cancel, tolerate Stop-button + tool-result message shapes
Three small fixes that fall under "do not let the audit findings
become user-visible papercuts".
routes/inference.py - image-decode error redaction (the audit hit
this with a 0-byte / malformed / wrong-extension image upload). The
three image-normalise sites previously raised HTTPException(400,
detail=f"Failed to process image: {e}"). When PIL raised
UnidentifiedImageError(io.BytesIO(raw)) the message string included
"<_io.BytesIO object at 0x7e40a5d7bf60>", leaking both the Python
class name (confirming the PIL/io stack) and a heap address (mildly
useful for ASLR-bypass chaining if another memory-corruption bug is
ever found). Each site now catches UnidentifiedImageError and
returns the generic "Unsupported or corrupt image format"; the
fall-through generic except returns "Failed to process image". No
exception-repr is interpolated into a response body anywhere along
these paths.
core/training/training.py - checkpoint cleanup on cancel. When a
user clicks Cancel Training, the trainer flips _cancel_requested=True
and the supervisor force-terminates the subprocess. The trainer
writes checkpoint-<step> directories under output_dir every
save_steps; previously these survived the cancel and accumulated on
disk (the audit recorded ~67 MB stuck after a 200-step cancel with
save_steps=20). New helper _cleanup_cancelled_checkpoints(output_dir)
globs checkpoint-<int> entries and removes them. It is gated by a
realpath containment check against outputs_root() so it cannot
accidentally rmtree anything outside the configured outputs root.
force_terminate() invokes the helper after the subprocess join when
_cancel_requested is true. Stop-and-Save runs are unaffected because
that path keeps _cancel_requested=False.
models/inference.py - chat message shape tolerance. Two related
frontend interactions used to crash the request validator:
- After the Stop button truncates a generation, the frontend
retained {role:"assistant", content:""} in the conversation
history and replayed it on the next send. ChatMessage previously
required role="assistant" to have non-empty content or tool_calls,
so the next message returned 422 and the thread was permanently
broken. The validator now normalises empty assistant content to
None so the request round-trips and the trailing empty turn can
be ignored downstream.
- The frontend's second-round tool POST drops the streamed
tool_call_id, hitting the strict-spec check "role=tool requires
tool_call_id". The validator now synthesises an opaque id
(call_<8 hex>) when missing, so the request reaches the handler
and the model's final summarising response gets generated. The
proper fix lives in the frontend (carry the streamed id through
the second POST) and will follow.
Verified end to end with curl: HTTP 400 (model not loaded) on both
the empty-assistant history shape and the tool-result-without-id
shape, instead of HTTP 422 from the schema validator.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten code comments from security-hardening pass
Trim verbose docstrings and inline finding references added in the
previous commits in this branch. Functionality unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: await get_current_subject in /api/health and make refresh-token consumption atomic
The /api/health auth probe called get_current_subject(creds) without
awaiting it. The coroutine object is truthy, so any caller presenting a
Bearer header (valid or not) received the full diagnostic payload
including version, device_type, studio_root_id, etc. Await the coroutine
and treat HTTPException as 'fall back to the minimal liveness payload'.
consume_refresh_token did SELECT then DELETE WHERE id under default
autocommit isolation. Two concurrent POST /api/auth/refresh requests
could both win the SELECT before either DELETE ran, defeating
single-use refresh-token rotation. Replace with a single
DELETE ... WHERE token_hash = ? AND expires_at >= ? RETURNING ...
statement so the validate-and-delete lands as one atomic op under
SQLite's write lock (3.45.1 supports RETURNING; min was 3.35).
* studio: enforce body cap on chunked uploads and drop unsafe-inline from script-src
MaxBodyMiddleware previously only inspected the declared Content-Length
header; clients omitting it or sending Transfer-Encoding: chunked
bypassed the cap and could still drive an OOM via the downstream
JSON / file readers on /v1/chat/completions, /api/inference, /api/data-recipe,
/api/datasets, /api/train, /api/export. Rewrite as a raw ASGI middleware
that drains and counts http.request frames, replies 413 once the running
total exceeds UNSLOTH_STUDIO_MAX_BODY_MB before invoking the FastAPI
handler, and replays the buffered body to downstream so route code that
calls request.json() / await request.body() works unchanged.
CSP previously included 'unsafe-inline' on script-src, which defeats the
main XSS protection. The frontend bundle does not need inline scripts;
the only inline <script> the backend ever emits is _inject_bootstrap,
which is opt-in via UNSLOTH_STUDIO_INJECT_BOOTSTRAP. Drop 'unsafe-inline'
from script-src by default; when _inject_bootstrap fires, generate a
per-response nonce, embed it on the inlined <script>, and have
SecurityHeadersMiddleware splice 'nonce-XXX' into the CSP for that one
response (the internal x-internal-script-nonce header is popped before
the response leaves the server). 'unsafe-inline' stays on style-src for
Vite-injected styles.
* studio: drop empty assistant sentinel before passthrough
ChatMessage._validate_role_shape normalises role="assistant", content=""
(the post-Stop sentinel emitted by the frontend) to content=None so the
in-process path can drop it via _extract_content_parts. The passthrough
path then ran m.model_dump(exclude_none=True), which strips the now-None
content key entirely, sending {"role":"assistant"} to llama-server / the
OpenAI-compat backend. That fails upstream and leaves the user without a
recoverable Stop->resume.
Add _drop_empty_assistant_sentinels and call it at both passthrough
message origins: _openai_messages_for_passthrough (covers
/v1/chat/completions and the Responses API which routes through it) and
the anthropic_messages_to_openai output before
_anthropic_passthrough_*. Assistant messages that carry only tool_calls
(no content) are preserved.
* studio/tests: cover audit-fix surfaces and rebase pre-existing tests
Adds and updates pytest coverage for the four bot-flagged audit fixes
landed earlier in this branch and rebases two pre-existing tests that
were broken by the relaxed-validator and /api/health auth-gate changes.
studio/backend/tests/test_middleware.py (new)
MaxBodyMiddleware: small protected, large declared, unprotected
passthrough, chunked-upload-over-cap rejection (the regression for
the original Content-Length-only gap), and chunked-under-cap replay.
SecurityHeadersMiddleware: script-src no longer carries
'unsafe-inline', style-src still does, default headers
(XFO/XCTO/Referrer-Policy/Permissions-Policy/server), and the
internal x-internal-script-nonce header is consumed by the
middleware and converted to 'nonce-XXX' in the CSP.
/api/health: no auth -> minimal, invalid Bearer -> minimal
(the await regression), valid Bearer -> full diagnostic payload.
studio/backend/tests/test_desktop_auth.py
consume_refresh_token: second-call returns None, expired returns
None, and a 64-thread concurrent pile-up against the same hash
produces exactly one successful consumer (regression for the
SELECT-then-DELETE race).
test_health_response_reports_desktop_capability_fields: rebase
against the new health_check(request) signature by going through
TestClient with a real bearer instead of asyncio.run-ing the
handler directly.
studio/backend/tests/test_openai_tool_passthrough.py
Pin the new ChatMessage tolerance: assistant without content or
tool_calls is tolerated (normalises content -> None), empty-string
and empty-list assistant content normalise to None, and a missing
/ empty tool_call_id on role='tool' is synthesised as call_<hex>
rather than raising. Tests for _drop_empty_assistant_sentinels
cover the three drop shapes (empty string, empty list, missing
content key), preservation of assistant text and tool_calls-only
messages, and end-to-end through
_openai_messages_for_passthrough.
studio/backend/main.py
SecurityHeadersMiddleware.dispatch used response.headers.pop(...)
for the nonce-header handoff; Starlette's MutableHeaders has no
pop. Read-then-del so the internal handoff header is still
stripped before the response leaves the server.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/tests: rebase three more pre-existing CI tests against this branch
CI on PR #5375 was red on three tests that were tuned for behaviour
predating this branch. Updates each so the assertions match what the
audit fixes intentionally changed; no production code touched.
studio/backend/tests/test_trained_model_scan.py
test_scan_trained_models_includes_lora_and_full_finetune_outputs
passed an absolute tmp_path through scan_trained_models, which now
runs resolve_output_dir / _assert_contained against outputs_root().
Repoint outputs_root() at tmp_path via monkeypatch so the fixture
dirs land under the configured root and the realpath containment
check passes.
tests/test_studio_install_workspace_guard.py
test_health_endpoint_exposes_studio_root_id_not_raw_path read
the first 1500 bytes after @app.get("/api/health") and asserted on
the studio_root_id literal. The handler grew (unauth short-circuit
+ await dependency gate) and the literal slid past the byte window.
Replace the fixed window with a slice up to the next top-level
@app.* decorator so the test surveys the whole handler regardless
of size.
tests/studio/studio_api_smoke.py
The "login burst (5x wrong pw) -> 401 each" assertion was tagged
"When/if we add one, this assertion updates in the same PR." We
added the per-IP rate-limit in routes/auth.py
(_LOGIN_MAX_FAILS=5/60s) but missed the assertion update. Rewrite
the burst probe to observe the new invariant: at least one 401,
eventual transition to 429, and Retry-After present on the 429.
Adds a small _login_with_headers helper since the existing login()
helper drops response headers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci(studio-ui): set UNSLOTH_STUDIO_INJECT_BOOTSTRAP=1 for Playwright Studios
The Chat UI Playwright test drives the first-boot change-password
form, which (per playwright_chat_ui.py step "1. Change-password
through the UI") pre-seeds the hidden current_password field from
window.__UNSLOTH_BOOTSTRAP__. That global is only emitted when the
backend's _inject_bootstrap path fires, which since the security
pass on this branch is gated behind UNSLOTH_STUDIO_INJECT_BOOTSTRAP
and defaults to off. Without the global, the React form's
current_password validator never satisfies, the submit button stays
disabled, and the composer.wait_for() probe times out on
/change-password.
Re-enable injection only for the CI Studios that drive the chat UI
across linux/mac/windows. Production deployments are unaffected: the
env var has to be explicitly opted into, and the on-disk
auth/.bootstrap_password remains the source of truth for human users
typing the password in by hand.
Covers all eight Studio launch sites: the primary chat-ui boot and
the "extra UI tests" boot for each of the three OSes, plus the
pipeTransport JSON-crash retry relaunches in the macOS workflow that
re-spawn Studio mid-job.
A follow-up frontend PR will add a visible current_password input so
the form satisfies its own validator without needing the bootstrap
auto-fill at all; once that lands this CI knob can come back out.
* studio/sandbox: drop unshare(CLONE_NEWNET); add trusted-host allowlist; block sandbox file uploads; raise CPU rlimit default to 600 s
CLONE_NEWNET inside _sandbox_preexec silently killed every outbound
HTTP request from sandboxed Python whenever the kernel allowed
unprivileged user namespaces. requests.get('https://huggingface.co'),
urllib.request.urlopen('https://en.wikipedia.org/wiki/...'),
socket.connect(('arxiv.org', 443)) all failed despite the AST visitor
intending to allow them. The bash blocklist (curl / wget / nc / ssh /
scp / sftp / rsync / socat / eval / source) plus the AST-level
metadata-host denylist still carry the network policy after this
change; CLONE_NEWNET was redundant with both.
Add _TRUSTED_PUBLIC_HOST_LITERALS + _TRUSTED_PUBLIC_HOST_SUFFIXES
(~100 informational hosts: Wikipedia language subdomains, Wikimedia,
Wikidata, Google search, Bing, DuckDuckGo, HuggingFace, GitHub,
raw.githubusercontent.com, arXiv, StackOverflow / Stack Exchange,
MDN, docs.python.org, PyTorch / TensorFlow / NumPy / pandas docs,
pypi / files.pythonhosted.org / npmjs / crates.io, ReadTheDocs,
arXiv, Britannica, BBC / Reuters / Nature / Science, NASA / CDC /
NIH / WHO open data, api.weather.gov). The visitor now blocks
literal hosts that are neither metadata nor trusted with a short
LLM-readable string so the model can retry with an allowed source
instead of choking on a multi-line error.
Block upload-shape calls regardless of host: requests.post / put /
patch / delete / request with files= or data=open(...) /
data=bytes_literal; httpx equivalents; urllib.request.urlopen /
Request with data=...; HuggingFace upload_file / upload_folder /
upload_large_folder / create_commit (module-level FQ paths AND
method-name match on any receiver). Message: "Blocked: file upload
disallowed in sandbox".
Bump UNSLOTH_STUDIO_SANDBOX_CPU_S default 300 -> 600 s so long
agentic chains that span multiple tool calls don't get SIGXCPU'd
mid-stride. Env-var override path is unchanged.
Host normalisation now strips trailing dot, userinfo @, and explicit
port before allowlist / denylist comparison so trailing-DNS-dot,
userinfo-smuggling, and explicit-:443 URLs are decided correctly.
* studio: raise default request-body cap from 100 MB to 500 MB
UNSLOTH_STUDIO_MAX_BODY_MB default goes 100 -> 500 to comfortably
cover vision + audio + multi-recipe-batch JSON payloads. The
MaxBodyMiddleware stream-counting logic from this branch's earlier
06ec088 already handles chunked bodies up to the new cap; env-var
override path is unchanged for callers that want a tighter limit.
* studio/auth: restore /api/auth/status.default_username to 'unsloth'
This branch's earlier b39e9a4 changed default_username to None on the
public /api/auth/status endpoint so the username field didn't leak to
unauthenticated callers. In practice this regressed third-party
clients (and the in-tree React login form's pre-fill UX) without
adding meaningful security: the bootstrap password is the actual
secret, and the username 'unsloth' is the documented default.
Pin default_username to storage.DEFAULT_ADMIN_USERNAME ('unsloth')
and tighten the response model so the field is required rather than
Optional. Anyone who needs anonymisation can still reach for an
allow-list deployment with auth disabled.
* studio/training: raise max_seq_length / batch_size / lora_r / lora_alpha caps
This branch's 7102815 introduced field validators with conservative
caps. The follow-up loosens them so long-context experiments and
high-rank LoRA exploration aren't gated at the schema layer:
_MAX_BATCH_SIZE 1024 -> 4096
_MAX_SEQ_LENGTH 131_072 -> 2_000_000 (2M tokens)
lora_r cap 512 -> 16_384 (_MAX_LORA_R)
lora_alpha cap 1024 -> 32_768 (_MAX_LORA_ALPHA)
_MAX_GRAD_ACCUM / _MAX_STEPS / _MAX_EPOCHS / lora_dropout /
warmup_ratio / weight_decay are unchanged. Hardware (VRAM, host
RAM, kernel launch latency) is now the binding constraint at the
new caps, which is the correct ordering -- the validator stays a
sanity check on -1 / 0 / 'abc' style garbage, not a usability gate.
* studio/tests: cover sandbox allowlist + upload block + raised training caps
studio/backend/tests/test_sandbox_tools.py (new):
TestMetadataHostDenylist -- short "Blocked: cloud-metadata host"
message on AWS IMDS, GCP metadata,
Alibaba ECS, AWS IPv6 IMDS, 169.254/16.
TestTrustedHostAllowlist -- Wikipedia (any language subdomain),
Google, DuckDuckGo, HF, raw GitHub,
arXiv, StackOverflow / family,
MDN, docs.python.org, pypi, BBC,
api.weather.gov, NumPy / PyTorch docs.
TestUntrustedHostBlock -- example.com / random unlisted host
rejected with the short "Blocked: host
not in sandbox allowlist; use an
allowed informational source" message.
Dynamic URLs (computed var) still pass
-- documented limit of static analysis.
TestHostNormalization -- trailing dot, explicit :443, uppercase,
userinfo-@-smuggle all decided
correctly without false-block /
false-pass.
TestUploadDenylist -- requests / httpx / urllib.urlopen with
files= / data=open / data=bytes,
HfApi().upload_file / upload_folder /
create_commit, module-level
huggingface_hub.upload_folder. POST
json= to trusted host still passes.
TestSandboxCpuRlimitDefault -- pin UNSLOTH_STUDIO_SANDBOX_CPU_S=600
default and confirm CLONE_NEWNET
source line is gone.
TestMaxBodyDefault -- pin UNSLOTH_STUDIO_MAX_BODY_MB=500
default.
studio/backend/tests/test_studio_train_validation.py (new):
Pin at-cap-accepts / over-cap-rejects boundaries for
max_seq_length=2_000_000, batch_size=4_096, lora_r=16_384,
lora_alpha=32_768 so a future regression that tightens them back
without explicit user opt-in is caught.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten code comments across the security-hardening pass
* studio: always inject bootstrap credentials on first boot
The UNSLOTH_STUDIO_INJECT_BOOTSTRAP gate added an extra
terminal-to-browser copy-paste on every fresh install. In practice
the LAN credential leak it guarded against is narrow: the password
is one-time, the user rotates it on the very next click, the
default Studio bind is 127.0.0.1, and -H 0.0.0.0 already exposes
the entire API surface. Drop the gate so the inject fires whenever
a bootstrap password is still pending. The CSP nonce wiring stays
in place; the inline script remains the only inline script the
backend ever emits.
The three Playwright UI smoke workflows lose their
UNSLOTH_STUDIO_INJECT_BOOTSTRAP=1 lines along with the explanatory
comment blocks since the inject now happens by default.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* scripts/scan_*: add Mini Shai-Hulud May-12 IOC strings and pin-blocklists
Append the May-12 2026 wave indicators (git-tanstack.com, transformers.pyz,
/tmp/transformers.pyz, "With Love TeamPCP", "We've been online over 2 hours")
to all three scanner IOC tables, add BLOCKED_NPM_VERSIONS (42 TanStack pkgs,
4 opensearch versions, 3 squawk pkgs) in scan_npm_packages.py and
lockfile_supply_chain_audit.py (kept byte-identical), add BLOCKED_PYPI_VERSIONS
(guardrails-ai 0.10.1, mistralai 2.4.6, lightning 2.6.2/2.6.3) plus
RE_MAY12_IOC wiring across check_py_file/check_shell_file/check_workflow_file
in scan_packages.py. The npm orchestrator and the lockfile auditor now
short-circuit on a blocked entry before fetching the tarball, and the
PyPI download pipeline drops blocked specs before pip download is invoked.
* tests/security: regression suite for supply-chain scanners
Adds offline fixture corpus and pytest coverage for scan_npm_packages,
scan_packages, and lockfile_supply_chain_audit so future IOC-table
drift surfaces at PR time. Pytest scope narrowed to tests/security so
GPU smoke tests are not picked up by default.
* ci(security-audit): drop continue-on-error on pip-scan and npm-scan jobs
Promote three harden-runner blocks to egress-policy: block with per-job allowlists.
Add tests-security job running pytest tests/security as a hard gate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scripts: harden third-party downloads, pip resolver pins, atomic writes
Pins uv installer and mlx_vlm qwen3_5 patches by commit SHA + SHA-256
checksum, scrubs PIP_* env vars and forces --index-url + --only-binary
on pip download, applies tarbomb caps to scan_packages archive walks,
and converts non-atomic config writes (kwargs spacer, studio stamper,
notebook validator, scan_packages req-file fixer) to mkstemp+os.replace.
Also adds host allowlist to notebook_to_python downloader, threads an
--allow-shell flag through its shell=True emission with reviewer warning
comments, locks both MLX installer scripts to set -euo pipefail, and
extends CODEOWNERS so colab snapshot data files require notebook-owner
review.
* ci(workflows): harden release-desktop / smoke / notebooks workflows
Pin dtolnay/rust-toolchain to a 40-char SHA, scope release-desktop permissions to read at workflow level with job-level write only on the build job, append --ignore-scripts to every npm ci / npm install in studio-frontend-ci / wheel-smoke / studio-tauri-smoke / release-desktop, validate client_payload.ref shape via an env-var-isolated regex on every notebooks-ci job, and add step-security/harden-runner in audit mode as the first step of release-desktop and mlx-ci.
* scripts: promote silent scanner failures to non-zero exit codes
scan_packages now returns 2 on pip-download failure and emits a CRITICAL archive_corrupted finding on truncated wheels/sdists.
notebook_to_python exits 1 on per-notebook failures; notebook_validator wraps the stash/pop in try/finally; lockfile audit rejects bare UNSLOTH_LOCKFILE_AUDIT_SKIP=1 with a loud GitHub Actions warning.
* Add npm cooldown + new-install-script gate + Dependabot cooldown
Pins min-release-age=7 (npm 11.10+) in repo-root and studio/frontend
.npmrc, adds scripts/check_new_install_scripts.py to fail PRs that
add a postinstall dep, ships a new security-audit job for npm audit
signatures plus the diff, and extends .github/dependabot.yml with
cooldown stanzas. Pin @tanstack/react-router to 1.169.9 per GHSA-
g7cv-rxg3-hmpx; lockfile regen deferred until that release lands on
npm. tests/security gains 4 new tests; full suite 26/26 green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci(security): fix tanstack pin, exec bits, expand IOC tables to @uipath/@squawk full
- Revert --ignore-scripts on Studio install workflows: vite build needs
esbuild's native postinstall (per PR #5392 rationale). Keep
--ignore-scripts on security-audit.yml's standalone npm audit job.
- Pin @tanstack/react-router to the actual published 1.169.2 (was a
forward-looking 1.169.9 that does not exist on npm; broke npm ci).
- Drop redundant repo-root .npmrc; studio/frontend/.npmrc covers the
only npm project today (root cooldown re-instate via dependabot.yml).
- Restore exec bits on 7 files my filesystem stripped during cherry-pick.
- Expand BLOCKED_NPM_VERSIONS with full safedep.io + Aikido enumeration:
22 @squawk/* packages with 5 versions each (110 entries; previously
3 entries with 1 version each), and 66 @uipath/* packages (entirely
missing before). Mirror in scripts/lockfile_supply_chain_audit.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests/security: suppress CodeQL py/incomplete-url-substring-sanitization
The two flagged 'X' in Y assertions are NOT URL sanitization checks.
They verify our scanner WROTE a known IOC literal into its stdout /
Finding.evidence, which is the opposite of an attack surface --
matching the scanner's output is precisely what catches the worm.
Inline lgtm[] suppression with a 4-line rationale comment above each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scripts/scan_*: expand IOC tables with Aikido full 169-pkg enumeration
Per Aikido 2026-05-12 disclosure (373 malicious package-version entries
across 169 npm package names), add to BLOCKED_NPM_VERSIONS:
- @mistralai/* npm scope (3 packages, 9 versions) -- separate from
the PyPI mistralai package already in BLOCKED_PYPI_VERSIONS
- @tallyui/* (10 packages, 30 entries)
- @beproduct/nestjs-auth (18 versions 0.1.2..0.1.19)
- @draftlab/* + @draftauth/* (5 packages)
- @taskflow-corp/cli, @tolka/cli, @ml-toolkit-ts/*, @mesadev/*,
@dirigible-ai/sdk, @supersurkhet/*
- 10 unscoped packages (safe-action, ts-dna, cross-stitch,
cmux-agent-mcp, agentwork-cli, git-branch-selector, wot-api,
git-git-git, nextmove-mcp, ml-toolkit-ts)
Also add to KNOWN_IOC_STRINGS / NPM_IOC_STRINGS:
- router_init.js SHA-256 ab4fcadaec49c03278063dd269ea5eef82d24f2124a8e15d7b90f2fa8601266c
- tanstack_runner.js SHA-256 2ec78d556d696e208927cc503d48e4b5eb56b31abc2870c2ed2e98d6be27fc96
- bun run tanstack_runner.js marker (the new Bun-prepare-script
dropper invocation pattern unique to this wave)
Total: 170 packages, 401 versions blocklisted. Studio lockfile still
scans clean (0 findings, 0 hard errors).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scripts/scan_*: web-verification additions (@tanstack/setup, intercom-client)
Two findings from cross-checking BLOCKED_NPM_VERSIONS / KNOWN_IOC_STRINGS
against GHSA-g7cv-rxg3-hmpx + Aikido + safedep.io + Socket + Semgrep.
- Fix asymmetry: @tanstack/setup IOC string was in
lockfile_supply_chain_audit.py's NPM_IOC_STRINGS but missing from
scan_npm_packages.py's KNOWN_IOC_STRINGS. The literal is the malicious
optional-dependency name used by the May-12 TanStack wave; no
legitimate npm package of this name exists.
- Add intercom-client@7.0.4: the npm counterpart of the lightning
2.6.2/2.6.3 PyPI compromise (Apr-30 wave). Same threat actor
(TeamPCP). Confirmed by Semgrep, Aikido, OX Security, Resecurity,
Kodem. Safe version is 7.0.3 and earlier.
Total BLOCKED_NPM_VERSIONS: 171 packages / 402 versions. Both files
remain byte-identical. Studio lockfile still scans clean.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci(security): add workflow-trigger lint refusing pull_request_target + cache-poisoning vectors
The two patterns that together powered GHSA-g7cv-rxg3-hmpx (TanStack
Mini Shai-Hulud) are now gated at PR time:
1. pull_request_target -- the worm chain started with a fork PR that
ran in the base-repo context. Every workflow in this repo today
uses 'pull_request' (safe); the lint refuses any new
pull_request_target additions outright. workflow_run is
restricted, allowed only with an explicit allow-comment.
2. Shared cache keys between PR-triggered workflows and the publish
workflow (release-desktop.yml). The TanStack attack chain poisoned
a shared Actions cache from a fork PR; the legitimate release
workflow then restored the poisoned cache. The lint refuses any
cache key that appears in both a PR-triggered workflow and a
workflow_dispatch-only / publish workflow.
Current tree is clean: 0 pull_request_target, 0 workflow_run, 0
PR-publish cache-key collisions across all 24 workflows. The lint
locks that invariant in place.
Files:
+ scripts/lint_workflow_triggers.py (~200 LOC, stdlib + PyYAML)
+ tests/security/test_lint_workflow_triggers.py (5 tests covering
current-tree pass, pull_request_target reject, workflow_run
restricted, justified workflow_run accept, cache-key collision
reject)
~ .github/workflows/security-audit.yml: new workflow-trigger-lint
job, no continue-on-error, harden-runner block-mode, PyYAML only
runtime dep.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* security: fix tests-security CI job + CodeQL false-positives
Two CI failures on the prior push:
1. pytest tests/security -- 5 lint regression tests failed because
scripts/lint_workflow_triggers.py imports PyYAML which is not in
the bare runner's Python env. Added pyyaml==6.0.2 to the pip
install step alongside pytest. (29 scanner tests already passed.)
2. CodeQL py/incomplete-url-substring-sanitization fired on two
test assertions that check the scanner WROTE the IOC literal
to its own stdout/stderr. The rule pattern-matches on
`"<host>" in <var>` and cannot distinguish a URL sanitizer from
a regression-test evidence check. Previous `# lgtm[...]` inline
suppressions were detached from the operator when pre-commit
reformatted the assert across multiple lines. Rebuilt the IOC
literals at runtime (`"git-tanstack." + "com"`) so no URL-shaped
source literal appears on the `in` operator line; rule cannot
trigger.
Verified locally: `pytest tests/security -v` -> 34 passed in 2.70s.
* security(studio): defensive .npmrc cooldown aliases + save-exact
Two additions to studio/frontend/.npmrc to harden the existing
`min-release-age=7` (Mini Shai-Hulud defence):
1. `minimum-release-age=10080` (minutes) -- defensive alias for the
same 7-day floor. Some npm versions / wrappers consult one key but
not the other; setting both prevents a single upstream setting-name
parse change from silently disabling the cooldown. The two keys
MUST agree (do not let them drift).
2. `save-exact=true` -- refuses to write back `^x.y.z` ranges into
package.json when a maintainer runs `npm install <pkg>` locally.
Does NOT rewrite already-present ranges; stops NEW carets from
creeping into the manifest as patch-version footguns.
Verified: pytest tests/security -> 34 passed in 2.63s.
* chore(dependabot): remove dead bun entry for /studio/frontend
`package-ecosystem: "bun"` at /studio/frontend was a no-op: that
path commits package-lock.json, not bun.lock / bun.lockb, so
Dependabot's bun ecosystem silently skipped it. The actual
behaviour is unchanged -- the npm entry below the cargo block
already owns npm_and_yarn security advisories for /studio/frontend
with `open-pull-requests-limit: 0` (version-update PRs suppressed,
security PRs flow through).
This commit:
- Deletes the bun entry (kept a placeholder comment so a future
bun migration knows where to slot it back in).
- Rewrites the npm /studio/frontend entry comment to explain the
real intent: lockfile is the authoritative pin, .npmrc
`min-release-age=7` already blocks fresh tarballs at install
time, dependabot only needs to surface security advisories.
No functional change: same set of dependabot PRs as before (zero
version updates, security advisories grouped weekly with cooldown).
Verified: pytest tests/security -> 34 passed in 2.67s; YAML
parses cleanly via PyYAML.
* fix(dependabot): drop unsupported semver-* cooldown keys on github-actions
Dependabot's validator rejected the config with:
The property '#/updates/0/cooldown/semver-minor-days' is not
supported for the package ecosystem 'github-actions'.
The property '#/updates/0/cooldown/semver-patch-days' is not
supported for the package ecosystem 'github-actions'.
The `semver-minor-days` / `semver-patch-days` cooldown knobs are
only valid for semver-aware ecosystems (npm, cargo, etc.). The
github-actions ecosystem pins via git tags / SHAs, not semver, so
only `default-days` is honored. Pre-existing bug on main; surfaced
on this PR because the prior commit re-validated the file.
Behaviour: github-actions PRs now respect the 7-day cooldown floor
(was already the intent), without the no-op semver bands.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Multi Image GRPO
* try matching trl semantics
* attn mask for multi image grpo
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix multi-image GRPO chunking and zoo guard in rl_replacements
image_sizes is now sliced on the image axis (img_start:img_end) when
the processor emits one row per image and num_images is provided;
sample-axis slicing is kept as the fallback. This restores correct
per-batch image_sizes alignment for multi-image VLM processors.
pixel_attention_mask now uses a three-way layout check: image-axis
when shape[0] matches image_grid_thw rows, pixel-row when shape[0]
matches pixel_values rows and is distinct from total_samples,
otherwise sample-axis. Prevents misalignment with image-axis grid
slicing for per-image masks and ambiguity when single-image-per-sample
shapes coincide.
cum_imgs slice indices materialize via .item to match the existing
cum_rows pattern in the same loop and avoid 0-dim tensors flowing
into a CUDA-tensor slice.
cum_rows is materialized on CPU once after construction; the
per-chunk loop uses .item on it, so keeping it on device caused a
GPU->CPU sync per iteration.
Add a one-time fail-loud guard in compute_loss when num_images is
provided but the resolved grpo_accumulated_loss source has no
num_images handling, pointing users at the corresponding unsloth_zoo
upgrade. The active GRPO path goes through grpo_accumulated_loss
(the local _get_per_token_logps and _get_per_token_logps_and_entropies
return None on the efficient path), so without this guard a stale
unsloth_zoo silently mis-slices multi-image batches.
* Tighten multi-image GRPO zoo compatibility guard
Only raise the zoo upgrade error when at least one entry in
num_images is not 1. Upstream TRL emits num_images=[1,1,...] for
any vision batch (one image per sample), and old unsloth_zoo
builds chunk those correctly because sample-axis and image-axis
slicing coincide for all-ones counts. Restricting the check to
batches with a real multi-image sample stops single-image VLM
GRPO from being needlessly broken on pre-companion zoo installs.
Prefer inspect.signature(grpo_accumulated_loss).parameters for
the num_images contract. Fall back to inspect.getsource string
matching only when the signature does not declare num_images
(e.g. the companion zoo wires it through **kwargs). The previous
try/except (TypeError, OSError) over getsource turned the guard
into a silent no-op when source files were absent; the new flow
raises in that case because the signature check will not have
proven support either.
* Consolidate multi-image GRPO chunking and zoo guard tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Harden Tauri backend preflight and startup
Require managed Studio root IDs to match before attaching to existing backends, close the concurrent backend-start window, and tighten frontend Tauri detection to Tauri-specific signals.
* Add Tauri backend manageability guards
Gate desktop backend compatibility on explicit manageability fields, add external-conflict handling for unsafe backend states, and protect update/repair paths from mutating active non-owned Studio backends. Track Tauri-owned backends with local owner metadata for verified orphan cleanup only.
* Split Tauri preflight probes into modules
Move preflight types, version checks, managed install probing, and backend probing into focused submodules while preserving behavior and keeping implementation files under the release-readiness size target.
* Use desktop-specific Tauri updater channel
Point the desktop updater at a same-repo desktop-latest manifest and publish that channel from non-draft desktop releases after validating the Tauri-generated latest.json.
* Add Linux desktop update policy
* Add owned backend lifecycle guards
* Adopt verified desktop-owned backends
* Validate desktop backend readiness
* Trim Tauri release hardening code
* Require desktop backend 2026.5.3
* Handle desktop backend edge cases
* Fail stalled desktop backend startup
* Fix desktop update edge cases
* Avoid secret-gating adopted watchdog
* Fix desktop update comparison guards
* Automate desktop release versioning
* Serialize desktop release workflow
* tests: follow preflight.rs split into preflight/{backend,managed,types,version}.rs
PR #5341 splits studio/src-tauri/src/preflight.rs into a directory of
submodules. The cmd.env_remove("UNSLOTH_STUDIO_HOME") + STUDIO_HOME
calls now live in preflight/managed.rs instead of preflight.rs, so
test_tauri_preflight_scrubs_studio_home_env counted zero matches in
the old single-file location and failed with "assert 0 >= 2".
Read whichever shape is on disk: preflight.rs at the old path plus
every *.rs under preflight/ (current PR has 2 occurrences in
preflight/managed.rs). The guard intent is unchanged: at least 2
env_remove calls covering run_cli_probe and probe_cli_capability,
plus the single commands.rs scrub in check_install_status. Verified
locally: pytest tests/test_studio_install_workspace_guard.py::test_tauri_preflight_scrubs_studio_home_env passes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Avoid browser Tauri hostname detection
* Restore shutdown flag after failed stop
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/tests: AbortSignal-bound in-page fetches + wall-clock watchdog
Run 25696797934 / job 75446949358 on PR #5387 cancelled the
"Chat UI Tests" macos-14 job at 30 min: studio.log went idle after
the chat surface mounted, no further requests reached the server,
and Playwright silently sat on a `page.evaluate(async () => fetch(
/api/inference/load))` for 27+ minutes before the runner-level
timeout fired. The two other Chat UI Tests jobs on the same SHA
passed in 5-17 min, so this was a transient renderer wedge under
--single-process Chromium, not a regression from the security
bumps in that PR.
Root cause: Playwright's `page.evaluate(...)` has no `timeout=`
argument. If the JS body awaits a fetch whose promise never
settles (the renderer's network thread stalls behind the busy main
thread on the free macos-14 runner), the entire Python script
hangs until something external kills it.
Add two helpers in `_playwright_robust.py`:
- `evaluate_fetch(page, url, *, method, headers, body, timeout_ms)`
wraps `fetch()` in an `AbortController` so the JS resolves
either with a real response or with `{status: 0, error:
"AbortError..."}` after the budget elapses. Callers fail loud
on a non-None `error` and the wedge surfaces as a one-line
diagnostic instead of a 30-min cancel.
- `install_wall_clock_watchdog(deadline_s)` starts a daemon
Timer that hard-exits the process at the deadline. Belt-and-
suspenders for any wedge inside the browser that the per-
action timeouts cannot bound. Default 720s (12 min); healthy
runs measure 5-9 min on macos-14 so the headroom is small
without amplifying a wedge to the 30-min runner cap.
Wire both into `playwright_chat_ui.py` and `playwright_extra_ui.py`:
- Replace every `page.evaluate(async () => fetch(...))` site
with `evaluate_fetch(...)`: refresh-token exchange, defaults
fetch, inference load, health probe, post-rotation refresh.
Five sites in chat_ui, two in extra_ui.
- Arm the watchdog at the top of `with sync_playwright()` and
cancel it on clean exit.
Knobs (all default-safe, override only for slow runners):
STUDIO_UI_WALL_TIMEOUT_S (default 720s)
STUDIO_UI_FETCH_TIMEOUT_MS (default 30000ms)
STUDIO_UI_LOAD_TIMEOUT_MS (default 180000ms)
Verified locally with `python -c "ast.parse(...)"` on all three
files and a unit smoke that confirms `evaluate_fetch`'s JS argument
shape and that `install_wall_clock_watchdog` returns a daemonised
Timer that responds to `.cancel()`.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
danielhanchen