unsloth/tests
Leo Borcherding bf2cd745b1
Fix installer selecting ROCm torch on NVIDIA Linux hosts (#6174)
* fix: prevent ROCm torch from installing on NVIDIA Linux hosts

NVIDIA's open kernel module (driver 560+) registers GPU topology nodes in
the KFD sysfs hierarchy with non-zero gpu_id values. The _has_amd_rocm_gpu
(install.sh) and _has_rocm_gpu (install_python_stack.py) sysfs fallbacks
previously treated any non-zero gpu_id as proof of an AMD GPU, so an
NVIDIA-only host with the open kernel driver was misrouted to the ROCm
install path, replacing the correctly-installed CUDA torch with ROCm wheels.

Fixes:

1. install.sh _has_amd_rocm_gpu sysfs fallback: require vendor_id 4098
   (AMD 0x1002) in the KFD node properties file before declaring an AMD
   GPU present. NVIDIA KFD nodes carry vendor_id 4318 (0x10DE) and are
   now skipped.

2. install_python_stack.py _has_rocm_gpu sysfs fallback: same vendor_id
   guard. Also preserves the existing fallback for older kernels that
   don't ship a properties file (trusts gpu_id alone there).

3. install.sh now exports UNSLOTH_TORCH_BACKEND ("cuda"/"rocm"/"cpu")
   immediately after get_torch_index_url() resolves the wheel family.
   install_python_stack.py reads this as _TORCH_BACKEND and short-circuits
   _ensure_rocm_torch() entirely on cuda/cpu hosts, providing a second
   layer of defense that is independent of subprocess GPU detection.

Tests: 9 new cases in TestHasRocmGpuKfdVendorGuard,
TestEnsureRocmTorch, and TestInstallShStructure cover all three changes.
Full test_rocm_support.py suite: 289 passed, 2 skipped, 0 failed.

Closes #6172

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: show actual torch backend in progress step labels

The 'ROCm torch check' and 'ROCm torch (final)' step labels were
hardcoded regardless of whether the installer was targeting CUDA, ROCm,
or CPU. On NVIDIA hosts they showed 'ROCm' even though no ROCm wheels
were being installed, which was misleading.

Add _torch_step_label(suffix) which reads UNSLOTH_TORCH_BACKEND (set by
install.sh) and formats the label as e.g. 'torch check (cuda)' or
'torch final (rocm)'. Falls back to live GPU detection for standalone
studio update runs that bypass install.sh.

* fix: make KFD sysfs vendor check conservative -- skip if no properties file

The previous implementation fell through to `return True` when the KFD
node's properties file was missing (OSError), intending to support older
kernels. But NVIDIA open driver KFD nodes can also lack a properties file
on some kernel versions, so the fallback still produced a false positive.

Change the `except OSError: pass` to `continue` so any node without a
readable properties file is skipped rather than trusted. KFD properties
files exist on every kernel version that actually exposes /sys/class/kfd,
so this does not regress real AMD GPU detection -- if the directory exists
at all, properties files will be present for genuine GPU nodes.

* fix: bulletproof NVIDIA vs AMD GPU detection

Four changes that together ensure ROCm torch can never be installed on an
NVIDIA host regardless of which detection path fires:

1. _has_rocm_gpu() (Python): NVIDIA guard at the top -- returns False
   immediately when _has_usable_nvidia_gpu() is True, blocking rocminfo,
   amd-smi, and KFD sysfs from producing a false positive even when ROCm
   tools are co-installed alongside the NVIDIA driver.

2. _has_amd_rocm_gpu() (install.sh): same NVIDIA guard -- calls
   _has_usable_nvidia_gpu first and returns 1 if it succeeds.

3. _has_usable_nvidia_gpu() (Python): adds /proc/driver/nvidia/gpus/
   sysfs fallback. The NVIDIA driver populates this directory on Linux
   regardless of nvidia-smi state, so a subprocess PATH gap, timeout, or
   driver initialisation race can no longer silence NVIDIA detection.

4. _has_usable_nvidia_gpu() (install.sh): same /proc/driver/nvidia/gpus
   fallback, tried after nvidia-smi -L rather than instead of it.

Together: NVIDIA wins at every decision point. If nvidia-smi works, it
confirms NVIDIA. If it fails, /proc/driver/nvidia confirms NVIDIA. If
somehow both fail, _has_rocm_gpu still checks NVIDIA first before any AMD
path runs.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: two KFD/proc-only corner cases from Codex review

1. KFD awk state not reset per node file (Ryzen+NVIDIA false positive):
   The awk glob processes all topology node properties files in one pass.
   Without FNR==1 reset, a Ryzen+NVIDIA host where an AMD CPU-agent node
   sets amd=1 (vendor_id 4098, gpu_id 0) can combine with a later NVIDIA
   node setting gpu=1 (gpu_id > 0), triggering found=1 before vendor_id
   4318 is seen. Added FNR==1{ gpu=0; amd=0 } to reset per file.

2. proc-only NVIDIA not reaching CUDA wheel selection:
   _has_usable_nvidia_gpu returning true via /proc/driver/nvidia fallback
   left _smi empty, so get_torch_index_url entered the AMD/CPU branch and
   selected CPU wheels despite NVIDIA being confirmed. Introduced
   _nvidia_detected flag (separate from _smi) so the AMD branch is skipped
   whenever NVIDIA is confirmed by any path, while _cuda_ver reads from
   _smi when available (with the existing cu126 fallback when _smi is absent).

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-06-10 21:00:11 -07:00
..
notebooks CI: scope GITHUB_TOKEN permissions, add MLX CI, unblock ~60 skipped tests (#5312) 2026-05-11 03:19:13 -07:00
python feat: support text-only loading of Gemma 3 27B via FastLanguageModel (skip SiglipVisionModel) (#5816) 2026-06-09 22:52:39 -07:00
qlora Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
saving Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
security Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
sh test(install): cover the Apple Silicon venv arch rebuild guard 2026-05-31 06:35:43 -07:00
studio Fix installer selecting ROCm torch on NVIDIA Linux hosts (#6174) 2026-06-10 21:00:11 -07:00
utils Tests + CI guard: batched left-padded generation can never silently regress again (#1066, #3699) (#6145) 2026-06-10 08:00:28 -07:00
version_compat Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
vllm_compat Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
__init__.py Qwen 3, Bug Fixes (#2445) 2025-04-30 22:38:39 -07:00
_zoo_aggressive_cuda_spoof.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
conftest.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
run_all.sh studio/setup.sh: cope with fresh CUDA toolkits like 13.3 (#5826) 2026-05-29 05:09:20 -07:00
test_attention_implementation.py fix(gpt-oss): prefer flex attention over sdpa (#5701) 2026-05-22 08:38:38 -07:00
test_callback_signature_drift.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_cli_export_unpacking.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_enforce_kwargs_spacing.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_finetune_last_n_layers.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_gemma4_chat_template.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_get_model_name.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_import_fixes_drift.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_loader_glob_skip.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_model_registry.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_moe_lora_targets.py Merge moe-lora-target-fix CI fixes 2026-06-08 20:20:00 +05:30
test_multi_image_grpo_chunking.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_nvfp4_quant_load.py Merge nvfp4_load CI fixes 2026-06-08 23:23:53 +05:30
test_peft_weight_converter_compat.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_public_api_surface.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_raw_text.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_resolve_model_class.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_studio_install_workspace_guard.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_studio_root_resilience.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_tool_mask_zoo_compat.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_video_path_validation.py fix/validate dataset video paths before training (#5136) 2026-06-10 05:32:32 -07:00
test_windows_rocm_bnb_version.py Auto-set BNB_ROCM_VERSION from the installed wheel on Windows + ROCm (#5986) 2026-06-10 08:11:26 -07:00