* Faster safetensors weight loading on unified-memory (integrated) GPUs
On unified-memory GPUs (AMD APUs / "Strix Halo", NVIDIA GB10 "Spark", Intel
iGPUs) the GPU shares the system memory pool. PyTorch's fast pinned-DMA
host->device path does not recognize the Rust-allocated, mmap-backed buffers
that safetensors hands back, so a direct safetensors GPU load
(`safe_open(..., device=<cuda>)`) drops onto a slow per-tensor copy that, on
unified memory, additionally triggers page-attribute changes and page faults.
Wrap `transformers.modeling_utils.safe_open` so that, when transformers asks it
to load a shard directly onto a CUDA/HIP device, the shard is opened on CPU and
each tensor is `.clone()`-d into a normal torch allocation before `.to(device)`.
This restores the fast DMA path. Data, dtype and final device are unchanged, so
outputs are bit-identical -- only *how* the bytes reach the GPU changes.
Strictly gated to integrated/unified-memory GPUs via the standard `is_integrated`
device property (every visible device must be integrated): a hard no-op on
discrete NVIDIA/AMD GPUs, CPU, XPU and MLX, where the pinned-DMA path already
works. Only intercepts `framework="pt"` CUDA-device targets; CPU / disk-offload
loads are left untouched. Accuracy-neutral, idempotent, opt out with
UNSLOTH_DISABLE_UMA_CLONE_LOAD=1 (force the gate for tests with
UNSLOTH_FORCE_UMA=1/0).
This is the AMD/universal-UMA counterpart to the NVIDIA DGX Spark work in #5945
(which deliberately left the H2D clone-then-move out): gating on `is_integrated`
covers AMD Strix Halo, Intel iGPUs and Spark-class parts alike.
Verified on an AMD Radeon 8060S (gfx1151, Strix Halo) Windows ROCm box with
in-process, ordering-cancelled A/B benchmarks:
- H2D mechanism (safe_open device=0 vs cpu->clone->.to(0)): 2.08x faster
(1.076s -> 0.518s for a 988MB bf16 shard)
- full `from_pretrained`: 1.56x faster (1.552s -> 0.996s), saving 0.555s --
matching the H2D delta exactly
- max|logit diff| stock vs patched == 0.0 (bit-identical), generate + a LoRA
train step both verified
The absolute/relative win grows with bf16/fp16 weight volume (the same trick is
reported as ~2.3-2.75x on NVIDIA GB10 Spark for larger models).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: evaluate the integrated-GPU gate lazily, not at import (Gemini review)
patch_unified_memory_safetensors_load() called
is_integrated_unified_memory_gpu() at install time, and the gate queries
torch.cuda.get_device_properties() for every visible device -- initializing
the CUDA context during `import unsloth` on every CUDA machine (discrete
included). That (a) breaks fork-based multiprocessing, (b) runs BEFORE
patch_dgx_spark_memory_config can set PYTORCH_CUDA_ALLOC_CONF on Spark,
defeating that patch's expandable_segments config in the very environment
this PR targets, and (c) charges a CUDA context to CPU-only imports.
The gate now runs lazily inside the wrapper, ordered AFTER the
framework/device check so non-CUDA loads never trigger the property query;
a CUDA-target safe_open means the caller is initializing CUDA anyway, and
the gate is lru-cached so it is evaluated once. The wrapper installs
unconditionally (opt-out and idempotency unchanged) and passes through when
the gate is off.
Tests: install-time no-eval guarantee (gate raises if called during
install), wrapper passthrough with the gate off, all previous gating /
passthrough / CUDA correctness tests kept -- 16/16 pass. Verified on the
N1X (WSL2): module exec + patch install leave torch.cuda.is_initialized()
unchanged; CPU loads pass through; forced CUDA-target loads intercept and
land bit-identical on the GPU.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Compress PR comments to essentials (comment-only; AST-verified)
Docstrings and the _utils hook comment trimmed to their load-bearing
content (lazy-gate rationale, gating scope, opt-out env). AST dumps
with normalized docstrings are identical before/after for all three
files; the module's 16 unit tests pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: tighten the UMA-load import comment (no code change)
* Tighten and trim code comments
* Drop unused is_integrated_unified_memory_gpu import from _utils.py
The UMA hook only needs patch_unified_memory_safetensors_load(); the
gate symbol is imported and used from ._uma_safetensors directly, so the
hoisted alias here was dead and tripped the import-hoist safety-net lint.
* Scope the UMA loader docstring to CUDA/HIP direct-device loads
The module text claimed Intel iGPU coverage, but the gate and device check
are CUDA/HIP only, and the clone path only wraps safe_open calls that carry
a CUDA device. State the actual scope and name the deliberate exclusions
(Intel XPU, CPU-open + .to() flows like bnb/HQQ) until they can be validated
on real hardware. Comment-only change.
* Tighten UMA safetensors loader comments
Trim the inline comments in the UMA clone-then-move path and the
_utils.py install site to be shorter and clearer. No code changes.
* uma: fall back to the direct move when the clone cannot allocate
The clone-and-move fast path transiently doubles one tensor's CPU
footprint while the mmap source and the CUDA destination are live. On a
UMA box with little free shared memory a large tensor could OOM where
the stock direct safe_open path would have loaded it. Both move sites
now go through a helper that catches the allocation failure and falls
back to the direct (slow but allocation-free) move, so the load always
succeeds; a genuine non-memory error re-raises identically from the
fallback.
Added a test that forces the clone to fail and verifies the wrapper
still lands tensors on the device with intact values (17 tests pass on
a real GPU).
* tests: track the moved pass-through inheritance in the gguf order check
Main moved the llama_extra_args pass-through inheritance out of the
GGUF branch into _resolve_inherited_extra_args, which runs before it,
so the source-order assertion's "if request.llama_extra_args is None"
anchor no longer exists inside the branch and the check failed after
the main merge. The test now asserts the same property in the current
shape: inheritance before the GGUF branch (a carried --no-mmproj still
shapes the hub guard's companion requirement), and marker, hub guard,
unload in order within the branch. Full file passes (32 tests).
* tests: anchor the inheritance order check on the call, not the definition
source.index("_resolve_inherited_extra_args(") matched the function
definition, which always precedes the endpoint, so the ordering
assertion was vacuously true. Anchoring on "= _resolve_inherited_
extra_args(" pins the first call site inside the load endpoint (line
4505), which is the statement whose position relative to the GGUF
branch the test is meant to guard. 32 tests pass.
* tests: align the gguf order test with main
Main fixed the stale ordering assertion in PR 7252; adopting its
version verbatim removes this file from the branch diff entirely and
avoids a conflict on the next main merge. 32 tests pass.
* uma: tighten comments
* Relicense UMA safetensors module and test under AGPL-3.0
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
4759a5139d