install: honor exact CUDA/custom index URL pins in the torch-index marker

Address three Codex review findings on the torch-index marker mechanism:

- install.sh: after the ROCm CPU repair reinstalls torch from the generic
  $TORCH_INDEX_URL, record that as the marker source. A Radeon --find-links
  install set _TORCH_MARKER_INDEX_URL to its repo.radeon.com base earlier, so
  leaving it made the marker misreport Radeon wheels and a later Radeon pin would
  compare equal and skip a needed reinstall.

- install_python_stack.py: _ensure_cuda_torch now consults the exact-URL marker
  (_marker_pin_mismatch) when the installed +cuXXX tag matches the pinned leaf,
  so a same-leaf CUDA mirror change (official cu128 to an internal cu128 mirror)
  is reinstalled and re-recorded instead of skipped.

- _normalize_index_url / _normalize_family_leaf (install.sh, setup.ps1,
  install_python_stack.py): lowercase only KNOWN wheel-family leaves (rocm/gfx/
  cpu/cuXXX) so gfx120X-all still matches gfx120x-all, while a custom
  (unknown-family) leaf keeps its case so a verbatim URL pin like /Current does
  not compare equal to /current. Tests updated to assert the refined behavior.
This commit is contained in:
danielhanchen 2026-07-06 10:44:41 +00:00
commit 5ed073755c
5 changed files with 75 additions and 19 deletions

View file

@ -137,15 +137,31 @@ _PYTORCH_WHL_BASE = (
_TORCH_INDEX_MARKER_NAME = ".unsloth-torch-index"
def _normalize_family_leaf(leaf: str) -> str:
"""Lowercase ONLY a known wheel-family leaf (rocm* / gfx* / cpu / cuXXX).
The canonical gfx120X-all (capital X) must match AMD's lowercase gfx120x-all, so
known-family leaves are lowercased. A custom mirror leaf (/Current, /simple, ...)
keeps its case: an unknown-family URL pin is applied verbatim, so /Current and
/current must NOT compare equal. Same known-family set as
_explicit_unknown_family_torch_index_url. Mirrors the gate in install.sh /
setup.ps1. Pure function.
"""
low = leaf.lower()
if low.startswith(("rocm", "gfx")) or low == "cpu" or re.match(r"^cu[0-9]", low):
return low
return leaf
def _normalize_index_url(url: "str | None") -> "str | None":
"""Canonicalise a wheel index URL for exact marker/pin comparison.
Trims surrounding whitespace, strips ALL trailing slashes, and lowercases only
the FINAL path segment (the wheel-family leaf: cu128 / cpu / rocm7.2 / gfx1151 /
gfx120X-all). The host part is left untouched (it may be case-sensitive on some
mirrors); the leaf is lowercased so the canonical gfx120X-all (capital X) and
AMD's lowercase pip leaf gfx120x-all compare equal. MUST match the same
normalization in install.sh / setup.ps1 / install.ps1. Returns None for an
Trims surrounding whitespace, strips ALL trailing slashes, and lowercases the
FINAL path segment ONLY when it is a known wheel-family leaf (cu128 / cpu /
rocm7.2 / gfx1151 / gfx120X-all) -- see _normalize_family_leaf. The host part is
left untouched (case-sensitive on some mirrors), and a custom (unknown-family)
leaf keeps its case so a verbatim URL pin is not falsely matched equal. MUST
match the same normalization in install.sh / setup.ps1. Returns None for an
empty/whitespace-only input. Pure function.
"""
if url is None:
@ -158,8 +174,8 @@ def _normalize_index_url(url: "str | None") -> "str | None":
return None
head, sep, leaf = url.rpartition("/")
if sep:
return f"{head}/{leaf.lower()}"
return url.lower()
return f"{head}/{_normalize_family_leaf(leaf)}"
return _normalize_family_leaf(url)
def _torch_index_marker_path() -> Path:
@ -1504,6 +1520,13 @@ def _ensure_cuda_torch() -> None:
# build simply re-lands on the same family (idempotent).
_installed_desc = _installed_cu if _installed_cu else "an untagged CUDA build"
_why = f"torch is {_installed_desc} but the pinned CUDA index is {_pin_leaf}"
elif _marker == "cuda" and _pinned_cuda and _marker_pin_mismatch(_pin) is True:
# Same cuXXX leaf but the marker records a DIFFERENT full index URL (e.g. the
# official cu128 index vs an internal mirror's cu128). The +cuXXX tag cannot
# see the host change, so consult the exact-URL marker and reinstall from the
# pinned URL (via _detect_cuda_torch_index_url, which honours the override) so
# an explicit mirror pin is applied and re-recorded, not skipped.
_why = f"the pinned CUDA index URL differs from the recorded marker (leaf {_pin_leaf})"
else:
return # healthy CUDA torch matching the pin, or a deliberate CPU wheel