Route dense NemotronH models to the transformers 5.10 tier (#6541)
* Route dense NemotronH models to the transformers 5.10 tier
Dense NemotronH models (e.g. unsloth/NVIDIA-Nemotron-3-Nano-4B) describe their
layer stack with a hybrid_override_pattern that includes '-' (MLP) layers.
transformers only learned to parse that ('-' -> 'mlp' in pattern_mapping, 'mlp'
in valid_types and MIXER_TYPES) in 5.10; on 5.3/5.5 the config raises
KeyError: '-'. The model also ships auto_map remote code, so training and
inference that approve trust_remote_code load fine, but a native (TRC=False)
load such as export hits the built-in parser and fails with
'Failed to load checkpoint: -'.
Detect dense NemotronH from config.json (a '-' in hybrid_override_pattern, or
'mlp' in an expanded layers_block_type) and route it to the 5.10 tier, where the
model loads natively without remote code. Pure-MoE NemotronH configs are
unaffected and keep their existing tier.
Covers both the local config.json and the remote HF-id paths, and adds tests for
the detector and the resulting tier selection.
* Tighten _nemotron_h_needs_mlp_support docstring
* Detect dense NemotronH in nested, cached, and resolved-away configs
Three gaps could still route a dense NemotronH (MLP '-' layers) to a tier
below 5.10 and hit KeyError: '-':
- VL wrappers (e.g. NemotronH_Nano_VL_V2) keep the dense language model under
llm_config/text_config; the detector only checked the top-level model_type.
Recurse into nested language configs.
- Offline or blocked config fetches returned None for an already-downloaded
repo. Read config.json from the HF hub cache before any network.
- A local checkpoint resolves to its base before tiering, so an offline/private
base discarded the local config that revealed the dense pattern. Prefer the
higher tier of the resolved base and the original path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden NemotronH tier detection follow-ups
Address review of the nested/cached/resolved-away detection:
- The local re-check ran the full tier detector on the original path, so a bare
LoRA adapter under e.g. /runs/gemma-4-x/llama-lora could upgrade a default base
via directory-name substrings. Gate the re-check on a real local config.json so
it reads metadata, not path names.
- The HF hub cache was read before any network, so an online tier check could
serve stale config.json after the repo changed upstream. Consult the cache only
offline or after a failed fetch.
- Reading the cache imported huggingface_hub during tier detection, which runs
before a sidecar venv is activated and could pin the default-env hub into
sys.modules. Resolve the cache path with stdlib only.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments to be more succinct
* Select newest hub-cache snapshot by mtime and retry transient config fetches
The HF cache fallback in tier detection picked the lexicographically-first
snapshot when refs/main was absent (commit-pinned downloads), which can be an
older SHA than the Hub would load. Sort snapshots by mtime instead.
A transient online fetch failure cached the hub-cache fallback under the normal
(model_name, token) key, so a long-lived worker kept serving stale metadata even
after connectivity recovered. Return the fallback without memoizing it so the
next call retries the network.
* Harden config.json tier detection against auth failures and transient blips
- _load_config_json: a 401/403/404 from the raw Hub request is a definitive access
answer, not an outage. Return None instead of falling back to the HF hub cache, so
an unauthenticated or wrong-token request can never read another caller's cached
private metadata.
- _check_config_needs_510/550: only memoize the derived tier when the underlying
config read was definitive (local file, offline cache, or a completed fetch).
A transient fetch fallback is no longer pinned, so the tier is re-evaluated once
connectivity returns instead of staying stuck on the lower tier.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in tier-detection auth/cache paths
---------
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han
GitHub
parent
e9c574be48
commit
aeb5075121
2 changed files with 474 additions and 25 deletions
|
|
@ -34,6 +34,11 @@ from utils.transformers_version import (
|
|||
_check_tokenizer_config_needs_v5,
|
||||
_check_config_needs_510,
|
||||
_check_config_needs_550,
|
||||
_config_needs_510,
|
||||
_nemotron_h_needs_mlp_support,
|
||||
_config_json_from_hf_cache,
|
||||
_load_config_json,
|
||||
_higher_tier,
|
||||
_config_json_cache,
|
||||
_tokenizer_class_cache,
|
||||
_config_needs_510_cache,
|
||||
|
|
@ -382,6 +387,259 @@ class TestCheckConfigNeeds510:
|
|||
mock_urlopen.assert_not_called()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# NemotronH dense (MLP) models need the 5.10 tier
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestNemotronHNeedsMlpSupport:
|
||||
"""Dense NemotronH configs (MLP layers) require transformers >= 5.10."""
|
||||
|
||||
def test_hybrid_override_pattern_with_dash(self):
|
||||
cfg = {
|
||||
"model_type": "nemotron_h",
|
||||
"hybrid_override_pattern": "M-M-M*-M-",
|
||||
}
|
||||
assert _nemotron_h_needs_mlp_support(cfg) is True
|
||||
|
||||
def test_layers_block_type_with_mlp(self):
|
||||
cfg = {
|
||||
"model_type": "nemotron_h",
|
||||
"layers_block_type": ["mamba", "mlp", "attention", "mamba"],
|
||||
}
|
||||
assert _nemotron_h_needs_mlp_support(cfg) is True
|
||||
|
||||
def test_nemotron_h_moe_only_returns_false(self):
|
||||
"""A pure MoE NemotronH (no MLP) does not need the 5.10 tier."""
|
||||
cfg = {
|
||||
"model_type": "nemotron_h",
|
||||
"hybrid_override_pattern": "MEME*MEM",
|
||||
}
|
||||
assert _nemotron_h_needs_mlp_support(cfg) is False
|
||||
|
||||
def test_non_nemotron_with_dash_returns_false(self):
|
||||
"""The dash heuristic only applies to nemotron_h configs."""
|
||||
cfg = {"model_type": "llama", "hybrid_override_pattern": "M-M-"}
|
||||
assert _nemotron_h_needs_mlp_support(cfg) is False
|
||||
|
||||
def test_config_needs_510_includes_dense_nemotron_h(self):
|
||||
cfg = {
|
||||
"model_type": "nemotron_h",
|
||||
"hybrid_override_pattern": "M-M-M*-",
|
||||
}
|
||||
assert _config_needs_510(cfg) is True
|
||||
|
||||
def test_nested_llm_config_with_dash(self):
|
||||
# VL wrapper (e.g. NemotronH_Nano_VL_V2): dense LM is under llm_config.
|
||||
cfg = {
|
||||
"model_type": "NemotronH_Nano_VL_V2",
|
||||
"llm_config": {"model_type": "nemotron_h", "hybrid_override_pattern": "M-M*-"},
|
||||
}
|
||||
assert _nemotron_h_needs_mlp_support(cfg) is True
|
||||
assert _config_needs_510(cfg) is True
|
||||
|
||||
def test_nested_text_config_with_mlp(self):
|
||||
cfg = {
|
||||
"model_type": "wrapper",
|
||||
"text_config": {"model_type": "nemotron_h", "layers_block_type": ["mamba", "mlp"]},
|
||||
}
|
||||
assert _nemotron_h_needs_mlp_support(cfg) is True
|
||||
|
||||
def test_nested_non_nemotron_returns_false(self):
|
||||
cfg = {"model_type": "wrapper", "llm_config": {"model_type": "llama"}}
|
||||
assert _nemotron_h_needs_mlp_support(cfg) is False
|
||||
|
||||
def test_non_dict_and_missing_nested_do_not_raise(self):
|
||||
assert _nemotron_h_needs_mlp_support(None) is False
|
||||
assert _nemotron_h_needs_mlp_support({"model_type": "wrapper", "llm_config": None}) is False
|
||||
|
||||
|
||||
def _hf_response(cfg: dict):
|
||||
"""A urlopen() context-manager stand-in returning *cfg* as JSON bytes."""
|
||||
|
||||
class _Resp:
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *a):
|
||||
return False
|
||||
|
||||
def read(self):
|
||||
return json.dumps(cfg).encode()
|
||||
|
||||
return _Resp()
|
||||
|
||||
|
||||
class TestConfigJsonHfCacheFallback:
|
||||
"""HF hub cache is consulted only offline or after a failed fetch (never stale online)."""
|
||||
|
||||
def setup_method(self):
|
||||
_config_json_cache.clear()
|
||||
|
||||
@staticmethod
|
||||
def _seed_cache(
|
||||
hub: Path,
|
||||
repo_id: str,
|
||||
cfg: dict,
|
||||
commit: str = "deadbeef",
|
||||
):
|
||||
repo = hub / ("models--" + repo_id.replace("/", "--"))
|
||||
snap = repo / "snapshots" / commit
|
||||
snap.mkdir(parents = True)
|
||||
(snap / "config.json").write_text(json.dumps(cfg))
|
||||
(repo / "refs").mkdir(parents = True)
|
||||
(repo / "refs" / "main").write_text(commit)
|
||||
|
||||
def test_offline_reads_from_cache(self, tmp_path: Path, monkeypatch):
|
||||
cfg = {"model_type": "nemotron_h", "hybrid_override_pattern": "M-M*-"}
|
||||
self._seed_cache(tmp_path, "unsloth/NVIDIA-Nemotron-3-Nano-4B", cfg)
|
||||
monkeypatch.setenv("HF_HUB_CACHE", str(tmp_path))
|
||||
monkeypatch.setenv("HF_HUB_OFFLINE", "1")
|
||||
with patch("urllib.request.urlopen") as mock_url:
|
||||
assert _load_config_json("unsloth/NVIDIA-Nemotron-3-Nano-4B") == cfg
|
||||
mock_url.assert_not_called()
|
||||
|
||||
def test_online_prefers_network_over_cache(self, tmp_path: Path, monkeypatch):
|
||||
stale = {"model_type": "nemotron_h", "hybrid_override_pattern": "MMMM"}
|
||||
fresh = {"model_type": "nemotron_h", "hybrid_override_pattern": "M-M*-"}
|
||||
self._seed_cache(tmp_path, "org/model", stale)
|
||||
monkeypatch.setenv("HF_HUB_CACHE", str(tmp_path))
|
||||
monkeypatch.delenv("HF_HUB_OFFLINE", raising = False)
|
||||
monkeypatch.delenv("TRANSFORMERS_OFFLINE", raising = False)
|
||||
with patch("urllib.request.urlopen", return_value = _hf_response(fresh)):
|
||||
assert _load_config_json("org/model") == fresh # network wins, not stale cache
|
||||
|
||||
def test_network_failure_falls_back_to_cache(self, tmp_path: Path, monkeypatch):
|
||||
cfg = {"model_type": "nemotron_h", "hybrid_override_pattern": "M-M*-"}
|
||||
self._seed_cache(tmp_path, "org/model", cfg)
|
||||
monkeypatch.setenv("HF_HUB_CACHE", str(tmp_path))
|
||||
monkeypatch.delenv("HF_HUB_OFFLINE", raising = False)
|
||||
with patch("urllib.request.urlopen", side_effect = OSError("boom")):
|
||||
assert _load_config_json("org/model") == cfg
|
||||
|
||||
def test_offline_uncached_returns_none(self, tmp_path: Path, monkeypatch):
|
||||
monkeypatch.setenv("HF_HUB_CACHE", str(tmp_path))
|
||||
monkeypatch.setenv("HF_HUB_OFFLINE", "1")
|
||||
with patch("urllib.request.urlopen") as mock_url:
|
||||
assert _load_config_json("private/unknown") is None
|
||||
mock_url.assert_not_called()
|
||||
|
||||
def test_helper_ignores_local_paths(self, tmp_path: Path):
|
||||
# A filesystem path is not a repo id; never treat it as one.
|
||||
assert _config_json_from_hf_cache(str(tmp_path)) is None
|
||||
assert _config_json_from_hf_cache("plainname") is None
|
||||
|
||||
def test_no_refs_main_picks_newest_snapshot(self, tmp_path: Path, monkeypatch):
|
||||
# No refs/main (commit-pinned downloads): lexicographic order would pick the older
|
||||
# SHA; selection must follow mtime so the newest snapshot wins.
|
||||
repo = tmp_path / "models--org--model"
|
||||
old = repo / "snapshots" / "0000old"
|
||||
new = repo / "snapshots" / "ffffnew"
|
||||
old.mkdir(parents = True)
|
||||
new.mkdir(parents = True)
|
||||
(old / "config.json").write_text(json.dumps({"model_type": "stale"}))
|
||||
(new / "config.json").write_text(json.dumps({"model_type": "fresh"}))
|
||||
os.utime(old / "config.json", (1000, 1000))
|
||||
os.utime(new / "config.json", (2000, 2000))
|
||||
monkeypatch.setenv("HF_HUB_CACHE", str(tmp_path))
|
||||
assert _config_json_from_hf_cache("org/model") == {"model_type": "fresh"}
|
||||
|
||||
def test_transient_failure_does_not_cache_fallback(self, tmp_path: Path, monkeypatch):
|
||||
stale = {"model_type": "nemotron_h", "hybrid_override_pattern": "MMMM"}
|
||||
fresh = {"model_type": "nemotron_h", "hybrid_override_pattern": "M-M*-"}
|
||||
self._seed_cache(tmp_path, "org/model", stale)
|
||||
monkeypatch.setenv("HF_HUB_CACHE", str(tmp_path))
|
||||
monkeypatch.delenv("HF_HUB_OFFLINE", raising = False)
|
||||
# Network fails -> serve the cached snapshot, but it must not be memoized.
|
||||
with patch("urllib.request.urlopen", side_effect = OSError("boom")):
|
||||
assert _load_config_json("org/model") == stale
|
||||
# Connectivity returns: the next call must hit the network for the fresh config.
|
||||
with patch("urllib.request.urlopen", return_value = _hf_response(fresh)):
|
||||
assert _load_config_json("org/model") == fresh
|
||||
|
||||
def test_auth_failure_does_not_serve_cache(self, tmp_path: Path, monkeypatch):
|
||||
import urllib.error
|
||||
|
||||
# config.json cached from an earlier authorized session; an unauthenticated 4xx
|
||||
# must not be handed that private metadata.
|
||||
cfg = {"model_type": "nemotron_h", "hybrid_override_pattern": "M-M*-"}
|
||||
self._seed_cache(tmp_path, "private/model", cfg)
|
||||
monkeypatch.setenv("HF_HUB_CACHE", str(tmp_path))
|
||||
monkeypatch.delenv("HF_HUB_OFFLINE", raising = False)
|
||||
for code in (401, 403, 404):
|
||||
_config_json_cache.clear()
|
||||
err = urllib.error.HTTPError("url", code, "denied", {}, None)
|
||||
with patch("urllib.request.urlopen", side_effect = err):
|
||||
assert _load_config_json("private/model") is None
|
||||
|
||||
def test_server_error_still_falls_back_to_cache(self, tmp_path: Path, monkeypatch):
|
||||
import urllib.error
|
||||
|
||||
cfg = {"model_type": "nemotron_h", "hybrid_override_pattern": "M-M*-"}
|
||||
self._seed_cache(tmp_path, "org/model", cfg)
|
||||
monkeypatch.setenv("HF_HUB_CACHE", str(tmp_path))
|
||||
monkeypatch.delenv("HF_HUB_OFFLINE", raising = False)
|
||||
# A 5xx is transient, not an access decision: keep serving the cache.
|
||||
err = urllib.error.HTTPError("url", 503, "busy", {}, None)
|
||||
with patch("urllib.request.urlopen", side_effect = err):
|
||||
assert _load_config_json("org/model") == cfg
|
||||
|
||||
|
||||
class TestTierCheckTransientRetry:
|
||||
"""tier-needs checks must not memoize a transient fetch fallback."""
|
||||
|
||||
def setup_method(self):
|
||||
_config_json_cache.clear()
|
||||
_config_needs_510_cache.clear()
|
||||
_config_needs_550_cache.clear()
|
||||
|
||||
@staticmethod
|
||||
def _seed_cache(
|
||||
hub: Path,
|
||||
repo_id: str,
|
||||
cfg: dict,
|
||||
commit: str = "deadbeef",
|
||||
):
|
||||
repo = hub / ("models--" + repo_id.replace("/", "--"))
|
||||
snap = repo / "snapshots" / commit
|
||||
snap.mkdir(parents = True)
|
||||
(snap / "config.json").write_text(json.dumps(cfg))
|
||||
(repo / "refs").mkdir(parents = True)
|
||||
(repo / "refs" / "main").write_text(commit)
|
||||
|
||||
def test_transient_fallback_not_memoized_then_retries(self, tmp_path: Path, monkeypatch):
|
||||
stale = {"model_type": "llama"} # does not need 510
|
||||
fresh = {"architectures": ["Gemma4UnifiedForConditionalGeneration"]} # needs 510
|
||||
self._seed_cache(tmp_path, "org/model", stale)
|
||||
monkeypatch.setenv("HF_HUB_CACHE", str(tmp_path))
|
||||
monkeypatch.delenv("HF_HUB_OFFLINE", raising = False)
|
||||
# Network blip -> serve the cache, but do NOT pin the tier result.
|
||||
with patch("urllib.request.urlopen", side_effect = OSError("boom")):
|
||||
assert _check_config_needs_510("org/model") is False
|
||||
assert "org/model" not in _config_needs_510_cache
|
||||
# Connectivity returns: the next call re-fetches and sees the higher tier.
|
||||
with patch("urllib.request.urlopen", return_value = _hf_response(fresh)):
|
||||
assert _check_config_needs_510("org/model") is True
|
||||
assert _config_needs_510_cache["org/model"] is True # definitive read is memoized
|
||||
|
||||
def test_definitive_network_read_is_memoized(self, tmp_path: Path, monkeypatch):
|
||||
fresh = {"architectures": ["Gemma4ForConditionalGeneration"]} # needs 550
|
||||
monkeypatch.setenv("HF_HUB_CACHE", str(tmp_path))
|
||||
monkeypatch.delenv("HF_HUB_OFFLINE", raising = False)
|
||||
with patch("urllib.request.urlopen", return_value = _hf_response(fresh)) as mock_url:
|
||||
assert _check_config_needs_550("org/model") is True
|
||||
assert _check_config_needs_550("org/model") is True
|
||||
assert mock_url.call_count == 1 # second call served from the tier cache
|
||||
|
||||
|
||||
class TestHigherTier:
|
||||
def test_picks_stronger_tier(self):
|
||||
assert _higher_tier("default", "510") == "510"
|
||||
assert _higher_tier("530", "550") == "550"
|
||||
assert _higher_tier("510", "default") == "510"
|
||||
assert _higher_tier("default", "default") == "default"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# get_transformers_tier — tier detection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -438,6 +696,43 @@ class TestGetTransformersTier:
|
|||
|
||||
assert get_transformers_tier(str(tmp_path)) == "510"
|
||||
|
||||
def test_dense_nemotron_h_config_json_returns_510(self, tmp_path: Path):
|
||||
"""Local dense NemotronH checkpoint → 510 (MLP layers need >= 5.10)."""
|
||||
cfg = {
|
||||
"model_type": "nemotron_h",
|
||||
"hybrid_override_pattern": "M-M-M*-M-",
|
||||
}
|
||||
(tmp_path / "config.json").write_text(json.dumps(cfg))
|
||||
# A v5 tokenizer would otherwise route this to 530; 510 must win.
|
||||
(tmp_path / "tokenizer_config.json").write_text(
|
||||
json.dumps({"tokenizer_class": "TokenizersBackend"})
|
||||
)
|
||||
|
||||
with patch("urllib.request.urlopen") as mock_urlopen:
|
||||
assert get_transformers_tier(str(tmp_path)) == "510"
|
||||
mock_urlopen.assert_not_called()
|
||||
|
||||
def test_dense_nemotron_h_remote_config_returns_510(self):
|
||||
"""Remote dense NemotronH (HF id) → 510 via config.json fetch, not 530."""
|
||||
|
||||
class _Response:
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, tb):
|
||||
return False
|
||||
|
||||
def read(self):
|
||||
return json.dumps(
|
||||
{
|
||||
"model_type": "nemotron_h",
|
||||
"hybrid_override_pattern": "M-M-M*-M-",
|
||||
}
|
||||
).encode()
|
||||
|
||||
with patch("urllib.request.urlopen", return_value = _Response()):
|
||||
assert get_transformers_tier("unsloth/NVIDIA-Nemotron-3-Nano-4B") == "510"
|
||||
|
||||
def test_local_config_json_short_circuits_path_substrings(self, tmp_path: Path):
|
||||
"""Local config.json should prevent false matches from parent directory names."""
|
||||
model_dir = tmp_path / "gemma-4-12b-experiment" / "llama-checkpoint"
|
||||
|
|
@ -686,6 +981,67 @@ class TestActivateLoggingClarity:
|
|||
"sys.path" in text or "path only" in text
|
||||
), f"early activation log does not clarify it is path-prepend only: {text!r}"
|
||||
|
||||
def test_activate_prefers_local_checkpoint_tier_over_resolved_base(self, caplog, tmp_path):
|
||||
# Base resolves to an offline/private id (default tier); the local config.json wins.
|
||||
(tmp_path / "config.json").write_text(json.dumps({"model_type": "llama"}))
|
||||
local = str(tmp_path)
|
||||
caplog.set_level(logging.INFO)
|
||||
snap = self._snapshot_env()
|
||||
tiers = {local: "510", "private/base": "default"}
|
||||
try:
|
||||
with (
|
||||
patch(
|
||||
"utils.transformers_version._resolve_base_model",
|
||||
return_value = "private/base",
|
||||
),
|
||||
patch(
|
||||
"utils.transformers_version.get_transformers_tier",
|
||||
side_effect = lambda m: tiers[m],
|
||||
),
|
||||
patch(
|
||||
"utils.transformers_version._ensure_venv_t5_510_exists",
|
||||
return_value = True,
|
||||
),
|
||||
):
|
||||
activate_transformers_for_subprocess(local)
|
||||
finally:
|
||||
self._restore_env(snap)
|
||||
|
||||
text = " ".join(r.getMessage() for r in caplog.records).lower()
|
||||
assert "5.10.2" in text, f"local checkpoint tier did not win: {text!r}"
|
||||
|
||||
def test_activate_adapter_without_config_skips_path_name_recheck(self, caplog, tmp_path):
|
||||
# Adapter dir named 'gemma-4' but no config.json: the path-name re-check must not run.
|
||||
adapter = tmp_path / "gemma-4-experiment" / "llama-lora"
|
||||
adapter.mkdir(parents = True)
|
||||
local = str(adapter)
|
||||
caplog.set_level(logging.INFO)
|
||||
snap = self._snapshot_env()
|
||||
seen = []
|
||||
|
||||
def fake_tier(m):
|
||||
seen.append(m)
|
||||
return "550" if "gemma-4" in m else "default"
|
||||
|
||||
try:
|
||||
with (
|
||||
patch(
|
||||
"utils.transformers_version._resolve_base_model",
|
||||
return_value = "meta/llama",
|
||||
),
|
||||
patch(
|
||||
"utils.transformers_version.get_transformers_tier",
|
||||
side_effect = fake_tier,
|
||||
),
|
||||
):
|
||||
activate_transformers_for_subprocess(local)
|
||||
finally:
|
||||
self._restore_env(snap)
|
||||
|
||||
assert seen == ["meta/llama"], f"adapter path was re-checked via substrings: {seen!r}"
|
||||
text = " ".join(r.getMessage() for r in caplog.records).lower()
|
||||
assert "default transformers" in text, f"adapter wrongly upgraded: {text!r}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _venv_dir_is_valid — issue #6103
|
||||
|
|
|
|||
|
|
@ -5,7 +5,9 @@
|
|||
|
||||
Some newer model architectures (Ministral-3, GLM-4.7-Flash, Qwen3-30B-A3B MoE,
|
||||
tiny_qwen3_moe) require transformers>=5.3.0, while Gemma 4 models require a
|
||||
newer 5.x sidecar. Everything else needs the default 4.57.x that ships with
|
||||
newer 5.x sidecar. Dense NemotronH models (e.g. NVIDIA-Nemotron-3-Nano-4B) use
|
||||
MLP layers that only transformers>=5.10 can parse natively, so they go on the
|
||||
5.10 sidecar too. Everything else needs the default 4.57.x that ships with
|
||||
Unsloth.
|
||||
|
||||
Two separate target directories are maintained:
|
||||
|
|
@ -135,6 +137,13 @@ _VENV_T5_510_DIR = str(_studio_root() / ".venv_t5_510")
|
|||
# Backwards-compat alias
|
||||
_VENV_T5_DIR = _VENV_T5_550_DIR
|
||||
|
||||
# Tier precedence: higher rank wins in _higher_tier.
|
||||
_TIER_RANK = {"default": 0, "530": 1, "550": 2, "510": 3}
|
||||
|
||||
|
||||
def _higher_tier(a: str, b: str) -> str:
|
||||
return a if _TIER_RANK.get(a, 0) >= _TIER_RANK.get(b, 0) else b
|
||||
|
||||
|
||||
def activate_transformers_for_subprocess(model_name: str) -> None:
|
||||
"""Activate the correct transformers version in a subprocess worker.
|
||||
|
|
@ -146,6 +155,10 @@ def activate_transformers_for_subprocess(model_name: str) -> None:
|
|||
"""
|
||||
resolved = _resolve_base_model(model_name)
|
||||
tier = get_transformers_tier(resolved)
|
||||
if model_name != resolved and (Path(model_name) / "config.json").is_file():
|
||||
# Gate on a real local config.json: a checkpoint carries config the base may not
|
||||
# surface, but path names alone must not upgrade a plain adapter.
|
||||
tier = _higher_tier(tier, get_transformers_tier(model_name))
|
||||
|
||||
if tier == "510":
|
||||
if not _ensure_venv_t5_510_exists():
|
||||
|
|
@ -328,12 +341,56 @@ def _check_tokenizer_config_needs_v5(model_name: str) -> bool:
|
|||
return False
|
||||
|
||||
|
||||
def _safe_mtime(path: Path) -> float:
|
||||
try:
|
||||
return path.stat().st_mtime
|
||||
except OSError:
|
||||
return 0.0
|
||||
|
||||
|
||||
def _config_json_from_hf_cache(model_name: str) -> dict | None:
|
||||
"""Parsed ``config.json`` from the local HF hub cache, or None.
|
||||
|
||||
Stdlib-only path resolution (no ``huggingface_hub`` import) so tier detection never
|
||||
loads the default-env hub before a sidecar venv is activated.
|
||||
"""
|
||||
# Only a canonical ``owner/repo`` Hub id maps to a cache dir; reject local paths.
|
||||
if not model_name or model_name.count("/") != 1 or model_name[0] in "/.~" or "\\" in model_name:
|
||||
return None
|
||||
hub = (
|
||||
os.environ.get("HF_HUB_CACHE")
|
||||
or os.environ.get("HUGGINGFACE_HUB_CACHE")
|
||||
or os.path.join(
|
||||
os.environ.get("HF_HOME") or os.path.expanduser("~/.cache/huggingface"), "hub"
|
||||
)
|
||||
)
|
||||
repo_dir = Path(hub) / ("models--" + model_name.replace("/", "--"))
|
||||
candidates = []
|
||||
ref_main = repo_dir / "refs" / "main"
|
||||
try:
|
||||
if ref_main.is_file():
|
||||
candidates.append(repo_dir / "snapshots" / ref_main.read_text().strip() / "config.json")
|
||||
# No refs/main (e.g. commit-pinned downloads): newest snapshot by mtime, not a stale
|
||||
# lexicographically-first SHA, matching what the Hub cache would actually load.
|
||||
candidates += sorted(
|
||||
repo_dir.glob("snapshots/*/config.json"), key = _safe_mtime, reverse = True
|
||||
)
|
||||
for cfg_path in candidates:
|
||||
if cfg_path.is_file():
|
||||
with open(cfg_path) as f:
|
||||
return json.load(f)
|
||||
except Exception as exc:
|
||||
logger.debug("HF cache config.json lookup failed for '%s': %s", model_name, exc)
|
||||
return None
|
||||
|
||||
|
||||
def _load_config_json(model_name: str, hf_token: str | None = None) -> dict | None:
|
||||
"""Return parsed ``config.json`` for *model_name*, checking local files first.
|
||||
|
||||
``hf_token`` authenticates the raw fetch so gated/private repos resolve. The
|
||||
cache is keyed on the token so an unauthenticated miss never poisons a later
|
||||
authenticated read.
|
||||
authenticated read. The HF hub cache is consulted only offline or after a failed
|
||||
network fetch, so an online read never serves stale metadata.
|
||||
"""
|
||||
import hashlib
|
||||
|
||||
|
|
@ -355,9 +412,12 @@ def _load_config_json(model_name: str, hf_token: str | None = None) -> dict | No
|
|||
return None
|
||||
|
||||
if _env_offline():
|
||||
_config_json_cache[cache_key] = None
|
||||
return None
|
||||
# No network: a previously downloaded repo can still tier from the hub cache.
|
||||
cfg = _config_json_from_hf_cache(model_name)
|
||||
_config_json_cache[cache_key] = cfg
|
||||
return cfg
|
||||
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
url = f"https://huggingface.co/{model_name}/raw/main/config.json"
|
||||
|
|
@ -370,10 +430,24 @@ def _load_config_json(model_name: str, hf_token: str | None = None) -> dict | No
|
|||
cfg = json.loads(resp.read().decode())
|
||||
_config_json_cache[cache_key] = cfg
|
||||
return cfg
|
||||
except urllib.error.HTTPError as exc:
|
||||
# 401/403/404 is a definitive access answer: never serve another caller's cached
|
||||
# private metadata to an unauthenticated/wrong-token request.
|
||||
if exc.code in (401, 403, 404):
|
||||
logger.debug("config.json access denied for '%s': %s", model_name, exc)
|
||||
return None
|
||||
logger.debug("Could not fetch config.json for '%s': %s", model_name, exc)
|
||||
return _config_json_from_hf_cache(model_name)
|
||||
except Exception as exc:
|
||||
logger.debug("Could not fetch config.json for '%s': %s", model_name, exc)
|
||||
_config_json_cache[cache_key] = None
|
||||
return None
|
||||
# Transient: serve the hub cache uncached so the next call retries the network.
|
||||
return _config_json_from_hf_cache(model_name)
|
||||
|
||||
|
||||
def _config_json_is_definitive(model_name: str) -> bool:
|
||||
"""True if the last unauthenticated ``_load_config_json`` read was cached (definitive),
|
||||
not a transient fallback (deliberately not stored, so callers re-check next call)."""
|
||||
return (model_name, None) in _config_json_cache
|
||||
|
||||
|
||||
def _config_matches_tier(cfg: dict, architectures: set[str], model_types: set[str]) -> bool:
|
||||
|
|
@ -393,30 +467,47 @@ def _config_needs_550(cfg: dict) -> bool:
|
|||
)
|
||||
|
||||
|
||||
_NESTED_CONFIG_KEYS = ("llm_config", "text_config", "language_config", "thinker_config")
|
||||
|
||||
|
||||
def _nemotron_h_needs_mlp_support(cfg: dict) -> bool:
|
||||
"""True for a dense NemotronH config using MLP (``-``) layers.
|
||||
|
||||
transformers only gained ``-`` -> ``mlp`` in 5.10; 5.3/5.5 raise ``KeyError: '-'``.
|
||||
Read from ``hybrid_override_pattern`` or ``layers_block_type``, recursing into nested
|
||||
language configs (VL wrappers hold the dense LM under ``llm_config``/``text_config``).
|
||||
"""
|
||||
if not isinstance(cfg, dict):
|
||||
return False
|
||||
if cfg.get("model_type") == "nemotron_h":
|
||||
pattern = cfg.get("hybrid_override_pattern")
|
||||
if isinstance(pattern, str) and "-" in pattern:
|
||||
return True
|
||||
block_types = cfg.get("layers_block_type")
|
||||
if isinstance(block_types, (list, tuple)) and "mlp" in block_types:
|
||||
return True
|
||||
return any(_nemotron_h_needs_mlp_support(cfg.get(key)) for key in _NESTED_CONFIG_KEYS)
|
||||
|
||||
|
||||
def _config_needs_510(cfg: dict) -> bool:
|
||||
return _config_matches_tier(
|
||||
if _config_matches_tier(
|
||||
cfg,
|
||||
_TRANSFORMERS_510_ARCHITECTURES,
|
||||
_TRANSFORMERS_510_MODEL_TYPES,
|
||||
)
|
||||
):
|
||||
return True
|
||||
return _nemotron_h_needs_mlp_support(cfg)
|
||||
|
||||
|
||||
def _check_config_needs_550(model_name: str) -> bool:
|
||||
"""True if ``config.json`` has architectures/model_type needing transformers
|
||||
5.5.0 (e.g. Gemma 4).
|
||||
|
||||
Checks locally first, else fetches from HuggingFace. Cached in
|
||||
``_config_needs_550_cache``. Returns False on any error (fail-open to lower tier).
|
||||
"""True if ``config.json`` needs transformers 5.5.0 (e.g. Gemma 4). Local first, else
|
||||
fetched; cached only for a definitive read so a transient miss retries. False on error.
|
||||
"""
|
||||
if model_name in _config_needs_550_cache:
|
||||
return _config_needs_550_cache[model_name]
|
||||
|
||||
cfg = _load_config_json(model_name)
|
||||
if cfg is None:
|
||||
_config_needs_550_cache[model_name] = False
|
||||
return False
|
||||
|
||||
result = _config_needs_550(cfg)
|
||||
result = bool(cfg) and _config_needs_550(cfg)
|
||||
if result:
|
||||
logger.info(
|
||||
"config.json check: %s needs transformers %s (architectures=%s, model_type=%s)",
|
||||
|
|
@ -425,7 +516,8 @@ def _check_config_needs_550(model_name: str) -> bool:
|
|||
cfg.get("architectures", []),
|
||||
cfg.get("model_type"),
|
||||
)
|
||||
_config_needs_550_cache[model_name] = result
|
||||
if _config_json_is_definitive(model_name):
|
||||
_config_needs_550_cache[model_name] = result
|
||||
return result
|
||||
|
||||
|
||||
|
|
@ -435,11 +527,7 @@ def _check_config_needs_510(model_name: str) -> bool:
|
|||
return _config_needs_510_cache[model_name]
|
||||
|
||||
cfg = _load_config_json(model_name)
|
||||
if cfg is None:
|
||||
_config_needs_510_cache[model_name] = False
|
||||
return False
|
||||
|
||||
result = _config_needs_510(cfg)
|
||||
result = bool(cfg) and _config_needs_510(cfg)
|
||||
if result:
|
||||
logger.info(
|
||||
"config.json check: %s needs transformers %s (architectures=%s, model_type=%s)",
|
||||
|
|
@ -448,7 +536,8 @@ def _check_config_needs_510(model_name: str) -> bool:
|
|||
cfg.get("architectures", []),
|
||||
cfg.get("model_type"),
|
||||
)
|
||||
_config_needs_510_cache[model_name] = result
|
||||
if _config_json_is_definitive(model_name):
|
||||
_config_needs_510_cache[model_name] = result
|
||||
return result
|
||||
|
||||
|
||||
|
|
@ -824,6 +913,10 @@ def ensure_transformers_version(model_name: str) -> None:
|
|||
# Resolve LoRA adapters to their base model for accurate detection.
|
||||
resolved = _resolve_base_model(model_name)
|
||||
tier = get_transformers_tier(resolved)
|
||||
if model_name != resolved and (Path(model_name) / "config.json").is_file():
|
||||
# Gate on a real local config.json: a checkpoint carries config the base may not
|
||||
# surface, but path names alone must not upgrade a plain adapter.
|
||||
tier = _higher_tier(tier, get_transformers_tier(model_name))
|
||||
|
||||
if tier == "510":
|
||||
target_version = TRANSFORMERS_510_VERSION
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue