18f8869829* Load repo-code VLMs that register AutoModel in auto_map FastModel.from_pretrained already falls back from the VLM auto class to AutoModelForCausalLM for repo-code VL models that register only that class in their auto_map (e.g. Nemotron-VL). Models like DeepSeek-OCR and DeepSeek-OCR-2 instead register their architecture under AutoModel, so they fell through to AutoModelForImageTextToText and raised "Unrecognized configuration class ... for AutoModelForImageTextToText". Generalize the guard: when neither vision auto class is registered, fall back to whichever generic auto class the repo actually registered (AutoModelForCausalLM, else AutoModel). * Do not hard-error on a newly initialized position_ids buffer RaiseUninitialized turns transformers' "some weights of ... were not initialized" warning into a hard error. position_ids is a deterministic arange buffer that transformers itself lists in _keys_to_ignore_on_load_missing, so re-initializing it is correct rather than a sign of a corrupt checkpoint. Some VLMs (e.g. DeepSeek-OCR) ship it non-persistently, which tripped the guard. Allowlist position_ids alongside the existing classifier/predictions head weights. * Only ignore missing-weight records that are exclusively position_ids The previous substring check skipped the whole "Some weights of ..." record whenever position_ids appeared anywhere in it. Transformers reports every missing key in one record, so a corrupt or incompatible checkpoint missing a real parameter could load with randomly initialized weights as long as one missing key contained position_ids. Parse the "newly initialized: [...]" list and suppress only when every listed key is a position_ids buffer; otherwise raise as before. * Match the concrete VLM auto class name when checking auto_map Transformers resolves remote code by the exact auto class name being called, and AutoModelForVision2Seq aliases to AutoModelForImageTextToText on transformers >= 5. Checking for both spellings treated a config that only registers the legacy AutoModelForVision2Seq key as having a supported VLM class, skipping the AutoModelForCausalLM fallback that used to load it and failing as an unrecognized config under AutoModelForImageTextToText. Match only the concrete class name we would actually pass, keeping the AutoModel and AutoModelForCausalLM fallbacks. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Preserve VLM mode on the vLLM path when falling back to AutoModel A repo-code VLM that registers only AutoModel or AutoModelForCausalLM (DeepSeek-OCR, Nemotron-VL) routes to that generic class, so is_vlm, derived from the resolved auto class, is False. That is correct for processor selection (these repos ship no AutoProcessor) but wrong for the vLLM path, where is_vision_model=is_vlm made vLLM treat a vision_config model as text-only and skip the VLM guard and conversion. Add is_vlm_config, derived from the config vision_config (and gated on not text_only so a text-only resolve still wins), and use it for the fast_inference VLM guard and the is_vision_model flags passed to load_vllm, get_vllm_state_dict and convert_vllm_to_huggingface. Processor selection still uses is_vlm, so DeepSeek-OCR keeps loading via its tokenizer. DeepSeek-OCR with fast_inference now raises the clear 'Fast inference is only supported for ...' error instead of being mishandled as text-only. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
76 lines
2.5 KiB
Python
76 lines
2.5 KiB
Python
"""RaiseUninitialized must ignore a checkpoint that only re-initializes deterministic
|
|
position_ids buffers, but still raise when a real weight is missing -- even if the same
|
|
HF record also lists a benign position_ids buffer.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
|
|
import pytest
|
|
|
|
from unsloth.models._utils import (
|
|
_all_missing_keys_are_position_ids,
|
|
_RaiseUninitialized,
|
|
)
|
|
|
|
_TEMPLATE = (
|
|
"Some weights of DeepseekOCRForCausalLM were not initialized from the model "
|
|
"checkpoint at unsloth/DeepSeek-OCR and are newly initialized: {keys}\n"
|
|
"You should probably TRAIN this model on a down-stream task."
|
|
)
|
|
|
|
|
|
def _record(keys_repr: str) -> logging.LogRecord:
|
|
return logging.LogRecord(
|
|
name = "transformers.modeling_utils",
|
|
level = logging.WARNING,
|
|
pathname = "modeling_utils.py",
|
|
lineno = 1,
|
|
msg = _TEMPLATE.format(keys = keys_repr),
|
|
args = None,
|
|
exc_info = None,
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"keys_repr, expected",
|
|
[
|
|
("['model.vision_model.embeddings.position_ids']", True),
|
|
(
|
|
"['model.vision_model.embeddings.position_ids', "
|
|
"'vision_model.encoder.layers.0.position_ids']",
|
|
True,
|
|
),
|
|
# A real missing weight alongside position_ids must NOT be suppressed.
|
|
(
|
|
"['model.vision_model.embeddings.position_ids', 'model.layers.5.mlp.weight']",
|
|
False,
|
|
),
|
|
("['model.layers.5.mlp.weight']", False),
|
|
("[]", False),
|
|
],
|
|
)
|
|
def test_all_missing_keys_are_position_ids(keys_repr, expected):
|
|
assert _all_missing_keys_are_position_ids(_TEMPLATE.format(keys = keys_repr)) is expected
|
|
|
|
|
|
def test_emit_suppresses_position_ids_only_record():
|
|
# A record listing only position_ids buffers loads cleanly (no raise).
|
|
handler = _RaiseUninitialized()
|
|
handler.emit(_record("['model.vision_model.embeddings.position_ids']"))
|
|
|
|
|
|
def test_emit_raises_when_real_weight_missing_alongside_position_ids():
|
|
# The core fix: one benign position_ids key must not mask a real missing weight.
|
|
handler = _RaiseUninitialized()
|
|
with pytest.raises(Exception, match = "some weights are not initialized"):
|
|
handler.emit(
|
|
_record("['model.vision_model.embeddings.position_ids', 'model.layers.5.mlp.weight']")
|
|
)
|
|
|
|
|
|
def test_emit_raises_on_real_missing_weight():
|
|
handler = _RaiseUninitialized()
|
|
with pytest.raises(Exception, match = "some weights are not initialized"):
|
|
handler.emit(_record("['model.layers.5.mlp.weight']"))
|