* Load repo-code VLMs that register AutoModel in auto_map
FastModel.from_pretrained already falls back from the VLM auto class to
AutoModelForCausalLM for repo-code VL models that register only that class
in their auto_map (e.g. Nemotron-VL). Models like DeepSeek-OCR and
DeepSeek-OCR-2 instead register their architecture under AutoModel, so they
fell through to AutoModelForImageTextToText and raised "Unrecognized
configuration class ... for AutoModelForImageTextToText".
Generalize the guard: when neither vision auto class is registered, fall
back to whichever generic auto class the repo actually registered
(AutoModelForCausalLM, else AutoModel).
* Do not hard-error on a newly initialized position_ids buffer
RaiseUninitialized turns transformers' "some weights of ... were not
initialized" warning into a hard error. position_ids is a deterministic
arange buffer that transformers itself lists in
_keys_to_ignore_on_load_missing, so re-initializing it is correct rather
than a sign of a corrupt checkpoint. Some VLMs (e.g. DeepSeek-OCR) ship it
non-persistently, which tripped the guard. Allowlist position_ids alongside
the existing classifier/predictions head weights.
* Only ignore missing-weight records that are exclusively position_ids
The previous substring check skipped the whole "Some weights of ..." record
whenever position_ids appeared anywhere in it. Transformers reports every
missing key in one record, so a corrupt or incompatible checkpoint missing a
real parameter could load with randomly initialized weights as long as one
missing key contained position_ids. Parse the "newly initialized: [...]" list
and suppress only when every listed key is a position_ids buffer; otherwise
raise as before.
* Match the concrete VLM auto class name when checking auto_map
Transformers resolves remote code by the exact auto class name being called,
and AutoModelForVision2Seq aliases to AutoModelForImageTextToText on
transformers >= 5. Checking for both spellings treated a config that only
registers the legacy AutoModelForVision2Seq key as having a supported VLM
class, skipping the AutoModelForCausalLM fallback that used to load it and
failing as an unrecognized config under AutoModelForImageTextToText. Match
only the concrete class name we would actually pass, keeping the AutoModel
and AutoModelForCausalLM fallbacks.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Preserve VLM mode on the vLLM path when falling back to AutoModel
A repo-code VLM that registers only AutoModel or AutoModelForCausalLM (DeepSeek-OCR, Nemotron-VL) routes to that generic class, so is_vlm, derived from the resolved auto class, is False. That is correct for processor selection (these repos ship no AutoProcessor) but wrong for the vLLM path, where is_vision_model=is_vlm made vLLM treat a vision_config model as text-only and skip the VLM guard and conversion.
Add is_vlm_config, derived from the config vision_config (and gated on not text_only so a text-only resolve still wins), and use it for the fast_inference VLM guard and the is_vision_model flags passed to load_vllm, get_vllm_state_dict and convert_vllm_to_huggingface. Processor selection still uses is_vlm, so DeepSeek-OCR keeps loading via its tokenizer. DeepSeek-OCR with fast_inference now raises the clear 'Fast inference is only supported for ...' error instead of being mishandled as text-only.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Daniel Han