vision: enable fast_inference for Gemma 4 via vLLM nightly
Add "gemma4" to VLLM_SUPPORTED_VLM so FastLanguageModel(fast_inference=True) stops rejecting unsloth/gemma-4-E2B-it / gemma-4-26b-a4b-it at the vision-model arch gate. vLLM nightly (vllm#39291, merged 2026-04-17) registers Gemma4ForConditionalGeneration as SupportsLoRA; the unsloth-zoo#603 vLLM patches (patch_gemma4_vllm_lora_support + patch_gemma4_vllm_k_eq_v_support) wire LoRA + attention_k_eq_v support on top of that. Also force compilation_config=0 and enforce_eager=True for any gemma4 variant. vLLM's -O3 backend currently hits "Tried to erase Node size_N but it still had N users" in _decompose_size_nodes when splitting the Gemma 4 FX graph (both dense E2B audio path and MoE 26B-A4B language path). Until the upstream FX splitter grows support for these size() users, eager execution preserves correctness — bitwise token parity with HF naive confirmed at 25/25, 4/4, 5/5 on 3 chat prompts × 32 tokens.
This commit is contained in:
parent
749fb75bd0
commit
f950338330
1 changed files with 11 additions and 0 deletions
|
|
@ -245,6 +245,7 @@ VLLM_SUPPORTED_VLM = [
|
|||
"mistral3",
|
||||
"qwen3_vl",
|
||||
"qwen3_vl_moe",
|
||||
"gemma4",
|
||||
]
|
||||
VLLM_NON_LORA_VLM = [
|
||||
"mllama",
|
||||
|
|
@ -1053,6 +1054,16 @@ class FastBaseModel:
|
|||
is_vision_model = is_vlm,
|
||||
fp8_mode = fp8_mode,
|
||||
)
|
||||
# why: vLLM's -O3 compile backend (compilation_config=3) hits
|
||||
# _decompose_size_nodes "Tried to erase Node size_N but it still
|
||||
# had N users" on Gemma 4 multimodal models (both dense E2B and
|
||||
# MoE 26B-A4B). Until the upstream FX splitter is fixed, fall
|
||||
# back to compilation_config=0 (no piecewise compile) for gemma4
|
||||
# so fast_inference still loads. Runtime is slower than -O3 but
|
||||
# correctness is preserved.
|
||||
if any(arch == "gemma4" for arch in (model_types or [])):
|
||||
load_vllm_kwargs.setdefault("compilation_config", 0)
|
||||
load_vllm_kwargs.setdefault("enforce_eager", True)
|
||||
for allowed_arg in allowed_args:
|
||||
if allowed_arg not in load_vllm_kwargs and allowed_arg in kwargs:
|
||||
load_vllm_kwargs[allowed_arg] = kwargs[allowed_arg]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue