The GitHub issue check had issues:
1. Network latency on import
2. Issue being closed does not mean the fix is in the installed vLLM version
Now skip the PDL workaround if vLLM version > 0.13.2, which is when
the upstream fix is expected to be included.
- Patch vllm.lora.ops.triton_ops.utils directly where supports_pdl is defined
- Clear lru_cache before patching to prevent stale cached results
- Add fused_moe_lora_op to consumer modules list
- Use *args, **kwargs in fake function for compatibility
- Add _spec_exists helper function to reduce duplication
- Scan all GPUs for SM100 instead of just device 0
- Use loop for module patching to improve maintainability
When using base models with custom chat templates applied after loading,
vLLM's internal tokenizer may not have the chat_template set. This causes
issues during RL training with vLLM inference.
This fix syncs the chat_template from the processing_class (the tokenizer
you loaded and configured) to vLLM's internal tokenizer during trainer
initialization, but only if vLLM's tokenizer does not already have one set.
vLLM's LoRA Triton kernels use tl.extra.cuda.gdc_wait() for PDL
optimization on SM90+ GPUs. This fails on SM100 (Blackwell) during
CUDA graph capture because Triton's pipeliner cannot handle gdc_wait
in complex kernels.
This fix:
- Detects SM100 GPUs and applies the workaround automatically
- Sets TRITON_DISABLE_PDL=1 environment variable
- Monkey-patches supports_pdl to return False in lora_expand_op and
lora_shrink_op
- Checks GitHub issue #30872 status (with 3s timeout) to auto-disable
the workaround once the upstream fix is merged
- Includes quick internet connectivity check (0.5s) to avoid delays
when offline
Fixes the error:
'tt.elementwise_inline_asm' op pipeliner doesn't know how to predicate this op
LLVM ERROR: Fatal pipeliner error
See: https://github.com/vllm-project/vllm/issues/30872
When users load a model with fast_inference=False but then try to use
vLLM-style arguments with fast_generate, they previously got confusing
errors. This adds a wrapper that detects common mistakes and provides
helpful guidance:
- Using sampling_params: explains to use HF generate args instead
- Using lora_request: explains LoRA weights are already merged
- Passing text strings: shows how to tokenize input first
Changes:
- Add make_fast_generate_wrapper to _utils.py
- Apply wrapper in llama.py when fast_inference=False
- Apply wrapper in vision.py when fast_inference=False
Gemma3 models have a large vocabulary (262144 tokens) which causes
training loss to explode when using int8 embedding quantization.
This fix auto-detects Gemma3 models and switches from int8-int4
(phone-deployment) to int4 weight-only QAT for stable training.
1. cohere.py:347-348 - Fixed wrong variable names in QK normalization.
Used `Q`/`K` but variables were named `Qn`/`Kn`. This caused NameError
when `use_qk_norm=True` (e.g., c4ai-command-r-plus models).
2. cohere.py:482 - Fixed wrong object reference in inference loop.
Used `self.mlp` but should be `decoder_layer.mlp` since we're
iterating through decoder layers. Caused AttributeError during inference.
3. falcon_h1.py:459,461 - Fixed wrong attribute names in inference path.
Used `post_attention_layernorm` and `mlp` but Falcon H1 uses
`pre_ff_layernorm` and `feed_forward`. Caused AttributeError during generation.
4. qwen3_moe.py:210 - Fixed wrong module path with incorrect capitalization.
Used `transformers.models.Qwen3Moe` but should be `transformers.models.qwen3_moe`.
Caused AttributeError when patching rotary embeddings.
5. qwen3_moe.py:239 - Fixed wrong model_patcher class.
Used `FastQwen3Model` but should be `FastQwen3MoeModel` for MoE models.
Caused incorrect patching for Qwen3 MoE models.
6. hf_hub.py:21-22 - Fixed floor division and missing return for billion values.
Used `//` instead of `/` for millions, and had no return for values >= 1B.
Caused incorrect formatting and None return for large numbers.
7. save.py:550 - Fixed self-assignment that did nothing.
`sharded_ram_usage = sharded_ram_usage` should be `= max_shard_size`.
Caused integer shard sizes to be ignored.
8. rl.py:562-567 - Fixed orphan string not included in length_check.
The elif branch for max_seq_length validation was a standalone string
expression, not concatenated to length_check. Caused silent skip of
the max_seq_length > model_max_seq_length warning.
9. granite.py:49-52 - Fixed wrong model name and version in error message.
Said "Gemma2" and "4.42.3" but should be "Granite" and "4.45.0".
* Fix correctness bugs in rl.py, rl_replacements.py, and vision.py
1. rl_replacements.py (lines 864, 870): Fixed undefined `nanmin`/`nanmax`
functions by using `.nan_to_num(nan=inf/-inf).min()/.max()` pattern.
PyTorch doesn't have torch.nanmin/nanmax, so we replace NaN values
before computing min/max.
2. vision.py (line 150): Fixed bug where code checked for "input" key
but then accessed kwargs["input_ids"] instead of kwargs["input"].
3. vision.py (line 159): Fixed bug where literal string "key" was used
instead of the variable `key` when accessing kwargs.
4. rl.py (lines 903, 905): Fixed non-existent `MathError` exception
by replacing with `ValueError`.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
