When using base models with custom chat templates applied after loading,
vLLM's internal tokenizer may not have the chat_template set. This causes
issues during RL training with vLLM inference.
This fix syncs the chat_template from the processing_class (the tokenizer
you loaded and configured) to vLLM's internal tokenizer during trainer
initialization, but only if vLLM's tokenizer does not already have one set.
vLLM's LoRA Triton kernels use tl.extra.cuda.gdc_wait() for PDL
optimization on SM90+ GPUs. This fails on SM100 (Blackwell) during
CUDA graph capture because Triton's pipeliner cannot handle gdc_wait
in complex kernels.
This fix:
- Detects SM100 GPUs and applies the workaround automatically
- Sets TRITON_DISABLE_PDL=1 environment variable
- Monkey-patches supports_pdl to return False in lora_expand_op and
lora_shrink_op
- Checks GitHub issue #30872 status (with 3s timeout) to auto-disable
the workaround once the upstream fix is merged
- Includes quick internet connectivity check (0.5s) to avoid delays
when offline
Fixes the error:
'tt.elementwise_inline_asm' op pipeliner doesn't know how to predicate this op
LLVM ERROR: Fatal pipeliner error
See: https://github.com/vllm-project/vllm/issues/30872
When users load a model with fast_inference=False but then try to use
vLLM-style arguments with fast_generate, they previously got confusing
errors. This adds a wrapper that detects common mistakes and provides
helpful guidance:
- Using sampling_params: explains to use HF generate args instead
- Using lora_request: explains LoRA weights are already merged
- Passing text strings: shows how to tokenize input first
Changes:
- Add make_fast_generate_wrapper to _utils.py
- Apply wrapper in llama.py when fast_inference=False
- Apply wrapper in vision.py when fast_inference=False
Gemma3 models have a large vocabulary (262144 tokens) which causes
training loss to explode when using int8 embedding quantization.
This fix auto-detects Gemma3 models and switches from int8-int4
(phone-deployment) to int4 weight-only QAT for stable training.
1. cohere.py:347-348 - Fixed wrong variable names in QK normalization.
Used `Q`/`K` but variables were named `Qn`/`Kn`. This caused NameError
when `use_qk_norm=True` (e.g., c4ai-command-r-plus models).
2. cohere.py:482 - Fixed wrong object reference in inference loop.
Used `self.mlp` but should be `decoder_layer.mlp` since we're
iterating through decoder layers. Caused AttributeError during inference.
3. falcon_h1.py:459,461 - Fixed wrong attribute names in inference path.
Used `post_attention_layernorm` and `mlp` but Falcon H1 uses
`pre_ff_layernorm` and `feed_forward`. Caused AttributeError during generation.
4. qwen3_moe.py:210 - Fixed wrong module path with incorrect capitalization.
Used `transformers.models.Qwen3Moe` but should be `transformers.models.qwen3_moe`.
Caused AttributeError when patching rotary embeddings.
5. qwen3_moe.py:239 - Fixed wrong model_patcher class.
Used `FastQwen3Model` but should be `FastQwen3MoeModel` for MoE models.
Caused incorrect patching for Qwen3 MoE models.
6. hf_hub.py:21-22 - Fixed floor division and missing return for billion values.
Used `//` instead of `/` for millions, and had no return for values >= 1B.
Caused incorrect formatting and None return for large numbers.
7. save.py:550 - Fixed self-assignment that did nothing.
`sharded_ram_usage = sharded_ram_usage` should be `= max_shard_size`.
Caused integer shard sizes to be ignored.
8. rl.py:562-567 - Fixed orphan string not included in length_check.
The elif branch for max_seq_length validation was a standalone string
expression, not concatenated to length_check. Caused silent skip of
the max_seq_length > model_max_seq_length warning.
9. granite.py:49-52 - Fixed wrong model name and version in error message.
Said "Gemma2" and "4.42.3" but should be "Granite" and "4.45.0".
* Fix correctness bugs in rl.py, rl_replacements.py, and vision.py
1. rl_replacements.py (lines 864, 870): Fixed undefined `nanmin`/`nanmax`
functions by using `.nan_to_num(nan=inf/-inf).min()/.max()` pattern.
PyTorch doesn't have torch.nanmin/nanmax, so we replace NaN values
before computing min/max.
2. vision.py (line 150): Fixed bug where code checked for "input" key
but then accessed kwargs["input_ids"] instead of kwargs["input"].
3. vision.py (line 159): Fixed bug where literal string "key" was used
instead of the variable `key` when accessing kwargs.
4. rl.py (lines 903, 905): Fixed non-existent `MathError` exception
by replacing with `ValueError`.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Add "corda" as an allowed value for the init_lora_weights parameter
in FastLanguageModel.get_peft_model() and FastBaseModel.get_peft_model().
This enables users to use CorDA (Correlation-aware Decomposed Adaptation)
initialization from PEFT, which provides an alternative LoRA initialization
strategy for improved finetuning performance.
Fixes#3693
Signed-off-by: majiayu000 <1835304752@qq.com>
* Fix is_contiguous() method call and remove duplicate imports
- Fix bug in rope_embedding.py where is_contiguous was used without
parentheses, causing the method object (always truthy) to be evaluated
instead of calling the method. This fixes issue #3781 where fast rope
backpropagation was broken for zero strided/non-contiguous tensors.
- Remove duplicate `import torch` in rl.py (lines 20 and 25)
- Remove duplicate `import functools` and `import types` in vision.py
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix Boolean value of Tensor ambiguity error in mistral.py
Replace `or` operator with explicit `is None` check when getting
n_items from kwargs. The `or` operator fails when the value is a
Tensor because Python cannot determine the boolean value of a
multi-element tensor.
Fixes#3766🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Update rope_embedding.py
---------
Co-authored-by: yurekami <yurekami@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Guard optional trl.experimental.openenv usage in RL patches
* Simplify optional trl.openenv import handling
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(trainer): import psutil to prevent NameError in _prepare_dataset
Fixes#3777
* Update rl.py
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
