Fix notebook compatibility for transformers 4.57.6 and TRL 0.22-0.27 (#3998)
* Patch before compile? * Fix notebook compatibility for transformers 4.57.6 and TRL 0.22-0.27 Fixes several notebook failures discovered during testing all 125 notebooks with transformers==4.57.6 + tRL 0.22.2 and TRL 0.27.1. Warning suppression (import_fixes.py): - Suppress torch 2.9+ pin_memory/is_pinned device deprecation warnings - Suppress cuda.cudart/cuda.nvrtc module deprecation FutureWarning - Filter vllm "Level is deprecated" stderr noise - Filter PydanticSerializationUnexpectedValue warnings - Filter Triton "df: No such file" stderr noise VLM tokenizer loading (vision.py): - Add _construct_vlm_processor_fallback() for models where AutoProcessor.from_pretrained fails (e.g., ERNIE 4.5 VL, LFM2.5-VL) - Wrap processor loading in try/except with fallback to manual construction from separate image_processor + tokenizer components - Add fallback to AutoTokenizer/PreTrainedTokenizerFast when tokenizer loading or patching fails TRL 0.27.1 trainer compatibility (trainer.py): - Add _resolve_trainer_params() to handle thin wrapper trainers that only have def __init__(self, *args, **kwargs) (e.g., ORPOTrainer in TRL 0.27.1) by walking MRO for real parameter signature VLM _is_vlm detection (rl.py): - Replace blanket _is_vlm=False override with model-architecture-based detection that checks vision_config or ForConditionalGeneration class name, fixing VLM training when bare tokenizer is passed as processing_class ModernBERT SDPA compatibility (loader.py, sentence_transformer.py): - Add "modernbert" to DISABLE_SDPA_MODEL_NAMES to avoid stride alignment issues with torch.compile backward pass - Add DISABLE_SDPA check for sentence transformer models Other fixes (_utils.py): - Suppress false uninitialized weight warnings for VLM multi_modal_projector.layer_norm Tested: 92/125 notebooks pass with TRL 0.22.2, 94/125 with TRL 0.27.1. Remaining failures are infra (missing FFmpeg, network timeouts, GPU arch) not code bugs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix KTO shape mismatch on TRL 0.27.2+ and truncation alignment - Patch KTO get_batch_logps to auto-align logits and labels when Unsloth model forward truncates input_ids beyond max_seq_length. TRL 0.27.2 changed _process_tokens to only truncate completions (not prompts), so sequences with long prompts exceed max_seq_length and trigger model-side truncation. The original ValueError is replaced with min-length alignment. - Also truncate attention_mask in LlamaModel forward when input_ids are truncated to max_seq_length, preventing shape mismatches in attention. - Widen except clause in rl_replacements.py openenv import from `except ImportError` to `except (ImportError, NameError, Exception)` to handle vllm SamplingParams NameError in TRL 0.27.2. * Fix TRL 0.26+ thin wrapper resolution, enable ModernBERT SDPA, clean up warning filters TRL 0.26+ thin wrapper resolution (rl.py): - Filter _-prefixed private imports when discovering Trainer/Config classes - Look up Config in separate *_config.py module when not found in trainer module - Detect thin wrappers (<1000 chars source) and resolve to experimental parent via MRO walk; use resolved module for imports and create_new_function - Enables all 15 trainers to patch successfully (was 5/15 before) ModernBERT SDPA (loader.py): - Remove "modernbert" from DISABLE_SDPA_MODEL_NAMES - SDPA works correctly for both classification and sentence transformers - Verified: 88.9% accuracy on emotion classification, correct domain-specific embeddings after sentence transformer fine-tuning Warning filter cleanup (import_fixes.py): - Remove cuda.cudart/cuda.nvrtc FutureWarning filters (no such warnings exist in torch 2.9.1+; proactive suppression is unnecessary) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove multi_modal_projector.layer_norm from uninitialized weight guard The LFM2.5-VL projector LayerNorm is properly initialized by transformers and does not need to be excluded from the uninitialized weight check. The original exclusion was added as a workaround but is no longer needed after the upstream fix. * Add transformers 5.0 compat: rope_theta helper, config-as-dim detection, BatchEncoding guard, try/except for TRL trainer source, push_to_hub_token compiler fix - llama.py: Add _get_rope_theta() helper handling both config.rope_theta and rope_parameters dict - llama.py: Handle BatchEncoding in unsloth_fast_generate (transformers 5.0+ returns BatchEncoding from apply_chat_template) - gemma.py: Detect config passed as dim arg in GemmaFixedRotaryEmbedding - tokenizer_utils.py: Add try/except for TRL trainer getsource in patch_sft_trainer_tokenizer - rl_replacements.py: Add compiler fix replacing bare pop("push_to_hub_token") with pop(..., None) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Use trl.experimental string check instead of char-count heuristic for thin wrapper detection The <1000 / >1000 char threshold was fragile -- XPOConfig's parent is only 994 chars and would be skipped. All thin wrappers in TRL 0.26+ contain "trl.experimental" in their deprecation warning, while no real trainer or config class does, making it a reliable detection marker. * Move DISABLE_SDPA_MODEL_NAMES import to module level in sentence_transformer The function-level import was redundant since loader.py is already imported at module level. Move it to the existing loader import line. --------- Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com> Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han
GitHub
parent
884ce4601f
commit
ba7366be53
10 changed files with 428 additions and 41 deletions
|
|
@ -164,6 +164,39 @@ if os.environ.get("UNSLOTH_ENABLE_LOGGING", "0") != "1":
|
|||
"ignore", message = r"unclosed file.*dev/null", category = ResourceWarning
|
||||
)
|
||||
|
||||
# torch 2.9+ pin_memory/is_pinned device arg deprecation
|
||||
warnings.filterwarnings(
|
||||
"ignore",
|
||||
message = r"The `device` argument is deprecated",
|
||||
category = DeprecationWarning,
|
||||
)
|
||||
warnings.filterwarnings(
|
||||
"ignore",
|
||||
message = r".*pin_memory.*device.*deprecated",
|
||||
category = DeprecationWarning,
|
||||
)
|
||||
warnings.filterwarnings(
|
||||
"ignore",
|
||||
message = r".*is_pinned.*device.*deprecated",
|
||||
category = DeprecationWarning,
|
||||
)
|
||||
|
||||
# vllm "Level is deprecated" stderr noise
|
||||
sys.stderr.add_filter("Level is deprecated")
|
||||
|
||||
# PydanticSerializationUnexpectedValue warning
|
||||
warnings.filterwarnings(
|
||||
"ignore",
|
||||
message = r".*PydanticSerializationUnexpectedValue",
|
||||
)
|
||||
warnings.filterwarnings(
|
||||
"ignore",
|
||||
message = r"Expected.*but got.*with value.*is not.*subclass",
|
||||
)
|
||||
|
||||
# Triton "df: No such file or directory" stderr noise
|
||||
sys.stderr.add_filter("df: No such file")
|
||||
|
||||
|
||||
# Fix up AttributeError: 'MessageFactory' object has no attribute 'GetPrototype'
|
||||
# MUST do this at the start primarily due to tensorflow causing issues
|
||||
|
|
|
|||
|
|
@ -1966,6 +1966,12 @@ def unsloth_compile_transformers(
|
|||
return model_types, False
|
||||
|
||||
supports_sdpa = [True]
|
||||
|
||||
# Run patches BEFORE compiler so class replacements (e.g. GptOssTopKRouter,
|
||||
# GptOssExperts) are in place before the compiler caches references to them.
|
||||
for temporary_patch in TEMPORARY_PATCHES:
|
||||
temporary_patch()
|
||||
|
||||
for model_type in model_types:
|
||||
_unsloth_compile_transformers(
|
||||
model_type,
|
||||
|
|
|
|||
|
|
@ -13,6 +13,7 @@
|
|||
# limitations under the License.
|
||||
|
||||
from .llama import *
|
||||
from .llama import _get_rope_theta
|
||||
from ._utils import __version__
|
||||
from unsloth_zoo.utils import _get_dtype, Version
|
||||
from unsloth_zoo.hf_utils import dtype_from_config
|
||||
|
|
@ -256,9 +257,17 @@ class GemmaFixedRotaryEmbedding(torch.nn.Module):
|
|||
config = None, # [TODO] Hack to pass in config - need to remove later
|
||||
):
|
||||
super().__init__()
|
||||
# In transformers 5.0+, RotaryEmbedding(config) passes config as first positional arg (dim)
|
||||
if (
|
||||
config is None
|
||||
and dim is not None
|
||||
and hasattr(dim, "max_position_embeddings")
|
||||
):
|
||||
config = dim
|
||||
dim = None
|
||||
if config is not None:
|
||||
# [TODO] Hack to pass in config - need to remove later
|
||||
base = config.rope_theta
|
||||
base = _get_rope_theta(config, default = base)
|
||||
partial_rotary_factor = (
|
||||
config.partial_rotary_factor
|
||||
if hasattr(config, "partial_rotary_factor")
|
||||
|
|
|
|||
|
|
@ -867,6 +867,11 @@ def LlamaModel_fast_forward(
|
|||
input_ids = input_ids[:, : self.max_seq_length]
|
||||
elif inputs_embeds is not None:
|
||||
inputs_embeds = inputs_embeds[:, : self.max_seq_length, :]
|
||||
if (
|
||||
attention_mask is not None
|
||||
and attention_mask.shape[-1] > self.max_seq_length
|
||||
):
|
||||
attention_mask = attention_mask[:, : self.max_seq_length]
|
||||
|
||||
past_key_values_length = 0
|
||||
|
||||
|
|
@ -1582,6 +1587,18 @@ def PeftModel_fast_forward(
|
|||
)
|
||||
|
||||
|
||||
def _get_rope_theta(config, default = 10000.0):
|
||||
"""Get rope_theta from config, handling both transformers 4.x and 5.x."""
|
||||
try:
|
||||
return config.rope_theta
|
||||
except (AttributeError, KeyError):
|
||||
pass
|
||||
rp = getattr(config, "rope_parameters", None)
|
||||
if isinstance(rp, dict):
|
||||
return rp.get("rope_theta", default)
|
||||
return default
|
||||
|
||||
|
||||
# Solves https://github.com/unslothai/unsloth/issues/168
|
||||
# Static KV Cache was introduced in 4.38.0, causing training to be much slower.
|
||||
# Inference can now be CUDAGraphed, but we shall retain the old rotary embeddings.
|
||||
|
|
@ -1602,11 +1619,7 @@ class LlamaRotaryEmbedding(torch.nn.Module):
|
|||
super().__init__()
|
||||
if config is not None:
|
||||
# [TODO] Hack to pass in config - need to remove later
|
||||
try:
|
||||
base = config.rope_theta
|
||||
except:
|
||||
base = getattr(config, "rope_parameters", {})
|
||||
base = base["rope_theta"]
|
||||
base = _get_rope_theta(config, default = base)
|
||||
partial_rotary_factor = (
|
||||
config.partial_rotary_factor
|
||||
if hasattr(config, "partial_rotary_factor")
|
||||
|
|
@ -1757,7 +1770,7 @@ class LlamaExtendedRotaryEmbedding(torch.nn.Module):
|
|||
super().__init__()
|
||||
if config is not None:
|
||||
# [TODO] Hack to pass in config - need to remove later
|
||||
base = config.rope_theta
|
||||
base = _get_rope_theta(config, default = base)
|
||||
partial_rotary_factor = (
|
||||
config.partial_rotary_factor
|
||||
if hasattr(config, "partial_rotary_factor")
|
||||
|
|
@ -1893,7 +1906,7 @@ class LongRopeRotaryEmbedding(torch.nn.Module):
|
|||
|
||||
if config is not None:
|
||||
# [TODO] Hack to pass in config - need to remove later
|
||||
base = config.rope_theta
|
||||
base = _get_rope_theta(config, default = base)
|
||||
partial_rotary_factor = (
|
||||
config.partial_rotary_factor
|
||||
if hasattr(config, "partial_rotary_factor")
|
||||
|
|
@ -2056,12 +2069,16 @@ def unsloth_fast_generate(
|
|||
and kwargs["input_ids"] is not None
|
||||
and "max_new_tokens" in kwargs
|
||||
):
|
||||
if (
|
||||
kwargs["input_ids"].shape[-1] + kwargs["max_new_tokens"]
|
||||
_ids = kwargs["input_ids"]
|
||||
# Handle BatchEncoding from transformers 5.0+ (no .shape attribute)
|
||||
if hasattr(_ids, "input_ids"):
|
||||
_ids = _ids["input_ids"]
|
||||
if hasattr(_ids, "shape") and (
|
||||
_ids.shape[-1] + kwargs["max_new_tokens"]
|
||||
> self.config.max_position_embeddings
|
||||
):
|
||||
raise ValueError(
|
||||
f"Unsloth: input length {kwargs['input_ids'].shape[-1]} + max_new_tokens {kwargs['max_new_tokens']} exceeds the maximum sequence length of {self.config.max_position_embeddings}!\n"
|
||||
f"Unsloth: input length {_ids.shape[-1]} + max_new_tokens {kwargs['max_new_tokens']} exceeds the maximum sequence length of {self.config.max_position_embeddings}!\n"
|
||||
"You will need to do long context extension by increasing the `max_seq_length` in `FastLanguageModel.from_pretrained`."
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -435,6 +435,7 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
|
|||
for x in dir(trainer)
|
||||
if x.endswith("Trainer")
|
||||
and x != "Trainer"
|
||||
and not x.startswith("_")
|
||||
and trainer_file.split("_")[0] in x.lower()
|
||||
]
|
||||
config = [
|
||||
|
|
@ -442,6 +443,7 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
|
|||
for x in dir(trainer)
|
||||
if x.endswith("Config")
|
||||
and x != "Config"
|
||||
and not x.startswith("_")
|
||||
and trainer_file.split("_")[0] in x.lower()
|
||||
]
|
||||
if len(name) != 1:
|
||||
|
|
@ -449,6 +451,21 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
|
|||
f"Unsloth: Could not find Trainer class in trl.trainer.{trainer_file}. Found: {name}"
|
||||
)
|
||||
return
|
||||
if len(config) != 1:
|
||||
# TRL 0.26+: Config may be in a separate *_config.py module
|
||||
config_module_name = trainer_file.replace("_trainer", "_config")
|
||||
try:
|
||||
config_mod = eval(f"trl.trainer.{config_module_name}")
|
||||
config = [
|
||||
x
|
||||
for x in dir(config_mod)
|
||||
if x.endswith("Config")
|
||||
and x != "Config"
|
||||
and not x.startswith("_")
|
||||
and trainer_file.split("_")[0] in x.lower()
|
||||
]
|
||||
except Exception:
|
||||
pass
|
||||
if len(config) != 1:
|
||||
logger.info(
|
||||
f"Unsloth: Could not find Config class in trl.trainer.{trainer_file}. Found: {config}"
|
||||
|
|
@ -467,11 +484,14 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
|
|||
return
|
||||
try:
|
||||
RLConfig = eval(f"trl.trainer.{trainer_file}.{RLConfig_name}")
|
||||
except Exception as e:
|
||||
logger.info(
|
||||
f"Unsloth: Could not load {RLConfig_name} from trl.trainer.{trainer_file}: {e}"
|
||||
)
|
||||
return
|
||||
except Exception:
|
||||
# TRL 0.26+: Config may be in a separate *_config.py module
|
||||
try:
|
||||
config_module_name = trainer_file.replace("_trainer", "_config")
|
||||
RLConfig = eval(f"trl.trainer.{config_module_name}.{RLConfig_name}")
|
||||
except Exception as e:
|
||||
logger.info(f"Unsloth: Could not load {RLConfig_name}: {e}")
|
||||
return
|
||||
|
||||
# Check name
|
||||
if RLTrainer.__name__.startswith("Unsloth"):
|
||||
|
|
@ -481,11 +501,49 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
|
|||
print(f"Unsloth: {RLConfig.__name__} is already patched.")
|
||||
return
|
||||
|
||||
# TRL 0.26+: Resolve thin wrappers to their experimental parent class.
|
||||
# Thin wrappers are deprecation shims that contain "trl.experimental" in
|
||||
# their source and just forward *args/**kwargs to the real implementation.
|
||||
_trainer_resolved_module = None
|
||||
try:
|
||||
_trainer_src = inspect.getsource(RLTrainer)
|
||||
if "trl.experimental" in _trainer_src:
|
||||
for _parent in RLTrainer.__mro__[1:]:
|
||||
if _parent is object:
|
||||
continue
|
||||
try:
|
||||
if "trl.experimental" not in inspect.getsource(_parent):
|
||||
RLTrainer = _parent
|
||||
_trainer_resolved_module = inspect.getmodule(_parent)
|
||||
break
|
||||
except Exception:
|
||||
continue
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
_config_src = inspect.getsource(RLConfig)
|
||||
if "trl.experimental" in _config_src:
|
||||
for _parent in RLConfig.__mro__[1:]:
|
||||
if _parent is object:
|
||||
continue
|
||||
try:
|
||||
if "trl.experimental" not in inspect.getsource(_parent):
|
||||
RLConfig = _parent
|
||||
break
|
||||
except Exception:
|
||||
continue
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Get old source
|
||||
old_RLTrainer_source = inspect.getsource(RLTrainer)
|
||||
old_RLConfig_source = inspect.getsource(RLConfig)
|
||||
|
||||
all_imports = dir(trainer)
|
||||
if _trainer_resolved_module is not None:
|
||||
all_imports = dir(_trainer_resolved_module)
|
||||
else:
|
||||
all_imports = dir(trainer)
|
||||
# Fix _deprecate_arguments not getting imported so stop __ but not _
|
||||
imports = [x for x in all_imports if not x.startswith("__")]
|
||||
|
||||
|
|
@ -1191,13 +1249,32 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
|
|||
new_text = 'self._signature_columns = ["input_ids", "attention_mask", "completion_mask","labels"]'
|
||||
RLTrainer_source = RLTrainer_source.replace(original_text, new_text)
|
||||
|
||||
# Temporary patch _is_vlm to False
|
||||
# as of 0.22 it only exists in sfttrainer
|
||||
original_is_vlm_text = "self._is_vlm = True"
|
||||
new_is_vlm_text = "self._is_vlm = False"
|
||||
RLTrainer_source = RLTrainer_source.replace(
|
||||
original_is_vlm_text, new_is_vlm_text
|
||||
# Do NOT override _is_vlm -- let TRL detect VLM models naturally.
|
||||
# In TRL 0.27.1+, forcing _is_vlm=False causes a ValueError when
|
||||
# vision datasets are used with VLM models.
|
||||
#
|
||||
# However, some notebooks pass a bare tokenizer (processor.tokenizer) as
|
||||
# processing_class. TRL then sets _is_vlm=False even for VLM models.
|
||||
# Add a model-architecture-based override before the validation check.
|
||||
_vlm_check_original = (
|
||||
' self._is_vision_dataset = "image" in dataset_sample or "images" in dataset_sample\n'
|
||||
" if self._is_vision_dataset and not self._is_vlm:"
|
||||
)
|
||||
_vlm_check_patched = (
|
||||
' self._is_vision_dataset = "image" in dataset_sample or "images" in dataset_sample\n'
|
||||
" # Unsloth: override _is_vlm for VLM models that pass a bare tokenizer\n"
|
||||
" if not self._is_vlm and self._is_vision_dataset:\n"
|
||||
" _m = model\n"
|
||||
' if hasattr(_m, "model"): _m = _m.model\n'
|
||||
' if hasattr(getattr(_m, "config", None), "vision_config") or \\\n'
|
||||
' _m.__class__.__name__.endswith("ForConditionalGeneration"):\n'
|
||||
" self._is_vlm = True\n"
|
||||
" if self._is_vision_dataset and not self._is_vlm:"
|
||||
)
|
||||
if _vlm_check_original in RLTrainer_source:
|
||||
RLTrainer_source = RLTrainer_source.replace(
|
||||
_vlm_check_original, _vlm_check_patched
|
||||
)
|
||||
|
||||
# Remove multiple doc strings
|
||||
if __RLConfig_doc__ != "" and RLTrainer_source.count(__RLTrainer_doc__) == 2:
|
||||
|
|
@ -1207,10 +1284,15 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
|
|||
RLTrainer_source = re.sub(r"[\n]{3,}", "\n", RLTrainer_source)
|
||||
|
||||
# Create new function
|
||||
_model_location = (
|
||||
_trainer_resolved_module.__name__
|
||||
if _trainer_resolved_module is not None
|
||||
else f"trl.trainer.{trainer_file}"
|
||||
)
|
||||
created_module = create_new_function(
|
||||
f"Unsloth{RLTrainer_name}",
|
||||
RLTrainer_source,
|
||||
f"trl.trainer.{trainer_file}",
|
||||
_model_location,
|
||||
imports,
|
||||
overwrite = False,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -217,6 +217,19 @@ def sft_trainer_compute_loss(function_name, function):
|
|||
RL_FUNCTIONS["sft_trainer"].append(sft_trainer_compute_loss)
|
||||
|
||||
|
||||
# Fix bare pop("push_to_hub_token") in compiled SFT/IterativeSFT trainer __init__
|
||||
# On transformers 5.0+, to_dict() no longer includes push_to_hub_token, so bare pop KeyErrors
|
||||
def sft_trainer_push_to_hub_token(function_name, function):
|
||||
if function_name != "__init__":
|
||||
return function
|
||||
return function.replace(
|
||||
'dict_args.pop("push_to_hub_token")', 'dict_args.pop("push_to_hub_token", None)'
|
||||
)
|
||||
|
||||
|
||||
RL_FUNCTIONS["sft_trainer"].append(sft_trainer_push_to_hub_token)
|
||||
|
||||
|
||||
# Autocast precision for GRPO
|
||||
def grpo_trainer__prepare_inputs(function_name, function):
|
||||
if function_name != "_prepare_inputs":
|
||||
|
|
@ -1193,6 +1206,29 @@ def grpo_trainer_compute_loss(function_name, function):
|
|||
RL_FUNCTIONS["grpo_trainer"].append(grpo_trainer_compute_loss)
|
||||
|
||||
|
||||
# Fix KTO shape mismatch when Unsloth model forward truncates input_ids
|
||||
# but labels aren't truncated. TRL 0.27.2+ _process_tokens only truncates
|
||||
# completions, not prompts -- so prompts exceeding max_seq_length cause the
|
||||
# model to produce shorter logits than the labels expect.
|
||||
def kto_trainer_get_batch_logps(function_name, function):
|
||||
if function_name != "get_batch_logps":
|
||||
return function
|
||||
# The raise is inside an if block inside the method, so we need
|
||||
# to preserve the exact indentation of the raise statement.
|
||||
old = 'raise ValueError("Logits (batch and sequence length dim) and labels must have the same shape.")'
|
||||
new = (
|
||||
"# Unsloth: auto-truncate to shorter sequence length (model may have truncated input_ids)\n"
|
||||
" _min_len = min(logits.shape[1], labels.shape[1])\n"
|
||||
" logits = logits[:, :_min_len, :]\n"
|
||||
" labels = labels[:, :_min_len]"
|
||||
)
|
||||
function = function.replace(old, new)
|
||||
return function
|
||||
|
||||
|
||||
RL_FUNCTIONS["kto_trainer"].append(kto_trainer_get_batch_logps)
|
||||
|
||||
|
||||
# https://github.com/huggingface/trl/blob/main/trl/trainer/grpo_trainer.py#L356
|
||||
# TRL warns if batch size is not a multiple of num_generations -> fix this.
|
||||
def grpo_trainer_fix_batch_size(RLTrainer_source, RLConfig_source):
|
||||
|
|
@ -1267,7 +1303,7 @@ def openenv_vllm_reload_weights():
|
|||
try:
|
||||
import trl.experimental.openenv.utils as openenv_utils
|
||||
import trl.experimental.openenv as openenv
|
||||
except ImportError as e:
|
||||
except (ImportError, NameError, Exception) as e:
|
||||
logger.info(f"Unsloth: Failed to import trl openenv: {e}")
|
||||
logger.info(
|
||||
"Unsloth: trl.experimental.openenv not available — skipping RL openenv patches."
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@
|
|||
|
||||
import logging
|
||||
|
||||
from .loader import FastModel
|
||||
from .loader import FastModel, DISABLE_SDPA_MODEL_NAMES
|
||||
from ._utils import SUPPORTS_BFLOAT16
|
||||
import inspect
|
||||
import json
|
||||
|
|
@ -1461,8 +1461,17 @@ class FastSentenceTransformer(FastModel):
|
|||
model_kwargs = {"torch_dtype": dtype}
|
||||
|
||||
# Enable SDPA if supported (1.2x extra speedup on top of torch.compile)
|
||||
# But disable for models with known SDPA + torch.compile backward issues
|
||||
_force_eager = False
|
||||
for _sdpa_model in DISABLE_SDPA_MODEL_NAMES:
|
||||
if _sdpa_model in model_type.lower():
|
||||
supports_sdpa = False
|
||||
_force_eager = True
|
||||
break
|
||||
if supports_sdpa:
|
||||
model_kwargs["attn_implementation"] = "sdpa"
|
||||
elif _force_eager:
|
||||
model_kwargs["attn_implementation"] = "eager"
|
||||
|
||||
# Print optimization status
|
||||
sdpa_str = " + SDPA" if supports_sdpa else ""
|
||||
|
|
|
|||
|
|
@ -317,6 +317,91 @@ def unsloth_base_fast_generate(
|
|||
return output
|
||||
|
||||
|
||||
def _construct_vlm_processor_fallback(
|
||||
tokenizer_name, model_type, token, trust_remote_code
|
||||
):
|
||||
"""Construct a VLM processor manually when AutoProcessor.from_pretrained fails.
|
||||
|
||||
Some VLMs (e.g., LFM2.5-VL) have tokenizer_class entries that AutoTokenizer
|
||||
cannot resolve. This function loads the image processor and tokenizer separately,
|
||||
sets required special token attributes, and constructs the processor.
|
||||
"""
|
||||
try:
|
||||
from transformers import AutoImageProcessor, PreTrainedTokenizerFast, AutoConfig
|
||||
from transformers.models.auto.processing_auto import PROCESSOR_MAPPING_NAMES
|
||||
import json
|
||||
|
||||
# Load image processor
|
||||
image_processor = AutoImageProcessor.from_pretrained(
|
||||
tokenizer_name,
|
||||
token = token,
|
||||
trust_remote_code = trust_remote_code,
|
||||
)
|
||||
# Load tokenizer via PreTrainedTokenizerFast (bypasses tokenizer_class check)
|
||||
tok = PreTrainedTokenizerFast.from_pretrained(
|
||||
tokenizer_name,
|
||||
padding_side = "left",
|
||||
token = token,
|
||||
trust_remote_code = trust_remote_code,
|
||||
)
|
||||
# Read tokenizer_config.json for model-specific special tokens
|
||||
try:
|
||||
from huggingface_hub import hf_hub_download
|
||||
|
||||
config_path = hf_hub_download(
|
||||
tokenizer_name, "tokenizer_config.json", token = token
|
||||
)
|
||||
with open(config_path, "r", encoding = "utf-8") as f:
|
||||
tok_config = json.load(f)
|
||||
# Set model-specific special tokens and their IDs
|
||||
for key in (
|
||||
"image_token",
|
||||
"image_start_token",
|
||||
"image_end_token",
|
||||
"image_thumbnail",
|
||||
"video_token",
|
||||
):
|
||||
if key in tok_config and not hasattr(tok, key):
|
||||
setattr(tok, key, tok_config[key])
|
||||
id_key = key + "_id" if not key.endswith("_id") else key
|
||||
token_id = tok.convert_tokens_to_ids(tok_config[key])
|
||||
if not hasattr(tok, id_key):
|
||||
setattr(tok, id_key, token_id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Find the processor class - try model_type first, then top-level config model_type
|
||||
proc_class_name = PROCESSOR_MAPPING_NAMES.get(model_type)
|
||||
if proc_class_name is None:
|
||||
# model_type might be a sub-model type (e.g. "lfm2" instead of "lfm2_vl").
|
||||
# Try the top-level config.model_type which often has the processor mapping.
|
||||
try:
|
||||
config = AutoConfig.from_pretrained(
|
||||
tokenizer_name,
|
||||
token = token,
|
||||
trust_remote_code = trust_remote_code,
|
||||
)
|
||||
proc_class_name = PROCESSOR_MAPPING_NAMES.get(config.model_type)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if proc_class_name is not None:
|
||||
import transformers
|
||||
|
||||
proc_class = getattr(transformers, proc_class_name, None)
|
||||
if proc_class is not None:
|
||||
processor = proc_class(image_processor = image_processor, tokenizer = tok)
|
||||
# Copy chat_template from tokenizer to processor if needed
|
||||
if not getattr(processor, "chat_template", None) and getattr(
|
||||
tok, "chat_template", None
|
||||
):
|
||||
processor.chat_template = tok.chat_template
|
||||
return processor
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
class FastBaseModel:
|
||||
@staticmethod
|
||||
def from_pretrained(
|
||||
|
|
@ -826,14 +911,17 @@ class FastBaseModel:
|
|||
if (whisper_language and whisper_task) or auto_model.__name__.endswith(
|
||||
"ForConditionalGeneration"
|
||||
):
|
||||
tokenizer = auto_processor.from_pretrained(
|
||||
tokenizer_name,
|
||||
padding_side = "left",
|
||||
token = token,
|
||||
language = whisper_language,
|
||||
task = whisper_task,
|
||||
trust_remote_code = trust_remote_code,
|
||||
)
|
||||
try:
|
||||
tokenizer = auto_processor.from_pretrained(
|
||||
tokenizer_name,
|
||||
padding_side = "left",
|
||||
token = token,
|
||||
language = whisper_language,
|
||||
task = whisper_task,
|
||||
trust_remote_code = trust_remote_code,
|
||||
)
|
||||
except Exception:
|
||||
tokenizer = None
|
||||
else:
|
||||
try:
|
||||
tokenizer = auto_processor.from_pretrained(
|
||||
|
|
@ -849,6 +937,23 @@ class FastBaseModel:
|
|||
token = token,
|
||||
trust_remote_code = trust_remote_code,
|
||||
)
|
||||
|
||||
# If processor loading failed (e.g., tokenizer class not found),
|
||||
# try constructing the processor manually from separate components.
|
||||
if tokenizer is None and is_vlm:
|
||||
tokenizer = _construct_vlm_processor_fallback(
|
||||
tokenizer_name,
|
||||
model_type_arch,
|
||||
token,
|
||||
trust_remote_code,
|
||||
)
|
||||
if tokenizer is None:
|
||||
import sys
|
||||
|
||||
print(
|
||||
f"Unsloth: Warning - VLM processor fallback returned None for model_type={model_type_arch}",
|
||||
file = sys.stderr,
|
||||
)
|
||||
if hasattr(tokenizer, "tokenizer"):
|
||||
__tokenizer = tokenizer.tokenizer
|
||||
# Add padding side as well
|
||||
|
|
@ -872,7 +977,29 @@ class FastBaseModel:
|
|||
do_forced_float32 = do_forced_float32,
|
||||
correct_dtype = correct_dtype,
|
||||
)
|
||||
model, tokenizer = patch_tokenizer(model, tokenizer)
|
||||
try:
|
||||
model, tokenizer = patch_tokenizer(model, tokenizer)
|
||||
except Exception as _patch_err:
|
||||
# Some VLM processors (e.g., ERNIE VL) may fail during tokenizer patching.
|
||||
# Try loading tokenizer separately via AutoTokenizer as fallback.
|
||||
try:
|
||||
from transformers import AutoTokenizer as _AutoTokenizer
|
||||
|
||||
_fallback_tok = _AutoTokenizer.from_pretrained(
|
||||
tokenizer_name,
|
||||
padding_side = "left",
|
||||
token = token,
|
||||
trust_remote_code = trust_remote_code,
|
||||
)
|
||||
model, _fallback_tok = patch_tokenizer(model, _fallback_tok)
|
||||
# Re-attach as processor wrapper if original was a processor
|
||||
if hasattr(tokenizer, "image_processor"):
|
||||
tokenizer.tokenizer = _fallback_tok
|
||||
else:
|
||||
tokenizer = _fallback_tok
|
||||
except Exception:
|
||||
# If fallback also fails, raise the original error
|
||||
raise _patch_err
|
||||
model = post_patch_loss_function(model)
|
||||
|
||||
# Log Unsloth version for future fastpaths for inference
|
||||
|
|
@ -880,10 +1007,31 @@ class FastBaseModel:
|
|||
model.config.update({"unsloth_version": __version__})
|
||||
patch_saving_functions(model, vision = True)
|
||||
if tokenizer is None:
|
||||
del model
|
||||
raise RuntimeError(
|
||||
"Unsloth: The tokenizer is weirdly not loaded? Please check if there is one."
|
||||
)
|
||||
# Last resort: try loading tokenizer via AutoTokenizer, then PreTrainedTokenizerFast
|
||||
try:
|
||||
from transformers import AutoTokenizer as _AutoTokenizer
|
||||
|
||||
tokenizer = _AutoTokenizer.from_pretrained(
|
||||
tokenizer_name,
|
||||
padding_side = "left",
|
||||
token = token,
|
||||
trust_remote_code = trust_remote_code,
|
||||
)
|
||||
except Exception:
|
||||
try:
|
||||
from transformers import PreTrainedTokenizerFast
|
||||
|
||||
tokenizer = PreTrainedTokenizerFast.from_pretrained(
|
||||
tokenizer_name,
|
||||
padding_side = "left",
|
||||
token = token,
|
||||
trust_remote_code = trust_remote_code,
|
||||
)
|
||||
except Exception:
|
||||
del model
|
||||
raise RuntimeError(
|
||||
"Unsloth: The tokenizer is weirdly not loaded? Please check if there is one."
|
||||
)
|
||||
patch_saving_functions(tokenizer, vision = True)
|
||||
|
||||
# Fix gradient accumulation
|
||||
|
|
|
|||
|
|
@ -1021,7 +1021,10 @@ def patch_sft_trainer_tokenizer():
|
|||
"kto_trainer.KTOTrainer",
|
||||
):
|
||||
function_name, replacer = "train", "if resume_from_checkpoint is False:"
|
||||
function = getsource(eval(f"trl.trainer.{path_to_trainer}.{function_name}"))
|
||||
try:
|
||||
function = getsource(eval(f"trl.trainer.{path_to_trainer}.{function_name}"))
|
||||
except Exception:
|
||||
continue
|
||||
where = function.find("def")
|
||||
function = function.split("\n")
|
||||
function = "\n".join(x[where:] for x in function)
|
||||
|
|
|
|||
|
|
@ -200,13 +200,57 @@ class UnslothTrainer(SFTTrainer):
|
|||
|
||||
# From `trl>=0.13.0`, they changed how to pass several params to the trainer
|
||||
# We need to patch to make the transition smooth
|
||||
def _resolve_trainer_params(trainer_class, init_fn):
|
||||
"""Resolve the real named parameters for a trainer __init__.
|
||||
|
||||
Some TRL trainers (e.g., ORPOTrainer in TRL 0.27.1) are thin wrappers
|
||||
with only ``def __init__(self, *args, **kwargs)``. For those, walk the
|
||||
MRO and return the first parent class that has real named parameters.
|
||||
"""
|
||||
params = inspect.signature(init_fn).parameters
|
||||
named = {
|
||||
k
|
||||
for k, v in params.items()
|
||||
if v.kind
|
||||
in (inspect.Parameter.POSITIONAL_OR_KEYWORD, inspect.Parameter.KEYWORD_ONLY)
|
||||
and k != "self"
|
||||
}
|
||||
if named:
|
||||
return set(params.keys())
|
||||
|
||||
# Thin wrapper detected - walk MRO for real signature
|
||||
for cls in trainer_class.__mro__[1:]:
|
||||
if cls is object:
|
||||
continue
|
||||
parent_init = cls.__dict__.get("__init__")
|
||||
if parent_init is None:
|
||||
continue
|
||||
try:
|
||||
parent_params = inspect.signature(parent_init).parameters
|
||||
parent_named = {
|
||||
k
|
||||
for k, v in parent_params.items()
|
||||
if v.kind
|
||||
in (
|
||||
inspect.Parameter.POSITIONAL_OR_KEYWORD,
|
||||
inspect.Parameter.KEYWORD_ONLY,
|
||||
)
|
||||
and k != "self"
|
||||
}
|
||||
if parent_named:
|
||||
return set(parent_params.keys())
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
return set(params.keys())
|
||||
|
||||
|
||||
def _backwards_compatible_trainer(trainer_class, config_class):
|
||||
original_init = trainer_class.__init__
|
||||
|
||||
@wraps(original_init)
|
||||
def new_init(self, *args, **kwargs):
|
||||
# All Trainer tokenizer are now called processing_class
|
||||
trainer_params = set(inspect.signature(original_init).parameters.keys())
|
||||
trainer_params = _resolve_trainer_params(trainer_class, original_init)
|
||||
|
||||
if "processing_class" in trainer_params and "tokenizer" in kwargs:
|
||||
kwargs["processing_class"] = kwargs.pop("tokenizer")
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue