From ba7366be533fbf1b873e20d414e9e26776c70561 Mon Sep 17 00:00:00 2001 From: Daniel Han Date: Mon, 9 Feb 2026 05:11:50 -0800 Subject: [PATCH] Fix notebook compatibility for transformers 4.57.6 and TRL 0.22-0.27 (#3998) * Patch before compile? * Fix notebook compatibility for transformers 4.57.6 and TRL 0.22-0.27 Fixes several notebook failures discovered during testing all 125 notebooks with transformers==4.57.6 + tRL 0.22.2 and TRL 0.27.1. Warning suppression (import_fixes.py): - Suppress torch 2.9+ pin_memory/is_pinned device deprecation warnings - Suppress cuda.cudart/cuda.nvrtc module deprecation FutureWarning - Filter vllm "Level is deprecated" stderr noise - Filter PydanticSerializationUnexpectedValue warnings - Filter Triton "df: No such file" stderr noise VLM tokenizer loading (vision.py): - Add _construct_vlm_processor_fallback() for models where AutoProcessor.from_pretrained fails (e.g., ERNIE 4.5 VL, LFM2.5-VL) - Wrap processor loading in try/except with fallback to manual construction from separate image_processor + tokenizer components - Add fallback to AutoTokenizer/PreTrainedTokenizerFast when tokenizer loading or patching fails TRL 0.27.1 trainer compatibility (trainer.py): - Add _resolve_trainer_params() to handle thin wrapper trainers that only have def __init__(self, *args, **kwargs) (e.g., ORPOTrainer in TRL 0.27.1) by walking MRO for real parameter signature VLM _is_vlm detection (rl.py): - Replace blanket _is_vlm=False override with model-architecture-based detection that checks vision_config or ForConditionalGeneration class name, fixing VLM training when bare tokenizer is passed as processing_class ModernBERT SDPA compatibility (loader.py, sentence_transformer.py): - Add "modernbert" to DISABLE_SDPA_MODEL_NAMES to avoid stride alignment issues with torch.compile backward pass - Add DISABLE_SDPA check for sentence transformer models Other fixes (_utils.py): - Suppress false uninitialized weight warnings for VLM multi_modal_projector.layer_norm Tested: 92/125 notebooks pass with TRL 0.22.2, 94/125 with TRL 0.27.1. Remaining failures are infra (missing FFmpeg, network timeouts, GPU arch) not code bugs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix KTO shape mismatch on TRL 0.27.2+ and truncation alignment - Patch KTO get_batch_logps to auto-align logits and labels when Unsloth model forward truncates input_ids beyond max_seq_length. TRL 0.27.2 changed _process_tokens to only truncate completions (not prompts), so sequences with long prompts exceed max_seq_length and trigger model-side truncation. The original ValueError is replaced with min-length alignment. - Also truncate attention_mask in LlamaModel forward when input_ids are truncated to max_seq_length, preventing shape mismatches in attention. - Widen except clause in rl_replacements.py openenv import from `except ImportError` to `except (ImportError, NameError, Exception)` to handle vllm SamplingParams NameError in TRL 0.27.2. * Fix TRL 0.26+ thin wrapper resolution, enable ModernBERT SDPA, clean up warning filters TRL 0.26+ thin wrapper resolution (rl.py): - Filter _-prefixed private imports when discovering Trainer/Config classes - Look up Config in separate *_config.py module when not found in trainer module - Detect thin wrappers (<1000 chars source) and resolve to experimental parent via MRO walk; use resolved module for imports and create_new_function - Enables all 15 trainers to patch successfully (was 5/15 before) ModernBERT SDPA (loader.py): - Remove "modernbert" from DISABLE_SDPA_MODEL_NAMES - SDPA works correctly for both classification and sentence transformers - Verified: 88.9% accuracy on emotion classification, correct domain-specific embeddings after sentence transformer fine-tuning Warning filter cleanup (import_fixes.py): - Remove cuda.cudart/cuda.nvrtc FutureWarning filters (no such warnings exist in torch 2.9.1+; proactive suppression is unnecessary) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove multi_modal_projector.layer_norm from uninitialized weight guard The LFM2.5-VL projector LayerNorm is properly initialized by transformers and does not need to be excluded from the uninitialized weight check. The original exclusion was added as a workaround but is no longer needed after the upstream fix. * Add transformers 5.0 compat: rope_theta helper, config-as-dim detection, BatchEncoding guard, try/except for TRL trainer source, push_to_hub_token compiler fix - llama.py: Add _get_rope_theta() helper handling both config.rope_theta and rope_parameters dict - llama.py: Handle BatchEncoding in unsloth_fast_generate (transformers 5.0+ returns BatchEncoding from apply_chat_template) - gemma.py: Detect config passed as dim arg in GemmaFixedRotaryEmbedding - tokenizer_utils.py: Add try/except for TRL trainer getsource in patch_sft_trainer_tokenizer - rl_replacements.py: Add compiler fix replacing bare pop("push_to_hub_token") with pop(..., None) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Use trl.experimental string check instead of char-count heuristic for thin wrapper detection The <1000 / >1000 char threshold was fragile -- XPOConfig's parent is only 994 chars and would be skipped. All thin wrappers in TRL 0.26+ contain "trl.experimental" in their deprecation warning, while no real trainer or config class does, making it a reliable detection marker. * Move DISABLE_SDPA_MODEL_NAMES import to module level in sentence_transformer The function-level import was redundant since loader.py is already imported at module level. Move it to the existing loader import line. --------- Co-authored-by: Datta Nimmaturi Co-authored-by: Daniel Hanchen Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- unsloth/import_fixes.py | 33 +++++ unsloth/models/_utils.py | 6 + unsloth/models/gemma.py | 11 +- unsloth/models/llama.py | 37 ++++-- unsloth/models/rl.py | 108 +++++++++++++-- unsloth/models/rl_replacements.py | 38 +++++- unsloth/models/sentence_transformer.py | 11 +- unsloth/models/vision.py | 174 +++++++++++++++++++++++-- unsloth/tokenizer_utils.py | 5 +- unsloth/trainer.py | 46 ++++++- 10 files changed, 428 insertions(+), 41 deletions(-) diff --git a/unsloth/import_fixes.py b/unsloth/import_fixes.py index cd8875b5bf..4a2211f9a0 100644 --- a/unsloth/import_fixes.py +++ b/unsloth/import_fixes.py @@ -164,6 +164,39 @@ if os.environ.get("UNSLOTH_ENABLE_LOGGING", "0") != "1": "ignore", message = r"unclosed file.*dev/null", category = ResourceWarning ) + # torch 2.9+ pin_memory/is_pinned device arg deprecation + warnings.filterwarnings( + "ignore", + message = r"The `device` argument is deprecated", + category = DeprecationWarning, + ) + warnings.filterwarnings( + "ignore", + message = r".*pin_memory.*device.*deprecated", + category = DeprecationWarning, + ) + warnings.filterwarnings( + "ignore", + message = r".*is_pinned.*device.*deprecated", + category = DeprecationWarning, + ) + + # vllm "Level is deprecated" stderr noise + sys.stderr.add_filter("Level is deprecated") + + # PydanticSerializationUnexpectedValue warning + warnings.filterwarnings( + "ignore", + message = r".*PydanticSerializationUnexpectedValue", + ) + warnings.filterwarnings( + "ignore", + message = r"Expected.*but got.*with value.*is not.*subclass", + ) + + # Triton "df: No such file or directory" stderr noise + sys.stderr.add_filter("df: No such file") + # Fix up AttributeError: 'MessageFactory' object has no attribute 'GetPrototype' # MUST do this at the start primarily due to tensorflow causing issues diff --git a/unsloth/models/_utils.py b/unsloth/models/_utils.py index 48e5683076..b01cb4113b 100644 --- a/unsloth/models/_utils.py +++ b/unsloth/models/_utils.py @@ -1966,6 +1966,12 @@ def unsloth_compile_transformers( return model_types, False supports_sdpa = [True] + + # Run patches BEFORE compiler so class replacements (e.g. GptOssTopKRouter, + # GptOssExperts) are in place before the compiler caches references to them. + for temporary_patch in TEMPORARY_PATCHES: + temporary_patch() + for model_type in model_types: _unsloth_compile_transformers( model_type, diff --git a/unsloth/models/gemma.py b/unsloth/models/gemma.py index 1789a9cd92..7173c03495 100644 --- a/unsloth/models/gemma.py +++ b/unsloth/models/gemma.py @@ -13,6 +13,7 @@ # limitations under the License. from .llama import * +from .llama import _get_rope_theta from ._utils import __version__ from unsloth_zoo.utils import _get_dtype, Version from unsloth_zoo.hf_utils import dtype_from_config @@ -256,9 +257,17 @@ class GemmaFixedRotaryEmbedding(torch.nn.Module): config = None, # [TODO] Hack to pass in config - need to remove later ): super().__init__() + # In transformers 5.0+, RotaryEmbedding(config) passes config as first positional arg (dim) + if ( + config is None + and dim is not None + and hasattr(dim, "max_position_embeddings") + ): + config = dim + dim = None if config is not None: # [TODO] Hack to pass in config - need to remove later - base = config.rope_theta + base = _get_rope_theta(config, default = base) partial_rotary_factor = ( config.partial_rotary_factor if hasattr(config, "partial_rotary_factor") diff --git a/unsloth/models/llama.py b/unsloth/models/llama.py index 1b831ecb26..61771e4567 100644 --- a/unsloth/models/llama.py +++ b/unsloth/models/llama.py @@ -867,6 +867,11 @@ def LlamaModel_fast_forward( input_ids = input_ids[:, : self.max_seq_length] elif inputs_embeds is not None: inputs_embeds = inputs_embeds[:, : self.max_seq_length, :] + if ( + attention_mask is not None + and attention_mask.shape[-1] > self.max_seq_length + ): + attention_mask = attention_mask[:, : self.max_seq_length] past_key_values_length = 0 @@ -1582,6 +1587,18 @@ def PeftModel_fast_forward( ) +def _get_rope_theta(config, default = 10000.0): + """Get rope_theta from config, handling both transformers 4.x and 5.x.""" + try: + return config.rope_theta + except (AttributeError, KeyError): + pass + rp = getattr(config, "rope_parameters", None) + if isinstance(rp, dict): + return rp.get("rope_theta", default) + return default + + # Solves https://github.com/unslothai/unsloth/issues/168 # Static KV Cache was introduced in 4.38.0, causing training to be much slower. # Inference can now be CUDAGraphed, but we shall retain the old rotary embeddings. @@ -1602,11 +1619,7 @@ class LlamaRotaryEmbedding(torch.nn.Module): super().__init__() if config is not None: # [TODO] Hack to pass in config - need to remove later - try: - base = config.rope_theta - except: - base = getattr(config, "rope_parameters", {}) - base = base["rope_theta"] + base = _get_rope_theta(config, default = base) partial_rotary_factor = ( config.partial_rotary_factor if hasattr(config, "partial_rotary_factor") @@ -1757,7 +1770,7 @@ class LlamaExtendedRotaryEmbedding(torch.nn.Module): super().__init__() if config is not None: # [TODO] Hack to pass in config - need to remove later - base = config.rope_theta + base = _get_rope_theta(config, default = base) partial_rotary_factor = ( config.partial_rotary_factor if hasattr(config, "partial_rotary_factor") @@ -1893,7 +1906,7 @@ class LongRopeRotaryEmbedding(torch.nn.Module): if config is not None: # [TODO] Hack to pass in config - need to remove later - base = config.rope_theta + base = _get_rope_theta(config, default = base) partial_rotary_factor = ( config.partial_rotary_factor if hasattr(config, "partial_rotary_factor") @@ -2056,12 +2069,16 @@ def unsloth_fast_generate( and kwargs["input_ids"] is not None and "max_new_tokens" in kwargs ): - if ( - kwargs["input_ids"].shape[-1] + kwargs["max_new_tokens"] + _ids = kwargs["input_ids"] + # Handle BatchEncoding from transformers 5.0+ (no .shape attribute) + if hasattr(_ids, "input_ids"): + _ids = _ids["input_ids"] + if hasattr(_ids, "shape") and ( + _ids.shape[-1] + kwargs["max_new_tokens"] > self.config.max_position_embeddings ): raise ValueError( - f"Unsloth: input length {kwargs['input_ids'].shape[-1]} + max_new_tokens {kwargs['max_new_tokens']} exceeds the maximum sequence length of {self.config.max_position_embeddings}!\n" + f"Unsloth: input length {_ids.shape[-1]} + max_new_tokens {kwargs['max_new_tokens']} exceeds the maximum sequence length of {self.config.max_position_embeddings}!\n" "You will need to do long context extension by increasing the `max_seq_length` in `FastLanguageModel.from_pretrained`." ) diff --git a/unsloth/models/rl.py b/unsloth/models/rl.py index 2d17e70d3a..6bf7abda97 100755 --- a/unsloth/models/rl.py +++ b/unsloth/models/rl.py @@ -435,6 +435,7 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"): for x in dir(trainer) if x.endswith("Trainer") and x != "Trainer" + and not x.startswith("_") and trainer_file.split("_")[0] in x.lower() ] config = [ @@ -442,6 +443,7 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"): for x in dir(trainer) if x.endswith("Config") and x != "Config" + and not x.startswith("_") and trainer_file.split("_")[0] in x.lower() ] if len(name) != 1: @@ -449,6 +451,21 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"): f"Unsloth: Could not find Trainer class in trl.trainer.{trainer_file}. Found: {name}" ) return + if len(config) != 1: + # TRL 0.26+: Config may be in a separate *_config.py module + config_module_name = trainer_file.replace("_trainer", "_config") + try: + config_mod = eval(f"trl.trainer.{config_module_name}") + config = [ + x + for x in dir(config_mod) + if x.endswith("Config") + and x != "Config" + and not x.startswith("_") + and trainer_file.split("_")[0] in x.lower() + ] + except Exception: + pass if len(config) != 1: logger.info( f"Unsloth: Could not find Config class in trl.trainer.{trainer_file}. Found: {config}" @@ -467,11 +484,14 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"): return try: RLConfig = eval(f"trl.trainer.{trainer_file}.{RLConfig_name}") - except Exception as e: - logger.info( - f"Unsloth: Could not load {RLConfig_name} from trl.trainer.{trainer_file}: {e}" - ) - return + except Exception: + # TRL 0.26+: Config may be in a separate *_config.py module + try: + config_module_name = trainer_file.replace("_trainer", "_config") + RLConfig = eval(f"trl.trainer.{config_module_name}.{RLConfig_name}") + except Exception as e: + logger.info(f"Unsloth: Could not load {RLConfig_name}: {e}") + return # Check name if RLTrainer.__name__.startswith("Unsloth"): @@ -481,11 +501,49 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"): print(f"Unsloth: {RLConfig.__name__} is already patched.") return + # TRL 0.26+: Resolve thin wrappers to their experimental parent class. + # Thin wrappers are deprecation shims that contain "trl.experimental" in + # their source and just forward *args/**kwargs to the real implementation. + _trainer_resolved_module = None + try: + _trainer_src = inspect.getsource(RLTrainer) + if "trl.experimental" in _trainer_src: + for _parent in RLTrainer.__mro__[1:]: + if _parent is object: + continue + try: + if "trl.experimental" not in inspect.getsource(_parent): + RLTrainer = _parent + _trainer_resolved_module = inspect.getmodule(_parent) + break + except Exception: + continue + except Exception: + pass + + try: + _config_src = inspect.getsource(RLConfig) + if "trl.experimental" in _config_src: + for _parent in RLConfig.__mro__[1:]: + if _parent is object: + continue + try: + if "trl.experimental" not in inspect.getsource(_parent): + RLConfig = _parent + break + except Exception: + continue + except Exception: + pass + # Get old source old_RLTrainer_source = inspect.getsource(RLTrainer) old_RLConfig_source = inspect.getsource(RLConfig) - all_imports = dir(trainer) + if _trainer_resolved_module is not None: + all_imports = dir(_trainer_resolved_module) + else: + all_imports = dir(trainer) # Fix _deprecate_arguments not getting imported so stop __ but not _ imports = [x for x in all_imports if not x.startswith("__")] @@ -1191,13 +1249,32 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"): new_text = 'self._signature_columns = ["input_ids", "attention_mask", "completion_mask","labels"]' RLTrainer_source = RLTrainer_source.replace(original_text, new_text) - # Temporary patch _is_vlm to False - # as of 0.22 it only exists in sfttrainer - original_is_vlm_text = "self._is_vlm = True" - new_is_vlm_text = "self._is_vlm = False" - RLTrainer_source = RLTrainer_source.replace( - original_is_vlm_text, new_is_vlm_text + # Do NOT override _is_vlm -- let TRL detect VLM models naturally. + # In TRL 0.27.1+, forcing _is_vlm=False causes a ValueError when + # vision datasets are used with VLM models. + # + # However, some notebooks pass a bare tokenizer (processor.tokenizer) as + # processing_class. TRL then sets _is_vlm=False even for VLM models. + # Add a model-architecture-based override before the validation check. + _vlm_check_original = ( + ' self._is_vision_dataset = "image" in dataset_sample or "images" in dataset_sample\n' + " if self._is_vision_dataset and not self._is_vlm:" ) + _vlm_check_patched = ( + ' self._is_vision_dataset = "image" in dataset_sample or "images" in dataset_sample\n' + " # Unsloth: override _is_vlm for VLM models that pass a bare tokenizer\n" + " if not self._is_vlm and self._is_vision_dataset:\n" + " _m = model\n" + ' if hasattr(_m, "model"): _m = _m.model\n' + ' if hasattr(getattr(_m, "config", None), "vision_config") or \\\n' + ' _m.__class__.__name__.endswith("ForConditionalGeneration"):\n' + " self._is_vlm = True\n" + " if self._is_vision_dataset and not self._is_vlm:" + ) + if _vlm_check_original in RLTrainer_source: + RLTrainer_source = RLTrainer_source.replace( + _vlm_check_original, _vlm_check_patched + ) # Remove multiple doc strings if __RLConfig_doc__ != "" and RLTrainer_source.count(__RLTrainer_doc__) == 2: @@ -1207,10 +1284,15 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"): RLTrainer_source = re.sub(r"[\n]{3,}", "\n", RLTrainer_source) # Create new function + _model_location = ( + _trainer_resolved_module.__name__ + if _trainer_resolved_module is not None + else f"trl.trainer.{trainer_file}" + ) created_module = create_new_function( f"Unsloth{RLTrainer_name}", RLTrainer_source, - f"trl.trainer.{trainer_file}", + _model_location, imports, overwrite = False, ) diff --git a/unsloth/models/rl_replacements.py b/unsloth/models/rl_replacements.py index 410eee66e6..27f00f10f3 100755 --- a/unsloth/models/rl_replacements.py +++ b/unsloth/models/rl_replacements.py @@ -217,6 +217,19 @@ def sft_trainer_compute_loss(function_name, function): RL_FUNCTIONS["sft_trainer"].append(sft_trainer_compute_loss) +# Fix bare pop("push_to_hub_token") in compiled SFT/IterativeSFT trainer __init__ +# On transformers 5.0+, to_dict() no longer includes push_to_hub_token, so bare pop KeyErrors +def sft_trainer_push_to_hub_token(function_name, function): + if function_name != "__init__": + return function + return function.replace( + 'dict_args.pop("push_to_hub_token")', 'dict_args.pop("push_to_hub_token", None)' + ) + + +RL_FUNCTIONS["sft_trainer"].append(sft_trainer_push_to_hub_token) + + # Autocast precision for GRPO def grpo_trainer__prepare_inputs(function_name, function): if function_name != "_prepare_inputs": @@ -1193,6 +1206,29 @@ def grpo_trainer_compute_loss(function_name, function): RL_FUNCTIONS["grpo_trainer"].append(grpo_trainer_compute_loss) +# Fix KTO shape mismatch when Unsloth model forward truncates input_ids +# but labels aren't truncated. TRL 0.27.2+ _process_tokens only truncates +# completions, not prompts -- so prompts exceeding max_seq_length cause the +# model to produce shorter logits than the labels expect. +def kto_trainer_get_batch_logps(function_name, function): + if function_name != "get_batch_logps": + return function + # The raise is inside an if block inside the method, so we need + # to preserve the exact indentation of the raise statement. + old = 'raise ValueError("Logits (batch and sequence length dim) and labels must have the same shape.")' + new = ( + "# Unsloth: auto-truncate to shorter sequence length (model may have truncated input_ids)\n" + " _min_len = min(logits.shape[1], labels.shape[1])\n" + " logits = logits[:, :_min_len, :]\n" + " labels = labels[:, :_min_len]" + ) + function = function.replace(old, new) + return function + + +RL_FUNCTIONS["kto_trainer"].append(kto_trainer_get_batch_logps) + + # https://github.com/huggingface/trl/blob/main/trl/trainer/grpo_trainer.py#L356 # TRL warns if batch size is not a multiple of num_generations -> fix this. def grpo_trainer_fix_batch_size(RLTrainer_source, RLConfig_source): @@ -1267,7 +1303,7 @@ def openenv_vllm_reload_weights(): try: import trl.experimental.openenv.utils as openenv_utils import trl.experimental.openenv as openenv - except ImportError as e: + except (ImportError, NameError, Exception) as e: logger.info(f"Unsloth: Failed to import trl openenv: {e}") logger.info( "Unsloth: trl.experimental.openenv not available — skipping RL openenv patches." diff --git a/unsloth/models/sentence_transformer.py b/unsloth/models/sentence_transformer.py index 6a908482be..a3ea950420 100644 --- a/unsloth/models/sentence_transformer.py +++ b/unsloth/models/sentence_transformer.py @@ -14,7 +14,7 @@ import logging -from .loader import FastModel +from .loader import FastModel, DISABLE_SDPA_MODEL_NAMES from ._utils import SUPPORTS_BFLOAT16 import inspect import json @@ -1461,8 +1461,17 @@ class FastSentenceTransformer(FastModel): model_kwargs = {"torch_dtype": dtype} # Enable SDPA if supported (1.2x extra speedup on top of torch.compile) + # But disable for models with known SDPA + torch.compile backward issues + _force_eager = False + for _sdpa_model in DISABLE_SDPA_MODEL_NAMES: + if _sdpa_model in model_type.lower(): + supports_sdpa = False + _force_eager = True + break if supports_sdpa: model_kwargs["attn_implementation"] = "sdpa" + elif _force_eager: + model_kwargs["attn_implementation"] = "eager" # Print optimization status sdpa_str = " + SDPA" if supports_sdpa else "" diff --git a/unsloth/models/vision.py b/unsloth/models/vision.py index 93811c2668..2c1371a4a2 100644 --- a/unsloth/models/vision.py +++ b/unsloth/models/vision.py @@ -317,6 +317,91 @@ def unsloth_base_fast_generate( return output +def _construct_vlm_processor_fallback( + tokenizer_name, model_type, token, trust_remote_code +): + """Construct a VLM processor manually when AutoProcessor.from_pretrained fails. + + Some VLMs (e.g., LFM2.5-VL) have tokenizer_class entries that AutoTokenizer + cannot resolve. This function loads the image processor and tokenizer separately, + sets required special token attributes, and constructs the processor. + """ + try: + from transformers import AutoImageProcessor, PreTrainedTokenizerFast, AutoConfig + from transformers.models.auto.processing_auto import PROCESSOR_MAPPING_NAMES + import json + + # Load image processor + image_processor = AutoImageProcessor.from_pretrained( + tokenizer_name, + token = token, + trust_remote_code = trust_remote_code, + ) + # Load tokenizer via PreTrainedTokenizerFast (bypasses tokenizer_class check) + tok = PreTrainedTokenizerFast.from_pretrained( + tokenizer_name, + padding_side = "left", + token = token, + trust_remote_code = trust_remote_code, + ) + # Read tokenizer_config.json for model-specific special tokens + try: + from huggingface_hub import hf_hub_download + + config_path = hf_hub_download( + tokenizer_name, "tokenizer_config.json", token = token + ) + with open(config_path, "r", encoding = "utf-8") as f: + tok_config = json.load(f) + # Set model-specific special tokens and their IDs + for key in ( + "image_token", + "image_start_token", + "image_end_token", + "image_thumbnail", + "video_token", + ): + if key in tok_config and not hasattr(tok, key): + setattr(tok, key, tok_config[key]) + id_key = key + "_id" if not key.endswith("_id") else key + token_id = tok.convert_tokens_to_ids(tok_config[key]) + if not hasattr(tok, id_key): + setattr(tok, id_key, token_id) + except Exception: + pass + + # Find the processor class - try model_type first, then top-level config model_type + proc_class_name = PROCESSOR_MAPPING_NAMES.get(model_type) + if proc_class_name is None: + # model_type might be a sub-model type (e.g. "lfm2" instead of "lfm2_vl"). + # Try the top-level config.model_type which often has the processor mapping. + try: + config = AutoConfig.from_pretrained( + tokenizer_name, + token = token, + trust_remote_code = trust_remote_code, + ) + proc_class_name = PROCESSOR_MAPPING_NAMES.get(config.model_type) + except Exception: + pass + + if proc_class_name is not None: + import transformers + + proc_class = getattr(transformers, proc_class_name, None) + if proc_class is not None: + processor = proc_class(image_processor = image_processor, tokenizer = tok) + # Copy chat_template from tokenizer to processor if needed + if not getattr(processor, "chat_template", None) and getattr( + tok, "chat_template", None + ): + processor.chat_template = tok.chat_template + return processor + except Exception: + pass + return None + + class FastBaseModel: @staticmethod def from_pretrained( @@ -826,14 +911,17 @@ class FastBaseModel: if (whisper_language and whisper_task) or auto_model.__name__.endswith( "ForConditionalGeneration" ): - tokenizer = auto_processor.from_pretrained( - tokenizer_name, - padding_side = "left", - token = token, - language = whisper_language, - task = whisper_task, - trust_remote_code = trust_remote_code, - ) + try: + tokenizer = auto_processor.from_pretrained( + tokenizer_name, + padding_side = "left", + token = token, + language = whisper_language, + task = whisper_task, + trust_remote_code = trust_remote_code, + ) + except Exception: + tokenizer = None else: try: tokenizer = auto_processor.from_pretrained( @@ -849,6 +937,23 @@ class FastBaseModel: token = token, trust_remote_code = trust_remote_code, ) + + # If processor loading failed (e.g., tokenizer class not found), + # try constructing the processor manually from separate components. + if tokenizer is None and is_vlm: + tokenizer = _construct_vlm_processor_fallback( + tokenizer_name, + model_type_arch, + token, + trust_remote_code, + ) + if tokenizer is None: + import sys + + print( + f"Unsloth: Warning - VLM processor fallback returned None for model_type={model_type_arch}", + file = sys.stderr, + ) if hasattr(tokenizer, "tokenizer"): __tokenizer = tokenizer.tokenizer # Add padding side as well @@ -872,7 +977,29 @@ class FastBaseModel: do_forced_float32 = do_forced_float32, correct_dtype = correct_dtype, ) - model, tokenizer = patch_tokenizer(model, tokenizer) + try: + model, tokenizer = patch_tokenizer(model, tokenizer) + except Exception as _patch_err: + # Some VLM processors (e.g., ERNIE VL) may fail during tokenizer patching. + # Try loading tokenizer separately via AutoTokenizer as fallback. + try: + from transformers import AutoTokenizer as _AutoTokenizer + + _fallback_tok = _AutoTokenizer.from_pretrained( + tokenizer_name, + padding_side = "left", + token = token, + trust_remote_code = trust_remote_code, + ) + model, _fallback_tok = patch_tokenizer(model, _fallback_tok) + # Re-attach as processor wrapper if original was a processor + if hasattr(tokenizer, "image_processor"): + tokenizer.tokenizer = _fallback_tok + else: + tokenizer = _fallback_tok + except Exception: + # If fallback also fails, raise the original error + raise _patch_err model = post_patch_loss_function(model) # Log Unsloth version for future fastpaths for inference @@ -880,10 +1007,31 @@ class FastBaseModel: model.config.update({"unsloth_version": __version__}) patch_saving_functions(model, vision = True) if tokenizer is None: - del model - raise RuntimeError( - "Unsloth: The tokenizer is weirdly not loaded? Please check if there is one." - ) + # Last resort: try loading tokenizer via AutoTokenizer, then PreTrainedTokenizerFast + try: + from transformers import AutoTokenizer as _AutoTokenizer + + tokenizer = _AutoTokenizer.from_pretrained( + tokenizer_name, + padding_side = "left", + token = token, + trust_remote_code = trust_remote_code, + ) + except Exception: + try: + from transformers import PreTrainedTokenizerFast + + tokenizer = PreTrainedTokenizerFast.from_pretrained( + tokenizer_name, + padding_side = "left", + token = token, + trust_remote_code = trust_remote_code, + ) + except Exception: + del model + raise RuntimeError( + "Unsloth: The tokenizer is weirdly not loaded? Please check if there is one." + ) patch_saving_functions(tokenizer, vision = True) # Fix gradient accumulation diff --git a/unsloth/tokenizer_utils.py b/unsloth/tokenizer_utils.py index 087a9a7f8a..1c107bd84f 100644 --- a/unsloth/tokenizer_utils.py +++ b/unsloth/tokenizer_utils.py @@ -1021,7 +1021,10 @@ def patch_sft_trainer_tokenizer(): "kto_trainer.KTOTrainer", ): function_name, replacer = "train", "if resume_from_checkpoint is False:" - function = getsource(eval(f"trl.trainer.{path_to_trainer}.{function_name}")) + try: + function = getsource(eval(f"trl.trainer.{path_to_trainer}.{function_name}")) + except Exception: + continue where = function.find("def") function = function.split("\n") function = "\n".join(x[where:] for x in function) diff --git a/unsloth/trainer.py b/unsloth/trainer.py index 858dcf2cd3..cb36b8639d 100644 --- a/unsloth/trainer.py +++ b/unsloth/trainer.py @@ -200,13 +200,57 @@ class UnslothTrainer(SFTTrainer): # From `trl>=0.13.0`, they changed how to pass several params to the trainer # We need to patch to make the transition smooth +def _resolve_trainer_params(trainer_class, init_fn): + """Resolve the real named parameters for a trainer __init__. + + Some TRL trainers (e.g., ORPOTrainer in TRL 0.27.1) are thin wrappers + with only ``def __init__(self, *args, **kwargs)``. For those, walk the + MRO and return the first parent class that has real named parameters. + """ + params = inspect.signature(init_fn).parameters + named = { + k + for k, v in params.items() + if v.kind + in (inspect.Parameter.POSITIONAL_OR_KEYWORD, inspect.Parameter.KEYWORD_ONLY) + and k != "self" + } + if named: + return set(params.keys()) + + # Thin wrapper detected - walk MRO for real signature + for cls in trainer_class.__mro__[1:]: + if cls is object: + continue + parent_init = cls.__dict__.get("__init__") + if parent_init is None: + continue + try: + parent_params = inspect.signature(parent_init).parameters + parent_named = { + k + for k, v in parent_params.items() + if v.kind + in ( + inspect.Parameter.POSITIONAL_OR_KEYWORD, + inspect.Parameter.KEYWORD_ONLY, + ) + and k != "self" + } + if parent_named: + return set(parent_params.keys()) + except (ValueError, TypeError): + continue + return set(params.keys()) + + def _backwards_compatible_trainer(trainer_class, config_class): original_init = trainer_class.__init__ @wraps(original_init) def new_init(self, *args, **kwargs): # All Trainer tokenizer are now called processing_class - trainer_params = set(inspect.signature(original_init).parameters.keys()) + trainer_params = _resolve_trainer_params(trainer_class, original_init) if "processing_class" in trainer_params and "tokenizer" in kwargs: kwargs["processing_class"] = kwargs.pop("tokenizer")