ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix: unwrap ProcessorMixin to raw tokenizer for text-only SFTTrainer on VLM-architecture models

Browse source
This commit is contained in:
Roland Tannous 2026-02-18 19:13:20 +00:00
commit e33920974b
13 changed files with 2431338 additions and 113 deletions
Download patch file Download diff file Expand all files Collapse all files

210
gemma_3_lora/README.md Normal file
View file

@ -0,0 +1,210 @@
---
base_model: unsloth/gemma-3-4b-it-unsloth-bnb-4bit
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:unsloth/gemma-3-4b-it-unsloth-bnb-4bit
- lora
- sft
- transformers
- trl
- unsloth
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed]
### Framework versions
- PEFT 0.18.1

42
gemma_3_lora/adapter_config.json Normal file
View file

@ -0,0 +1,42 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": {
"base_model_class": "Gemma3ForConditionalGeneration",
"parent_library": "transformers.models.gemma3.modeling_gemma3",
"unsloth_fixed": true
},
"base_model_name_or_path": "unsloth/gemma-3-4b-it-unsloth-bnb-4bit",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 8,
"lora_bias": false,
"lora_dropout": 0.0,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": null,
"peft_type": "LORA",
"peft_version": "0.18.1",
"qalora_group_size": 16,
"r": 8,
"rank_pattern": {},
"revision": null,
"target_modules": "(?:.*?(?:vision|image|visual|patch|language|text).*?(?:self_attn|attention|attn|mlp|feed_forward|ffn|dense).*?(?:k_proj|v_proj|q_proj|out_proj|fc1|fc2|o_proj|gate_proj|up_proj|down_proj).*?)|(?:\\bmodel\\.layers\\.[\\d]{1,}\\.(?:self_attn|attention|attn|mlp|feed_forward|ffn|dense)\\.(?:(?:k_proj|v_proj|q_proj|out_proj|fc1|fc2|o_proj|gate_proj|up_proj|down_proj)))",
"target_parameters": null,
"task_type": "CAUSAL_LM",
"trainable_token_indices": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}

3
gemma_3_lora/added_tokens.json Normal file
View file

@ -0,0 +1,3 @@
{
"<image_soft_token>": 262144
}

47
gemma_3_lora/chat_template.jinja Normal file
View file

@ -0,0 +1,47 @@
{{ bos_token }}
{%- if messages[0]['role'] == 'system' -%}
{%- if messages[0]['content'] is string -%}
{%- set first_user_prefix = messages[0]['content'] + '
' -%}
{%- else -%}
{%- set first_user_prefix = messages[0]['content'][0]['text'] + '
' -%}
{%- endif -%}
{%- set loop_messages = messages[1:] -%}
{%- else -%}
{%- set first_user_prefix = "" -%}
{%- set loop_messages = messages -%}
{%- endif -%}
{%- for message in loop_messages -%}
{%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
{{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
{%- endif -%}
{%- if (message['role'] == 'assistant') -%}
{%- set role = "model" -%}
{%- else -%}
{%- set role = message['role'] -%}
{%- endif -%}
{{ '<start_of_turn>' + role + '
' + (first_user_prefix if loop.first else "") }}
{%- if message['content'] is string -%}
{{ message['content'] | trim }}
{%- elif message['content'] is iterable -%}
{%- for item in message['content'] -%}
{%- if item['type'] == 'image' -%}
{{ '<start_of_image>' }}
{%- elif item['type'] == 'text' -%}
{{ item['text'] | trim }}
{%- endif -%}
{%- endfor -%}
{%- else -%}
{{ raise_exception("Invalid content type") }}
{%- endif -%}
{{ '<end_of_turn>
' }}
{%- endfor -%}
{%- if add_generation_prompt -%}
{{ '<start_of_turn>model
' }}
{%- endif -%}

29
gemma_3_lora/preprocessor_config.json Normal file
View file

@ -0,0 +1,29 @@
{
"do_convert_rgb": null,
"do_normalize": true,
"do_pan_and_scan": null,
"do_rescale": true,
"do_resize": true,
"image_mean": [
0.5,
0.5,
0.5
],
"image_processor_type": "Gemma3ImageProcessor",
"image_seq_length": 256,
"image_std": [
0.5,
0.5,
0.5
],
"pan_and_scan_max_num_crops": null,
"pan_and_scan_min_crop_size": null,
"pan_and_scan_min_ratio_to_activate": null,
"processor_class": "_Unsloth_Patched_Gemma3Processor",
"resample": 2,
"rescale_factor": 0.00392156862745098,
"size": {
"height": 896,
"width": 896
}
}

4
gemma_3_lora/processor_config.json Normal file
View file

@ -0,0 +1,4 @@
{
"image_seq_length": 256,
"processor_class": "_Unsloth_Patched_Gemma3Processor"
}

33
gemma_3_lora/special_tokens_map.json Normal file
View file

@ -0,0 +1,33 @@
{
"boi_token": "<start_of_image>",
"bos_token": {
"content": "<bos>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eoi_token": "<end_of_image>",
"eos_token": {
"content": "<end_of_turn>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"image_token": "<image_soft_token>",
"pad_token": {
"content": "<pad>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"unk_token": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

2379611
gemma_3_lora/tokenizer.json Normal file
View file

File diff suppressed because it is too large Load diff

BIN
gemma_3_lora/tokenizer.model Normal file
View file

Binary file not shown.

51346
gemma_3_lora/tokenizer_config.json Normal file
View file

File diff suppressed because it is too large Load diff

66
studio/backend/core/training/trainer.py
View file

@ -268,12 +268,6 @@ class UnslothTrainer:
print(f"Configuring LoRA adapters (r={lora_r}, alpha={lora_alpha})...\n")
print(f"Gradient checkpointing: {use_gradient_checkpointing} (type: {type(use_gradient_checkpointing).__name__})\n")
# Normalize ["all-linear"] (list from frontend/YAML) → "all-linear" (string)
# Unsloth/PEFT expect the string form for this shorthand
if target_modules == ["all-linear"]:
target_modules = "all-linear"
print(f" Normalized target_modules from list to string: '{target_modules}'")
# Branch based on vision vs text
if self.is_vlm:
# Vision model LoRA
@ -445,7 +439,6 @@ class UnslothTrainer:
dataset,
model_name=self.model_name,
tokenizer=self.tokenizer,
model=self.model,
is_vlm=self.is_vlm,
format_type=format_type,
dataset_name=dataset_source,
@ -468,7 +461,6 @@ class UnslothTrainer:
eval_dataset,
model_name=self.model_name,
tokenizer=self.tokenizer,
model=self.model,
is_vlm=self.is_vlm,
format_type=format_type,
dataset_name=dataset_source,
@ -795,51 +787,6 @@ class UnslothTrainer:
print(f"The configuration is: {config_args}")
print("Training configuration prepared\n")
# ========== DEBUG: Dataset & Model Routing Info ==========
print("=" * 60)
print("DEBUG: Pre-Training Diagnostics")
print("=" * 60)
print(f" Route taken: {'VLM' if self.is_vlm else 'LLM (text)'}")
print(f" Model name: {self.model_name}")
print(f" Model class: {type(self.model).__name__}")
print(f" is_vlm flag: {self.is_vlm}")
# Dataset info
train_ds = dataset['dataset']
if hasattr(train_ds, 'column_names'):
print(f" Dataset columns: {train_ds.column_names}")
print(f" Dataset size: {len(train_ds)} rows")
# Print first sample
try:
sample = train_ds[0]
print(f" First sample keys: {list(sample.keys())}")
for key, val in sample.items():
val_str = str(val)
if len(val_str) > 200:
val_str = val_str[:200] + "..."
print(f" {key}: {val_str}")
except Exception as e:
print(f" Could not read first sample: {e}")
elif isinstance(train_ds, list):
print(f" Dataset type: list ({len(train_ds)} items)")
if train_ds:
print(f" First sample keys: {list(train_ds[0].keys()) if isinstance(train_ds[0], dict) else 'N/A'}")
sample_str = str(train_ds[0])
if len(sample_str) > 300:
sample_str = sample_str[:300] + "..."
print(f" First sample: {sample_str}")
# Model forward signature
try:
import inspect
sig = inspect.signature(self.model.forward)
fwd_params = list(sig.parameters.keys())
print(f" model.forward() params: {fwd_params}")
except Exception as e:
print(f" Could not inspect model.forward(): {e}")
print("=" * 60)
# ========== TRAINER INITIALIZATION ==========
if self.is_vlm:
trainer_kwargs = {
@ -853,9 +800,20 @@ class UnslothTrainer:
trainer_kwargs["eval_dataset"] = eval_dataset
self.trainer = SFTTrainer(**trainer_kwargs)
else:
# For text-only training, if the tokenizer is actually a Processor
# (e.g., Gemma-3 returns a ProcessorMixin even for text), we must
# unwrap to the raw tokenizer. Otherwise Unsloth's SFTTrainer detects
# ProcessorMixin → sets _is_vlm=True → skips _prepare_dataset entirely,
# and the 'text' column never gets tokenized to 'input_ids'.
from transformers import ProcessorMixin
sft_tokenizer = self.tokenizer
if isinstance(self.tokenizer, ProcessorMixin) and hasattr(self.tokenizer, 'tokenizer'):
print(f" ⚠️ Unwrapping Processor → raw tokenizer for text-only SFTTrainer")
sft_tokenizer = self.tokenizer.tokenizer
trainer_kwargs = {
"model": self.model,
"tokenizer": self.tokenizer,
"tokenizer": sft_tokenizer,
"train_dataset": dataset['dataset'],
"data_collator": data_collator,
"args": SFTConfig(**config_args),

23
studio/backend/utils/datasets/dataset_utils.py
View file

@ -28,8 +28,6 @@ from .format_conversion import (
convert_alpaca_to_chatml,
convert_to_vlm_format,
convert_llava_to_vlm_format,
get_expected_chat_column,
rename_chat_column_in_list,
)
from .chat_templates import (
apply_chat_template_to_dataset,
@ -549,7 +547,6 @@ def format_and_template_dataset(
dataset,
model_name,
tokenizer,
model=None,
is_vlm = False,
format_type="auto",
# VLM-specific parameters
@ -738,30 +735,12 @@ def format_and_template_dataset(
dataset = [sample for sample in dataset]
warnings.append("Dataset already in standard VLM messages format")
# Defensive: rename chat column if model expects a different name
expected_col = get_expected_chat_column(model) if model is not None else None
# VLM data is a list of dicts — check what key the first item uses
current_col = "messages" # default from our converters
if isinstance(dataset, list) and len(dataset) > 0:
sample_keys = dataset[0].keys()
if "conversations" in sample_keys:
current_col = "conversations"
elif "messages" in sample_keys:
current_col = "messages"
if expected_col and expected_col != current_col:
warnings.append(
f"Model expects '{expected_col}' but dataset has '{current_col}' — renaming."
)
dataset = rename_chat_column_in_list(dataset, current_col, expected_col)
current_col = expected_col
# Return as list
return {
"dataset": dataset,
"detected_format": vlm_structure["format"],
"final_format": "vlm_messages",
"chat_column": current_col,
"chat_column": "messages",
"is_vlm": True,
"is_multimodal": multimodal_info["is_multimodal"],
"multimodal_info": multimodal_info,

37
studio/backend/utils/datasets/format_conversion.py
View file

@ -8,43 +8,6 @@ This module contains functions for converting between dataset formats
from datasets import IterableDataset
def get_expected_chat_column(model):
"""
Inspect the model's forward() signature to determine if it expects
'messages' or 'conversations' as a column name.
Returns:
str or None: 'messages', 'conversations', or None if neither found.
"""
import inspect
try:
sig = inspect.signature(model.forward)
params = list(sig.parameters.keys())
if "messages" in params:
return "messages"
elif "conversations" in params:
return "conversations"
except (ValueError, TypeError):
pass
return None
def rename_chat_column_in_list(data, from_col, to_col):
"""
Rename a chat column key in a list of dicts (for VLM data).
"""
if from_col == to_col:
return data
renamed = []
for item in data:
new_item = {}
for k, v in item.items():
new_item[to_col if k == from_col else k] = v
renamed.append(new_item)
return renamed
def standardize_chat_format(
dataset,
tokenizer=None,