ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Gemma 3 (#1986)

Browse source 
* Update llama.py

* GRPO optimized

* Update rl.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Selective Log softmax

* Fix GRPO bsz

* Update rl.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Fix TRL

* Metrics GRPO

* Update rl_replacements.py

* Update rl_replacements.py

* No compile

* Update rl.py

* Remove docs

* Update rl.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl_replacements.py

* Update rl_replacements.py

* llama-quantize on WINDOWS WSL error fix - edit save.py (gguf saving breaks) (#1649)

* edit save.py to fix gguf saving breaks.

* add check for .exe or not exe file extension for linux and windows

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update llama.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update llama.py

* Update llama.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* unsloth_num_chunks

* Update rl.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py (#1754)

Fix typo in comment: know -> now.

This was printed when running the Llama3.1_(8B)-GRPO.ipynb example notebook, so I'd expect others to run into it as well.

* Optional logits

* Update rl.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update rl.py

* fix an import error (#1767)

* fix an import error

* Delete .gitignore

* Update loader.py

* Update save.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* SamplingParams

* Convert mask to float (#1762)

* [Windows Support] Add latest `xformers` wheels to pyproject.toml (#1753)

* Add latest xformers

* Add a couple of lines to docs

* vLLMSamplingParams

* Update __init__.py

* default num_chunks == -1

* Versioning

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update _utils.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update pyproject.toml

* Update pyproject.toml

* Export Model to ollama.com  (#1648)

* Ollama Export Model to ollama.com

Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in>

* Check for model_name

Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in>

* subprocess use instead of requests | added check for ollama server

Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in>

* create_ollama_model

Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in>

* create_ollama_model | fix

Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in>

* Push to Ollama

Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in>

---------

Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in>

* Update cross_entropy_loss.py

* torch_cuda_device

* Update utils.py

* Update utils.py

* Update utils.py

* device

* device

* Update loader.py

* Update llama.py

* Update README.md

* Update llama.py

* Update llama.py

* Update _utils.py

* Update utils.py

* Update utils.py

* Update utils.py

* Update utils.py

* Update utils.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update utils.py

* Update utils.py

* Update utils.py

* Update utils.py

* __version__

* Update rl.py

* Bug fixes

* Bug fixes

* Update llama.py

* Update _utils.py

* _wrap_fast_inference

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update _utils.py

* SFT dataset prepare

* Update pyproject.toml

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl_replacements.py

* Update rl.py

* Update llama.py

* Update llama.py

* Update utils.py

* bug fix

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update __init__.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update _utils.py

* Update __init__.py

* Update _utils.py

* Version

* versioning

* Update _utils.py

* Update llama.py

* Update llama.py

* Bug fixes

* FastModel

* __doc__

* Update vision.py

* Update loader.py

* Update loader.py

* Update loader.py

* version

* move use_modelscope to _utils (#1938)

* move use_modelscope to _utils

* Update _utils.py

* Update loader.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Don't use revision when loading model_config and is_peft=True (#1949)

* More syntax warnings (#1944)

* move use_modelscope to _utils

* fix

* Update _utils.py

* Update loader.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Full finetuning and other fixes

* UNSLOTH_ENABLE_FULL_FINETUNING

* Update loader.py

* Update loader.py

* Update loader.py

* Update vision.py

* Update vision.py

* full finetuning

* Update loader.py

* Update loader.py

* Update loader.py

* Update _utils.py

* max_seq_length

* Update rl.py

* Update rl.py

* Update rl.py

* Update pyproject.toml

* AutoModelForImageTextToText

* Update mapper.py

* Update pyproject.toml

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Batch samples

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update _utils.py

* Update loader.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update vision.py

* Update mapper.py

---------

Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in>
Co-authored-by: Gennadii Manzhos <105049664+everythingisc00l@users.noreply.github.com>
Co-authored-by: Seth Weidman <seth@sethweidman.com>
Co-authored-by: Nino Risteski <95188570+NinoRisteski@users.noreply.github.com>
Co-authored-by: Edd <68678137+Erland366@users.noreply.github.com>
Co-authored-by: Ben <6579034+versipellis@users.noreply.github.com>
Co-authored-by: Jyotin Goel <120490013+gjyotin305@users.noreply.github.com>
Co-authored-by: Kareem <81531392+KareemMusleh@users.noreply.github.com>
Co-authored-by: Wilson Wu <140025193+wiwu2390@users.noreply.github.com>
This commit is contained in:
Daniel Han 2025-03-12 01:23:34 -07:00 • committed by GitHub
commit f35d5977d6
9 changed files with 329 additions and 163 deletions
Download patch file Download diff file Expand all files Collapse all files

4
pyproject.toml
View file

@ -40,7 +40,7 @@ triton = [
]
huggingface = [
"unsloth_zoo>=2025.3.8",
"unsloth_zoo>=2025.3.9",
"packaging",
"tyro",
"transformers>=4.46.1,!=4.47.0",
@ -354,7 +354,7 @@ colab-ampere-torch220 = [
"flash-attn>=2.6.3",
]
colab-new = [
"unsloth_zoo>=2025.3.8",
"unsloth_zoo>=2025.3.9",
"packaging",
"tyro",
"transformers>=4.46.1,!=4.47.0",

17
unsloth/__init__.py
View file

@ -198,14 +198,19 @@ pass
# Check for unsloth_zoo
try:
unsloth_zoo_version = importlib_version("unsloth_zoo")
if Version(unsloth_zoo_version) < Version("2025.3.8"):
try:
os.system("pip install --upgrade --no-cache-dir --no-deps unsloth_zoo")
except:
if Version(unsloth_zoo_version) < Version("2025.3.9"):
print(
"Unsloth: Updating Unsloth-Zoo utilies to the latest version.\n"\
"To disable this, set os.environ['UNSLOTH_DISABLE_AUTO_UPDATES'] = '1'"
)
if os.environ.get("UNSLOTH_DISABLE_AUTO_UPDATES", "0") == "0":
try:
os.system("pip install --upgrade --no-cache-dir --no-deps --user unsloth_zoo")
os.system("pip install --upgrade --no-cache-dir --no-deps unsloth_zoo")
except:
raise ImportError("Unsloth: Please update unsloth_zoo via `pip install --upgrade --no-cache-dir --no-deps unsloth_zoo`")
try:
os.system("pip install --upgrade --no-cache-dir --no-deps --user unsloth_zoo")
except:
raise ImportError("Unsloth: Please update unsloth_zoo via `pip install --upgrade --no-cache-dir --no-deps unsloth_zoo`")
import unsloth_zoo
except:
raise ImportError("Unsloth: Please install unsloth_zoo via `pip install unsloth_zoo`")

130
unsloth/models/_utils.py
View file

@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
__version__ = "2025.3.9"
__version__ = "2025.3.10"
__all__ = [
"SUPPORTS_BFLOAT16",
@ -25,6 +25,7 @@ __all__ = [
"__version__",
"HAS_FLASH_ATTENTION",
"HAS_FLASH_ATTENTION_SOFTCAPPING",
"USE_MODELSCOPE",
"platform_system",
"patch_tokenizer",
"get_statistics",
@ -100,6 +101,7 @@ from unsloth_zoo.gradient_checkpointing import (
from unsloth_zoo.loss_utils import (
HAS_CUT_CROSS_ENTROPY,
fused_linear_cross_entropy,
_unsloth_get_batch_samples,
)
from unsloth_zoo.vision_utils import (
process_vision_info,
@ -108,6 +110,9 @@ from unsloth_zoo.compiler import (
get_transformers_model_type,
unsloth_compile_transformers as _unsloth_compile_transformers,
)
from unsloth_zoo.training_utils import (
prepare_model_for_training,
)
# =============================================
# Disable some warnings which can get annoying
@ -508,67 +513,16 @@ def prepare_model_for_kbit_training(
use_gradient_checkpointing : Optional = True,
use_reentrant : Optional[bool] = True,
) -> Any:
"""
Calculates where to place the gradient checkpoints given n_layers.
We also freeze all other layers's gradients
Args:
model: Any LlamaModel with layers.
use_gradient_checkpointing (`bool`, *optional*):
Default enabled. Provides memory savings by not saving all activations,
but only some.
use_reentrant (`bool`, *optional*):
https://github.com/pytorch/pytorch/blob/main/torch/utils/checkpoint.py#L354
Optimal gradient checkpointing algorithm which will be the default in
future Pytorch versions.
"""
# Freeze all parameters except LoRA
with torch.no_grad():
for name, param in model.named_parameters():
if ".lora_A." in name or ".lora_B." in name or ".lora_magnitude_vector" in name:
param.requires_grad_(True)
# Also must be in float32!
if param.dtype != torch.float32:
name = name.replace("base_model", "model", 1)
layer_number = re.search(r"\.[\d]{1,}\.", name).group(0)
name = name.replace(layer_number, f"[{layer_number[1:-1]}].")
name = name.replace(".weight", "", 1)
exec(f"{name}.to(torch.float32)")
pass
else:
param.requires_grad_(False)
pass
pass
# Gradient checkpointing!
if use_gradient_checkpointing == "unsloth":
# Saves VRAM!
original_model = model
while hasattr(original_model, "model"):
original_model._offloaded_gradient_checkpointing = True
original_model = original_model.model
pass
original_model._offloaded_gradient_checkpointing = True
model.gradient_checkpointing_enable()
elif use_gradient_checkpointing == True:
model.gradient_checkpointing_enable()
pass
# If use_reentrant = True which is the Pytorch default, we just make the input requires_grad.
if use_reentrant:
if hasattr(model, "enable_input_require_grads"):
model.enable_input_require_grads()
else:
def make_inputs_require_grad(module, input, output):
output.requires_grad_(True)
model.get_input_embeddings().register_forward_hook(make_inputs_require_grad)
pass
return model
return prepare_model_for_training(
model = model,
use_gradient_checkpointing = use_gradient_checkpointing,
use_reentrant = use_reentrant,
full_finetuning = False,
train_layernorms = False,
train_embedding = False,
train_lm_head = False,
float32_mixed_precision = True,
)
pass
# =============================================
@ -999,44 +953,6 @@ def test_mask_creation():
pass
def _unsloth_get_batch_samples(self, epoch_iterator, num_batches):
batch_samples = []
num_items_in_batch = None
# Check if model allows **kwargs
model = self.model
f = model.base_model.model.forward if hasattr(model, "base_model") else model.forward
has_kwargs = tuple(inspect.signature(f).parameters.values())[-1].kind == inspect._VAR_KEYWORD
# Iterate to find all batches
for _ in range(num_batches):
try:
batch_samples += [next(epoch_iterator)]
except StopIteration:
break
pass
# Get num_items_in_batch
if has_kwargs and len(batch_samples) > 0 and "labels" in batch_samples[0]:
try:
num_items_in_batch = sum(
[(x["labels"][..., 1:] != -100).sum() for x in batch_samples]
)
if self.args.average_tokens_across_devices:
num_items_in_batch = self.accelerator.gather(num_items_in_batch).sum().item()
if torch.is_tensor(num_items_in_batch):
num_items_in_batch = num_items_in_batch.item()
except Exception as exception:
logger.warning_once(exception)
pass
return batch_samples, num_items_in_batch
pass
def _unsloth_pre_compute_loss(self, model, inputs, *args, **kwargs):
num_items_in_batch = None
@ -1053,7 +969,12 @@ def _unsloth_pre_compute_loss(self, model, inputs, *args, **kwargs):
# Get gradient accumulation steps if possible
if num_items_in_batch is None and \
getattr(getattr(self, "args", self), "gradient_accumulation_steps", 1) != 1:
name = (model.base_model.model if hasattr(model, "base_model") else model).__class__.__name__
inner_model = model
if hasattr(inner_model, "base_model"): inner_model = inner_model. base_model
if hasattr(inner_model, "model"): inner_model = inner_model.model
name = inner_model.__class__.__name__
logger.warning_once(
f"Unsloth: Not an error, but {name} does not accept `num_items_in_batch`.\n"\
"Using gradient accumulation will be very slightly less accurate.\n"\
@ -1271,3 +1192,10 @@ for j, function in enumerate(functions):
try: exec(f"EMPTY_LOGITS.{function} = raise_{j}", globals(), locals())
except: continue
pass
USE_MODELSCOPE = os.environ.get("UNSLOTH_USE_MODELSCOPE", "0") == "1"
if USE_MODELSCOPE:
if importlib.util.find_spec("modelscope") is None:
raise ImportError(f'You are using the modelscope hub, please install modelscope by `pip install modelscope -U`')
pass
pass

14
unsloth/models/llama.py
View file

@ -1913,12 +1913,12 @@ class FastLlamaModel:
# Save max_seq_length
model.max_seq_length = max_seq_length
internal_model = model
while hasattr(internal_model, "model"):
internal_model.max_seq_length = max_seq_length
internal_model = internal_model.model
m = model
while hasattr(m, "model"):
m.max_seq_length = max_seq_length
m = m.model
pass
internal_model.max_seq_length = max_seq_length
m.max_seq_length = max_seq_length
# We check the tokenizer first for errors
if fix_tokenizer:
@ -2016,6 +2016,10 @@ class FastLlamaModel:
temporary_location = "_unsloth_temporary_saved_buffers",
**kwargs,
):
if os.environ.get("UNSLOTH_ENABLE_FULL_FINETUNING", "0") == "1":
print("Unsloth: Full finetuning is enabled, so .get_peft_model has no effect")
return model
pass
transformers_set_seed(random_state)
if use_gradient_checkpointing == "unsloth":

129
unsloth/models/loader.py
View file

@ -12,7 +12,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.
from ._utils import is_bfloat16_supported, HAS_FLASH_ATTENTION, HAS_FLASH_ATTENTION_SOFTCAPPING
from ._utils import (
is_bfloat16_supported,
HAS_FLASH_ATTENTION,
HAS_FLASH_ATTENTION_SOFTCAPPING,
USE_MODELSCOPE,
)
from .granite import FastGraniteModel
from .llama import FastLlamaModel, logger
from .mistral import FastMistralModel
@ -36,14 +41,6 @@ pass
from huggingface_hub import HfFileSystem
import importlib.util
# [TODO] Move USE_MODELSCOPE to utils
USE_MODELSCOPE = os.environ.get("UNSLOTH_USE_MODELSCOPE", "0") == "1"
if USE_MODELSCOPE:
if importlib.util.find_spec("modelscope") is None:
raise ImportError(f'You are using the modelscope hub, please install modelscope by `pip install modelscope -U`')
pass
pass
# https://github.com/huggingface/transformers/pull/26037 allows 4 bit loading!
from unsloth_zoo.utils import Version, _get_dtype
transformers_version = Version(transformers_version)
@ -76,6 +73,8 @@ class FastLanguageModel(FastLlamaModel):
max_seq_length = None,
dtype = None,
load_in_4bit = True,
load_in_8bit = False,
full_finetuning = False,
token = None,
device_map = "sequential",
rope_scaling = None,
@ -94,6 +93,29 @@ class FastLanguageModel(FastLlamaModel):
disable_log_stats = True,
*args, **kwargs,
):
if load_in_8bit or full_finetuning:
return FastModel.from_pretrained(
model_name = model_name,
max_seq_length = max_seq_length, # [TODO] No effect
dtype = dtype,
load_in_4bit = load_in_4bit,
load_in_8bit = load_in_8bit,
full_finetuning = full_finetuning,
token = token,
device_map = device_map,
rope_scaling = rope_scaling, # [TODO] No effect
fix_tokenizer = fix_tokenizer, # [TODO] No effect
trust_remote_code = trust_remote_code,
use_gradient_checkpointing = use_gradient_checkpointing,
resize_model_vocab = resize_model_vocab, # [TODO] No effect
revision = revision,
return_logits = False, # Return logits
fullgraph = True, # No graph breaks
use_exact_model_name = use_exact_model_name,
*args, **kwargs,
)
pass
if token is None: token = get_token()
assert (dtype is None or dtype == torch.float16 or dtype == torch.bfloat16)
@ -153,7 +175,7 @@ class FastLanguageModel(FastLlamaModel):
# Old transformers versions check
both_exist = (is_model and is_peft) and not SUPPORTS_LLAMA32
# New transformers need to check manually.
if SUPPORTS_LLAMA32:
# Check if folder exists locally
@ -202,7 +224,6 @@ class FastLanguageModel(FastLlamaModel):
model_config = AutoConfig.from_pretrained(
model_name,
token = token,
revision = revision,
trust_remote_code = trust_remote_code,
)
pass
@ -265,15 +286,32 @@ class FastLanguageModel(FastLlamaModel):
dispatch_model = FastGemma2Model
elif model_type == "qwen2":
dispatch_model = FastQwen2Model
elif model_type == "cohere":
dispatch_model = FastCohereModel
elif model_type == "granite":
dispatch_model = FastGraniteModel
# Temporary disable optimized Cohere until errors match
# elif model_type == "cohere":
# dispatch_model = FastCohereModel
# Temporary disable optimized Granite until errors match
# elif model_type == "granite":
# dispatch_model = FastGraniteModel
else:
raise NotImplementedError(
f"Unsloth: {model_name} not supported yet!\n"\
"Maybe you're doing vision finetuning? Please use FastVisionModel instead!\n"\
"Otherwise, make an issue to https://github.com/unslothai/unsloth!",
return FastModel.from_pretrained(
model_name = model_name,
max_seq_length = max_seq_length, # [TODO] No effect
dtype = dtype,
load_in_4bit = load_in_4bit,
load_in_8bit = load_in_8bit,
full_finetuning = full_finetuning,
token = token,
device_map = device_map,
rope_scaling = rope_scaling, # [TODO] No effect
fix_tokenizer = fix_tokenizer, # [TODO] No effect
trust_remote_code = trust_remote_code,
use_gradient_checkpointing = use_gradient_checkpointing,
resize_model_vocab = resize_model_vocab, # [TODO] No effect
revision = revision,
return_logits = False, # Return logits
fullgraph = True, # No graph breaks
use_exact_model_name = use_exact_model_name,
*args, **kwargs,
)
pass
@ -288,6 +326,11 @@ class FastLanguageModel(FastLlamaModel):
pass
if fast_inference:
import platform
if platform.system().lower() == 'windows':
print("Unsloth: vLLM does not work in Windows! Will use Unsloth inference!")
fast_inference = False
pass
from unsloth_zoo.vllm_utils import (
patch_vllm,
vllm_dynamic_quant_supported,
@ -385,9 +428,15 @@ from ..kernels import (
)
from .vision import FastBaseModel
from transformers import (
AutoModelForVision2Seq,
AutoModelForCausalLM,
)
try:
from transformers import AutoModelForImageTextToText
AutoModelForVision2Seq = AutoModelForImageTextToText
except:
from transformers import AutoModelForVision2Seq
pass
class FastModel(FastBaseModel):
@staticmethod
@ -396,6 +445,8 @@ class FastModel(FastBaseModel):
max_seq_length = None, # [TODO] No effect
dtype = None,
load_in_4bit = True,
load_in_8bit = False,
full_finetuning = False,
token = None,
device_map = "sequential",
rope_scaling = None, # [TODO] No effect
@ -417,10 +468,40 @@ class FastModel(FastBaseModel):
if use_gradient_checkpointing == "unsloth":
patch_unsloth_smart_gradient_checkpointing(dtype = dtype)
if full_finetuning and (load_in_4bit or load_in_8bit):
print("Unsloth: You selected full finetuning support, but 4bit / 8bit is enabled - disabling LoRA / QLoRA.")
load_in_4bit = False
load_in_8bit = False
pass
if load_in_4bit and load_in_8bit:
raise RuntimeError(
"Unsloth: Can only load in 4bit or 8bit, not both!\n"\
"Also, we by default set `load_in_4bit = True`.\n"\
"If you want 8bit finetuning, set both `load_in_4bit = False` and `load_in_8bit = True`"
)
if load_in_4bit: pass
elif load_in_8bit: pass
elif not load_in_4bit and not load_in_8bit and not full_finetuning:
print("Unsloth: LoRA, QLoRA and full finetuning all not selected. Switching to QLoRA.")
load_in_4bit = True
pass
old_model_name = model_name
if not use_exact_model_name:
model_name = get_model_name(model_name, load_in_4bit)
# Check versions
LATEST = '\nPlease use transformers via `pip install --no-deps git+https://github.com/huggingface/transformers.git`'
NIGHTLY = '\nPlease use nightly transformers via pip install --upgrade "transformers>=4.49.0"`'
if "pixtral" in model_name.lower() and transformers_version < Version("4.49.0"):
raise RuntimeError("Unsloth: Pixtral only works on transformers >= 4.49.0." + LATEST)
elif "qwen2.5" in model_name.lower() and transformers_version < Version("4.49.0"):
raise RuntimeError("Unsloth: Qwen 2.5 only works on transformers >= 4.49.0." + LATEST)
elif "aya-vision" in model_name.lower() and transformers_version < Version("4.50.0.dev0"):
raise RuntimeError("Unsloth: Aya Vision only works on transformers >= 4.50.0." + NIGHTLY)
pass
if USE_MODELSCOPE and not os.path.exists(model_name):
from modelscope import snapshot_download
model_name = snapshot_download(model_name)
@ -510,7 +591,6 @@ class FastModel(FastBaseModel):
model_config = AutoConfig.from_pretrained(
model_name,
token = token,
revision = revision,
trust_remote_code = trust_remote_code,
)
pass
@ -565,7 +645,7 @@ class FastModel(FastBaseModel):
pass
# Check if VLM
is_vlm = (x.endswith("ForConditionalGeneration") for x in model_config.architectures)
is_vlm = any(x.endswith("ForConditionalGeneration") for x in model_config.architectures)
is_vlm = is_vlm or hasattr(model_config, "vision_config")
auto_model = AutoModelForVision2Seq if is_vlm else AutoModelForCausalLM
@ -574,6 +654,8 @@ class FastModel(FastBaseModel):
max_seq_length = max_seq_length,
dtype = _get_dtype(dtype),
load_in_4bit = load_in_4bit,
load_in_8bit = load_in_8bit,
full_finetuning = full_finetuning,
token = token,
device_map = device_map,
trust_remote_code = trust_remote_code,
@ -581,6 +663,7 @@ class FastModel(FastBaseModel):
model_types = model_types,
tokenizer_name = tokenizer_name,
auto_model = auto_model,
use_gradient_checkpointing = use_gradient_checkpointing,
*args, **kwargs,
)
@ -628,7 +711,7 @@ class FastModel(FastBaseModel):
trust_remote_code = trust_remote_code,
)
# Patch it as well!
model = FastBaseModel.patch_peft_model(model, use_gradient_checkpointing)
model = FastBaseModel.post_patch_model(model, use_gradient_checkpointing)
pass
return model, tokenizer
pass

44
unsloth/models/mapper.py
View file

@ -492,6 +492,18 @@ __INT_TO_FLOAT_MAPPER = \
"unsloth/Qwen2-VL-72B-Instruct",
"Qwen/Qwen2-VL-72B-Instruct",
),
"unsloth/Qwen2-VL-2B-bnb-4bit" : (
"unsloth/Qwen2-VL-2B",
"Qwen/Qwen2-VL-2B",
),
"unsloth/Qwen2-VL-7B-bnb-4bit" : (
"unsloth/Qwen2-VL-7B",
"Qwen/Qwen2-VL-7B",
),
"unsloth/Qwen2-VL-72B-bnb-4bit" : (
"unsloth/Qwen2-VL-72B",
"Qwen/Qwen2-VL-72B",
),
"unsloth/Llama-3.2-11B-Vision-Instruct-unsloth-bnb-4bit" : (
"unsloth/Llama-3.2-11B-Vision-Instruct",
"meta-llama/Llama-3.2-11B-Vision-Instruct",
@ -626,6 +638,38 @@ __INT_TO_FLOAT_MAPPER = \
"Qwen/QwQ-32B",
"unsloth/QwQ-32B-bnb-4bit",
),
"unsloth/gemma-3-1b-it" : (
"unsloth/gemma-3-1b-it",
"google/gemma-3-1b-it",
),
"unsloth/gemma-3-4b-it" : (
"unsloth/gemma-3-4b-it",
"google/gemma-3-4b-it",
),
"unsloth/gemma-3-12b-it" : (
"unsloth/gemma-3-12b-it",
"google/gemma-3-12b-it",
),
"unsloth/gemma-3-27b-it" : (
"unsloth/gemma-3-27b-it",
"google/gemma-3-27b-it",
),
"unsloth/gemma-3-1b-pt" : (
"unsloth/gemma-3-1b-pt",
"google/gemma-3-1b-pt",
),
"unsloth/gemma-3-4b-pt" : (
"unsloth/gemma-3-4b-pt",
"google/gemma-3-4b-pt",
),
"unsloth/gemma-3-12b-pt" : (
"unsloth/gemma-3-12b-pt",
"google/gemma-3-12b-pt",
),
"unsloth/gemma-3-27b-pt" : (
"unsloth/gemma-3-27b-pt",
"google/gemma-3-27b-pt",
),
}
INT_TO_FLOAT_MAPPER = {}

32
unsloth/models/rl.py
View file

@ -106,6 +106,8 @@ import torch
import numpy as np
from contextlib import nullcontext
from torch.nn import functional as F
from transformers import DataCollatorForSeq2Seq, DataCollatorForLanguageModeling
torch_compile_options = {{
"epilogue_fusion" : True,
"max_autotune" : False,
@ -234,6 +236,7 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
mixed_precision = \
"use_bf16 = getattr(args, 'bf16', False)\n"\
"use_fp16 = getattr(args, 'fp16', False)\n"\
"mixed_precision_dtype = os.environ.get('UNSLOTH_MIXED_PRECISION', 'float32')\n"\
"dtype = getattr(model.config, 'torch_dtype', None)\n"\
"if dtype is None: dtype = model.get_input_embeddings().dtype\n"\
"from unsloth_zoo.utils import _get_dtype\n"\
@ -241,10 +244,14 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
"float16 = dtype == torch.float16\n"\
"if float16 and use_bf16: raise TypeError('Unsloth: Model is in float16 precision but you want to use bfloat16 precision. Set fp16 to `True` and bf16 to `False`')\n"\
"if not float16 and use_fp16: raise TypeError('Unsloth: Model is in bfloat16 precision but you want to use float16 precision. Set fp16 to `False` and bf16 to `True`')\n"\
"if not use_bf16 and not use_fp16:\n"\
"if (not use_bf16 and not use_fp16) and mixed_precision_dtype == 'float32':\n"\
" args.fp16 = float16\n"\
" args.bf16 = not float16\n"\
" os.environ['ACCELERATE_MIXED_PRECISION'] = 'fp16' if float16 else 'bf16'\n"
"elif mixed_precision_dtype == 'bfloat16':\n"\
" args.fp16 = False\n"\
" args.bf16 = False\n"\
" os.environ['ACCELERATE_MIXED_PRECISION'] = 'no'\n"
extra_args += mixed_precision
pass
@ -280,7 +287,12 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
"bf16_full_eval = getattr(args, 'bf16_full_eval', False)\n"\
"if args.fp16 and bf16_full_eval: args.bf16_full_eval = False; args.fp16_full_eval = True\n"\
"if args.bf16 and fp16_full_eval: args.bf16_full_eval = True; args.fp16_full_eval = False\n"\
"if not bf16_full_eval and not fp16_full_eval: args.bf16_full_eval = args.bf16; args.fp16_full_eval = args.fp16\n"
"if os.environ.get('UNSLOTH_MIXED_PRECISION', 'float32') == 'bfloat16':\n"\
" args.bf16_full_eval = True\n"\
" args.fp16_full_eval = False\n"\
"elif not bf16_full_eval and not fp16_full_eval:\n"\
" args.bf16_full_eval = args.bf16\n"\
" args.fp16_full_eval = args.fp16\n"
extra_args += eval_changes
pass
@ -327,6 +339,20 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
extra_args += training_check
pass
# Check data collator if it's correct!
if "data_collator" in call_args and "train_dataset" in call_args:
data_collator_check = \
"if isinstance(data_collator, DataCollatorForSeq2Seq) and 'labels' not in train_dataset.column_names:\n"\
" print('Unsloth: Changing data collator to `DataCollatorForLanguageModeling` since `labels` not found.')\n"\
" data_collator = DataCollatorForLanguageModeling("\
"tokenizer = processing_class if 'processing_class' in locals() else tokenizer, mlm = False)\n"\
"elif isinstance(data_collator, DataCollatorForLanguageModeling) and 'labels' in train_dataset.column_names:\n"\
" print('Unsloth: Changing data collator to `DataCollatorForSeq2Seq` since `labels` found.')\n"\
" data_collator = DataCollatorForSeq2Seq("\
"tokenizer = processing_class if 'processing_class' in locals() else tokenizer)\n"
extra_args += data_collator_check
pass
# Check NEFTune
if "model" in call_args:
neftune_check = \
@ -536,7 +562,7 @@ def patch_functions(RLTrainer, trainer_file, RLTrainer_name, all_imports, import
if "args.use_vllm" in init and "model" in init and "args" in init:
# .*? matches first match. .+? matches final match.
replacer = re.findall(
"def __init__\(.*?\).*?\:\n",
r"def __init__\(.*?\).*?\:\n",
init,
flags = re.MULTILINE | re.DOTALL,
)

120
unsloth/models/vision.py
View file

@ -15,17 +15,22 @@
import torch
from transformers import (
BitsAndBytesConfig,
AutoModelForVision2Seq,
AutoProcessor,
AutoTokenizer,
AutoModelForCausalLM,
)
try:
from transformers import AutoModelForImageTextToText
AutoModelForVision2Seq = AutoModelForImageTextToText
except:
from transformers import AutoModelForVision2Seq
pass
from .llama import *
from ..kernels import (
post_patch_loss_function,
)
from ._utils import __version__
from peft import LoraConfig, TaskType, get_peft_model
from peft import LoraConfig, TaskType, get_peft_model as _get_peft_model
from transformers import set_seed as transformers_set_seed
from unsloth_zoo.peft_utils import (
get_peft_regex,
@ -35,6 +40,7 @@ from unsloth_zoo.peft_utils import (
from triton import __version__ as triton_version
from unsloth_zoo.utils import _get_dtype
from unsloth_zoo.patching_utils import patch_model_and_tokenizer
from unsloth_zoo.training_utils import prepare_model_for_training
import types
import functools
@ -52,14 +58,21 @@ def unsloth_base_fast_generate(
dtype = _get_dtype(self.config.torch_dtype)
# Check if VLM
is_vlm = (x.endswith("ForConditionalGeneration") for x in self.config.architectures)
is_vlm = (
x.endswith(("ForConditionalGeneration", "ForVisionText2Text"))
for x in self.config.architectures
)
is_vlm = is_vlm or hasattr(self.config, "vision_config")
# Remove token_type_ids
kwargs.pop("token_type_ids", None)
# VLMs do not allow logits_to_keep
if not is_vlm: kwargs["logits_to_keep"] = 1
if not is_vlm:
kwargs["logits_to_keep"] = 1
else:
kwargs.pop("logits_to_keep", None)
kwargs.pop("num_logits_to_keep", None)
# Check pad_token
model_eos_token_id = getattr(self.config, "eos_token_id", None)
@ -90,12 +103,15 @@ class FastBaseModel:
max_seq_length = None,
dtype = None,
load_in_4bit = True,
load_in_8bit = False,
full_finetuning = False,
token = None,
device_map = "sequential",
trust_remote_code = False,
model_types = None,
tokenizer_name = None,
auto_model = AutoModelForVision2Seq,
use_gradient_checkpointing = "unsloth",
**kwargs,
):
if trust_remote_code:
@ -141,6 +157,14 @@ class FastBaseModel:
assert(dtype == torch.float16 or dtype == torch.bfloat16 or dtype == torch.float32)
bnb_config = None
if full_finetuning and (load_in_4bit or load_in_8bit):
print("Unsloth: You selected full finetuning support, but 4bit / 8bit is enabled - disabling LoRA / QLoRA.")
load_in_4bit = False
load_in_8bit = False
pass
if load_in_4bit and load_in_8bit:
raise RuntimeError("Unsloth: Can only load in 4bit or 8bit, not both!")
if load_in_4bit:
bnb_config = BitsAndBytesConfig(
load_in_4bit = True,
@ -149,6 +173,31 @@ class FastBaseModel:
bnb_4bit_compute_dtype = dtype,
llm_int8_skip_modules = SKIP_QUANTIZATION_MODULES,
)
elif load_in_8bit:
bnb_config = BitsAndBytesConfig(
load_in_8bit = True,
llm_int8_skip_modules = SKIP_QUANTIZATION_MODULES,
)
elif not load_in_4bit and not load_in_8bit and not full_finetuning:
print("Unsloth: LoRA, QLoRA and full finetuning all not selected. Switching to QLoRA.")
load_in_4bit = True
bnb_config = BitsAndBytesConfig(
load_in_4bit = True,
bnb_4bit_use_double_quant = True,
bnb_4bit_quant_type = "nf4",
bnb_4bit_compute_dtype = dtype,
llm_int8_skip_modules = SKIP_QUANTIZATION_MODULES,
)
pass
if full_finetuning:
os.environ["UNSLOTH_ENABLE_FULL_FINETUNING"] = "1"
if dtype == torch.bfloat16:
print("Unsloth: Using bfloat16 full finetuning which cuts memory usage by 50%.")
else:
print("Unsloth: Float16 full finetuning uses more memory since we upcast weights to float32.")
else:
os.environ["UNSLOTH_ENABLE_FULL_FINETUNING"] = "0"
pass
kwargs.pop("attn_implementation", None); # No need since we auto call it
@ -204,23 +253,37 @@ class FastBaseModel:
# Save tokenizer for inference purposes
tokenizer.padding_side = "left" # Force inference
tokenizer.tokenizer.padding_side = "left" # Force inference
if hasattr(tokenizer, "tokenizer"):
tokenizer.tokenizer.padding_side = "left" # Force inference
m = model
while hasattr(m, "model"):
m.max_seq_length = max_seq_length
m._saved_temp_tokenizer = tokenizer
# Also set is_loaded_in_8bit to disable incorrect DDP
m.is_loaded_in_8bit = True
m.is_loaded_in_8bit = True if not full_finetuning else False
m = m.model
pass
m.max_seq_length = max_seq_length
m._saved_temp_tokenizer = tokenizer
# Also set is_loaded_in_8bit to disable incorrect DDP
m.is_loaded_in_8bit = True
m.is_loaded_in_8bit = True if not full_finetuning else False
# Patch generate
if model.generate.__name__ != "unsloth_base_fast_generate":
model._old_generate = model.generate
unsloth_base_fast_generate.__doc__ = model._old_generate.__doc__
model.generate = types.MethodType(unsloth_base_fast_generate, model)
# Post patches
model = FastBaseModel.post_patch_model(
model,
use_gradient_checkpointing = use_gradient_checkpointing,
)
# Clear deleted GPU items
for _ in range(3):
gc.collect()
torch.cuda.empty_cache()
pass
return model, tokenizer
pass
@ -249,6 +312,10 @@ class FastBaseModel:
temporary_location = "_unsloth_temporary_saved_buffers",
**kwargs,
):
if os.environ.get("UNSLOTH_ENABLE_FULL_FINETUNING", "0") == "1":
print("Unsloth: Full finetuning is enabled, so .get_peft_model has no effect")
return model
pass
transformers_set_seed(random_state)
if type(r) is not int:
@ -282,7 +349,7 @@ class FastBaseModel:
gc.collect()
torch.cuda.empty_cache()
pass
max_seq_length = model.max_seq_length
lora_config = LoraConfig(
r = r,
lora_alpha = lora_alpha,
@ -295,11 +362,12 @@ class FastBaseModel:
model,
use_gradient_checkpointing = use_gradient_checkpointing,
)
model = get_peft_model(model, lora_config)
model = _get_peft_model(model, lora_config)
# Enable gradients on modules which are trainable
requires_grad_for_gradient_checkpointing(model)
model = FastBaseModel.patch_peft_model(model, use_gradient_checkpointing)
model = FastBaseModel.post_patch_model(model, use_gradient_checkpointing)
model.max_seq_length = max_seq_length
# Clear deleted GPU items
for _ in range(3):
@ -316,20 +384,26 @@ class FastBaseModel:
@staticmethod
def patch_peft_model(
def post_patch_model(
model,
use_gradient_checkpointing = True,
):
if not isinstance(model, PeftModelForCausalLM):
raise TypeError(
"Unsloth: Your model needs to call `.get_peft_model` first!"
)
pass
full_finetuning = os.environ.get("UNSLOTH_ENABLE_FULL_FINETUNING", "0") == "1"
model = prepare_model_for_kbit_training(
float32_mixed_precision = True
if _get_dtype(model.config.torch_dtype) == torch.bfloat16:
# Use bfloat16 precision for full finetuning
float32_mixed_precision = False
model = prepare_model_for_training(
model,
use_gradient_checkpointing = use_gradient_checkpointing,
use_reentrant = True,
use_reentrant = True,
full_finetuning = full_finetuning,
train_layernorms = full_finetuning,
train_embedding = full_finetuning,
train_lm_head = full_finetuning,
float32_mixed_precision = float32_mixed_precision,
)
from transformers.trainer import Trainer
@ -347,17 +421,19 @@ class FastBaseModel:
m = model
while hasattr(m, "model"):
if hasattr(m, "_saved_temp_tokenizer"):
m._saved_temp_tokenizer.tokenizer.padding_side = "right"
if hasattr(m._saved_temp_tokenizer, "tokenizer"):
m._saved_temp_tokenizer.tokenizer.padding_side = "right"
pass
# Also set is_loaded_in_8bit to disable incorrect DDP
m.is_loaded_in_8bit = True
m.is_loaded_in_8bit = True if not full_finetuning else False
m = m.model
pass
if hasattr(m, "_saved_temp_tokenizer"):
m._saved_temp_tokenizer.tokenizer.padding_side = "right"
if hasattr(m._saved_temp_tokenizer, "tokenizer"):
m._saved_temp_tokenizer.tokenizer.padding_side = "right"
pass
# Also set is_loaded_in_8bit to disable incorrect DDP
m.is_loaded_in_8bit = True
m.is_loaded_in_8bit = True if not full_finetuning else False
# Clear deleted GPU items
for _ in range(3):

2
unsloth/tokenizer_utils.py
View file

@ -945,7 +945,7 @@ def patch_sft_trainer_tokenizer():
if replacer is None:
# .*? matches first match. .+? matches final match.
replacer = re.findall(
f"def {function_name}\(.*?\).*?\:\n",
f"def {function_name}" + r"\(.*?\).*?\:\n",
function,
flags = re.MULTILINE | re.DOTALL,
)