ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Vision support (#1315)

Browse source 
* Fix pad token

* Update llama.py

* Typo

* ignored labels

* Revert "ignored labels"

This reverts commit 4b25138ac7.

* More patching

* Update _utils.py

* Update _utils.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Feat/all tmp (#1219)

* Update save.py

Check whether path is in /tmp dir for Kaggle environment

* Update save.py

Move temporary_location to /tmp in Kaggle

* Enhance Kaggle environment support in save and tokenizer utilities

---------

Co-authored-by: dendarrion <37800703+dendarrion@users.noreply.github.com>
Co-authored-by: Erland366 <erland.pg366@gmail.com>

* Bug fixes

* Update pyproject.toml

* Update _utils.py

* Update __init__.py

* Update __init__.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Tied weights

* Revert "Tied weights"

This reverts commit 820cd4efef.

* Tied weights

* Utils

* CE Loss patching

* Update __init__.py

* Update __init__.py

* Patching

* Update cross_entropy_loss.py

* CE Loss

* Update _utils.py

* Update _utils.py

* CE Loss

* Update _utils.py

* Update _utils.py

* Layernorm

* Update _utils.py

* Update _utils.py

* Post patch

* Update _utils.py

* Update llama.py

* Update _utils.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* typing

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* int64

* Update _utils.py

* Update cross_entropy_loss.py

* constexpr

* constexpr

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* CE

* Update cross_entropy_loss.py

* Update _utils.py

* Update llama.py

* Update _utils.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* Update utils.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* Update rms_layernorm.py

* typing

* Update rope_embedding.py

* types

* Disable compiling

* Update _utils.py

* Update _utils.py

* Forward hook

* Update _utils.py

* Update llama.py

* Update _utils.py

* Update llama.py

* Update llama.py

* Update _utils.py

* Update pyproject.toml

* Update _utils.py

* Update llama.py

* CE Loss

* Update cross_entropy_loss.py

* Update _utils.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update cross_entropy_loss.py

* Update llama.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Fix: cast logits to float32 in cross_entropy_forward to prevent errors (#1254)

* Fix: cast logits to float32 in cross_entropy_forward to prevent errors

* Update cross_entropy_loss.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Throw error when inferencing longer than max_popsition_embeddings (#1236)

* Throw error when inferencing longer than max_popsition_embeddings without rope scaling

* Update llama.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* CLI now handles user input strings for dtype correctly (#1235)

Co-authored-by: root <root@ieeres.chu.cam.ac.uk>

* Update flex_attention.py

* Update _utils.py

* Update _utils.py

* Update flex_attention.py

* Update flex_attention.py

* Update loader.py

* Update loader.py

* Update flex_attention.py

* Update flex_attention.py

* Update flex_attention.py

* Update flex_attention.py

* Update _utils.py

* Update cross_entropy_loss.py

* Update _utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* triton_cast

* Update utils.py

* Qwen 2.5 Coder

* Fix/export mistral (#1281)

* Enhance install_python_non_blocking to handle protobuf installation and process management

* Revert "Enhance install_python_non_blocking to handle protobuf installation and process management"

This reverts commit a3b796a05841fb8d93c652c845591e12cf81ea93.

* Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION to 'python' to address issue #1266

* Revert "Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION to 'python' to address issue #1266"

This reverts commit f00fbf5eac7ad4f5d48c70b98d770255d1a9ef58.

* Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION to 'python' to address issue #1266

* Update __init__.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* DOC Update - Update README.md with os.environ in example (#1269)

* Update README.md with os.environ in example

Added OS Environ in example to avoid device conflicts , for a user at least in jupyter notebook this allows to select GPU in a multi GPU setup. 
As currently the  unsloth init checks all GPU's and takes the first in the order which can be a issue when some GPU's are in use and the list still shows them. So to manually avoid this, this os config is required.
Small change but a bit time saver for those who straight away copies the tutorials

* Update README.md

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* fix/get_chat_template (#1246)

* Refactor `get_chat_template` to now support system message instead. It supposed to fix ollama tokenizer chattemplate to

* Remove type hinting

* Update chat_templates.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* fix/sft-trainer (#1276)

* Add patch for SFTTrainer to maintain backward compatibility with TRL changes

* Update trainer.py

* Update trainer.py

* Refactor trainer patch to maintain backward compatibility with TRL changes

* Update trainer.py

* Refactor trainer.py to exclude non-convertible trainers from backward compatibility patch

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update __init__.py

* Update trainer.py

* Update trainer.py

* Update trainer.py

* Update tokenizer_utils.py

* Update llama.py

* Fix #853

* fix/sfttrainer-compatibility (#1293)

* Refactor trainer.py to import SFTConfig directly and update UnslothTrainingArguments class inheritance

* Update trainer.py

* Update trainer.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update rms_layernorm.py

* Update rms_layernorm.py

* Gemma

* Update rms_layernorm.py

* Update gemma2.py

* Cut Cross Entropy

* Update llama.py

* Cut Cross Entropy

* Update llama.py

* Update llama.py

* Update llama.py

* Update __init__.py

* Update __init__.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update mapper.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* patch_fast_lora

* vision

* Update fast_lora.py

* Update _utils.py

* Update _utils.py

* Vision

* Update trainer.py

* Update save.py

* FastBaseVisionModel

* Update loader_utils.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update _utils.py

* tokenizer_name

* Update loader.py

* Update vision.py

* Update save.py

* Update save.py

* Update vision.py

* Update vision.py

* Update vision.py

* Update vision.py

* Update vision.py

* Update vision.py

* Update _utils.py

---------

Co-authored-by: dendarrion <37800703+dendarrion@users.noreply.github.com>
Co-authored-by: Erland366 <erland.pg366@gmail.com>
Co-authored-by: Edd <68678137+Erland366@users.noreply.github.com>
Co-authored-by: Datta Nimmaturi <datta.nimmaturi@nutanix.com>
Co-authored-by: Edwin Fennell <edwinfennell1@gmail.com>
Co-authored-by: root <root@ieeres.chu.cam.ac.uk>
Co-authored-by: Uday Girish Maradana <einsteingirish@gmail.com>
This commit is contained in:
Daniel Han 2024-11-21 05:01:44 -08:00 • committed by GitHub
commit ddf118a8fc
14 changed files with 1083 additions and 519 deletions
Download patch file Download diff file Expand all files Collapse all files

2
unsloth/__init__.py
View file

@ -55,7 +55,7 @@ else:
pass pass
# Reduce VRAM usage by reducing fragmentation # Reduce VRAM usage by reducing fragmentation
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True" os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True,roundup_power2_divisions:[64:128,256:64,>:32]"
# Hugging Face Hub faster downloads # Hugging Face Hub faster downloads
if "HF_HUB_ENABLE_HF_TRANSFER" not in os.environ: if "HF_HUB_ENABLE_HF_TRANSFER" not in os.environ:

1
unsloth/kernels/__init__.py
View file

@ -42,6 +42,7 @@ from .fast_lora import (
apply_lora_mlp_geglu_approx, apply_lora_mlp_geglu_approx,
apply_lora_qkv, apply_lora_qkv,
apply_lora_o, apply_lora_o,
fast_lora_forward,
) )
from .utils import fast_dequantize, fast_gemv, QUANT_STATE, fast_linear_forward, matmul_lora from .utils import fast_dequantize, fast_gemv, QUANT_STATE, fast_linear_forward, matmul_lora

78
unsloth/kernels/fast_lora.py
View file

@ -410,3 +410,81 @@ def apply_lora_o(self, X):
O = LoRA_W.apply(X, OW, OW_quant, OA, OB, OS) O = LoRA_W.apply(X, OW, OW_quant, OA, OB, OS)
return O return O
pass pass
IDENTITY_DROPOUT = torch.nn.Identity
@torch._disable_dynamo
def fast_lora_forward(self, x: torch.Tensor, *args, **kwargs) -> torch.Tensor:
raise NotImplementedError(
"Unsloth: Currently not supported yet - reshaping done incorrectly"
)
self._check_forward_args(x, *args, **kwargs)
adapter_names = kwargs.pop("adapter_names", None)
if self.disable_adapters:
if self.merged:
self.unmerge()
result = self.base_layer(x, *args, **kwargs)
elif adapter_names is not None:
result = self._mixed_batch_forward(x, *args, adapter_names=adapter_names, **kwargs)
elif self.merged:
result = self.base_layer(x, *args, **kwargs)
else:
# Fastpath
if len(self.active_adapters) == 1:
active_adapter = self.active_adapters[0]
if active_adapter not in self.lora_A.keys(): return self.base_layer(x, *args, **kwargs)
dropout = self.lora_dropout[active_adapter]
if isinstance(dropout, IDENTITY_DROPOUT) and not self.use_dora[active_adapter]:
lora_A = self.lora_A[active_adapter].weight
lora_B = self.lora_B[active_adapter].weight
scaling = self.scaling[active_adapter]
W = self.base_layer.weight
return LoRA_W.apply(x, W, QUANT_STATE(W), lora_A, lora_B, scaling)
pass
pass
result = self.base_layer(x, *args, **kwargs)
# As per Tim Dettmers, for 4bit, we need to defensively clone here.
# The reason is that in some cases, an error can occur that backprop
# does not work on a manipulated view. This issue may be solved with
# newer PyTorch versions but this would need extensive testing to be
# sure.
result = result.clone()
for active_adapter in self.active_adapters:
if active_adapter not in self.lora_A.keys():
continue
lora_A = self.lora_A[active_adapter]
lora_B = self.lora_B[active_adapter]
dropout = self.lora_dropout[active_adapter]
scaling = self.scaling[active_adapter]
requires_conversion = not torch.is_autocast_enabled()
if requires_conversion:
expected_dtype = result.dtype
x = x.to(lora_A.weight.dtype)
if not self.use_dora[active_adapter]:
result = result + lora_B(lora_A(dropout(x))) * scaling
else:
if isinstance(dropout, torch.nn.Identity) or not self.training:
base_result = result
else:
x = dropout(x)
base_result = None
result = result + self.lora_magnitude_vector[active_adapter](
x,
lora_A=lora_A,
lora_B=lora_B,
scaling=scaling,
base_layer=self.get_base_layer(),
base_result=base_result,
)
if requires_conversion:
result = result.to(expected_dtype)
return result
pass

10
unsloth/kernels/rms_layernorm.py
View file

@ -57,6 +57,7 @@ pass
@triton.jit @triton.jit
def _rms_layernorm_backward( def _rms_layernorm_backward(
dY, dY_row_stride, dY, dY_row_stride,
dX, dX_row_stride,
X, X_row_stride, X, X_row_stride,
W, W_row_stride, W, W_row_stride,
r, r_row_stride, r, r_row_stride,
@ -78,6 +79,9 @@ def _rms_layernorm_backward(
X += row_idx * X_row_stride X += row_idx * X_row_stride
r += row_idx * r_row_stride r += row_idx * r_row_stride
if GEMMA: dX += row_idx * dY_row_stride
else: dX = dY
dY_row = tl.load(dY + col_offsets, mask = mask, other = 0).to(tl.float32) dY_row = tl.load(dY + col_offsets, mask = mask, other = 0).to(tl.float32)
X_row = tl.load(X + col_offsets, mask = mask, other = 0).to(tl.float32) X_row = tl.load(X + col_offsets, mask = mask, other = 0).to(tl.float32)
W_row = tl.load(W + col_offsets, mask = mask, other = 0).to(tl.float32) W_row = tl.load(W + col_offsets, mask = mask, other = 0).to(tl.float32)
@ -91,7 +95,7 @@ def _rms_layernorm_backward(
rowsum_dY_normed = tl.sum(dY_W * normed, axis = 0) rowsum_dY_normed = tl.sum(dY_W * normed, axis = 0)
output = inv_var/n_cols * (n_cols*dY_W - normed*rowsum_dY_normed) output = inv_var/n_cols * (n_cols*dY_W - normed*rowsum_dY_normed)
tl.store(dY + col_offsets, output, mask = mask) tl.store(dX + col_offsets, output, mask = mask)
pass pass
@ -172,9 +176,11 @@ class Fast_RMS_Layernorm(torch.autograd.Function):
n_cols : int n_cols : int
n_rows, n_cols = dY.shape n_rows, n_cols = dY.shape
# dW = X # dW = X
dX = torch.empty_like(dY, device = "cuda:0") if ctx.GEMMA else dY
_rms_layernorm_backward[(n_rows,)]( _rms_layernorm_backward[(n_rows,)](
dY, dY.stride(0), dY, dY.stride(0),
dX, dX.stride(0),
X, X .stride(0), X, X .stride(0),
W, W .stride(0), W, W .stride(0),
r, r .stride(0), r, r .stride(0),
@ -184,7 +190,7 @@ class Fast_RMS_Layernorm(torch.autograd.Function):
BLOCK_SIZE = ctx.BLOCK_SIZE, BLOCK_SIZE = ctx.BLOCK_SIZE,
num_warps = ctx.num_warps, num_warps = ctx.num_warps,
) )
dX = dY.view(*shape) dX = dX.view(*shape)
return dX, None, None, None return dX, None, None, None
pass pass
pass pass

2
unsloth/models/__init__.py
View file

@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from .loader import FastLanguageModel from .loader import FastLanguageModel, FastVisionModel
from .llama import FastLlamaModel from .llama import FastLlamaModel
from .mistral import FastMistralModel from .mistral import FastMistralModel
from .qwen2 import FastQwen2Model from .qwen2 import FastQwen2Model

211
unsloth/models/_utils.py
View file

@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
__version__ = "2024.11.7" __version__ = "2024.11.8"
__all__ = [ __all__ = [
"prepare_model_for_kbit_training", "prepare_model_for_kbit_training",
@ -52,6 +52,17 @@ __all__ = [
"unpatch_unsloth_gradient_checkpointing", "unpatch_unsloth_gradient_checkpointing",
"patch_gradient_checkpointing", "patch_gradient_checkpointing",
"unpatch_gradient_checkpointing", "unpatch_gradient_checkpointing",
"HAS_CUT_CROSS_ENTROPY",
"fused_linear_cross_entropy",
"patch_unsloth_smart_gradient_checkpointing",
"unpatch_unsloth_smart_gradient_checkpointing",
"create_gradient_checkpointing_buffer",
"patch_compiled_autograd",
"process_vision_info",
"unsloth_compile_transformers",
"patch_fast_lora",
] ]
import torch import torch
@ -70,6 +81,7 @@ from unsloth_zoo.patching_utils import (
patch_layernorm, patch_layernorm,
patch_torch_compile, patch_torch_compile,
patch_model_and_tokenizer, patch_model_and_tokenizer,
patch_compiled_autograd,
) )
from unsloth_zoo.gradient_checkpointing import ( from unsloth_zoo.gradient_checkpointing import (
Unsloth_Offloaded_Gradient_Checkpointer, Unsloth_Offloaded_Gradient_Checkpointer,
@ -81,6 +93,21 @@ from unsloth_zoo.gradient_checkpointing import (
unsloth_gradient_checkpoint, unsloth_gradient_checkpoint,
patch_gradient_checkpointing, patch_gradient_checkpointing,
unpatch_gradient_checkpointing, unpatch_gradient_checkpointing,
patch_unsloth_smart_gradient_checkpointing,
unpatch_unsloth_smart_gradient_checkpointing,
create_gradient_checkpointing_buffer,
)
from unsloth_zoo.loss_utils import (
HAS_CUT_CROSS_ENTROPY,
fused_linear_cross_entropy,
)
from unsloth_zoo.vision_utils import (
process_vision_info,
)
from unsloth_zoo.compiler import (
get_transformers_model_type,
unsloth_compile_transformers as _unsloth_compile_transformers,
) )
# ============================================= # =============================================
@ -120,6 +147,22 @@ try:
except: except:
pass pass
# The model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function.
try:
from accelerate.utils.modeling import logger as accelerate_utils_modeling_logger
accelerate_utils_modeling_logger.addFilter(HideLoggingMessage("The model weights are not tied"))
del accelerate_utils_modeling_logger
except:
pass
# Setting `pad_token_id` to `eos_token_id`
try:
from transformers.generation.utils import logger as transformers_generation_utils_logger
transformers_generation_utils_logger.addFilter(HideLoggingMessage("Setting `pad_token_id` to `eos_token_id`"))
del transformers_generation_utils_logger
except:
pass
# ============================================= # =============================================
# ============================================= # =============================================
@ -282,54 +325,60 @@ from transformers.models.llama.modeling_llama import logger
# ============================================= # =============================================
# Get Xformers # Get Xformers
from xformers import __version__ as xformers_version
# Temporarily disable 0.0.27 and higher - inference issues
if False: #Version(xformers_version) >= Version("0.0.27"):
raise ImportError(
"Unsloth: If you are in Colab, we updated the top cell install instructions - please change it to below "\
"then press Disconnect Runtime and then Restart it.\n"\
"\n"\
"%%capture\n"
"# Installs Unsloth, Xformers (Flash Attention) and all other packages!\n"
'!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"\n'
'!pip install --no-deps "xformers<=0.0.27" trl peft accelerate bitsandbytes\n'\
'\n'\
f"Otherwise in local machines, your xformers version of {xformers_version} is too new.\n"\
'Please downgrade xformers via `pip install --force-reinstall "xformers<=0.0.27"'
)
pass
if Version(torch_version) < Version("2.2.0") and Version(xformers_version) >= Version("0.0.24"):
raise ImportError(
f"Unsloth: You have torch = {torch_version} but xformers = {xformers_version}.\n"\
f"Please install xformers < 0.0.24 for torch = {torch_version}."
)
elif Version(torch_version) < Version("2.3.0") and Version(xformers_version) >= Version("0.0.26"):
raise ImportError(
f"Unsloth: You have torch = {torch_version} but xformers = {xformers_version}.\n"\
f"Please install xformers < 0.0.26 for torch = {torch_version}."
)
elif Version(torch_version) < Version("2.4.0") and Version(xformers_version) > Version("0.0.27"):
raise ImportError(
f"Unsloth: You have torch = {torch_version} but xformers = {xformers_version}.\n"\
f"Please install xformers <= 0.0.27 for torch = {torch_version}."
)
pass
from xformers._cpp_lib import _register_extensions
try: try:
_register_extensions() # Check if C++ modules are loaded correctly from xformers import __version__ as xformers_version
except Exception as error: # Temporarily disable 0.0.27 and higher - inference issues
raise ImportError( if False: #Version(xformers_version) >= Version("0.0.27"):
"Unsloth: Xformers was not installed correctly.\n"\ raise ImportError(
"Please install xformers separately first.\n"\ "Unsloth: If you are in Colab, we updated the top cell install instructions - please change it to below "\
"Then confirm if it's correctly installed by running:\n"\ "then press Disconnect Runtime and then Restart it.\n"\
"python -m xformers.info\n\n" "\n"\
"Longer error message:\n" + str(error) "%%capture\n"
) "# Installs Unsloth, Xformers (Flash Attention) and all other packages!\n"
'!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"\n'
'!pip install --no-deps "xformers<=0.0.27" trl peft accelerate bitsandbytes\n'\
'\n'\
f"Otherwise in local machines, your xformers version of {xformers_version} is too new.\n"\
'Please downgrade xformers via `pip install --force-reinstall "xformers<=0.0.27"'
)
pass
if Version(torch_version) < Version("2.2.0") and Version(xformers_version) >= Version("0.0.24"):
raise ImportError(
f"Unsloth: You have torch = {torch_version} but xformers = {xformers_version}.\n"\
f"Please install xformers < 0.0.24 for torch = {torch_version}."
)
elif Version(torch_version) < Version("2.3.0") and Version(xformers_version) >= Version("0.0.26"):
raise ImportError(
f"Unsloth: You have torch = {torch_version} but xformers = {xformers_version}.\n"\
f"Please install xformers < 0.0.26 for torch = {torch_version}."
)
elif Version(torch_version) < Version("2.4.0") and Version(xformers_version) > Version("0.0.27"):
raise ImportError(
f"Unsloth: You have torch = {torch_version} but xformers = {xformers_version}.\n"\
f"Please install xformers <= 0.0.27 for torch = {torch_version}."
)
pass
from xformers._cpp_lib import _register_extensions
try:
_register_extensions() # Check if C++ modules are loaded correctly
except Exception as error:
raise ImportError(
"Unsloth: Xformers was not installed correctly.\n"\
"Please install xformers separately first.\n"\
"Then confirm if it's correctly installed by running:\n"\
"python -m xformers.info\n\n"
"Longer error message:\n" + str(error)
)
pass
import xformers.ops.fmha as xformers
xformers_attention = xformers.memory_efficient_attention
except:
xformers = None
xformers_attention = None
xformers_version = None
pass pass
import xformers.ops.fmha as xformers
xformers_attention = xformers.memory_efficient_attention
# Check TRL version # Check TRL version
from trl import __version__ as trl_version from trl import __version__ as trl_version
@ -658,7 +707,7 @@ BitsAndBytesConfig__init__ = BitsAndBytesConfig__init__.replace(
) )
def _prepare_backend( def _prepare_backend(
self, cpu: bool = False, sagemaker_dp = False, backend: str = None, self, cpu = False, sagemaker_dp = False, backend: str = None,
) -> tuple[str, DistributedType]: ) -> tuple[str, DistributedType]:
return None, DistributedType.NO return None, DistributedType.NO
pass pass
@ -1047,3 +1096,69 @@ def patch_tokenizer(model, tokenizer):
model.config.update({"unsloth_version" : __version__}) model.config.update({"unsloth_version" : __version__})
return model, tokenizer return model, tokenizer
pass pass
def patch_fast_lora():
import peft.tuners.lora.bnb
peft.tuners.lora.bnb.Linear4bit.forward = fast_lora_forward
pass
def unsloth_compile_transformers(
model_name,
token = None,
revision = None,
trust_remote_code = False,
sdpa_dynamic_mask = True,
sdpa_bool_masks = True,
sdpa_gqa_replace = True,
sdpa_dynamic_compile = True,
compile_attention = True,
disable_causal_masks = True,
compile_torch_modules = True,
compile_custom_modules = True,
compile_function_calls = True,
fuse_lm_head = True,
gradient_checkpointing = True,
manual_replacements = True,
epilogue_fusion = True,
max_autotune = False,
shape_padding = True,
cudagraphs = False,
debug = False,
import_from_cache = False,
disable = False,
):
if disable: return
model_types = get_transformers_model_type(
model_name = model_name,
token = token,
revision = revision,
trust_remote_code = trust_remote_code,
)
for model_type in model_types:
_unsloth_compile_transformers(
model_type,
sdpa_dynamic_mask = sdpa_dynamic_mask,
sdpa_bool_masks = sdpa_bool_masks,
sdpa_gqa_replace = sdpa_gqa_replace,
sdpa_dynamic_compile = sdpa_dynamic_compile,
compile_attention = compile_attention,
disable_causal_masks = disable_causal_masks,
compile_torch_modules = compile_torch_modules,
compile_custom_modules = compile_custom_modules,
compile_function_calls = compile_function_calls,
fuse_lm_head = fuse_lm_head,
gradient_checkpointing = gradient_checkpointing,
manual_replacements = manual_replacements,
epilogue_fusion = epilogue_fusion,
max_autotune = max_autotune,
shape_padding = shape_padding,
cudagraphs = cudagraphs,
debug = debug,
import_from_cache = import_from_cache,
disable = disable,
)
pass
return model_types
pass

11
unsloth/models/gemma2.py
View file

@ -60,8 +60,7 @@ if HAS_FLASH_ATTENTION_SOFTCAPPING:
from flash_attn import flash_attn_func from flash_attn import flash_attn_func
# [TODO] We must randomnly use torch.compile? # [TODO] We must randomnly use torch.compile?
# I checked the gradients and formulas and I'm sure it's correct. # Gemma 2 uses double RMS Layernorms, so the backward passes should not overwrite the gradients!
# I'm stumped :(
@torch.compile(fullgraph = False, dynamic = True, options = torch_compile_options) @torch.compile(fullgraph = False, dynamic = True, options = torch_compile_options)
def fast_rms_layernorm_gemma2_compiled(layernorm, X, gemma = True): def fast_rms_layernorm_gemma2_compiled(layernorm, X, gemma = True):
old_dtype = X.dtype old_dtype = X.dtype
@ -207,7 +206,7 @@ def Gemma2DecoderLayer_fast_forward(
hidden_states += residual hidden_states += residual
else: else:
residual = hidden_states residual = hidden_states
hidden_states = fast_rms_layernorm_gemma2_compiled(self.input_layernorm, hidden_states, gemma = True) hidden_states = fast_rms_layernorm(self.input_layernorm, hidden_states, gemma = True)
hidden_states, self_attn_weights, present_key_value = self.self_attn( hidden_states, self_attn_weights, present_key_value = self.self_attn(
hidden_states=hidden_states, hidden_states=hidden_states,
causal_mask=causal_mask, causal_mask=causal_mask,
@ -218,14 +217,14 @@ def Gemma2DecoderLayer_fast_forward(
use_cache=use_cache, use_cache=use_cache,
padding_mask=padding_mask, padding_mask=padding_mask,
) )
hidden_states = fast_rms_layernorm_gemma2_compiled(self.post_attention_layernorm, hidden_states, gemma = True) hidden_states = fast_rms_layernorm(self.post_attention_layernorm, hidden_states, gemma = True)
hidden_states = residual + hidden_states hidden_states = residual + hidden_states
# Fully Connected # Fully Connected
residual = hidden_states residual = hidden_states
hidden_states = fast_rms_layernorm_gemma2_compiled(self. pre_feedforward_layernorm, hidden_states, gemma = True) hidden_states = fast_rms_layernorm(self. pre_feedforward_layernorm, hidden_states, gemma = True)
hidden_states = self.mlp(hidden_states) hidden_states = self.mlp(hidden_states)
hidden_states = fast_rms_layernorm_gemma2_compiled(self.post_feedforward_layernorm, hidden_states, gemma = True) hidden_states = fast_rms_layernorm(self.post_feedforward_layernorm, hidden_states, gemma = True)
hidden_states = residual + hidden_states hidden_states = residual + hidden_states
pass pass

50
unsloth/models/llama.py
View file

@ -719,25 +719,33 @@ def LlamaModel_fast_forward(
pass pass
# Gemma2 has alternating SWA and global attn # Gemma2 has alternating SWA and global attn
use_static_mask = True
dynamic_SWA_mask = None
dynamic_GA_mask = None
if IS_GEMMA2: if IS_GEMMA2:
if HAS_FLASH_ATTENTION_SOFTCAPPING and attention_mask is None: if HAS_FLASH_ATTENTION_SOFTCAPPING and attention_mask is None:
self.SWA_mask = True self.SWA_mask = True
self.GA_mask = False self.GA_mask = False
elif attention_mask is not None: elif attention_mask is not None:
self.SWA_mask = _prepare_4d_causal_attention_mask_for_sdpa(
# Fixes https://github.com/unslothai/unsloth/issues/853
# Unsloth needs a 2D mask, not a [2, 1, n, n] mask!
dynamic_SWA_mask = _prepare_4d_causal_attention_mask_for_sdpa(
attention_mask, attention_mask,
(batch_size, seq_length), (batch_size, seq_length),
inputs_embeds, inputs_embeds,
past_key_values_length, past_key_values_length,
sliding_window = self.config.sliding_window, sliding_window = self.config.sliding_window,
) )[0][0]
self.GA_mask = _prepare_4d_causal_attention_mask_for_sdpa( dynamic_GA_mask = _prepare_4d_causal_attention_mask_for_sdpa(
attention_mask, attention_mask,
(batch_size, seq_length), (batch_size, seq_length),
inputs_embeds, inputs_embeds,
past_key_values_length, past_key_values_length,
sliding_window = None, sliding_window = None,
) )[0][0]
use_static_mask = False
elif not hasattr(self, "SWA_mask"): elif not hasattr(self, "SWA_mask"):
if HAS_FLEX_ATTENTION: if HAS_FLEX_ATTENTION:
# Use Flex Attention instead! # Use Flex Attention instead!
@ -772,7 +780,12 @@ def LlamaModel_fast_forward(
past_key_value = past_key_values[idx] if past_key_values is not None else None past_key_value = past_key_values[idx] if past_key_values is not None else None
mask = causal_mask mask = causal_mask
if IS_GEMMA2: mask = self.SWA_mask if (idx % 2 == 0) else self.GA_mask if IS_GEMMA2:
if (idx % 2 == 0):
mask = self.SWA_mask if use_static_mask else dynamic_SWA_mask
else:
mask = self. GA_mask if use_static_mask else dynamic_GA_mask
pass
if offloaded_gradient_checkpointing: if offloaded_gradient_checkpointing:
hidden_states = Unsloth_Offloaded_Gradient_Checkpointer.apply( hidden_states = Unsloth_Offloaded_Gradient_Checkpointer.apply(
@ -955,14 +968,39 @@ def CausalLM_fast_forward(fast_forward_inference):
) )
pass pass
hidden_states = outputs[0] hidden_states = outputs[0]
bsz, q_len, hd = hidden_states.shape bsz, q_len, hd = hidden_states.shape
lm_head = self.lm_head.weight lm_head = self.lm_head.weight
logit_softcapping = getattr(self.config, "final_logit_softcapping", 0)
logit_scaling = getattr(self.config, "logit_scale", 0)
if bsz == 1 and q_len == 1: if bsz == 1 and q_len == 1:
logits = torch.mv(lm_head, hidden_states.ravel().to(lm_head.dtype)) logits = torch.mv(lm_head, hidden_states.ravel().to(lm_head.dtype))
logits = logits.unsqueeze(0).unsqueeze(0) logits = logits.unsqueeze(0).unsqueeze(0)
elif num_logits_to_keep != 0: elif num_logits_to_keep != 0:
logits = self.lm_head(hidden_states[:, -num_logits_to_keep:, :].to(lm_head.dtype)) logits = self.lm_head(hidden_states[:, -num_logits_to_keep:, :].to(lm_head.dtype))
else: else:
if HAS_CUT_CROSS_ENTROPY and labels is not None:
n_items = kwargs.get("num_items_in_batch", None) or kwargs.get("n_items", None)
loss = fused_linear_cross_entropy(
hidden_states = hidden_states,
lm_weight = lm_head,
labels = labels,
num_items_in_batch = n_items,
logit_softcapping = logit_softcapping,
)
if not return_dict:
output = (logits,) + outputs[1:]
return (loss,) + output if loss is not None else output
return CausalLMOutputWithPast(
loss=loss,
logits=None,
past_key_values=outputs.past_key_values,
hidden_states=outputs.hidden_states,
attentions=outputs.attentions,
)
pass
logits = self.lm_head(hidden_states.to(lm_head.dtype)) logits = self.lm_head(hidden_states.to(lm_head.dtype))
pass pass
@ -974,8 +1012,6 @@ def CausalLM_fast_forward(fast_forward_inference):
pass pass
loss = None loss = None
logit_softcapping = getattr(self.config, "final_logit_softcapping", 0)
logit_scaling = getattr(self.config, "logit_scale", 0)
if labels is not None: if labels is not None:
shift_logits = logits shift_logits = logits
if not hasattr(self, "extra_ignored_labels"): if not hasattr(self, "extra_ignored_labels"):

340
unsloth/models/loader.py
View file

@ -20,8 +20,8 @@ from .cohere import FastCohereModel
from transformers import AutoConfig from transformers import AutoConfig
from transformers import __version__ as transformers_version from transformers import __version__ as transformers_version
from peft import PeftConfig, PeftModel from peft import PeftConfig, PeftModel
from .mapper import INT_TO_FLOAT_MAPPER, FLOAT_TO_INT_MAPPER, MAP_TO_UNSLOTH_16bit from .loader_utils import get_model_name
import os import os, contextlib, sys
try: try:
from huggingface_hub.utils import get_token from huggingface_hub.utils import get_token
except: except:
@ -63,105 +63,6 @@ def _get_dtype(dtype):
pass pass
def __get_model_name(
model_name,
load_in_4bit = True,
INT_TO_FLOAT_MAPPER = None,
FLOAT_TO_INT_MAPPER = None,
MAP_TO_UNSLOTH_16bit = None,
):
model_name = str(model_name)
lower_model_name = model_name.lower()
if not SUPPORTS_FOURBIT and lower_model_name in INT_TO_FLOAT_MAPPER:
model_name = INT_TO_FLOAT_MAPPER[lower_model_name]
logger.warning_once(
f"Unsloth: Your transformers version of {transformers_version} does not support native "\
f"4bit loading.\nThe minimum required version is 4.37.\n"\
f'Try `pip install --upgrade "transformers>=4.37"`\n'\
f"to obtain the latest transformers build, then restart this session.\n"\
f"For now, we shall load `{model_name}` instead (still 4bit, just slower downloading)."
)
return model_name
elif not load_in_4bit and lower_model_name in INT_TO_FLOAT_MAPPER:
new_model_name = INT_TO_FLOAT_MAPPER[lower_model_name]
# logger.warning_once(
# f"Unsloth: You passed in `{model_name}` which is a 4bit model, yet you set\n"\
# f"`load_in_4bit = False`. We shall load `{new_model_name}` instead."
# )
return new_model_name
elif not load_in_4bit and lower_model_name in MAP_TO_UNSLOTH_16bit:
new_model_name = MAP_TO_UNSLOTH_16bit[lower_model_name]
return new_model_name
elif load_in_4bit and SUPPORTS_FOURBIT and lower_model_name in FLOAT_TO_INT_MAPPER:
new_model_name = FLOAT_TO_INT_MAPPER[lower_model_name]
# logger.warning_once(
# f"Unsloth: You passed in `{model_name}` and `load_in_4bit = True`.\n"\
# f"We shall load `{new_model_name}` for 4x faster loading."
# )
return new_model_name
pass
return None
pass
def _get_new_mapper():
try:
import requests
new_mapper = "https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/models/mapper.py"
with requests.get(new_mapper, timeout = 3) as new_mapper: new_mapper = new_mapper.text
new_mapper = new_mapper[new_mapper.find("__INT_TO_FLOAT_MAPPER"):]
new_mapper = new_mapper\
.replace("INT_TO_FLOAT_MAPPER", "NEW_INT_TO_FLOAT_MAPPER")\
.replace("FLOAT_TO_INT_MAPPER", "NEW_FLOAT_TO_INT_MAPPER")\
.replace("MAP_TO_UNSLOTH_16bit", "NEW_MAP_TO_UNSLOTH_16bit")
exec(new_mapper, globals())
return NEW_INT_TO_FLOAT_MAPPER, NEW_FLOAT_TO_INT_MAPPER, NEW_MAP_TO_UNSLOTH_16bit
except:
return {}, {}, {}
pass
pass
def get_model_name(model_name, load_in_4bit = True):
new_model_name = __get_model_name(
model_name = model_name,
load_in_4bit = load_in_4bit,
INT_TO_FLOAT_MAPPER = INT_TO_FLOAT_MAPPER,
FLOAT_TO_INT_MAPPER = FLOAT_TO_INT_MAPPER,
MAP_TO_UNSLOTH_16bit = MAP_TO_UNSLOTH_16bit,
)
if new_model_name is None and model_name.count("/") == 1 and model_name[0].isalnum():
# Try checking if a new Unsloth version allows it!
NEW_INT_TO_FLOAT_MAPPER, NEW_FLOAT_TO_INT_MAPPER, NEW_MAP_TO_UNSLOTH_16bit = _get_new_mapper()
upgraded_model_name = __get_model_name(
model_name = model_name,
load_in_4bit = load_in_4bit,
INT_TO_FLOAT_MAPPER = NEW_INT_TO_FLOAT_MAPPER,
FLOAT_TO_INT_MAPPER = NEW_FLOAT_TO_INT_MAPPER,
MAP_TO_UNSLOTH_16bit = NEW_MAP_TO_UNSLOTH_16bit,
)
if upgraded_model_name is not None:
raise NotImplementedError(
f"Unsloth: {model_name} is not supported in your current Unsloth version! Please update Unsloth via:\n\n"\
'pip uninstall unsloth -y\n'\
'pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"'
)
pass
pass
return new_model_name if new_model_name is not None else model_name
pass
class FastLanguageModel(FastLlamaModel): class FastLanguageModel(FastLlamaModel):
@staticmethod @staticmethod
def from_pretrained( def from_pretrained(
@ -333,7 +234,8 @@ class FastLanguageModel(FastLlamaModel):
else: else:
raise NotImplementedError( raise NotImplementedError(
f"Unsloth: {model_name} not supported yet!\n"\ f"Unsloth: {model_name} not supported yet!\n"\
"Make an issue to https://github.com/unslothai/unsloth!", "Maybe you're doing vision finetuning? Please use FastVisionModel instead!\n"\
"Otherwise, make an issue to https://github.com/unslothai/unsloth!",
) )
pass pass
@ -411,4 +313,236 @@ class FastLanguageModel(FastLlamaModel):
pass pass
return model, tokenizer return model, tokenizer
pass pass
pass pass
from ._utils import (
patch_compiling_bitsandbytes,
patch_model_and_tokenizer,
prepare_model_for_kbit_training,
patch_unsloth_smart_gradient_checkpointing,
patch_compiled_autograd,
process_vision_info,
unsloth_compile_transformers,
)
from ..kernels import (
patch_loss_functions,
post_patch_loss_function,
)
from .vision import FastBaseVisionModel
class FastVisionModel(FastBaseVisionModel):
@staticmethod
def from_pretrained(
model_name = "unsloth/Llama-3.2-11B-Vision-Instruct-bnb-4bit",
max_seq_length = None, # [TODO] No effect
dtype = None,
load_in_4bit = True,
token = None,
device_map = "sequential",
rope_scaling = None, # [TODO] No effect
fix_tokenizer = True, # [TODO] No effect
trust_remote_code = False,
use_gradient_checkpointing = "unsloth",
resize_model_vocab = None, # [TODO] No effect
revision = None,
*args, **kwargs,
):
if token is None: token = get_token()
patch_compiled_autograd()
patch_compiling_bitsandbytes()
if use_gradient_checkpointing == "unsloth":
patch_unsloth_smart_gradient_checkpointing()
old_model_name = model_name
model_name = get_model_name(model_name, load_in_4bit)
with contextlib.redirect_stdout(open(os.devnull, "w")):
patch_loss_functions(torch_compile = False)
model_types = unsloth_compile_transformers(
model_name = model_name,
sdpa_dynamic_mask = True,
sdpa_bool_masks = True,
sdpa_gqa_replace = True,
sdpa_dynamic_compile = True,
compile_attention = True,
disable_causal_masks = True,
compile_torch_modules = True,
compile_custom_modules = True,
compile_function_calls = True,
fuse_lm_head = True,
gradient_checkpointing = True,
manual_replacements = True,
epilogue_fusion = True,
max_autotune = False,
shape_padding = True,
cudagraphs = False,
debug = False,
import_from_cache = False,
disable = False,
)
pass
# First check if it's a normal model via AutoConfig
from huggingface_hub.utils import disable_progress_bars, enable_progress_bars, are_progress_bars_disabled
was_disabled = are_progress_bars_disabled()
disable_progress_bars()
autoconfig_error = None
peft_error = None
try:
model_config = AutoConfig.from_pretrained(
model_name,
token = token,
revision = revision,
trust_remote_code = trust_remote_code,
)
is_model = True
except Exception as error:
autoconfig_error = str(error)
is_model = False
try:
peft_config = PeftConfig.from_pretrained(
model_name,
token = token,
revision = revision,
trust_remote_code = trust_remote_code,
)
is_peft = True
except Exception as error:
peft_error = str(error)
is_peft = False
pass
# Both config.json and adapter_config.json should not exist!
# Old transformers versions check
both_exist = (is_model and is_peft) and not SUPPORTS_LLAMA32
# New transformers need to check manually.
if SUPPORTS_LLAMA32:
# Check if folder exists locally
if os.path.isdir(model_name):
exist_adapter_config = os.path.exists(os.path.join(model_name, "adapter_config.json"))
exist_config = os.path.exists(os.path.join(model_name, "config.json"))
both_exist = exist_adapter_config and exist_config
else:
files = HfFileSystem(token = token).glob(os.path.join(model_name, "*.json"))
files = (os.path.split(x)[-1] for x in files)
if sum(x == "adapter_config.json" or x == "config.json" for x in files) >= 2:
both_exist = True
pass
pass
pass
# Error out if both LoRA and normal model config exists.
if both_exist:
raise RuntimeError(
"Unsloth: Your repo has a LoRA adapter and a base model.\n"\
"You have 2 files `config.json` and `adapter_config.json`.\n"\
"We must only allow one config file.\n"\
"Please separate the LoRA and base models to 2 repos."
)
elif not is_model and not is_peft:
error = autoconfig_error or peft_error
# Old transformers version
if "rope_scaling" in error.lower() and not SUPPORTS_LLAMA31:
raise ImportError(
f"Unsloth: Your transformers version of {transformers_version} does not support new RoPE scaling methods.\n"\
f"This includes Llama 3.1. The minimum required version is 4.43.2\n"\
f'Try `pip install --upgrade "transformers>=4.43.2"`\n'\
f"to obtain the latest transformers build, then restart this session."\
)
raise RuntimeError(autoconfig_error or peft_error)
pass
# Get base model for PEFT:
if is_peft:
# Check base model again for PEFT
model_name = get_model_name(peft_config.base_model_name_or_path, load_in_4bit)
model_config = AutoConfig.from_pretrained(
model_name,
token = token,
revision = revision,
trust_remote_code = trust_remote_code,
)
pass
if not was_disabled: enable_progress_bars()
# Check if this is local model since the tokenizer gets overwritten
if os.path.exists(os.path.join(old_model_name, "tokenizer_config.json")) and \
os.path.exists(os.path.join(old_model_name, "tokenizer.json")) and \
os.path.exists(os.path.join(old_model_name, "special_tokens_map.json")):
tokenizer_name = old_model_name
else:
tokenizer_name = None
pass
model, tokenizer = FastBaseVisionModel.from_pretrained(
model_name = model_name,
max_seq_length = max_seq_length,
dtype = _get_dtype(dtype),
load_in_4bit = load_in_4bit,
token = token,
device_map = device_map,
trust_remote_code = trust_remote_code,
revision = revision if not is_peft else None,
model_types = model_types,
tokenizer_name = tokenizer_name,
*args, **kwargs,
)
if resize_model_vocab is not None:
model.resize_token_embeddings(resize_model_vocab)
pass
# In case the model supports tagging, add the unsloth tag.
if hasattr(model, "add_model_tags"):
model.add_model_tags(["unsloth",])
pass
if hasattr(tokenizer, "add_model_tags"):
tokenizer.add_model_tags(["unsloth",])
pass
if load_in_4bit:
# Fix up bitsandbytes config
quantization_config = \
{
# Sometimes torch_dtype is not a string!!
"bnb_4bit_compute_dtype" : model.config.to_dict()["torch_dtype"],
"bnb_4bit_quant_type" : "nf4",
"bnb_4bit_use_double_quant" : True,
"llm_int8_enable_fp32_cpu_offload" : False,
"llm_int8_has_fp16_weight" : False,
"llm_int8_skip_modules" : None,
"llm_int8_threshold" : 6.0,
"load_in_4bit" : True,
"load_in_8bit" : False,
"quant_method" : "bitsandbytes",
}
model.config.update({"quantization_config" : quantization_config})
pass
if is_peft:
# From https://github.com/huggingface/peft/issues/184
# Now add PEFT adapters
model.enable_input_require_grads()
model = PeftModel.from_pretrained(
model,
old_model_name,
token = token,
revision = revision,
is_trainable = True,
trust_remote_code = trust_remote_code,
)
# Patch it as well!
model = FastBaseVisionModel.patch_peft_model(model, use_gradient_checkpointing)
pass
return model, tokenizer
pass
pass

120
unsloth/models/loader_utils.py Normal file
View file

@ -0,0 +1,120 @@
# Copyright 2023-present Daniel Han-Chen & the Unsloth team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from .mapper import INT_TO_FLOAT_MAPPER, FLOAT_TO_INT_MAPPER, MAP_TO_UNSLOTH_16bit
# https://github.com/huggingface/transformers/pull/26037 allows 4 bit loading!
from packaging.version import Version
from transformers import __version__ as transformers_version
transformers_version = Version(transformers_version)
SUPPORTS_FOURBIT = transformers_version >= Version("4.37")
def __get_model_name(
model_name,
load_in_4bit = True,
INT_TO_FLOAT_MAPPER = None,
FLOAT_TO_INT_MAPPER = None,
MAP_TO_UNSLOTH_16bit = None,
):
model_name = str(model_name)
lower_model_name = model_name.lower()
if not SUPPORTS_FOURBIT and lower_model_name in INT_TO_FLOAT_MAPPER:
model_name = INT_TO_FLOAT_MAPPER[lower_model_name]
print(
f"Unsloth: Your transformers version of {transformers_version} does not support native "\
f"4bit loading.\nThe minimum required version is 4.37.\n"\
f'Try `pip install --upgrade "transformers>=4.37"`\n'\
f"to obtain the latest transformers build, then restart this session.\n"\
f"For now, we shall load `{model_name}` instead (still 4bit, just slower downloading)."
)
return model_name
elif not load_in_4bit and lower_model_name in INT_TO_FLOAT_MAPPER:
new_model_name = INT_TO_FLOAT_MAPPER[lower_model_name]
# logger.warning_once(
# f"Unsloth: You passed in `{model_name}` which is a 4bit model, yet you set\n"\
# f"`load_in_4bit = False`. We shall load `{new_model_name}` instead."
# )
return new_model_name
elif not load_in_4bit and lower_model_name in MAP_TO_UNSLOTH_16bit:
new_model_name = MAP_TO_UNSLOTH_16bit[lower_model_name]
return new_model_name
elif load_in_4bit and SUPPORTS_FOURBIT and lower_model_name in FLOAT_TO_INT_MAPPER:
new_model_name = FLOAT_TO_INT_MAPPER[lower_model_name]
# logger.warning_once(
# f"Unsloth: You passed in `{model_name}` and `load_in_4bit = True`.\n"\
# f"We shall load `{new_model_name}` for 4x faster loading."
# )
return new_model_name
pass
return None
pass
def _get_new_mapper():
try:
import requests
new_mapper = "https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/models/mapper.py"
with requests.get(new_mapper, timeout = 3) as new_mapper: new_mapper = new_mapper.text
new_mapper = new_mapper[new_mapper.find("__INT_TO_FLOAT_MAPPER"):]
new_mapper = new_mapper\
.replace("INT_TO_FLOAT_MAPPER", "NEW_INT_TO_FLOAT_MAPPER")\
.replace("FLOAT_TO_INT_MAPPER", "NEW_FLOAT_TO_INT_MAPPER")\
.replace("MAP_TO_UNSLOTH_16bit", "NEW_MAP_TO_UNSLOTH_16bit")
exec(new_mapper, globals())
return NEW_INT_TO_FLOAT_MAPPER, NEW_FLOAT_TO_INT_MAPPER, NEW_MAP_TO_UNSLOTH_16bit
except:
return {}, {}, {}
pass
pass
def get_model_name(model_name, load_in_4bit = True):
new_model_name = __get_model_name(
model_name = model_name,
load_in_4bit = load_in_4bit,
INT_TO_FLOAT_MAPPER = INT_TO_FLOAT_MAPPER,
FLOAT_TO_INT_MAPPER = FLOAT_TO_INT_MAPPER,
MAP_TO_UNSLOTH_16bit = MAP_TO_UNSLOTH_16bit,
)
if new_model_name is None and model_name.count("/") == 1 and model_name[0].isalnum():
# Try checking if a new Unsloth version allows it!
NEW_INT_TO_FLOAT_MAPPER, NEW_FLOAT_TO_INT_MAPPER, NEW_MAP_TO_UNSLOTH_16bit = _get_new_mapper()
upgraded_model_name = __get_model_name(
model_name = model_name,
load_in_4bit = load_in_4bit,
INT_TO_FLOAT_MAPPER = NEW_INT_TO_FLOAT_MAPPER,
FLOAT_TO_INT_MAPPER = NEW_FLOAT_TO_INT_MAPPER,
MAP_TO_UNSLOTH_16bit = NEW_MAP_TO_UNSLOTH_16bit,
)
if upgraded_model_name is not None:
raise NotImplementedError(
f"Unsloth: {model_name} is not supported in your current Unsloth version! Please update Unsloth via:\n\n"\
'pip uninstall unsloth unsloth_zoo -y\n'\
'pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"\n'\
'pip install --upgrade --no-cache-dir "git+https://github.com/unslothai/unsloth-zoo.git"\n'\
)
pass
pass
return new_model_name if new_model_name is not None else model_name
pass

48
unsloth/models/mapper.py
View file

@ -409,12 +409,12 @@ __INT_TO_FLOAT_MAPPER = \
"Qwen/Qwen2.5-Coder-32B", "Qwen/Qwen2.5-Coder-32B",
), ),
"unsloth/Qwen2.5-Coder-0.5B-Instruct-bnb-4bit" : ( "unsloth/Qwen2.5-Coder-0.5B-Instruct-bnb-4bit" : (
"unsloth/Qwen2.5-Coder-Instruct-0.5B", "unsloth/Qwen2.5-Coder-0.5B-Instruct",
"Qwen/Qwen2.5-Coder-Instruct-0.5B", "Qwen/Qwen2.5-Coder-0.5B-Instruct",
), ),
"unsloth/Qwen2.5-Coder-1.5B-Instruct-bnb-4bit" : ( "unsloth/Qwen2.5-Coder-1.5B-Instruct-bnb-4bit" : (
"unsloth/Qwen2.5-Coder-Instruct-1.5B", "unsloth/Qwen2.5-Coder-1.5B-Instruct",
"Qwen/Qwen2.5-Coder-Instruct-1.5B", "Qwen/Qwen2.5-Coder-1.5B-Instruct",
), ),
"unsloth/Qwen2.5-Coder-3B-Instruct-bnb-4bit" : ( "unsloth/Qwen2.5-Coder-3B-Instruct-bnb-4bit" : (
"unsloth/Qwen2.5-Coder-3B-Instruct", "unsloth/Qwen2.5-Coder-3B-Instruct",
@ -452,6 +452,46 @@ __INT_TO_FLOAT_MAPPER = \
"unsloth/Llama-3.1-Nemotron-70B-Instruct", "unsloth/Llama-3.1-Nemotron-70B-Instruct",
"nvidia/Llama-3.1-Nemotron-70B-Instruct-HF", "nvidia/Llama-3.1-Nemotron-70B-Instruct-HF",
), ),
"unsloth/Qwen2-VL-2B-Instruct-bnb-4bit" : (
"unsloth/Qwen2-VL-2B-Instruct",
"Qwen/Qwen2-VL-2B-Instruct",
),
"unsloth/Qwen2-VL-7B-Instruct-bnb-4bit" : (
"unsloth/Qwen2-VL-7B-Instruct",
"Qwen/Qwen2-VL-7B-Instruct",
),
"unsloth/Llama-3.2-11B-Vision-Instruct-bnb-4bit" : (
"unsloth/Llama-3.2-11B-Vision-Instruct",
"meta-llama/Llama-3.2-11B-Vision-Instruct",
),
"unsloth/Llama-3.2-90B-Vision-Instruct-bnb-4bit" : (
"unsloth/Llama-3.2-90B-Vision-Instruct",
"meta-llama/Llama-3.2-90B-Vision-Instruct",
),
"unsloth/Llama-3.2-11B-Vision-bnb-4bit" : (
"unsloth/Llama-3.2-11B-Vision",
"meta-llama/Llama-3.2-11B-Vision",
),
"unsloth/Llama-3.2-90B-Vision-bnb-4bit" : (
"unsloth/Llama-3.2-90B-Vision",
"meta-llama/Llama-3.2-90B-Vision",
),
"unsloth/Pixtral-12B-2409-bnb-4bit" : (
"unsloth/Pixtral-12B-2409",
"mistralai/Pixtral-12B-2409",
),
"unsloth/Pixtral-12B-2409-Base-bnb-4bit" : (
"unsloth/Pixtral-12B-Base-2409",
"mistralai/Pixtral-12B-Base-2409",
),
"unsloth/llava-1.5-7b-hf-bnb-4bit" : (
"unsloth/llava-1.5-7b-hf",
"llava-hf/llava-1.5-7b-hf",
),
"unsloth/llava-v1.6-mistral-7b-hf-bnb-4bit" : (
"unsloth/llava-v1.6-mistral-7b-hf",
"llava-hf/llava-v1.6-mistral-7b-hf",
),
} }
INT_TO_FLOAT_MAPPER = {} INT_TO_FLOAT_MAPPER = {}

535
unsloth/models/vision.py
View file

@ -1,58 +1,86 @@
# Unsloth Zoo - Utilities for Unsloth
# Copyright 2023-present Daniel Han-Chen & the Unsloth team. All rights reserved. # Copyright 2023-present Daniel Han-Chen & the Unsloth team. All rights reserved.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # This program is free software: you can redistribute it and/or modify
# you may not use this file except in compliance with the License. # it under the terms of the GNU Lesser General Public License as published by
# You may obtain a copy of the License at # the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# #
# http://www.apache.org/licenses/LICENSE-2.0 # This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# #
# Unless required by applicable law or agreed to in writing, software # You should have received a copy of the GNU Lesser General Public License
# distributed under the License is distributed on an "AS IS" BASIS, # along with this program. If not, see <https://www.gnu.org/licenses/>.
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import torch
from transformers import (
BitsAndBytesConfig,
AutoModelForVision2Seq,
AutoProcessor,
)
from .llama import * from .llama import *
from ..kernels import patch_layernorm, unpatch_layernorm from ..kernels import (
from ..kernels import patch_rms_layernorm, unpatch_rms_layernorm post_patch_loss_function,
from ..kernels import patch_llama_for_causal_lm, unpatch_llama_for_causal_lm )
from ._utils import patch_gradient_checkpointing from ._utils import __version__
from peft import LoraConfig, TaskType, get_peft_model
from transformers import set_seed as transformers_set_seed
from unsloth_zoo.peft_utils import (
get_peft_regex,
merge_and_overwrite_lora,
)
from transformers import AutoProcessor __all__ = [
try: "FastBaseVisionModel",
from transformers import MllamaForConditionalGeneration ]
except:
raise ImportError( def _wrap_fast_inference(generate, device_type, dtype, model):
"Unsloth: Please update your transformers version to 4.46.0 for Llama 3.2 support!" # Wraps inference with bfloat16 / float16
) @torch.inference_mode
def _fast_generate(*args, **kwargs):
# For num_logits_to_keep
kwargs["num_logits_to_keep"] = 1
# Remove token_type_ids
kwargs.pop("token_type_ids", None)
# Check pad_token
model_eos_token_id = getattr(model.config, "eos_token_id", None)
if model_eos_token_id is not None and hasattr(model_eos_token_id, "__iter__"):
model_eos_token_id = model_eos_token_id[0]
kwargs["pad_token_id"] = kwargs.pop("pad_token_id", model_eos_token_id)
try:
kwargs["pixel_values"] = kwargs["pixel_values"].to(model.dtype)
except:
pass
# Autocasted
with torch.autocast(device_type = device_type, dtype = dtype):
output = generate(*args, **kwargs)
pass
return output
pass
return _fast_generate
pass pass
class FastVisionModel:
def pre_patch(self):
patch_gradient_checkpointing()
patch_layernorm()
patch_rms_layernorm()
patch_llama_for_causal_lm()
pass
def post_unpatch(self):
unpatch_layernorm()
unpatch_rms_layernorm()
unpatch_llama_for_causal_lm()
pass
class FastBaseVisionModel:
@staticmethod @staticmethod
def from_pretrained( def from_pretrained(
model_name = "llava-hf/llava-1.5-7b-hf", model_name = "unsloth/llama-3-8b-bnb-4bit",
max_seq_length = None, max_seq_length = None,
dtype = None, dtype = None,
load_in_4bit = True, load_in_4bit = True,
token = None, token = None,
device_map = "sequential", device_map = "sequential",
rope_scaling = None,
trust_remote_code = False, trust_remote_code = False,
model_types = None,
tokenizer_name = None,
**kwargs, **kwargs,
): ):
if trust_remote_code: if trust_remote_code:
@ -67,7 +95,7 @@ class FastVisionModel:
max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3) max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)
statistics = \ statistics = \
f"==((====))== Unsloth {__version__}: Fast {model_patcher.__name__[4:-5]} patching. Transformers = {transformers_version}.\n"\ f"==((====))== Unsloth {__version__}: Fast {model_types[0].title()} vision patching. Transformers = {transformers_version}.\n"\
f" \\\ /| GPU: {gpu_stats.name}. Max memory: {max_memory} GB. Platform = {platform_system}.\n"\ f" \\\ /| GPU: {gpu_stats.name}. Max memory: {max_memory} GB. Platform = {platform_system}.\n"\
f"O^O/ \_/ \\ Pytorch: {torch.__version__}. CUDA = {gpu_stats.major}.{gpu_stats.minor}. CUDA Toolkit = {torch.version.cuda}.\n"\ f"O^O/ \_/ \\ Pytorch: {torch.__version__}. CUDA = {gpu_stats.major}.{gpu_stats.minor}. CUDA Toolkit = {torch.version.cuda}.\n"\
f"\ / Bfloat16 = {str(SUPPORTS_BFLOAT16).upper()}. FA [Xformers = {xformers_version}. FA2 = {HAS_FLASH_ATTENTION}]\n"\ f"\ / Bfloat16 = {str(SUPPORTS_BFLOAT16).upper()}. FA [Xformers = {xformers_version}. FA2 = {HAS_FLASH_ATTENTION}]\n"\
@ -81,6 +109,7 @@ class FastVisionModel:
pass pass
# Return old flag # Return old flag
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = old_hf_transfer os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = old_hf_transfer
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
get_statistics() # For debugging - we use a download counter to see if environments are not breaking get_statistics() # For debugging - we use a download counter to see if environments are not breaking
@ -105,160 +134,36 @@ class FastVisionModel:
) )
pass pass
kwargs.pop("attn_implementation", None); # No need since we auto call it
# Cannot be None, since HF now checks for the config # Cannot be None, since HF now checks for the config
if load_in_4bit: kwargs["quantization_config"] = bnb_config if load_in_4bit: kwargs["quantization_config"] = bnb_config
self.pre_patch() model = AutoModelForVision2Seq.from_pretrained(
model = MllamaForConditionalGeneration.from_pretrained(
model_name, model_name,
device_map = device_map, device_map = device_map,
torch_dtype = dtype, torch_dtype = dtype,
# quantization_config = bnb_config, # quantization_config = bnb_config,
token = token, token = token,
max_position_embeddings = max_position_embeddings,
trust_remote_code = trust_remote_code, trust_remote_code = trust_remote_code,
attn_implementation = "sdpa", # attn_implementation = "sdpa", [TODO] Pixtral for eg fails
**kwargs, **kwargs,
) )
self.post_unpatch()
# Return old flag # Return old flag
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = old_hf_transfer os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = old_hf_transfer
# We currently only support NVIDIA GPUs - AMD / Intel is a work in progress! # We currently only support NVIDIA GPUs - AMD / Intel is a work in progress!
post_check = check_nvidia() post_check = check_nvidia()
# Counteract saved tokenizers # Counteract saved tokenizers
tokenizer_name = model_name if tokenizer_name is None else tokenizer_name
tokenizer = AutoProcessor.from_pretrained( tokenizer = AutoProcessor.from_pretrained(
model_name, tokenizer_name,
) padding_side = "right",
model = FastVisionModel.post_patch(model) token = token,
# Patch Trainer
from transformers.trainer import Trainer
try:
if Trainer._inner_training_loop.__name__ != "_fast_inner_training_loop":
inner_training_loop = inspect.getsource(Trainer._inner_training_loop)
Trainer._original_training_loop = inner_training_loop
else:
inner_training_loop = Trainer._original_training_loop
except:
raise RuntimeError('Unsloth currently does not support multi GPU setups - but we are working on it!')
pass
if ((post_check - pre_check) >= 1).sum() > 1:
raise RuntimeError('Unsloth currently does not support multi GPU setups - but we are working on it!')
import transformers.trainer
items_in_trainer = dir(transformers.trainer)
good_items = []
for item in items_in_trainer:
# TODO: Support Deepspeed
if item.startswith(("deepspeed", "xm", "met", "smp")): continue
if item in inner_training_loop: good_items.append(item)
pass
exec("from transformers.trainer import (" + ", ".join(x for x in good_items) + ")", globals())
start = re.search('logger\.info\([\"\'].+?Running training', inner_training_loop).span(0)[0]
end = inner_training_loop.find("\n\n", start)
original_debug = inner_training_loop[start:end]
spaces = re.search('\n([\s\t]{1,})', original_debug).group(0)[1:]
front_spaces = re.match('([\s\t]{1,})', inner_training_loop).group(0)
debug_info = """debug_info = \\
f"==((====))== Unsloth - 2x faster free finetuning | Num GPUs = {args.world_size}\\n"\\
f" \\\\\\ /| Num examples = {num_examples:,} | Num Epochs = {num_train_epochs:,}\\n"\\
f"O^O/ \\_/ \\ Batch size per device = {self._train_batch_size:,} | Gradient Accumulation steps = {args.gradient_accumulation_steps}\\n"\\
f"\\ / Total batch size = {total_train_batch_size:,} | Total steps = {max_steps:,}\\n"\\
f' "-____-" Number of trainable parameters = {get_model_param_count(model, trainable_only=True):,}'
logger.warning(debug_info)
import subprocess, re, gc, numpy as np
a = np.array([0,])
try:
a = subprocess.check_output('nvidia-smi --query-gpu=memory.used --format=csv', shell = True)
a = re.findall(rb'([\\d]{1,})[\\s]{1,}M', a)
a = np.array([int(x.decode('utf-8'))/1024 for x in a])
except:
if not torch.cuda.is_available():
raise RuntimeError('Unsloth: We do not support AMD / Intel machines yet - it is a work in progress!')
if ((a - PRE_CHECK) >= 1).sum() > 1:
raise RuntimeError('Unsloth currently does not support multi GPU setups - but we are working on it!')
for _ in range(3):
gc.collect()
torch.cuda.empty_cache()"""
debug_info = debug_info.split('\n')
debug_info = "\n".join([debug_info[0]] + [spaces + x[8:] for x in debug_info[1:]])
inner_training_loop = inner_training_loop.replace(original_debug, debug_info)
debug_info = """n_total_devices = total_train_batch_size // \\
args.gradient_accumulation_steps // self._train_batch_size
if n_total_devices > 1:
logger.warning_once('Unsloth currently does not support multi GPU setups - but we are working on it!')
debug_info ="""
debug_info = debug_info.split('\n')
debug_info = "\n".join([debug_info[0]] + [spaces + x[8:] for x in debug_info[1:]])
inner_training_loop = inner_training_loop.replace("debug_info =", debug_info, 1)
front_spaces = re.match(r"[\t\s]{1,}", inner_training_loop).group(0)
inner_training_loop = re.sub(r"^" + front_spaces, "", inner_training_loop, flags = re.MULTILINE)
inner_training_loop = inner_training_loop.replace(
"train_dataloader = tpu_spmd_dataloader(train_dataloader)",
"raise RuntimeError('Unsloth: TPUs are not yet supported!')"
)
inner_training_loop = inner_training_loop.replace(
"self.accelerator.free_memory()",
"self.accelerator.free_memory()\n" + \
front_spaces + "if self.is_deepspeed_enabled:"\
"raise RuntimeError('Unsloth: Deepspeed is not yet supported!')\n", 1,
) )
check_batches = """train_dataloader = self.get_train_dataloader() model, tokenizer = patch_tokenizer(model, tokenizer)
ga = args.gradient_accumulation_steps model = post_patch_loss_function(model)
bsz = self._train_batch_size
total_batches = bsz * ga * args.world_size
n_total_devices = total_batches // ga // bsz
if n_total_devices > 1:
logger.warning_once('Unsloth currently does not support multi GPU setups - but we are working on it!')
divisor = n_total_devices / 1
bsz = self._train_batch_size = max(int(bsz / divisor), 1)
if total_batches // ga // bsz > 1:
divisor = n_total_devices / 1
ga = args.gradient_accumulation_steps = max(int(ga / divisor), 1)"""
check_batches = check_batches.split('\n')
check_batches = "\n".join([check_batches[0]] + [front_spaces + x[8:] for x in check_batches[1:]])
inner_training_loop = inner_training_loop.replace(
"train_dataloader = self.get_train_dataloader()",
check_batches, 1,
)
inner_training_loop = inner_training_loop.replace(
"_inner_training_loop",
"_fast_inner_training_loop", 1,
)
exec(inner_training_loop, globals())
Trainer._inner_training_loop = _fast_inner_training_loop
inner_training_loop = inner_training_loop.replace(
"is_torch_tpu_available()",
"False",
)
if "n_total_devices >" not in inner_training_loop:
raise RuntimeError('Unsloth currently does not support multi GPU setups - but we are working on it!')
pass
inner_training_loop = inner_training_loop.replace(
"is_sagemaker_mp_enabled()",
"False",
)
exec(inner_training_loop, globals())
Trainer._inner_training_loop = _fast_inner_training_loop
# Save max_seq_length
model.max_seq_length = max_position_embeddings
internal_model = model
while hasattr(internal_model, "model"):
internal_model.max_seq_length = max_position_embeddings
internal_model = internal_model.model
pass
internal_model.max_seq_length = max_position_embeddings
# Fix up config for transformers uploading PEFT # Fix up config for transformers uploading PEFT
# Not necessary anymore since we require transformers>=4.37! # Not necessary anymore since we require transformers>=4.37!
@ -271,115 +176,76 @@ class FastVisionModel:
pass pass
# Log Unsloth version for future fastpaths for inference # Log Unsloth version for future fastpaths for inference
model.config.update({"unsloth_version" : __version__}) if hasattr(model, "config"):
model.config.update({"unsloth_version" : __version__})
pass
patch_saving_functions(model, vision = True)
patch_saving_functions(tokenizer, vision = True)
# Add save modules # Save tokenizer for inference purposes
patch_saving_functions(model) tokenizer.padding_side = "left" # Force inference
Trainer._inner_training_loop = _fast_inner_training_loop
# Also fix torch_dtype
internal_model = model internal_model = model
while hasattr(internal_model, "model"): while hasattr(internal_model, "model"):
if hasattr(internal_model, "config"): internal_model._saved_temp_tokenizer = tokenizer
if internal_model.config.torch_dtype == "float32":
internal_model.config.torch_dtype = torch.float32
elif internal_model.config.torch_dtype == "bfloat16":
internal_model.config.torch_dtype = torch.bfloat16
elif internal_model.config.torch_dtype == "float16":
internal_model.config.torch_dtype = torch.float16
pass
pass
internal_model = internal_model.model internal_model = internal_model.model
pass pass
if hasattr(internal_model, "config"): internal_model._saved_temp_tokenizer = tokenizer
if internal_model.config.torch_dtype == "float32":
internal_model.config.torch_dtype = torch.float32
elif internal_model.config.torch_dtype == "bfloat16":
internal_model.config.torch_dtype = torch.bfloat16
elif internal_model.config.torch_dtype == "float16":
internal_model.config.torch_dtype = torch.float16
pass
pass
return model, tokenizer return model, tokenizer
pass pass
@staticmethod
def post_patch(model):
# Patch model
layers = model.model.layers
lm_head = model.get_output_embeddings().weight
# Also patch all dtypes - BnB seems to not allocate the correct type?
# BnB default dtype seems to be float16!
correct_dtype = lm_head.weight.dtype
for name, module in model.named_modules():
if isinstance(module, (Bnb_Linear4bit, Peft_Linear4bit)):
weight = module.weight
quant_state = weight.quant_state
if type(quant_state) is list:
# BnB seems to have float16 as default!
module.weight.quant_state[2] = correct_dtype # Cast to correct dtype
else:
# https://github.com/TimDettmers/bitsandbytes/pull/763/files
quant_state.dtype = correct_dtype
pass
pass
pass
# Clear deleted GPU items
for _ in range(3):
gc.collect()
torch.cuda.empty_cache()
return model
pass
@staticmethod @staticmethod
def get_peft_model( def get_peft_model(
model, model,
r = 16, r = 16,
target_modules = "all-linear", target_modules = None,
lora_alpha = 16, lora_alpha = 16,
lora_dropout = 0, lora_dropout = 0,
bias = "none", bias = "none",
layers_to_transform = None, finetune_vision_layers = True,
layers_pattern = None, finetune_language_layers = True,
finetune_attention_modules = True,
finetune_mlp_modules = True,
layers_to_transform = None,
layers_pattern = None,
use_gradient_checkpointing = True, use_gradient_checkpointing = True,
random_state = 3407, random_state = 3407,
max_seq_length = 2048, # not used anymore max_seq_length = 2048, # not used anymore
use_rslora = False, use_rslora = False,
modules_to_save = None, modules_to_save = None,
init_lora_weights = True, init_lora_weights = True,
loftq_config = {}, loftq_config = {},
temporary_location = "_unsloth_temporary_saved_buffers", temporary_location = "_unsloth_temporary_saved_buffers",
**kwargs, **kwargs,
): ):
transformers_set_seed(random_state) transformers_set_seed(random_state)
# Get LoRA if type(r) is not int:
arguments = dict( raise TypeError(f"Unsloth: Rank of {str(r)} must be an integer.")
r = r, if r <= 0:
lora_alpha = lora_alpha, raise TypeError(f"Unsloth: Rank of {str(r)} must be larger than 0.")
target_modules = target_modules,
lora_dropout = lora_dropout,
bias = bias,
layers_to_transform = layers_to_transform,
init_lora_weights = init_lora_weights,
# loftq_config = loftq_config,
# use_rslora = use_rslora,
modules_to_save = modules_to_save,
**kwargs,
)
lora_config = LoraConfig(**arguments) if isinstance(model, PeftModelForCausalLM):
raise RuntimeError("Unsloth: You already added LoRA adapters to your model!")
model = _get_peft_model(model, lora_config) if target_modules == "all-linear":
finetune_vision_layers = True
model = FastVisionModel.patch_peft_model(model, use_gradient_checkpointing) finetune_language_layers = True
finetune_attention_modules = True
finetune_mlp_modules = True
pass
if target_modules is None:
target_modules = get_peft_regex(
model,
finetune_vision_layers = finetune_vision_layers,
finetune_language_layers = finetune_language_layers,
finetune_attention_modules = finetune_attention_modules,
finetune_mlp_modules = finetune_mlp_modules,
)
else:
assert(type(target_modules) in (list, tuple,))
pass
# Clear deleted GPU items # Clear deleted GPU items
for _ in range(3): for _ in range(3):
@ -387,6 +253,29 @@ class FastVisionModel:
torch.cuda.empty_cache() torch.cuda.empty_cache()
pass pass
lora_config = LoraConfig(
r = r,
lora_alpha = lora_alpha,
target_modules = target_modules,
lora_dropout = lora_dropout,
bias = bias,
task_type = TaskType.CAUSAL_LM,
)
model = prepare_model_for_kbit_training(
model,
use_gradient_checkpointing = use_gradient_checkpointing,
)
model = get_peft_model(model, lora_config)
model = FastBaseVisionModel.patch_peft_model(model, use_gradient_checkpointing)
# Clear deleted GPU items
for _ in range(3):
gc.collect()
torch.cuda.empty_cache()
pass
patch_saving_functions(model, vision = True)
return model return model
pass pass
@ -396,6 +285,11 @@ class FastVisionModel:
model, model,
use_gradient_checkpointing = True, use_gradient_checkpointing = True,
): ):
if not isinstance(model, PeftModelForCausalLM):
raise TypeError(
"Unsloth: Your model needs to call `.get_peft_model` first!"
)
pass
model = prepare_model_for_kbit_training( model = prepare_model_for_kbit_training(
model, model,
@ -403,20 +297,6 @@ class FastVisionModel:
use_reentrant = True, use_reentrant = True,
) )
# Fix up config for transformers uploading PEFT
for active_adapter in model.peft_config.keys():
# Not necessary since we requires transformers >= 4.37
if False:
name = model.peft_config[active_adapter].base_model_name_or_path
if name.startswith("unsloth/") and name.endswith("-bnb-4bit"):
name = name[:len(name) - len("-bnb-4bit")]
model.peft_config[active_adapter].base_model_name_or_path = name
pass
# Add revision to enable future fast inference paths
# [TODO] Bugs out!see https://github.com/unslothai/unsloth/issues/492
# model.peft_config[active_adapter].revision = f"unsloth"
pass
from transformers.trainer import Trainer from transformers.trainer import Trainer
if Trainer._inner_training_loop.__name__ != "_fast_inner_training_loop": if Trainer._inner_training_loop.__name__ != "_fast_inner_training_loop":
raise RuntimeError( raise RuntimeError(
@ -426,24 +306,7 @@ class FastVisionModel:
'Thank you for your understanding and we appreciate it immensely!' 'Thank you for your understanding and we appreciate it immensely!'
) )
pass pass
patch_saving_functions(model, vision = True)
logger.warning_once(
f"Unsloth {__version__} patched {len(model.model.model.layers)} layers with "\
f"{n_qkv} QKV layers, {n_o} O layers and {n_mlp} MLP layers.",
)
patch_saving_functions(model)
# Patch cross entropy loss labels
# Fixes https://github.com/unslothai/unsloth/issues/10
max_seq_length = model.max_seq_length
extra_ignored_labels = torch.full((max_seq_length, 1), -100, device = "cuda:0")
model.model.extra_ignored_labels = extra_ignored_labels
internal_model = model
while hasattr(internal_model, "model"):
internal_model.max_seq_length = max_seq_length
internal_model = internal_model.model
pass
internal_model.max_seq_length = max_seq_length
# Patch tokenizer to pad to the right # Patch tokenizer to pad to the right
internal_model = model internal_model = model
@ -468,37 +331,40 @@ class FastVisionModel:
@staticmethod @staticmethod
def for_inference(model): def for_inference(model):
# if model.config.model_type == "qwen2": model.gradient_checkpointing = False
# FastLlamaModel.for_training(model) model.training = False
# return
# pass
internal_model = model for name, module in model.named_modules():
internal_model.gradient_checkpointing = False if hasattr(module, "gradient_checkpointing"):
internal_model.training = False module.gradient_checkpointing = False
if hasattr(module, "training"):
while hasattr(internal_model, "model"): module.training = False
internal_model = internal_model.model
internal_model.gradient_checkpointing = False
internal_model.training = False
pass
if hasattr(internal_model, "training"):
internal_model.training = False
pass pass
# Also check if lm_head / embeddings are trained
internal_model = model
while not hasattr(internal_model, "lm_head"):
internal_model = internal_model.model
pass
lm_head = internal_model.lm_head.weight
device_type = lm_head.device.type
dtype = model.config.torch_dtype dtype = model.config.torch_dtype
if type(dtype) is str: if type(dtype) is str:
if dtype == "float16": dtype = torch.float16 if dtype == "float16": dtype = torch.float16
elif dtype == "bfloat16": dtype = torch.bfloat16 elif dtype == "bfloat16": dtype = torch.bfloat16
pass pass
device_type = model.device.type
# Wrap model.generate
if model.generate.__name__ != "_fast_generate":
model._unwrapped_old_generate = model.generate
model.generate = _wrap_fast_inference(model.generate, device_type, dtype, model)
pass
# Patch tokenizer to pad to the left
internal_model = model
while hasattr(internal_model, "model"):
if hasattr(internal_model, "_saved_temp_tokenizer"):
internal_model._saved_temp_tokenizer.padding_side = "left"
pass
internal_model = internal_model.model
pass
if hasattr(internal_model, "_saved_temp_tokenizer"):
internal_model._saved_temp_tokenizer.padding_side = "left"
pass
# Also disable training for embeddings for NEFTune # Also disable training for embeddings for NEFTune
if hasattr(model, "get_input_embeddings"): if hasattr(model, "get_input_embeddings"):
@ -516,23 +382,32 @@ class FastVisionModel:
@staticmethod @staticmethod
def for_training(model, use_gradient_checkpointing = True): def for_training(model, use_gradient_checkpointing = True):
model.gradient_checkpointing = use_gradient_checkpointing
model.training = True
for name, module in model.named_modules():
if hasattr(module, "gradient_checkpointing"):
module.gradient_checkpointing = use_gradient_checkpointing
if hasattr(module, "training"):
module.training = True
pass
# Also revert model.generate
if hasattr(model, "_unwrapped_old_generate"):
model.generate = model._unwrapped_old_generate
del model._unwrapped_old_generate
pass
# Patch tokenizer to pad to the right
internal_model = model internal_model = model
internal_model.gradient_checkpointing = use_gradient_checkpointing
internal_model.training = True
# Delete all fast inference loras
for param in model.parameters():
if hasattr(param, "_fast_lora"):
del param._fast_lora
pass
while hasattr(internal_model, "model"): while hasattr(internal_model, "model"):
if hasattr(internal_model, "_saved_temp_tokenizer"):
internal_model._saved_temp_tokenizer.padding_side = "right"
pass
internal_model = internal_model.model internal_model = internal_model.model
internal_model.gradient_checkpointing = use_gradient_checkpointing
internal_model.training = True
pass pass
if hasattr(internal_model, "training"): if hasattr(internal_model, "_saved_temp_tokenizer"):
internal_model.training = True internal_model._saved_temp_tokenizer.padding_side = "right"
pass pass
# Also re-enable training for embeddings for NEFTune # Also re-enable training for embeddings for NEFTune
@ -548,3 +423,5 @@ class FastVisionModel:
return model return model
pass pass
pass pass

171
unsloth/save.py
View file

@ -2041,8 +2041,153 @@ def unsloth_convert_lora_to_ggml_and_save_locally(
print("Unsloth: Done.") print("Unsloth: Done.")
print(f"Unsloth: Conversion completed! Output file: {output_file}") print(f"Unsloth: Conversion completed! Output file: {output_file}")
print("\nThis GGML making function was made by Maheswar. Ping him @Maheswar on the Unsloth Discord or on HuggingFace (@mahiatlinux) if you like this!") print("\nThis GGML making function was made by Maheswar. Ping him @Maheswar on the Unsloth Discord or on HuggingFace (@mahiatlinux) if you like this!")
pass
def patch_saving_functions(model):
from unsloth_zoo.peft_utils import merge_and_overwrite_lora
from .models.loader_utils import get_model_name
@torch.inference_mode
def unsloth_generic_save(
model,
tokenizer,
save_directory : Union[str, os.PathLike] = "unsloth_finetuned_merge",
save_method : str = "lora", # ["lora", "merged_16bit", "merged_4bit"]
push_to_hub : bool = False,
token : Optional[Union[str, bool]] = None,
is_main_process : bool = True,
state_dict : Optional[dict] = None,
save_function : Callable = torch.save,
max_shard_size : Union[int, str] = "5GB",
safe_serialization : bool = True,
variant : Optional[str] = None,
save_peft_format : bool = True,
# Push to hub
use_temp_dir : Optional[bool] = None,
commit_message : Optional[str] = "Trained with Unsloth",
private : Optional[bool] = None,
create_pr : bool = False,
revision : str = None,
commit_description : str = "Upload model trained with Unsloth 2x faster",
tags : List[str] = None,
# Our functions
temporary_location : str = "_unsloth_temporary_saved_buffers",
maximum_memory_usage : float = 0.9,
):
if token is None and push_to_hub: token = get_token()
merge_and_overwrite_lora(
get_model_name,
create_huggingface_repo,
model,
save_location = save_directory,
push_to_hub = push_to_hub,
token = token,
upload_location = save_directory if push_to_hub else None,
low_disk_space_usage = True,
private = private,
)
return
pass
def unsloth_generic_save_pretrained_merged(
self,
save_directory : Union[str, os.PathLike],
tokenizer = None,
save_method : str = "merged_16bit", # ["lora", "merged_16bit", "merged_4bit"]
push_to_hub : bool = False,
token : Optional[Union[str, bool]] = None,
is_main_process : bool = True,
state_dict : Optional[dict] = None,
save_function : Callable = torch.save,
max_shard_size : Union[int, str] = "5GB",
safe_serialization : bool = True,
variant : Optional[str] = None,
save_peft_format : bool = True,
tags : List[str] = None,
temporary_location : str = "_unsloth_temporary_saved_buffers",
maximum_memory_usage : float = 0.75,
):
"""
Same as .push_to_hub(...) except 4bit weights are auto
converted to float16 with as few overhead as possible.
Choose for `save_method` to be either:
1. `16bit`: Merge LoRA into float16 weights. Useful for GGUF / llama.cpp.
2. `4bit`: Merge LoRA into int4 weights. Useful for DPO / HF inference.
3. `lora`: Save LoRA adapters with no merging. Useful for HF inference.
"""
if tokenizer is None:
logger.warning_once(
"Unsloth: You're not saving a tokenizer as well?\n"\
"You can do it separately via `tokenizer.save_pretrained(...)`"
)
pass
arguments = dict(locals())
arguments["model"] = self
del arguments["self"]
unsloth_generic_save(**arguments)
for _ in range(3):
gc.collect()
pass
def unsloth_generic_push_to_hub_merged(
self,
repo_id : str,
tokenizer = None,
save_method : str = "merged_16bit", # ["lora", "merged_16bit", "merged_4bit"]
use_temp_dir : Optional[bool] = None,
commit_message : Optional[str] = "Trained with Unsloth",
private : Optional[bool] = None,
token : Union[bool, str, None] = None,
max_shard_size : Union[int, str, None] = "5GB",
create_pr : bool = False,
safe_serialization : bool = True,
revision : str = None,
commit_description : str = "Upload model trained with Unsloth 2x faster",
tags : Optional[List[str]] = None,
temporary_location : str = "_unsloth_temporary_saved_buffers",
maximum_memory_usage : float = 0.75,
):
"""
Same as .push_to_hub(...) except 4bit weights are auto
converted to float16 with as few overhead as possible.
Choose for `save_method` to be either:
1. `16bit`: Merge LoRA into float16 weights. Useful for GGUF / llama.cpp.
2. `4bit`: Merge LoRA into int4 weights. Useful for DPO / HF inference.
3. `lora`: Save LoRA adapters with no merging. Useful for HF inference.
"""
if tokenizer is None:
logger.warning_once(
"Unsloth: You're not saving a tokenizer as well?\n"\
"You can do it separately via `tokenizer.push_to_hub(...)`"
)
pass
arguments = dict(locals())
arguments["model"] = self
arguments["save_directory"] = repo_id
arguments["push_to_hub"] = True
del arguments["self"]
del arguments["repo_id"]
unsloth_generic_save(**arguments)
for _ in range(3):
gc.collect()
pass
def not_implemented_save(*args, **kwargs):
raise NotImplementedError("Unsloth: Sorry GGUF is currently not supported for vision models!")
pass
def patch_saving_functions(model, vision = False):
import inspect import inspect
import types import types
from typing import Callable, Optional, Union, List from typing import Callable, Optional, Union, List
@ -2131,14 +2276,22 @@ def patch_saving_functions(model):
pass pass
# Add saving methods to top level model # Add saving methods to top level model
if hasattr(model, "config"): if not vision:
# Counteract tokenizers if hasattr(model, "config"):
model.push_to_hub_merged = types.MethodType(unsloth_push_to_hub_merged, model) # Counteract tokenizers
model.save_pretrained_merged = types.MethodType(unsloth_save_pretrained_merged, model) model.push_to_hub_merged = types.MethodType(unsloth_push_to_hub_merged, model)
model.push_to_hub_gguf = types.MethodType(unsloth_push_to_hub_gguf, model) model.save_pretrained_merged = types.MethodType(unsloth_save_pretrained_merged, model)
model.save_pretrained_gguf = types.MethodType(unsloth_save_pretrained_gguf, model) model.push_to_hub_gguf = types.MethodType(unsloth_push_to_hub_gguf, model)
model.push_to_hub_ggml = types.MethodType(unsloth_convert_lora_to_ggml_and_push_to_hub, model) model.save_pretrained_gguf = types.MethodType(unsloth_save_pretrained_gguf, model)
model.save_pretrained_ggml = types.MethodType(unsloth_convert_lora_to_ggml_and_save_locally, model) model.push_to_hub_ggml = types.MethodType(unsloth_convert_lora_to_ggml_and_push_to_hub, model)
model.save_pretrained_ggml = types.MethodType(unsloth_convert_lora_to_ggml_and_save_locally, model)
pass
else:
# Vision only 1 option
model.push_to_hub_merged = types.MethodType(unsloth_generic_push_to_hub_merged, model)
model.save_pretrained_merged = types.MethodType(unsloth_generic_save_pretrained_merged, model)
model.push_to_hub_gguf = types.MethodType(not_implemented_save, model)
model.save_pretrained_gguf = types.MethodType(not_implemented_save, model)
pass pass
return model return model
pass pass

31
unsloth/trainer.py
View file

@ -20,13 +20,13 @@ from functools import wraps
import trl import trl
import inspect import inspect
from trl import SFTTrainer from trl import SFTTrainer
try:
from trl import SFTConfig as TrainingArguments
except:
from transformers import TrainingArguments
pass
from . import is_bfloat16_supported from . import is_bfloat16_supported
from unsloth_zoo.training_utils import unsloth_train as _unsloth_train from unsloth_zoo.training_utils import (
unsloth_train as _unsloth_train,
)
from unsloth_zoo.vision_utils import (
UnslothVisionDataCollator,
)
from packaging.version import Version from packaging.version import Version
import dataclasses import dataclasses
@ -35,6 +35,7 @@ __all__ = [
"UnslothTrainer", "UnslothTrainer",
"unsloth_train", "unsloth_train",
"_patch_trl_trainer", "_patch_trl_trainer",
"UnslothVisionDataCollator",
] ]
# Unsloth gradient accumulation fix: # Unsloth gradient accumulation fix:
@ -60,7 +61,11 @@ else:
pass pass
pass pass
try:
from trl import SFTConfig as TrainingArguments
except:
from transformers import TrainingArguments
pass
@dataclass @dataclass
class UnslothTrainingArguments(TrainingArguments): class UnslothTrainingArguments(TrainingArguments):
embedding_learning_rate : Optional[float] = field( embedding_learning_rate : Optional[float] = field(
@ -134,7 +139,7 @@ pass
# From `trl>=0.13.0`, they changed how to pass several params to the trainer # From `trl>=0.13.0`, they changed how to pass several params to the trainer
# We need to patch to make the transition smooth # We need to patch to make the transition smooth
def create_backwards_compatible_trainer(trainer_class, config_class): def _backwards_compatible_trainer(trainer_class, config_class):
original_init = trainer_class.__init__ original_init = trainer_class.__init__
@wraps(original_init) @wraps(original_init)
@ -167,6 +172,7 @@ def create_backwards_compatible_trainer(trainer_class, config_class):
} }
# Get parameters that exist in Config but not in TrainingArguments # Get parameters that exist in Config but not in TrainingArguments
from transformers import TrainingArguments
moved_params = \ moved_params = \
set(inspect.signature(config_class) .parameters.keys()) - \ set(inspect.signature(config_class) .parameters.keys()) - \
set(inspect.signature(TrainingArguments).parameters.keys()) set(inspect.signature(TrainingArguments).parameters.keys())
@ -207,14 +213,13 @@ def _patch_trl_trainer():
import trl.trainer import trl.trainer
trl_classes = dir(trl.trainer) trl_classes = dir(trl.trainer)
trl_trainers = set(x[:-len("Trainer")] for x in trl_classes if x.endswith("Trainer"))
non_convertable_trainer = set(["PPOv2", "AlignProp"]) trl_configs = set(x[:-len("Config")] for x in trl_classes if x.endswith("Config"))
trl_trainers = set(x[:-len("Trainer")] for x in trl_classes if x.endswith("Trainer")) - non_convertable_trainer
trl_configs = set(x[:-len("Config")] for x in trl_classes if x.endswith("Config")) - non_convertable_trainer
trl_classes = list(trl_trainers & trl_configs) trl_classes = list(trl_trainers & trl_configs)
for x in trl_classes: for x in trl_classes:
exec(f"trl.{x}Trainer.__init__ = create_backwards_compatible_trainer(trl.{x}Trainer, trl.{x}Config)", globals()) try: exec(f"trl.{x}Trainer.__init__ = _backwards_compatible_trainer(trl.{x}Trainer, trl.{x}Config)", globals())
except: continue
pass pass
trl.__UNSLOTH_BACKWARDS_COMPATIBLE__ = True trl.__UNSLOTH_BACKWARDS_COMPATIBLE__ = True