Bug fixes (#1951)
* Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update llama.py * Update _utils.py * Update llama.py * Update _utils.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update rl_replacements.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * GRPO optimized * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Selective Log softmax * Fix GRPO bsz * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Fix TRL * Metrics GRPO * Update rl_replacements.py * Update rl_replacements.py * No compile * Update rl.py * Remove docs * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * llama-quantize on WINDOWS WSL error fix - edit save.py (gguf saving breaks) (#1649) * edit save.py to fix gguf saving breaks. * add check for .exe or not exe file extension for linux and windows * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update llama.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * unsloth_num_chunks * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py (#1754) Fix typo in comment: know -> now. This was printed when running the Llama3.1_(8B)-GRPO.ipynb example notebook, so I'd expect others to run into it as well. * Optional logits * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * fix an import error (#1767) * fix an import error * Delete .gitignore * Update loader.py * Update save.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * SamplingParams * Convert mask to float (#1762) * [Windows Support] Add latest `xformers` wheels to pyproject.toml (#1753) * Add latest xformers * Add a couple of lines to docs * vLLMSamplingParams * Update __init__.py * default num_chunks == -1 * Versioning * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update _utils.py * Update rl_replacements.py * Update rl_replacements.py * Update pyproject.toml * Update pyproject.toml * Export Model to ollama.com (#1648) * Ollama Export Model to ollama.com Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> * Check for model_name Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> * subprocess use instead of requests | added check for ollama server Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> * create_ollama_model Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> * create_ollama_model | fix Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> * Push to Ollama Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> --------- Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> * Update cross_entropy_loss.py * torch_cuda_device * Update utils.py * Update utils.py * Update utils.py * device * device * Update loader.py * Update llama.py * Update README.md * Update llama.py * Update llama.py * Update _utils.py * Update utils.py * Update utils.py * Update utils.py * Update utils.py * Update utils.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update utils.py * Update utils.py * Update utils.py * Update utils.py * __version__ * Update rl.py * Bug fixes * Bug fixes * Update llama.py * Update _utils.py * _wrap_fast_inference * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update _utils.py * SFT dataset prepare * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update llama.py * Update llama.py * Update utils.py * bug fix * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update __init__.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update rl.py * Update rl.py * Update rl.py * Update _utils.py * Update __init__.py * Update _utils.py * Version * versioning * Update _utils.py * Update llama.py * Update llama.py * Bug fixes * FastModel * __doc__ * Update vision.py * Update loader.py * Update loader.py * Update loader.py * version --------- Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> Co-authored-by: Gennadii Manzhos <105049664+everythingisc00l@users.noreply.github.com> Co-authored-by: Seth Weidman <seth@sethweidman.com> Co-authored-by: Nino Risteski <95188570+NinoRisteski@users.noreply.github.com> Co-authored-by: Edd <68678137+Erland366@users.noreply.github.com> Co-authored-by: Ben <6579034+versipellis@users.noreply.github.com> Co-authored-by: Jyotin Goel <120490013+gjyotin305@users.noreply.github.com>
This commit is contained in:
Daniel Han
GitHub
parent
81506a8f2f
commit
49186bee79
8 changed files with 196 additions and 134 deletions
|
|
@ -40,7 +40,7 @@ triton = [
|
|||
]
|
||||
|
||||
huggingface = [
|
||||
"unsloth_zoo>=2025.3.7",
|
||||
"unsloth_zoo>=2025.3.8",
|
||||
"packaging",
|
||||
"tyro",
|
||||
"transformers>=4.46.1,!=4.47.0",
|
||||
|
|
@ -354,7 +354,7 @@ colab-ampere-torch220 = [
|
|||
"flash-attn>=2.6.3",
|
||||
]
|
||||
colab-new = [
|
||||
"unsloth_zoo>=2025.3.7",
|
||||
"unsloth_zoo>=2025.3.8",
|
||||
"packaging",
|
||||
"tyro",
|
||||
"transformers>=4.46.1,!=4.47.0",
|
||||
|
|
|
|||
|
|
@ -198,7 +198,7 @@ pass
|
|||
# Check for unsloth_zoo
|
||||
try:
|
||||
unsloth_zoo_version = importlib_version("unsloth_zoo")
|
||||
if Version(unsloth_zoo_version) < Version("2025.3.7"):
|
||||
if Version(unsloth_zoo_version) < Version("2025.3.8"):
|
||||
try:
|
||||
os.system("pip install --upgrade --no-cache-dir --no-deps unsloth_zoo")
|
||||
except:
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@
|
|||
# limitations under the License.
|
||||
|
||||
from .llama import FastLlamaModel
|
||||
from .loader import FastLanguageModel, FastVisionModel
|
||||
from .loader import FastLanguageModel, FastVisionModel, FastTextModel, FastModel
|
||||
from .mistral import FastMistralModel
|
||||
from .qwen2 import FastQwen2Model
|
||||
from .granite import FastGraniteModel
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@
|
|||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
__version__ = "2025.3.8"
|
||||
__version__ = "2025.3.9"
|
||||
|
||||
__all__ = [
|
||||
"SUPPORTS_BFLOAT16",
|
||||
|
|
|
|||
|
|
@ -91,7 +91,7 @@ def original_apply_o(self, X):
|
|||
pass
|
||||
|
||||
from math import sqrt as math_sqrt
|
||||
KV_CACHE_INCREMENT = 256 # KV Cache update size
|
||||
KV_CACHE_INCREMENT = 512 # KV Cache update size
|
||||
torch_nn_functional_softmax = torch.nn.functional.softmax
|
||||
# SDPA has GQA internally
|
||||
SDPA_HAS_GQA = "enable_gqa" in scaled_dot_product_attention.__doc__
|
||||
|
|
@ -1656,6 +1656,13 @@ class FastLlamaModel:
|
|||
"Are you certain you want to do remote code execution?"
|
||||
)
|
||||
pass
|
||||
if fast_inference:
|
||||
import platform
|
||||
if platform.system().lower() == 'windows':
|
||||
print("Unsloth: vLLM does not work in Windows! Will use Unsloth inference!")
|
||||
fast_inference = False
|
||||
pass
|
||||
|
||||
if token is None: token = get_token()
|
||||
if model_patcher is None: model_patcher = FastLlamaModel
|
||||
SUPPORTS_BFLOAT16 = is_bfloat16_supported()
|
||||
|
|
@ -1966,12 +1973,17 @@ class FastLlamaModel:
|
|||
for layer in model.model.layers:
|
||||
layer.self_attn.rotary_emb = rotary_emb
|
||||
pass
|
||||
|
||||
|
||||
# Add for_inference and for_training
|
||||
model.for_training = functools.partial(FastLlamaModel.for_training, model)
|
||||
model.for_inference = functools.partial(FastLlamaModel.for_inference, model)
|
||||
|
||||
# Patch generate
|
||||
if model.generate.__name__ != "unsloth_fast_generate":
|
||||
model._old_generate = model.generate
|
||||
unsloth_fast_generate.__doc__ = model._old_generate.__doc__
|
||||
model.generate = types.MethodType(unsloth_fast_generate, model)
|
||||
pass
|
||||
return model, tokenizer
|
||||
pass
|
||||
|
||||
|
|
@ -2404,7 +2416,7 @@ class FastLlamaModel:
|
|||
# Add for_inference and for_training
|
||||
model.for_training = functools.partial(FastLlamaModel.for_training, model)
|
||||
model.for_inference = functools.partial(FastLlamaModel.for_inference, model)
|
||||
|
||||
|
||||
# Patch generate
|
||||
if model.generate.__name__ != "unsloth_fast_generate":
|
||||
model._old_generate = model.generate
|
||||
|
|
|
|||
|
|
@ -383,10 +383,13 @@ from ..kernels import (
|
|||
patch_loss_functions,
|
||||
post_patch_loss_function,
|
||||
)
|
||||
from .vision import FastBaseVisionModel
|
||||
from .vision import FastBaseModel
|
||||
from transformers import (
|
||||
AutoModelForVision2Seq,
|
||||
AutoModelForCausalLM,
|
||||
)
|
||||
|
||||
|
||||
class FastVisionModel(FastBaseVisionModel):
|
||||
class FastModel(FastBaseModel):
|
||||
@staticmethod
|
||||
def from_pretrained(
|
||||
model_name = "unsloth/Llama-3.2-11B-Vision-Instruct-bnb-4bit",
|
||||
|
|
@ -413,7 +416,7 @@ class FastVisionModel(FastBaseVisionModel):
|
|||
patch_compiling_bitsandbytes()
|
||||
if use_gradient_checkpointing == "unsloth":
|
||||
patch_unsloth_smart_gradient_checkpointing(dtype = dtype)
|
||||
|
||||
|
||||
old_model_name = model_name
|
||||
if not use_exact_model_name:
|
||||
model_name = get_model_name(model_name, load_in_4bit)
|
||||
|
|
@ -427,7 +430,7 @@ class FastVisionModel(FastBaseVisionModel):
|
|||
from huggingface_hub.utils import disable_progress_bars, enable_progress_bars, are_progress_bars_disabled
|
||||
was_disabled = are_progress_bars_disabled()
|
||||
disable_progress_bars()
|
||||
|
||||
|
||||
autoconfig_error = None
|
||||
peft_error = None
|
||||
try:
|
||||
|
|
@ -458,7 +461,7 @@ class FastVisionModel(FastBaseVisionModel):
|
|||
|
||||
# Old transformers versions check
|
||||
both_exist = (is_model and is_peft) and not SUPPORTS_LLAMA32
|
||||
|
||||
|
||||
# New transformers need to check manually.
|
||||
if SUPPORTS_LLAMA32:
|
||||
# Check if folder exists locally
|
||||
|
|
@ -515,9 +518,12 @@ class FastVisionModel(FastBaseVisionModel):
|
|||
if not was_disabled: enable_progress_bars()
|
||||
|
||||
do_logging = os.environ.get("UNSLOTH_ENABLE_LOGGING", "0") == "1"
|
||||
redirector = sys.stdout if do_logging else open(os.devnull, "w")
|
||||
if do_logging:
|
||||
redirector = contextlib.nullcontext()
|
||||
else:
|
||||
redirector = contextlib.redirect_stdout(open(os.devnull, "w"))
|
||||
|
||||
with contextlib.redirect_stdout(redirector):
|
||||
with redirector:
|
||||
patch_loss_functions(torch_compile = False)
|
||||
model_types = unsloth_compile_transformers(
|
||||
model_name = model_name,
|
||||
|
|
@ -547,7 +553,6 @@ class FastVisionModel(FastBaseVisionModel):
|
|||
return_logits = return_logits,
|
||||
)
|
||||
pass
|
||||
if do_logging: redirector.close()
|
||||
|
||||
# Check if this is local model since the tokenizer gets overwritten
|
||||
if os.path.exists(os.path.join(old_model_name, "tokenizer_config.json")) and \
|
||||
|
|
@ -559,7 +564,12 @@ class FastVisionModel(FastBaseVisionModel):
|
|||
tokenizer_name = None
|
||||
pass
|
||||
|
||||
model, tokenizer = FastBaseVisionModel.from_pretrained(
|
||||
# Check if VLM
|
||||
is_vlm = (x.endswith("ForConditionalGeneration") for x in model_config.architectures)
|
||||
is_vlm = is_vlm or hasattr(model_config, "vision_config")
|
||||
auto_model = AutoModelForVision2Seq if is_vlm else AutoModelForCausalLM
|
||||
|
||||
model, tokenizer = FastBaseModel.from_pretrained(
|
||||
model_name = model_name,
|
||||
max_seq_length = max_seq_length,
|
||||
dtype = _get_dtype(dtype),
|
||||
|
|
@ -570,6 +580,7 @@ class FastVisionModel(FastBaseVisionModel):
|
|||
revision = revision if not is_peft else None,
|
||||
model_types = model_types,
|
||||
tokenizer_name = tokenizer_name,
|
||||
auto_model = auto_model,
|
||||
*args, **kwargs,
|
||||
)
|
||||
|
||||
|
|
@ -617,8 +628,14 @@ class FastVisionModel(FastBaseVisionModel):
|
|||
trust_remote_code = trust_remote_code,
|
||||
)
|
||||
# Patch it as well!
|
||||
model = FastBaseVisionModel.patch_peft_model(model, use_gradient_checkpointing)
|
||||
model = FastBaseModel.patch_peft_model(model, use_gradient_checkpointing)
|
||||
pass
|
||||
return model, tokenizer
|
||||
pass
|
||||
pass
|
||||
|
||||
class FastVisionModel(FastModel):
|
||||
pass
|
||||
|
||||
class FastTextModel(FastModel):
|
||||
pass
|
||||
|
|
|
|||
|
|
@ -611,6 +611,21 @@ __INT_TO_FLOAT_MAPPER = \
|
|||
"open-thoughts/OpenThinker-7B",
|
||||
"unsloth/OpenThinker-7B-bnb-4bit",
|
||||
),
|
||||
"unsloth/granite-3.2-2b-instruct-unsloth-bnb-4bit" : (
|
||||
"unsloth/granite-3.2-2b-instruct",
|
||||
"ibm-granite/granite-3.2-2b-instruct",
|
||||
"unsloth/granite-3.2-2b-instruct-bnb-4bit",
|
||||
),
|
||||
"unsloth/granite-3.2-8b-instruct-unsloth-bnb-4bit" : (
|
||||
"unsloth/granite-3.2-8b-instruct",
|
||||
"ibm-granite/granite-3.2-8b-instruct",
|
||||
"unsloth/granite-3.2-8b-instruct-bnb-4bit",
|
||||
),
|
||||
"unsloth/QwQ-32B-unsloth-bnb-4bit" : (
|
||||
"unsloth/QwQ-32B",
|
||||
"Qwen/QwQ-32B",
|
||||
"unsloth/QwQ-32B-bnb-4bit",
|
||||
),
|
||||
}
|
||||
|
||||
INT_TO_FLOAT_MAPPER = {}
|
||||
|
|
|
|||
|
|
@ -17,6 +17,8 @@ from transformers import (
|
|||
BitsAndBytesConfig,
|
||||
AutoModelForVision2Seq,
|
||||
AutoProcessor,
|
||||
AutoTokenizer,
|
||||
AutoModelForCausalLM,
|
||||
)
|
||||
from .llama import *
|
||||
from ..kernels import (
|
||||
|
|
@ -31,48 +33,60 @@ from unsloth_zoo.peft_utils import (
|
|||
requires_grad_for_gradient_checkpointing,
|
||||
)
|
||||
from triton import __version__ as triton_version
|
||||
from unsloth_zoo.utils import _get_dtype
|
||||
from unsloth_zoo.patching_utils import patch_model_and_tokenizer
|
||||
import types
|
||||
import functools
|
||||
|
||||
__all__ = [
|
||||
"FastBaseVisionModel",
|
||||
"FastBaseModel",
|
||||
]
|
||||
|
||||
def _wrap_fast_inference(generate, device_type, dtype, model):
|
||||
# Wraps inference with bfloat16 / float16
|
||||
@torch.inference_mode
|
||||
def _fast_generate(*args, **kwargs):
|
||||
# For num_logits_to_keep
|
||||
# kwargs["num_logits_to_keep"] = 1
|
||||
|
||||
# Remove token_type_ids
|
||||
kwargs.pop("token_type_ids", None)
|
||||
def unsloth_base_fast_generate(
|
||||
self,
|
||||
*args,
|
||||
**kwargs,
|
||||
):
|
||||
FastBaseModel.for_inference(self)
|
||||
dtype = _get_dtype(self.config.torch_dtype)
|
||||
|
||||
# Check pad_token
|
||||
model_eos_token_id = getattr(model.config, "eos_token_id", None)
|
||||
if model_eos_token_id is not None and hasattr(model_eos_token_id, "__iter__"):
|
||||
model_eos_token_id = model_eos_token_id[0]
|
||||
# Check if VLM
|
||||
is_vlm = (x.endswith("ForConditionalGeneration") for x in self.config.architectures)
|
||||
is_vlm = is_vlm or hasattr(self.config, "vision_config")
|
||||
|
||||
kwargs["pad_token_id"] = kwargs.pop("pad_token_id", model_eos_token_id)
|
||||
# Remove token_type_ids
|
||||
kwargs.pop("token_type_ids", None)
|
||||
|
||||
try:
|
||||
kwargs["pixel_values"] = kwargs["pixel_values"].to(model.dtype)
|
||||
except:
|
||||
pass
|
||||
# VLMs do not allow logits_to_keep
|
||||
if not is_vlm: kwargs["logits_to_keep"] = 1
|
||||
|
||||
# Autocasted
|
||||
with torch.autocast(device_type = device_type, dtype = dtype):
|
||||
output = generate(*args, **kwargs)
|
||||
pass
|
||||
return output
|
||||
# Check pad_token
|
||||
model_eos_token_id = getattr(self.config, "eos_token_id", None)
|
||||
if model_eos_token_id is not None and hasattr(model_eos_token_id, "__iter__"):
|
||||
model_eos_token_id = model_eos_token_id[0]
|
||||
|
||||
kwargs["pad_token_id"] = kwargs.pop("pad_token_id", model_eos_token_id)
|
||||
|
||||
# Get pixel values for VLMs
|
||||
try: kwargs["pixel_values"] = kwargs["pixel_values"].to(dtype)
|
||||
except: pass
|
||||
|
||||
# Mixed precision autocast
|
||||
with torch.inference_mode(), torch.autocast(device_type = "cuda", dtype = dtype):
|
||||
output = self._old_generate(*args, **kwargs)
|
||||
pass
|
||||
return _fast_generate
|
||||
|
||||
FastBaseModel.for_training(self)
|
||||
return output
|
||||
pass
|
||||
|
||||
|
||||
class FastBaseVisionModel:
|
||||
class FastBaseModel:
|
||||
|
||||
@staticmethod
|
||||
def from_pretrained(
|
||||
model_name = "unsloth/llama-3-8b-bnb-4bit",
|
||||
model_name = "unsloth/Llama-3.2-1B-Instruct",
|
||||
max_seq_length = None,
|
||||
dtype = None,
|
||||
load_in_4bit = True,
|
||||
|
|
@ -81,6 +95,7 @@ class FastBaseVisionModel:
|
|||
trust_remote_code = False,
|
||||
model_types = None,
|
||||
tokenizer_name = None,
|
||||
auto_model = AutoModelForVision2Seq,
|
||||
**kwargs,
|
||||
):
|
||||
if trust_remote_code:
|
||||
|
|
@ -94,12 +109,16 @@ class FastBaseVisionModel:
|
|||
gpu_stats = torch.cuda.get_device_properties(0)
|
||||
max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)
|
||||
|
||||
from importlib.metadata import version as importlib_version
|
||||
try: vllm_version = f" vLLM: {importlib_version('vllm')}."
|
||||
except: vllm_version = ""
|
||||
|
||||
statistics = \
|
||||
f"==((====))== Unsloth {__version__}: Fast {model_types[0].title()} vision patching. Transformers: {transformers_version}.\n"\
|
||||
f" {chr(92)}{chr(92)} /| GPU: {gpu_stats.name}. Max memory: {max_memory} GB. Platform: {platform_system}.\n"\
|
||||
f"==((====))== Unsloth {__version__}: Fast {model_types[0].title()} patching. Transformers: {transformers_version}.{vllm_version}\n"\
|
||||
f" {chr(92)}{chr(92)} /| {gpu_stats.name}. Num GPUs = {torch.cuda.device_count()}. Max memory: {max_memory} GB. Platform: {platform_system}.\n"\
|
||||
f"O^O/ {chr(92)}_/ {chr(92)} Torch: {torch.__version__}. CUDA: {gpu_stats.major}.{gpu_stats.minor}. CUDA Toolkit: {torch.version.cuda}. Triton: {triton_version}\n"\
|
||||
f"{chr(92)} / Bfloat16 = {str(SUPPORTS_BFLOAT16).upper()}. FA [Xformers = {xformers_version}. FA2 = {HAS_FLASH_ATTENTION}]\n"\
|
||||
f' "-____-" Free Apache license: http://github.com/unslothai/unsloth'
|
||||
f' "-____-" Free license: http://github.com/unslothai/unsloth'
|
||||
print(statistics)
|
||||
|
||||
# Warn about fast transfers
|
||||
|
|
@ -136,8 +155,8 @@ class FastBaseVisionModel:
|
|||
|
||||
# Cannot be None, since HF now checks for the config
|
||||
if load_in_4bit: kwargs["quantization_config"] = bnb_config
|
||||
|
||||
model = AutoModelForVision2Seq.from_pretrained(
|
||||
|
||||
model = auto_model.from_pretrained(
|
||||
model_name,
|
||||
device_map = device_map,
|
||||
torch_dtype = dtype,
|
||||
|
|
@ -152,26 +171,25 @@ class FastBaseVisionModel:
|
|||
|
||||
# Counteract saved tokenizers
|
||||
tokenizer_name = model_name if tokenizer_name is None else tokenizer_name
|
||||
tokenizer = AutoProcessor.from_pretrained(
|
||||
auto_processor = AutoProcessor if auto_model is AutoModelForVision2Seq else AutoTokenizer
|
||||
tokenizer = auto_processor.from_pretrained(
|
||||
tokenizer_name,
|
||||
padding_side = "right",
|
||||
token = token,
|
||||
)
|
||||
# Add padding side as well
|
||||
tokenizer.tokenizer.padding_side = "right"
|
||||
if hasattr(tokenizer, "tokenizer"):
|
||||
tokenizer.tokenizer.padding_side = "right"
|
||||
|
||||
model, tokenizer = patch_tokenizer(model, tokenizer)
|
||||
model = post_patch_loss_function(model)
|
||||
|
||||
# Fix up config for transformers uploading PEFT
|
||||
# Not necessary anymore since we require transformers>=4.37!
|
||||
if False:
|
||||
name = model.config._name_or_path
|
||||
if name.startswith("unsloth/") and name.endswith("-bnb-4bit"):
|
||||
name = name[:len(name) - len("-bnb-4bit")]
|
||||
model.config.update({"_name_or_path" : name})
|
||||
pass
|
||||
pass
|
||||
# Fix other stuff like BnB compute data types
|
||||
model, tokenizer = patch_model_and_tokenizer(
|
||||
model,
|
||||
tokenizer,
|
||||
downcast_rope = False,
|
||||
fix_embeddings = False,
|
||||
)
|
||||
|
||||
# Log Unsloth version for future fastpaths for inference
|
||||
if hasattr(model, "config"):
|
||||
|
|
@ -187,13 +205,22 @@ class FastBaseVisionModel:
|
|||
# Save tokenizer for inference purposes
|
||||
tokenizer.padding_side = "left" # Force inference
|
||||
tokenizer.tokenizer.padding_side = "left" # Force inference
|
||||
internal_model = model
|
||||
while hasattr(internal_model, "model"):
|
||||
internal_model._saved_temp_tokenizer = tokenizer
|
||||
internal_model = internal_model.model
|
||||
m = model
|
||||
while hasattr(m, "model"):
|
||||
m._saved_temp_tokenizer = tokenizer
|
||||
# Also set is_loaded_in_8bit to disable incorrect DDP
|
||||
m.is_loaded_in_8bit = True
|
||||
m = m.model
|
||||
pass
|
||||
internal_model._saved_temp_tokenizer = tokenizer
|
||||
|
||||
m._saved_temp_tokenizer = tokenizer
|
||||
# Also set is_loaded_in_8bit to disable incorrect DDP
|
||||
m.is_loaded_in_8bit = True
|
||||
|
||||
# Patch generate
|
||||
if model.generate.__name__ != "unsloth_base_fast_generate":
|
||||
model._old_generate = model.generate
|
||||
unsloth_base_fast_generate.__doc__ = model._old_generate.__doc__
|
||||
model.generate = types.MethodType(unsloth_base_fast_generate, model)
|
||||
return model, tokenizer
|
||||
pass
|
||||
|
||||
|
|
@ -272,7 +299,7 @@ class FastBaseVisionModel:
|
|||
# Enable gradients on modules which are trainable
|
||||
requires_grad_for_gradient_checkpointing(model)
|
||||
|
||||
model = FastBaseVisionModel.patch_peft_model(model, use_gradient_checkpointing)
|
||||
model = FastBaseModel.patch_peft_model(model, use_gradient_checkpointing)
|
||||
|
||||
# Clear deleted GPU items
|
||||
for _ in range(3):
|
||||
|
|
@ -281,6 +308,9 @@ class FastBaseVisionModel:
|
|||
pass
|
||||
patch_saving_functions(model, vision = True)
|
||||
|
||||
# Add for_inference and for_training
|
||||
model.for_training = functools.partial(FastBaseModel.for_training, model)
|
||||
model.for_inference = functools.partial(FastBaseModel.for_inference, model)
|
||||
return model
|
||||
pass
|
||||
|
||||
|
|
@ -314,62 +344,57 @@ class FastBaseVisionModel:
|
|||
patch_saving_functions(model, vision = True)
|
||||
|
||||
# Patch tokenizer to pad to the right
|
||||
internal_model = model
|
||||
while hasattr(internal_model, "model"):
|
||||
if hasattr(internal_model, "_saved_temp_tokenizer"):
|
||||
internal_model._saved_temp_tokenizer.tokenizer.padding_side = "right"
|
||||
m = model
|
||||
while hasattr(m, "model"):
|
||||
if hasattr(m, "_saved_temp_tokenizer"):
|
||||
m._saved_temp_tokenizer.tokenizer.padding_side = "right"
|
||||
pass
|
||||
internal_model = internal_model.model
|
||||
# Also set is_loaded_in_8bit to disable incorrect DDP
|
||||
m.is_loaded_in_8bit = True
|
||||
m = m.model
|
||||
pass
|
||||
if hasattr(internal_model, "_saved_temp_tokenizer"):
|
||||
internal_model._saved_temp_tokenizer.tokenizer.padding_side = "right"
|
||||
if hasattr(m, "_saved_temp_tokenizer"):
|
||||
m._saved_temp_tokenizer.tokenizer.padding_side = "right"
|
||||
pass
|
||||
# Also set is_loaded_in_8bit to disable incorrect DDP
|
||||
m.is_loaded_in_8bit = True
|
||||
|
||||
# Clear deleted GPU items
|
||||
for _ in range(3):
|
||||
gc.collect()
|
||||
torch.cuda.empty_cache()
|
||||
pass
|
||||
# Add for_inference and for_training
|
||||
model.for_training = functools.partial(FastBaseModel.for_training, model)
|
||||
model.for_inference = functools.partial(FastBaseModel.for_inference, model)
|
||||
|
||||
# Patch generate
|
||||
if model.generate.__name__ != "unsloth_base_fast_generate":
|
||||
model._old_generate = model.generate
|
||||
unsloth_base_fast_generate.__doc__ = model._old_generate.__doc__
|
||||
model.generate = types.MethodType(unsloth_base_fast_generate, model)
|
||||
return model
|
||||
pass
|
||||
|
||||
|
||||
@staticmethod
|
||||
def for_inference(model):
|
||||
model.gradient_checkpointing = False
|
||||
model.training = False
|
||||
if not hasattr(model, "parameters"):
|
||||
raise TypeError("Unsloth: I think you're passing a tokenizer, not the model to for_inference!")
|
||||
|
||||
for name, module in model.named_modules():
|
||||
if hasattr(module, "gradient_checkpointing"):
|
||||
module.gradient_checkpointing = False
|
||||
if hasattr(module, "training"):
|
||||
module.training = False
|
||||
pass
|
||||
|
||||
dtype = model.config.torch_dtype
|
||||
if type(dtype) is str:
|
||||
if dtype == "float16": dtype = torch.float16
|
||||
elif dtype == "bfloat16": dtype = torch.bfloat16
|
||||
pass
|
||||
device_type = model.device.type
|
||||
|
||||
# Wrap model.generate
|
||||
if model.generate.__name__ != "_fast_generate":
|
||||
model._unwrapped_old_generate = model.generate
|
||||
model.generate = _wrap_fast_inference(model.generate, device_type, dtype, model)
|
||||
pass
|
||||
|
||||
# Patch tokenizer to pad to the left
|
||||
internal_model = model
|
||||
while hasattr(internal_model, "model"):
|
||||
if hasattr(internal_model, "_saved_temp_tokenizer"):
|
||||
internal_model._saved_temp_tokenizer.tokenizer.padding_side = "left"
|
||||
pass
|
||||
internal_model = internal_model.model
|
||||
pass
|
||||
if hasattr(internal_model, "_saved_temp_tokenizer"):
|
||||
internal_model._saved_temp_tokenizer.tokenizer.padding_side = "left"
|
||||
def _for_inference(m):
|
||||
if hasattr(m, "gradient_checkpointing"): m.gradient_checkpointing = False
|
||||
if hasattr(m, "training"): m.training = False
|
||||
# Pad tokenizer to the left
|
||||
if hasattr(m, "_saved_temp_tokenizer"): m._saved_temp_tokenizer.padding_side = "left"
|
||||
# Set a flag for generation!
|
||||
m._flag_for_generation = True
|
||||
pass
|
||||
m = model
|
||||
while hasattr(m, "model"):
|
||||
_for_inference(m)
|
||||
m = m.model
|
||||
_for_inference(m)
|
||||
|
||||
# Also disable training for embeddings for NEFTune
|
||||
if hasattr(model, "get_input_embeddings"):
|
||||
|
|
@ -380,40 +405,34 @@ class FastBaseVisionModel:
|
|||
embeddings = model.get_output_embeddings()
|
||||
if hasattr(embeddings, "training"): embeddings.training = False
|
||||
pass
|
||||
|
||||
return model
|
||||
pass
|
||||
|
||||
|
||||
@staticmethod
|
||||
def for_training(model, use_gradient_checkpointing = True):
|
||||
model.gradient_checkpointing = use_gradient_checkpointing
|
||||
model.training = True
|
||||
if not hasattr(model, "parameters"):
|
||||
raise TypeError("Unsloth: I think you're passing a tokenizer, not the model to for_training!")
|
||||
|
||||
for name, module in model.named_modules():
|
||||
if hasattr(module, "gradient_checkpointing"):
|
||||
module.gradient_checkpointing = use_gradient_checkpointing
|
||||
if hasattr(module, "training"):
|
||||
module.training = True
|
||||
# Delete all fast inference loras
|
||||
for param in model.parameters():
|
||||
if hasattr(param, "_fast_lora"):
|
||||
del param._fast_lora
|
||||
pass
|
||||
|
||||
# Also revert model.generate
|
||||
if hasattr(model, "_unwrapped_old_generate"):
|
||||
model.generate = model._unwrapped_old_generate
|
||||
del model._unwrapped_old_generate
|
||||
pass
|
||||
|
||||
# Patch tokenizer to pad to the right
|
||||
internal_model = model
|
||||
while hasattr(internal_model, "model"):
|
||||
if hasattr(internal_model, "_saved_temp_tokenizer"):
|
||||
internal_model._saved_temp_tokenizer.tokenizer.padding_side = "right"
|
||||
pass
|
||||
internal_model = internal_model.model
|
||||
pass
|
||||
if hasattr(internal_model, "_saved_temp_tokenizer"):
|
||||
internal_model._saved_temp_tokenizer.tokenizer.padding_side = "right"
|
||||
def _for_training(m):
|
||||
if hasattr(m, "gradient_checkpointing"): m.gradient_checkpointing = use_gradient_checkpointing
|
||||
if hasattr(m, "training"): m.training = True
|
||||
# Pad tokenizer to the left
|
||||
if hasattr(m, "_saved_temp_tokenizer"): m._saved_temp_tokenizer.padding_side = "right"
|
||||
# Set a flag for generation!
|
||||
if hasattr(m, "_flag_for_generation"): del m._flag_for_generation
|
||||
pass
|
||||
m = model
|
||||
while hasattr(m, "model"):
|
||||
_for_training(m)
|
||||
m = m.model
|
||||
_for_training(m)
|
||||
|
||||
# Also re-enable training for embeddings for NEFTune
|
||||
if hasattr(model, "get_input_embeddings"):
|
||||
|
|
@ -424,7 +443,6 @@ class FastBaseVisionModel:
|
|||
embeddings = model.get_output_embeddings()
|
||||
if hasattr(embeddings, "training"): embeddings.training = True
|
||||
pass
|
||||
|
||||
return model
|
||||
pass
|
||||
pass
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue