ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Torch dtype (#314)

Browse source 
* Update chat_templates.py

* Update save.py

* Update save.py

* Update save.py

* Update chat_templates.py

* Update llama.py

* model_name

* Update loader.py

* Tokenizer overwritten

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Accuracy

* Revert

* Update save.py

* Update fast_lora.py

* Update fast_lora.py

* Update fast_lora.py

* Update fast_lora.py

* Update fast_lora.py

* Update chat_templates.py

* Update save.py

* Update save.py

* Update llama.py

* Update llama.py

* Account for DoRA

* Update llama.py

* Update save.py

* GGUF incorrect

* Update save.py

* Update pyproject.toml

* kaggle new

* Update pyproject.toml

* Update pyproject.toml

* upcasting

* Fix Colab

* Update pyproject.toml

* Update pyproject.toml

* Update pyproject.toml

* Update pyproject.toml

* Update pyproject.toml

* Update pyproject.toml

* Update pyproject.toml

* Update pyproject.toml

* Update chat_templates.py

* Update chat_templates.py

* Update chat_templates.py

* Update chat_templates.py

* Update chat_templates.py

* Update pyproject.toml

* Update pyproject.toml

* Update pyproject.toml

* Update rope_embedding.py

* Update rope_embedding.py

* Fix bugs

* Update fast_lora.py

* Update fast_lora.py

* Update README.md

* Update README.md

* GGUF

* Update save.py

* Update save.py

* Update save.py

* Update save.py

* Update README.md

* Update README.md

* Bugs

* Update fast_lora.py

* Update pyproject.toml

* Update fast_lora.py

* Update __init__.py

* Update fast_lora.py

* dtype

* Update llama.py

* Update llama.py

* Update llama.py

* dtype

* Update mistral.py

* trust_remote_code

* lm_head

* Update llama.py

* save_pretrained_settings

* Update save.py

* Update save.py

* Update save.py

* Update save.py

* Update save.py

* Update save.py

* Update save.py

* Update save.py

* Update save.py

* Update save.py

* Update save.py

* Update save.py

* state_dict

* Update save.py

* whoami

* Update llama.py

* Update save.py

* Update llama.py

* Patch tokenizer

* Update chat_templates.py

* Heal tokenizers

* Update chat_templates.py

* Update mapper.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update chat_templates.py

* tokenizer patching

* patch_tokenizer

* Update chat_templates.py

* Update tokenizer_utils.py

* Update chat_templates.py

* Update chat_templates.py

* Update chat_templates.py

* Update tokenizer_utils.py

* Edit

* Update mistral.py

* Update mistral.py

* Stats

* Update mistral.py

* attention_mask

* Update llama.py

* Update llama.py

* batch

* Temp fix batch inference

* Update llama.py

* Update gemma.py

* Fix inference

* swiglu

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update mistral.py

* Update llama.py

* fast inference

* model

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update utils.py

* Update llama.py

* Update utils.py

* inference

* Update llama.py

* Update llama.py

* Update llama.py

* overhead

* Update llama.py

* Update llama.py

* compile

* Update gemma.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update utils.py

* Update utils.py

* lora mamtul

* Update llama.py

* Update llama.py

* Update llama.py

* offloaded checkpointing

* Update llama.py

* Update llama.py

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Update llama.py

* Update llama.py

* Update gemma.py

* Revert "Update gemma.py"

This reverts commit e3c3c5f3fa3d04a87f854056f6b547ced610d712.

* Update _utils.py

* Update _utils.py

* Update _utils.py

* Saving

* sentencepiece_model_pb2

* Update llama.py

* Update save.py

* Update llama.py

* padding side

* Update tokenizer_utils.py

* cache dir

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update pyproject.toml

* Update pyproject.toml

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update llama.py

* Update save.py

* Update save.py

* checkpoint

* Gemma 1.1

* more models

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* dtype
This commit is contained in:
Daniel Han 2024-04-08 23:19:46 +10:00 • committed by GitHub
commit ef99b2b019
2 changed files with 12 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

5
unsloth/models/llama.py
View file

@ -1702,6 +1702,11 @@ class FastLlamaModel:
lm_head = internal_model.lm_head.weight
device_type = lm_head.device.type
dtype = model.config.torch_dtype
if type(dtype) is str:
if dtype == "float16": dtype = torch.float16
elif dtype == "bfloat16": dtype = torch.bfloat16
pass
# Wrap model.generate
model._unwrapped_old_generate = model.generate

9
unsloth/save.py
View file

@ -183,7 +183,7 @@ def unsloth_save_model(
):
if token is None and "HF_TOKEN" in os.environ:
token = os.environ["HF_TOKEN"]
if token is None and "HUGGINGFACE_TOKEN" in os.environ:
token = os.environ["HUGGINGFACE_TOKEN"]
@ -489,7 +489,12 @@ def unsloth_save_model(
from collections import OrderedDict
state_dict = OrderedDict()
torch_dtype = model.config.torch_dtype
torch_dtype = internal_model.config.torch_dtype
if type(torch_dtype) is str:
if torch_dtype == "float16": torch_dtype = torch.float16
elif torch_dtype == "bfloat16": torch_dtype = torch.bfloat16
pass
# Check modules to save float32 dtype
state_dict["model.embed_tokens.weight"] = internal_model.model.embed_tokens.weight.data.to(torch_dtype)