Bug fixes (#1458)
* Update _utils.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update utils.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * typing * Update rope_embedding.py * types * Disable compiling * Update _utils.py * Update _utils.py * Forward hook * Update _utils.py * Update llama.py * Update _utils.py * Update llama.py * Update llama.py * Update _utils.py * Update pyproject.toml * Update _utils.py * Update llama.py * CE Loss * Update cross_entropy_loss.py * Update _utils.py * Update cross_entropy_loss.py * Update cross_entropy_loss.py * Update cross_entropy_loss.py * Update llama.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Fix: cast logits to float32 in cross_entropy_forward to prevent errors (#1254) * Fix: cast logits to float32 in cross_entropy_forward to prevent errors * Update cross_entropy_loss.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Throw error when inferencing longer than max_popsition_embeddings (#1236) * Throw error when inferencing longer than max_popsition_embeddings without rope scaling * Update llama.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * CLI now handles user input strings for dtype correctly (#1235) Co-authored-by: root <root@ieeres.chu.cam.ac.uk> * Update flex_attention.py * Update _utils.py * Update _utils.py * Update flex_attention.py * Update flex_attention.py * Update loader.py * Update loader.py * Update flex_attention.py * Update flex_attention.py * Update flex_attention.py * Update flex_attention.py * Update _utils.py * Update cross_entropy_loss.py * Update _utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * triton_cast * Update utils.py * Qwen 2.5 Coder * Fix/export mistral (#1281) * Enhance install_python_non_blocking to handle protobuf installation and process management * Revert "Enhance install_python_non_blocking to handle protobuf installation and process management" This reverts commit a3b796a05841fb8d93c652c845591e12cf81ea93. * Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION to 'python' to address issue #1266 * Revert "Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION to 'python' to address issue #1266" This reverts commit f00fbf5eac7ad4f5d48c70b98d770255d1a9ef58. * Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION to 'python' to address issue #1266 * Update __init__.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * DOC Update - Update README.md with os.environ in example (#1269) * Update README.md with os.environ in example Added OS Environ in example to avoid device conflicts , for a user at least in jupyter notebook this allows to select GPU in a multi GPU setup. As currently the unsloth init checks all GPU's and takes the first in the order which can be a issue when some GPU's are in use and the list still shows them. So to manually avoid this, this os config is required. Small change but a bit time saver for those who straight away copies the tutorials * Update README.md --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * fix/get_chat_template (#1246) * Refactor `get_chat_template` to now support system message instead. It supposed to fix ollama tokenizer chattemplate to * Remove type hinting * Update chat_templates.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * fix/sft-trainer (#1276) * Add patch for SFTTrainer to maintain backward compatibility with TRL changes * Update trainer.py * Update trainer.py * Refactor trainer patch to maintain backward compatibility with TRL changes * Update trainer.py * Refactor trainer.py to exclude non-convertible trainers from backward compatibility patch --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Update __init__.py * Update trainer.py * Update trainer.py * Update trainer.py * Update tokenizer_utils.py * Update llama.py * Fix #853 * fix/sfttrainer-compatibility (#1293) * Refactor trainer.py to import SFTConfig directly and update UnslothTrainingArguments class inheritance * Update trainer.py * Update trainer.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Update rms_layernorm.py * Update rms_layernorm.py * Gemma * Update rms_layernorm.py * Update gemma2.py * Cut Cross Entropy * Update llama.py * Cut Cross Entropy * Update llama.py * Update llama.py * Update llama.py * Update __init__.py * Update __init__.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update mapper.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * patch_fast_lora * vision * Update fast_lora.py * Update _utils.py * Update _utils.py * Vision * Update trainer.py * Update save.py * FastBaseVisionModel * Update loader_utils.py * Update vision.py * Update loader.py * Update vision.py * Update loader.py * Update vision.py * Update _utils.py * tokenizer_name * Update loader.py * Update vision.py * Update save.py * Update save.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update _utils.py * Update loader.py * kwargs * logits * Update llama.py * Update llama.py * Update llama.py * Update _utils.py * Update _utils.py * Update _utils.py * error * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update loader.py * Update llama.py * Update vision.py * Update loader.py * Old torch versions * Update loader.py * Update loader.py * prints * recheck * Update loader.py * Update loader.py * Update _utils.py * Update _utils.py * Update mapper.py * Feat/kto (#1316) * Add PatchKTOTrainer and update model imports * Update dpo.py * Update __init__.py * Delete unsloth/models/kto.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Fix orpo/dpo trainer (#1286) * change the colab notebook for dpo zephyr and orpo * use original tokenizer * Update README.md * Update README.md --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * skip modules * Update vision.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Fix llama.cpp * Update save.py * Update save.py * Update vision.py * Update save.py * Update save.py * Update save.py * Update save.py * Update save.py * Update save.py * Update save.py * Update _utils.py * Update save.py * Update save.py * Update mapper.py * modules * Fix vision model tokenizer padding side. (#1384) * Dynamic quants (#1379) * typing * Update cross_entropy_loss.py * Update cross_entropy_loss.py * Update cross_entropy_loss.py * Update cross_entropy_loss.py * Update cross_entropy_loss.py * Update cross_entropy_loss.py * Update cross_entropy_loss.py * Update cross_entropy_loss.py * Update cross_entropy_loss.py * int64 * Update _utils.py * Update cross_entropy_loss.py * constexpr * constexpr * Update cross_entropy_loss.py * Update cross_entropy_loss.py * Update _utils.py * Update _utils.py * Update _utils.py * CE * Update cross_entropy_loss.py * Update _utils.py * Update llama.py * Update _utils.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update utils.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * Update rms_layernorm.py * typing * Update rope_embedding.py * types * Disable compiling * Update _utils.py * Update _utils.py * Forward hook * Update _utils.py * Update llama.py * Update _utils.py * Update llama.py * Update llama.py * Update _utils.py * Update pyproject.toml * Update _utils.py * Update llama.py * CE Loss * Update cross_entropy_loss.py * Update _utils.py * Update cross_entropy_loss.py * Update cross_entropy_loss.py * Update cross_entropy_loss.py * Update llama.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Fix: cast logits to float32 in cross_entropy_forward to prevent errors (#1254) * Fix: cast logits to float32 in cross_entropy_forward to prevent errors * Update cross_entropy_loss.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Throw error when inferencing longer than max_popsition_embeddings (#1236) * Throw error when inferencing longer than max_popsition_embeddings without rope scaling * Update llama.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * CLI now handles user input strings for dtype correctly (#1235) Co-authored-by: root <root@ieeres.chu.cam.ac.uk> * Update flex_attention.py * Update _utils.py * Update _utils.py * Update flex_attention.py * Update flex_attention.py * Update loader.py * Update loader.py * Update flex_attention.py * Update flex_attention.py * Update flex_attention.py * Update flex_attention.py * Update _utils.py * Update cross_entropy_loss.py * Update _utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * Update tokenizer_utils.py * triton_cast * Update utils.py * Qwen 2.5 Coder * Fix/export mistral (#1281) * Enhance install_python_non_blocking to handle protobuf installation and process management * Revert "Enhance install_python_non_blocking to handle protobuf installation and process management" This reverts commit a3b796a05841fb8d93c652c845591e12cf81ea93. * Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION to 'python' to address issue #1266 * Revert "Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION to 'python' to address issue #1266" This reverts commit f00fbf5eac7ad4f5d48c70b98d770255d1a9ef58. * Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION to 'python' to address issue #1266 * Update __init__.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * DOC Update - Update README.md with os.environ in example (#1269) * Update README.md with os.environ in example Added OS Environ in example to avoid device conflicts , for a user at least in jupyter notebook this allows to select GPU in a multi GPU setup. As currently the unsloth init checks all GPU's and takes the first in the order which can be a issue when some GPU's are in use and the list still shows them. So to manually avoid this, this os config is required. Small change but a bit time saver for those who straight away copies the tutorials * Update README.md --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * fix/get_chat_template (#1246) * Refactor `get_chat_template` to now support system message instead. It supposed to fix ollama tokenizer chattemplate to * Remove type hinting * Update chat_templates.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * fix/sft-trainer (#1276) * Add patch for SFTTrainer to maintain backward compatibility with TRL changes * Update trainer.py * Update trainer.py * Refactor trainer patch to maintain backward compatibility with TRL changes * Update trainer.py * Refactor trainer.py to exclude non-convertible trainers from backward compatibility patch --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Update __init__.py * Update trainer.py * Update trainer.py * Update trainer.py * Update tokenizer_utils.py * Update llama.py * Fix #853 * fix/sfttrainer-compatibility (#1293) * Refactor trainer.py to import SFTConfig directly and update UnslothTrainingArguments class inheritance * Update trainer.py * Update trainer.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Update rms_layernorm.py * Update rms_layernorm.py * Gemma * Update rms_layernorm.py * Update gemma2.py * Cut Cross Entropy * Update llama.py * Cut Cross Entropy * Update llama.py * Update llama.py * Update llama.py * Update __init__.py * Update __init__.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update mapper.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * patch_fast_lora * vision * Update fast_lora.py * Update _utils.py * Update _utils.py * Vision * Update trainer.py * Update save.py * FastBaseVisionModel * Update loader_utils.py * Update vision.py * Update loader.py * Update vision.py * Update loader.py * Update vision.py * Update _utils.py * tokenizer_name * Update loader.py * Update vision.py * Update save.py * Update save.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update _utils.py * Update loader.py * kwargs * logits * Update llama.py * Update llama.py * Update llama.py * Update _utils.py * Update _utils.py * Update _utils.py * error * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update loader.py * Update llama.py * Update vision.py * Update loader.py * Old torch versions * Update loader.py * Update loader.py * prints * recheck * Update loader.py * Update loader.py * Update _utils.py * Update _utils.py * Update mapper.py * Feat/kto (#1316) * Add PatchKTOTrainer and update model imports * Update dpo.py * Update __init__.py * Delete unsloth/models/kto.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Fix orpo/dpo trainer (#1286) * change the colab notebook for dpo zephyr and orpo * use original tokenizer * Update README.md * Update README.md --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * skip modules * Update vision.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Fix llama.cpp * Update save.py * Update save.py * Update vision.py * Update save.py * Update save.py * Update save.py * Update save.py * Update save.py * Update save.py * Update save.py * Update _utils.py * Update save.py * Update save.py * Update mapper.py * modules --------- Co-authored-by: Edd <68678137+Erland366@users.noreply.github.com> Co-authored-by: Datta Nimmaturi <datta.nimmaturi@nutanix.com> Co-authored-by: Edwin Fennell <edwinfennell1@gmail.com> Co-authored-by: root <root@ieeres.chu.cam.ac.uk> Co-authored-by: Uday Girish Maradana <einsteingirish@gmail.com> Co-authored-by: cell-dame <122996026+dame-cell@users.noreply.github.com> * Update README.md Unsloth Dynamic 4-bit Quantization Update * Fix vision model tokenizer padding side. * Update vision.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Edd <68678137+Erland366@users.noreply.github.com> Co-authored-by: Datta Nimmaturi <datta.nimmaturi@nutanix.com> Co-authored-by: Edwin Fennell <edwinfennell1@gmail.com> Co-authored-by: root <root@ieeres.chu.cam.ac.uk> Co-authored-by: Uday Girish Maradana <einsteingirish@gmail.com> Co-authored-by: cell-dame <122996026+dame-cell@users.noreply.github.com> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> * Add citation section to README.md (#1377) * Add citation section to README.md * Update README.md --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Granite support (#1218) * [WIP] Support for Granite * Fixup inference * Cleanup flex attention * remove sliding window * Use torch.add for residual multiplier * Llama 3.3 * Update llama.py * Update llama.py * fullgraph * Fix loader.py to work on Windows (#1453) * Update README.md Llama 3.3 + Reddit * Update README.md Apple ML Cross Entropy * Update README.md Removing double citation * Fix loader.py to work on Windows --------- Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> * Update save.py warning message (#1425) * Update README.md Llama 3.3 + Reddit * Update README.md Apple ML Cross Entropy * Update README.md Removing double citation * Update save.py warning message --------- Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> * Change _fix_chat_template in case a template has both endif and endfor (#1388) * Update llama and derivatives to pass position embeddings explicitly for transformers v4.47+ (#1442) * Update save.py * Update llama.py * Update mistral.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Temp fix * Update _utils.py --------- Co-authored-by: Edd <68678137+Erland366@users.noreply.github.com> Co-authored-by: Datta Nimmaturi <datta.nimmaturi@nutanix.com> Co-authored-by: Edwin Fennell <edwinfennell1@gmail.com> Co-authored-by: root <root@ieeres.chu.cam.ac.uk> Co-authored-by: Uday Girish Maradana <einsteingirish@gmail.com> Co-authored-by: cell-dame <122996026+dame-cell@users.noreply.github.com> Co-authored-by: Zewen Shen <zewen.public@gmail.com> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> Co-authored-by: Scott Phillips <polygonguru@gmail.com> Co-authored-by: qingy1337 <qxli2@students.everettcc.edu> Co-authored-by: Giulia Baldini <44327645+giuliabaldini@users.noreply.github.com>
This commit is contained in:
Daniel Han
GitHub
parent
51d5e373c0
commit
b24c8fcd8c
8 changed files with 143 additions and 94 deletions
|
|
@ -36,7 +36,7 @@ huggingface = [
|
|||
"unsloth_zoo>=2024.11.8",
|
||||
"packaging",
|
||||
"tyro",
|
||||
"transformers>=4.46.1",
|
||||
"transformers>=4.46.1,<=4.46.3",
|
||||
"datasets>=2.16.0",
|
||||
"sentencepiece>=0.2.0",
|
||||
"tqdm",
|
||||
|
|
@ -247,7 +247,7 @@ colab-new = [
|
|||
"unsloth_zoo>=2024.11.8",
|
||||
"packaging",
|
||||
"tyro",
|
||||
"transformers>=4.46.1",
|
||||
"transformers>=4.46.1,<=4.46.3",
|
||||
"datasets>=2.16.0",
|
||||
"sentencepiece>=0.2.0",
|
||||
"tqdm",
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@
|
|||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
__version__ = "2024.12.4"
|
||||
__version__ = "2024.12.5"
|
||||
|
||||
__all__ = [
|
||||
"prepare_model_for_kbit_training",
|
||||
|
|
@ -1127,6 +1127,7 @@ def unsloth_compile_transformers(
|
|||
shape_padding = True,
|
||||
cudagraphs = False,
|
||||
debug = False,
|
||||
fullgraph = True,
|
||||
import_from_cache = False,
|
||||
disable = False,
|
||||
return_logits = False,
|
||||
|
|
@ -1170,6 +1171,7 @@ def unsloth_compile_transformers(
|
|||
shape_padding = shape_padding,
|
||||
cudagraphs = cudagraphs,
|
||||
debug = debug,
|
||||
fullgraph = fullgraph,
|
||||
import_from_cache = import_from_cache,
|
||||
disable = disable,
|
||||
return_logits = return_logits,
|
||||
|
|
|
|||
|
|
@ -75,6 +75,7 @@ def CohereAttention_fast_forward(
|
|||
output_attentions: bool = False,
|
||||
use_cache: bool = False,
|
||||
padding_mask: Optional[torch.LongTensor] = None,
|
||||
position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
|
||||
*args, **kwargs,
|
||||
) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
|
||||
|
||||
|
|
@ -112,12 +113,11 @@ def CohereAttention_fast_forward(
|
|||
if past_key_value is not None:
|
||||
kv_seq_len += past_key_value[0].shape[-2]
|
||||
|
||||
cos, sin = position_embeddings
|
||||
if position_ids is None:
|
||||
cos = self.rotary_emb.cos_cached
|
||||
sin = self.rotary_emb.sin_cached
|
||||
Q, K = fast_rope_embedding(Q, K, cos, sin)
|
||||
else:
|
||||
cos, sin = self.rotary_emb(V, seq_len = kv_seq_len)
|
||||
cos, sin = cos[position_ids], sin[position_ids]
|
||||
Q, K = inplace_rope_embedding(Q, K, cos, sin, position_ids)
|
||||
pass
|
||||
|
||||
|
|
@ -190,6 +190,7 @@ def CohereDecoderLayer_fast_forward(
|
|||
output_attentions: Optional[bool] = False,
|
||||
use_cache: Optional[bool] = False,
|
||||
padding_mask: Optional[torch.LongTensor] = None,
|
||||
position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
|
||||
*args, **kwargs,
|
||||
):
|
||||
if use_cache and hasattr(self, "_flag_for_generation"): #past_key_value is not None:
|
||||
|
|
|
|||
|
|
@ -329,14 +329,15 @@ pass
|
|||
# https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L320
|
||||
def LlamaAttention_fast_forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
causal_mask: Optional[xformers.attn_bias.BlockDiagonalCausalMask] = None,
|
||||
attention_mask: Optional[torch.Tensor] = None,
|
||||
position_ids: Optional[torch.LongTensor] = None,
|
||||
past_key_value: Optional[Tuple[torch.Tensor]] = None,
|
||||
output_attentions: bool = False,
|
||||
use_cache: bool = False,
|
||||
padding_mask: Optional[torch.LongTensor] = None,
|
||||
hidden_states: torch.Tensor,
|
||||
causal_mask: Optional[xformers.attn_bias.BlockDiagonalCausalMask] = None,
|
||||
attention_mask: Optional[torch.Tensor] = None,
|
||||
position_ids: Optional[torch.LongTensor] = None,
|
||||
past_key_value: Optional[Tuple[torch.Tensor]] = None,
|
||||
output_attentions: bool = False,
|
||||
use_cache: bool = False,
|
||||
padding_mask: Optional[torch.LongTensor] = None,
|
||||
position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
|
||||
*args, **kwargs,
|
||||
) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
|
||||
|
||||
|
|
@ -368,20 +369,24 @@ def LlamaAttention_fast_forward(
|
|||
if past_key_value is not None:
|
||||
kv_seq_len += past_key_value[0].shape[-2]
|
||||
|
||||
# Extend RoPE dynamically to fit in VRAM
|
||||
rotary_emb = self.rotary_emb
|
||||
rotary_emb.extend_rope_embedding(V, seq_len = kv_seq_len)
|
||||
|
||||
if position_ids is None:
|
||||
# Useful for LongRoPE
|
||||
cos, sin = rotary_emb.get_cached(kv_seq_len)
|
||||
# cos = self.rotary_emb.cos_cached
|
||||
# sin = self.rotary_emb.sin_cached
|
||||
Q, K = fast_rope_embedding(Q, K, cos, sin)
|
||||
if position_embeddings:
|
||||
cos, sin = position_embeddings
|
||||
else:
|
||||
cos, sin = rotary_emb(V, seq_len = kv_seq_len)
|
||||
Q, K = inplace_rope_embedding(Q, K, cos, sin, position_ids)
|
||||
pass
|
||||
# Extend RoPE dynamically to fit in VRA
|
||||
rotary_emb = self.rotary_emb
|
||||
rotary_emb.extend_rope_embedding(V, seq_len=kv_seq_len)
|
||||
|
||||
if position_ids is None:
|
||||
# Useful for LongRoPE
|
||||
cos, sin = rotary_emb.get_cached(kv_seq_len)
|
||||
else:
|
||||
cos, sin = rotary_emb(V, seq_len=kv_seq_len)
|
||||
|
||||
Q, K = (
|
||||
fast_rope_embedding(Q, K, cos, sin)
|
||||
if position_ids is None
|
||||
else inplace_rope_embedding(Q, K, cos, sin, position_ids)
|
||||
)
|
||||
|
||||
if past_key_value is not None:
|
||||
K = torch.cat([past_key_value[0], K], dim = 2)
|
||||
|
|
@ -444,14 +449,15 @@ pass
|
|||
# https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L590
|
||||
def LlamaDecoderLayer_fast_forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
causal_mask = None,
|
||||
attention_mask: Optional[torch.Tensor] = None,
|
||||
position_ids: Optional[torch.LongTensor] = None,
|
||||
past_key_value: Optional[Tuple[torch.Tensor]] = None,
|
||||
output_attentions: Optional[bool] = False,
|
||||
use_cache: Optional[bool] = False,
|
||||
padding_mask: Optional[torch.LongTensor] = None,
|
||||
hidden_states: torch.Tensor,
|
||||
causal_mask = None,
|
||||
attention_mask: Optional[torch.Tensor] = None,
|
||||
position_ids: Optional[torch.LongTensor] = None,
|
||||
past_key_value: Optional[Tuple[torch.Tensor]] = None,
|
||||
output_attentions: Optional[bool] = False,
|
||||
use_cache: Optional[bool] = False,
|
||||
padding_mask: Optional[torch.LongTensor] = None,
|
||||
position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
|
||||
*args, **kwargs,
|
||||
) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:
|
||||
"""
|
||||
|
|
@ -471,14 +477,15 @@ def LlamaDecoderLayer_fast_forward(
|
|||
residual = hidden_states
|
||||
hidden_states = fast_rms_layernorm_inference(self.input_layernorm, hidden_states)
|
||||
hidden_states, self_attn_weights, present_key_value = self.self_attn(
|
||||
hidden_states=hidden_states,
|
||||
causal_mask=causal_mask,
|
||||
attention_mask=attention_mask,
|
||||
position_ids=position_ids,
|
||||
past_key_value=past_key_value,
|
||||
output_attentions=output_attentions,
|
||||
use_cache=use_cache,
|
||||
padding_mask=padding_mask,
|
||||
hidden_states = hidden_states,
|
||||
causal_mask = causal_mask,
|
||||
attention_mask = attention_mask,
|
||||
position_ids = position_ids,
|
||||
past_key_value = past_key_value,
|
||||
output_attentions = output_attentions,
|
||||
use_cache = use_cache,
|
||||
padding_mask = padding_mask,
|
||||
position_embeddings = position_embeddings,
|
||||
)
|
||||
hidden_states += residual
|
||||
|
||||
|
|
@ -491,14 +498,15 @@ def LlamaDecoderLayer_fast_forward(
|
|||
residual = hidden_states
|
||||
hidden_states = fast_rms_layernorm(self.input_layernorm, hidden_states)
|
||||
hidden_states, self_attn_weights, present_key_value = self.self_attn(
|
||||
hidden_states=hidden_states,
|
||||
causal_mask=causal_mask,
|
||||
attention_mask=attention_mask,
|
||||
position_ids=position_ids,
|
||||
past_key_value=past_key_value,
|
||||
output_attentions=output_attentions,
|
||||
use_cache=use_cache,
|
||||
padding_mask=padding_mask,
|
||||
hidden_states = hidden_states,
|
||||
causal_mask = causal_mask,
|
||||
attention_mask = attention_mask,
|
||||
position_ids = position_ids,
|
||||
past_key_value = past_key_value,
|
||||
output_attentions = output_attentions,
|
||||
use_cache = use_cache,
|
||||
padding_mask = padding_mask,
|
||||
position_embeddings = position_embeddings,
|
||||
)
|
||||
hidden_states = residual + hidden_states
|
||||
|
||||
|
|
@ -776,9 +784,10 @@ def LlamaModel_fast_forward(
|
|||
pass
|
||||
pass
|
||||
|
||||
|
||||
if IS_GRANITE:
|
||||
position_embeddings = self.rotary_emb(hidden_states, position_ids, self.max_position_embeddings)
|
||||
if transformers_version > "4.47.1" and hasattr(self, "rotary_emb"):
|
||||
# Transformers main has made it mandatory to pass position_embeddings
|
||||
# https://github.com/huggingface/transformers/pull/34858
|
||||
position_embeddings = self.rotary_emb(hidden_states, position_ids, self.config.max_position_embeddings)
|
||||
else:
|
||||
position_embeddings = None
|
||||
|
||||
|
|
@ -832,13 +841,13 @@ def LlamaModel_fast_forward(
|
|||
layer_outputs = decoder_layer(
|
||||
hidden_states,
|
||||
causal_mask=mask,
|
||||
attention_mask=attention_mask,
|
||||
position_ids=position_ids,
|
||||
past_key_value=past_key_value,
|
||||
output_attentions=output_attentions,
|
||||
use_cache=use_cache,
|
||||
padding_mask=padding_mask,
|
||||
position_embeddings = position_embeddings
|
||||
attention_mask = attention_mask,
|
||||
position_ids = position_ids,
|
||||
past_key_value = past_key_value,
|
||||
output_attentions = output_attentions,
|
||||
use_cache = use_cache,
|
||||
padding_mask = padding_mask,
|
||||
position_embeddings = position_embeddings,
|
||||
)
|
||||
hidden_states = layer_outputs[0]
|
||||
pass
|
||||
|
|
@ -993,6 +1002,9 @@ def CausalLM_fast_forward(fast_forward_inference):
|
|||
logits = self.lm_head(hidden_states[:, -num_logits_to_keep:, :].to(lm_head.dtype))
|
||||
else:
|
||||
RETURN_LOGITS = os.environ.get("UNSLOTH_RETURN_LOGITS", "0") == "1"
|
||||
# < 1024 Normal Unsloth uses less VRAM!
|
||||
if bsz*q_len <= 1024: RETURN_LOGITS = True
|
||||
|
||||
if not RETURN_LOGITS and HAS_CUT_CROSS_ENTROPY and labels is not None:
|
||||
n_items = kwargs.get("num_items_in_batch", None) or kwargs.get("n_items", None)
|
||||
loss = fused_linear_cross_entropy(
|
||||
|
|
@ -1041,7 +1053,6 @@ def CausalLM_fast_forward(fast_forward_inference):
|
|||
# Fixes https://github.com/unslothai/unsloth/issues/10
|
||||
self.extra_ignored_labels = torch.full((self.max_seq_length, 1), -100, device = "cuda:0")
|
||||
pass
|
||||
|
||||
shift_labels = torch.hstack((labels[..., 1:], self.extra_ignored_labels[:labels.shape[0]]))
|
||||
loss = fast_cross_entropy_loss(
|
||||
logits = shift_logits,
|
||||
|
|
@ -1570,7 +1581,7 @@ class FastLlamaModel:
|
|||
max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)
|
||||
|
||||
statistics = \
|
||||
f"==((====))== Unsloth {__version__}: Fast {model_patcher.__name__[4:-5]} patching. Transformers:{transformers_version}.\n"\
|
||||
f"==((====))== Unsloth {__version__}: Fast {model_patcher.__name__[4:-5]} patching. Transformers: {transformers_version}.\n"\
|
||||
f" \\\ /| GPU: {gpu_stats.name}. Max memory: {max_memory} GB. Platform: {platform_system}.\n"\
|
||||
f"O^O/ \_/ \\ Torch: {torch.__version__}. CUDA: {gpu_stats.major}.{gpu_stats.minor}. CUDA Toolkit: {torch.version.cuda}. Triton: {triton_version}\n"\
|
||||
f"\ / Bfloat16 = {str(SUPPORTS_BFLOAT16).upper()}. FA [Xformers = {xformers_version}. FA2 = {HAS_FLASH_ATTENTION}]\n"\
|
||||
|
|
|
|||
|
|
@ -131,7 +131,8 @@ class FastLanguageModel(FastLlamaModel):
|
|||
exist_config = os.path.exists(os.path.join(model_name, "config.json"))
|
||||
both_exist = exist_adapter_config and exist_config
|
||||
else:
|
||||
files = HfFileSystem(token = token).glob(os.path.join(model_name, "*.json"))
|
||||
# Because HfFileSystem assumes linux paths, we need to set the path with forward slashes, even on Windows.
|
||||
files = HfFileSystem(token = token).glob(f"{model_name}/*.json")
|
||||
files = (os.path.split(x)[-1] for x in files)
|
||||
if sum(x == "adapter_config.json" or x == "config.json" for x in files) >= 2:
|
||||
both_exist = True
|
||||
|
|
@ -352,6 +353,7 @@ class FastVisionModel(FastBaseVisionModel):
|
|||
resize_model_vocab = None, # [TODO] No effect
|
||||
revision = None,
|
||||
return_logits = False, # Return logits
|
||||
fullgraph = True, # No graph breaks
|
||||
*args, **kwargs,
|
||||
):
|
||||
if token is None: token = get_token()
|
||||
|
|
@ -473,6 +475,7 @@ class FastVisionModel(FastBaseVisionModel):
|
|||
shape_padding = True,
|
||||
cudagraphs = False,
|
||||
debug = False,
|
||||
fullgraph = fullgraph,
|
||||
import_from_cache = False,
|
||||
disable = False,
|
||||
return_logits = return_logits,
|
||||
|
|
|
|||
|
|
@ -39,14 +39,15 @@ pass
|
|||
|
||||
def MistralAttention_fast_forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
causal_mask: Optional[xformers.attn_bias.BlockDiagonalCausalMask] = None,
|
||||
attention_mask: Optional[torch.Tensor] = None,
|
||||
position_ids: Optional[torch.LongTensor] = None,
|
||||
past_key_value: Optional[Tuple[torch.Tensor]] = None,
|
||||
output_attentions: bool = False,
|
||||
use_cache: bool = False,
|
||||
padding_mask: Optional[torch.LongTensor] = None,
|
||||
hidden_states: torch.Tensor,
|
||||
causal_mask: Optional[xformers.attn_bias.BlockDiagonalCausalMask] = None,
|
||||
attention_mask: Optional[torch.Tensor] = None,
|
||||
position_ids: Optional[torch.LongTensor] = None,
|
||||
past_key_value: Optional[Tuple[torch.Tensor]] = None,
|
||||
output_attentions: bool = False,
|
||||
use_cache: bool = False,
|
||||
padding_mask: Optional[torch.LongTensor] = None,
|
||||
position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
|
||||
*args, **kwargs,
|
||||
) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
|
||||
|
||||
|
|
|
|||
|
|
@ -577,7 +577,7 @@ def unsloth_save_model(
|
|||
# max_ram = max(max_ram - W.nbytes, 0)
|
||||
else:
|
||||
# Save to Disk
|
||||
logger.warning_once("We will save to Disk and not RAM now.")
|
||||
logger.warning_once("\nWe will save to Disk and not RAM now.")
|
||||
filename = os.path.join(temporary_location, f"{name}.pt")
|
||||
torch.save(W, filename, pickle_module = pickle, pickle_protocol = pickle.HIGHEST_PROTOCOL,)
|
||||
# weights_only = True weirdly fails?
|
||||
|
|
@ -2129,17 +2129,31 @@ def unsloth_generic_save(
|
|||
):
|
||||
if token is None and push_to_hub: token = get_token()
|
||||
|
||||
merge_and_overwrite_lora(
|
||||
get_model_name,
|
||||
create_huggingface_repo,
|
||||
model,
|
||||
save_location = save_directory,
|
||||
push_to_hub = push_to_hub,
|
||||
token = token,
|
||||
upload_location = save_directory if push_to_hub else None,
|
||||
low_disk_space_usage = True,
|
||||
private = private,
|
||||
)
|
||||
import unsloth_zoo
|
||||
if Version(unsloth_zoo.__version__) <= Version("2024.12.1"):
|
||||
merge_and_overwrite_lora(
|
||||
get_model_name,
|
||||
create_huggingface_repo,
|
||||
model,
|
||||
save_location = save_directory,
|
||||
push_to_hub = push_to_hub,
|
||||
token = token,
|
||||
upload_location = save_directory if push_to_hub else None,
|
||||
low_disk_space_usage = True,
|
||||
private = private,
|
||||
)
|
||||
else:
|
||||
merge_and_overwrite_lora(
|
||||
get_model_name,
|
||||
model,
|
||||
save_directory = save_directory,
|
||||
push_to_hub = push_to_hub,
|
||||
private = private,
|
||||
token = token,
|
||||
low_disk_space_usage = False,
|
||||
use_temp_file = False,
|
||||
)
|
||||
pass
|
||||
return
|
||||
pass
|
||||
|
||||
|
|
|
|||
|
|
@ -585,26 +585,43 @@ def load_correct_tokenizer(
|
|||
pass
|
||||
|
||||
|
||||
def _find_end_position(template, endfor, endif):
|
||||
where_endfor = template.find(endfor)
|
||||
where_endif = template.find(endif)
|
||||
if where_endfor == where_endif == -1:
|
||||
return None
|
||||
elif where_endfor > where_endif:
|
||||
return endfor
|
||||
else:
|
||||
return endif
|
||||
pass
|
||||
pass
|
||||
|
||||
|
||||
def _fix_chat_template(chat_template):
|
||||
endfor = "{% endif %}"
|
||||
where = chat_template.find(endfor)
|
||||
if where == -1:
|
||||
endfor = "{%- endif %}"
|
||||
where = chat_template.find(endfor)
|
||||
if where == -1:
|
||||
endfor = "{% endfor %}"
|
||||
endif = "{% endif %}"
|
||||
chosen_end = _find_end_position(chat_template, endfor, endif)
|
||||
if chosen_end is None:
|
||||
endfor = "{%- endfor %}"
|
||||
endif = "{%- endif %}"
|
||||
chosen_end = _find_end_position(chat_template, endfor, endif)
|
||||
if chosen_end is None:
|
||||
return chat_template
|
||||
|
||||
where = chat_template.find(chosen_end)
|
||||
|
||||
after_endfor = chat_template[where + len(endfor):]
|
||||
after_endfor = chat_template[where + len(chosen_end):]
|
||||
|
||||
dash = "-" if endfor.startswith("{%-") else ""
|
||||
dash = "-" if chosen_end.startswith("{%-") else ""
|
||||
|
||||
if "{%" + dash + " if" not in after_endfor and "{%" + dash + " set " not in after_endfor and \
|
||||
after_endfor.startswith("{{") and after_endfor.endswith("}}") and \
|
||||
after_endfor.count("{{") == 1 and after_endfor.count("}}") == 1:
|
||||
|
||||
after_endfor = "{%" + dash + " if add_generation_prompt %}" + after_endfor + endfor
|
||||
after_endfor = "{%" + dash + " if add_generation_prompt %}" + after_endfor + endif
|
||||
|
||||
chat_template = chat_template[:where + len(endfor)] + after_endfor
|
||||
chat_template = chat_template[:where + len(chosen_end)] + after_endfor
|
||||
pass
|
||||
return chat_template
|
||||
pass
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue