Daniel Han
b040e3407d
Merge branch 'main' into nightly
2024-10-27 16:24:20 -07:00
Daniel Han
9e9d6fe660
Merge branch 'main' of https://github.com/unslothai/unsloth
2024-10-27 15:09:42 -07:00
Daniel Han
55fd65a6ed
Update _utils.py
2024-10-27 15:09:35 -07:00
Edd
539fcea071
Fix/casting continue pretraining ( #1200 )
...
* Bring back float32 if float16 instead of bfloat16
* Refactor mixed precision handling for lm_head and embed_tokens to ensure correct dtype usage
* Fix dtype retrieval for embed_tokens and lm_head in mixed precision training
* Fix dtype retrieval for embed_tokens and lm_head to use weight dtype in mixed precision training
* Fix dtype handling for embed_tokens and lm_head to ensure correct float32 usage in mixed precision training
* Fix dtype assignment for lm_head modules to ensure correct weight dtype usage in mixed precision training
2024-10-27 15:06:45 -07:00
Daniel Han
9d5f58224d
Update pyproject.toml
2024-10-26 18:05:55 -07:00
Daniel Han
e7ede2f7db
Torch 2.5
2024-10-26 18:03:15 -07:00
Daniel Han
a4a724bab2
Merge branch 'main' into nightly
2024-10-26 01:22:21 -07:00
Daniel Han
c58dc701c8
Bug fixes ( #1195 )
...
* Fix TRL
* Update mistral.py
* Patch processing_class
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Installation guide (#1165 )
* chore: update chat_templates.py (#1166 )
orginal -> original
* Disable Flex Attention
* Update tokenizer_utils.py
* Update _utils.py
* n_items
* Update cross_entropy_loss.py
* Fix DPO, ORPO
* Update _utils.py
* Update _utils.py
* fix/transformers-unpack (#1180 )
* Fix DPO, ORPO (#1177 )
* Fix TRL
* Update mistral.py
* Patch processing_class
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Installation guide (#1165 )
* chore: update chat_templates.py (#1166 )
orginal -> original
* Disable Flex Attention
* Update tokenizer_utils.py
* Update _utils.py
* n_items
* Update cross_entropy_loss.py
* Fix DPO, ORPO
* Update _utils.py
---------
Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
* Add warning for missing Unpack and KwargsForCausalLM in older Transformers versions
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
* Update cross_entropy_loss.py
* Update _utils.py
* Update _utils.py
* donot upcast lm_head and embeddings to float32 (#1186 )
* Cleanup upcast logs (#1188 )
* Fix/phi-longrope (#1193 )
* Enhance rotary embedding handling in LlamaAttention and LongRopeRotaryEmbedding
* Typo
* Improve rotary embedding handling in LlamaAttention to prevent errors with short KV cache
* Update llama.py
* Update llama.py
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Update transformers
---------
Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
Co-authored-by: Edd <68678137+Erland366@users.noreply.github.com>
Co-authored-by: Datta Nimmaturi <datta.nimmaturi@nutanix.com>
2024-10-26 01:21:24 -07:00
Daniel Han
1e8980127c
Update transformers
2024-10-26 01:20:37 -07:00
Edd
c8c4cb3a6d
Fix/phi-longrope ( #1193 )
...
* Enhance rotary embedding handling in LlamaAttention and LongRopeRotaryEmbedding
* Typo
* Improve rotary embedding handling in LlamaAttention to prevent errors with short KV cache
* Update llama.py
* Update llama.py
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
2024-10-25 15:44:10 -07:00
Datta Nimmaturi
1ba5a0161d
Cleanup upcast logs ( #1188 )
2024-10-25 12:17:54 -07:00
Datta Nimmaturi
06050f1802
donot upcast lm_head and embeddings to float32 ( #1186 )
2024-10-25 01:28:12 -07:00
Daniel Han
dcf27bcca7
Merge branch 'main' into nightly
2024-10-24 12:17:57 -07:00
Daniel Han
519c0df00c
Update _utils.py
2024-10-24 12:17:48 -07:00
Daniel Han
06a5c752e3
Fix 4.47 issue ( #1182 )
...
* Fix TRL
* Update mistral.py
* Patch processing_class
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Installation guide (#1165 )
* chore: update chat_templates.py (#1166 )
orginal -> original
* Disable Flex Attention
* Update tokenizer_utils.py
* Update _utils.py
* n_items
* Update cross_entropy_loss.py
* Fix DPO, ORPO
* Update _utils.py
* Update _utils.py
* fix/transformers-unpack (#1180 )
* Fix DPO, ORPO (#1177 )
* Fix TRL
* Update mistral.py
* Patch processing_class
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Installation guide (#1165 )
* chore: update chat_templates.py (#1166 )
orginal -> original
* Disable Flex Attention
* Update tokenizer_utils.py
* Update _utils.py
* n_items
* Update cross_entropy_loss.py
* Fix DPO, ORPO
* Update _utils.py
---------
Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
* Add warning for missing Unpack and KwargsForCausalLM in older Transformers versions
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
* Update cross_entropy_loss.py
* Update _utils.py
* Update _utils.py
---------
Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
Co-authored-by: Edd <68678137+Erland366@users.noreply.github.com>
2024-10-24 12:17:21 -07:00
Daniel Han
e24b2db194
Update _utils.py
2024-10-24 12:17:09 -07:00
Daniel Han
6f34885c29
Update _utils.py
2024-10-24 12:14:14 -07:00
Daniel Han
8603d08f3b
Update cross_entropy_loss.py
2024-10-24 12:11:38 -07:00
Edd
79effae03d
fix/transformers-unpack ( #1180 )
...
* Fix DPO, ORPO (#1177 )
* Fix TRL
* Update mistral.py
* Patch processing_class
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Installation guide (#1165 )
* chore: update chat_templates.py (#1166 )
orginal -> original
* Disable Flex Attention
* Update tokenizer_utils.py
* Update _utils.py
* n_items
* Update cross_entropy_loss.py
* Fix DPO, ORPO
* Update _utils.py
---------
Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
* Add warning for missing Unpack and KwargsForCausalLM in older Transformers versions
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
2024-10-24 12:10:52 -07:00
Daniel Han
e3e4b7dfc3
Update _utils.py
2024-10-24 01:11:20 -07:00
Daniel Han
a6e4a8bf76
Fix DPO, ORPO ( #1177 )
...
* Fix TRL
* Update mistral.py
* Patch processing_class
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Installation guide (#1165 )
* chore: update chat_templates.py (#1166 )
orginal -> original
* Disable Flex Attention
* Update tokenizer_utils.py
* Update _utils.py
* n_items
* Update cross_entropy_loss.py
* Fix DPO, ORPO
* Update _utils.py
---------
Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
2024-10-24 00:36:37 -07:00
Daniel Han
0bd8517b1e
Update _utils.py
2024-10-24 00:25:28 -07:00
Daniel Han
d6382ca656
Merge branch 'main' into nightly
2024-10-24 00:24:27 -07:00
Daniel Han
ccf1f946f3
Fix DPO, ORPO
2024-10-24 00:17:26 -07:00
Daniel Han
7fa3179e88
Update cross_entropy_loss.py
2024-10-23 22:18:24 -07:00
Daniel Han
dd8487a63e
n_items
2024-10-23 22:13:45 -07:00
Daniel Han
aa48184c41
Update _utils.py
2024-10-23 12:39:58 -07:00
Edd
da8e547678
Fix/patch tokenizer ( #1171 )
...
* fix: correct tokenizer handling in patch_sft_trainer_tokenizer
* Revert "fix: correct tokenizer handling in patch_sft_trainer_tokenizer"
This reverts commit 7a98e465cbd4f980c8b364b0396d44f2d052090f.
* fix: correct condition for test_text assignment in patch_sft_trainer_tokenizer
2024-10-23 12:32:33 -07:00
Daniel Han
4c85177719
Many bug fixes ( #1162 )
...
* Fix TRL
* Update mistral.py
* Patch processing_class
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Update tokenizer_utils.py
* Installation guide (#1165 )
* chore: update chat_templates.py (#1166 )
orginal -> original
* Disable Flex Attention
* Update tokenizer_utils.py
* Update _utils.py
---------
Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
2024-10-23 03:14:57 -07:00
Daniel Han
8e4cd551e7
Update _utils.py
2024-10-23 03:14:48 -07:00
Daniel Han
f1d3a8ae6c
Update tokenizer_utils.py
2024-10-23 03:04:22 -07:00
Daniel Han
6a1bef2c4a
Disable Flex Attention
2024-10-23 02:58:40 -07:00
Ikko Eltociear Ashimine
5d1de9d42e
chore: update chat_templates.py ( #1166 )
...
orginal -> original
2024-10-23 00:59:02 -07:00
timothelaborie
8dbf2d5daa
Installation guide ( #1165 )
2024-10-23 00:55:26 -07:00
Daniel Han
1623c3242b
Update tokenizer_utils.py
2024-10-22 01:28:38 -07:00
Daniel Han
444bc97f13
Update tokenizer_utils.py
2024-10-22 01:22:13 -07:00
Daniel Han
e749d57859
Update tokenizer_utils.py
2024-10-22 01:09:20 -07:00
Daniel Han
48eda0ba34
Update tokenizer_utils.py
2024-10-22 01:05:41 -07:00
Daniel Han
ccf31a0285
Update tokenizer_utils.py
2024-10-22 00:57:36 -07:00
Daniel Han
e8bc00bd33
Update tokenizer_utils.py
2024-10-22 00:55:47 -07:00
Daniel Han
30230e6a02
Patch processing_class
2024-10-22 00:53:38 -07:00
Daniel Han
0b20946bcf
Update mistral.py
2024-10-22 00:29:30 -07:00
Daniel Han
4a51572765
Fix TRL
2024-10-21 01:02:53 -07:00
Daniel Han
a51a84f62d
Update save.py
2024-10-20 01:52:21 -07:00
Daniel Han
108fa8dfbe
Update _utils.py
2024-10-18 23:10:30 -07:00
Daniel Han
828ef9afc5
Fix get_token
2024-10-18 23:08:30 -07:00
vo1d-ai
f63a2a5026
fix: compute_loss bug ( #1151 )
...
Currently, Unsloth doesn't pass additional parameters to Trainer.compute_loss such as return_outputs. This leads to errors when calling trainer.evaluate(). This change fixes the bug by properly passing parameters to Trainer.compute_loss.
2024-10-18 20:46:07 -07:00
Daniel Han
cde7401259
Update _utils.py
2024-10-17 20:50:05 -07:00
Daniel Han
139c3b29b3
Update README.md
2024-10-17 20:46:11 -07:00
Daniel Han
3a33dad3c9
Update README.md
2024-10-17 20:45:40 -07:00
