ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
595 commits 391 branches 58 tags 304 MiB
Commit graph

595 commits

This branch
This branch
All branches
Author SHA1 Message Date
Daniel Han
2dfdba3493 More patching 2024-10-28 01:10:23 -07:00
Daniel Han
a8b37a320d Revert "ignored labels" 
This reverts commit 9d07be077b.
 2024-10-27 22:18:05 -07:00
Daniel Han
9d07be077b ignored labels 2024-10-27 22:10:59 -07:00
Daniel Han
02437a8391 Typo 2024-10-27 19:09:27 -07:00
Daniel Han
5286f19725 Update llama.py 2024-10-27 19:08:14 -07:00
Daniel Han
1c044da660 Fix pad token 2024-10-27 19:06:57 -07:00
Daniel Han
3acc5afad3 Update _utils.py 2024-10-27 17:34:33 -07:00
Daniel Han
7083a1d455 Unk token issues 2024-10-27 17:32:26 -07:00
Daniel Han
bf3b175d39 Merge branch 'main' into nightly 2024-10-27 16:24:20 -07:00
Daniel Han
a2f8db3e73 Merge branch 'main' of https://github.com/unslothai/unsloth 2024-10-27 15:09:42 -07:00
Daniel Han
007efc2751 Update _utils.py 2024-10-27 15:09:35 -07:00
Edd
fdf25b758a
 		Fix/casting continue pretraining (#1200) 
* Bring back float32 if float16 instead of bfloat16

* Refactor mixed precision handling for lm_head and embed_tokens to ensure correct dtype usage

* Fix dtype retrieval for embed_tokens and lm_head in mixed precision training

* Fix dtype retrieval for embed_tokens and lm_head to use weight dtype in mixed precision training

* Fix dtype handling for embed_tokens and lm_head to ensure correct float32 usage in mixed precision training

* Fix dtype assignment for lm_head modules to ensure correct weight dtype usage in mixed precision training
 2024-10-27 15:06:45 -07:00
Daniel Han
49ae619412 Update pyproject.toml 2024-10-26 18:05:55 -07:00
Daniel Han
8d46c0d4d6 Torch 2.5 2024-10-26 18:03:15 -07:00
Daniel Han
f94f7c1882 Merge branch 'main' into nightly 2024-10-26 01:22:21 -07:00
Daniel Han
d76eda4f66
 		Bug fixes (#1195) 
* Fix TRL

* Update mistral.py

* Patch processing_class

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Installation guide (#1165)

* chore: update chat_templates.py (#1166)

orginal -> original

* Disable Flex Attention

* Update tokenizer_utils.py

* Update _utils.py

* n_items

* Update cross_entropy_loss.py

* Fix DPO, ORPO

* Update _utils.py

* Update _utils.py

* fix/transformers-unpack (#1180)

* Fix DPO, ORPO (#1177)

* Fix TRL

* Update mistral.py

* Patch processing_class

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Installation guide (#1165)

* chore: update chat_templates.py (#1166)

orginal -> original

* Disable Flex Attention

* Update tokenizer_utils.py

* Update _utils.py

* n_items

* Update cross_entropy_loss.py

* Fix DPO, ORPO

* Update _utils.py

---------

Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>

* Add warning for missing Unpack and KwargsForCausalLM in older Transformers versions

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>

* Update cross_entropy_loss.py

* Update _utils.py

* Update _utils.py

* donot upcast lm_head and embeddings to float32 (#1186)

* Cleanup upcast logs (#1188)

* Fix/phi-longrope (#1193)

* Enhance rotary embedding handling in LlamaAttention and LongRopeRotaryEmbedding

* Typo

* Improve rotary embedding handling in LlamaAttention to prevent errors with short KV cache

* Update llama.py

* Update llama.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update transformers

---------

Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
Co-authored-by: Edd <68678137+Erland366@users.noreply.github.com>
Co-authored-by: Datta Nimmaturi <datta.nimmaturi@nutanix.com>
 2024-10-26 01:21:24 -07:00
Daniel Han
6f28d160b2 Update transformers 2024-10-26 01:20:37 -07:00
Edd
2bc189f490
 		Fix/phi-longrope (#1193) 
* Enhance rotary embedding handling in LlamaAttention and LongRopeRotaryEmbedding

* Typo

* Improve rotary embedding handling in LlamaAttention to prevent errors with short KV cache

* Update llama.py

* Update llama.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
 2024-10-25 15:44:10 -07:00
Datta Nimmaturi
625209e11f
 		Cleanup upcast logs (#1188) 2024-10-25 12:17:54 -07:00
Datta Nimmaturi
67760559e4
 		donot upcast lm_head and embeddings to float32 (#1186) 2024-10-25 01:28:12 -07:00
Daniel Han
44b480f41c Merge branch 'main' into nightly 2024-10-24 12:17:57 -07:00
Daniel Han
9ca13b836f Update _utils.py 2024-10-24 12:17:48 -07:00
Daniel Han
9ca0bba14b
 		Fix 4.47 issue (#1182) 
* Fix TRL

* Update mistral.py

* Patch processing_class

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Installation guide (#1165)

* chore: update chat_templates.py (#1166)

orginal -> original

* Disable Flex Attention

* Update tokenizer_utils.py

* Update _utils.py

* n_items

* Update cross_entropy_loss.py

* Fix DPO, ORPO

* Update _utils.py

* Update _utils.py

* fix/transformers-unpack (#1180)

* Fix DPO, ORPO (#1177)

* Fix TRL

* Update mistral.py

* Patch processing_class

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Installation guide (#1165)

* chore: update chat_templates.py (#1166)

orginal -> original

* Disable Flex Attention

* Update tokenizer_utils.py

* Update _utils.py

* n_items

* Update cross_entropy_loss.py

* Fix DPO, ORPO

* Update _utils.py

---------

Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>

* Add warning for missing Unpack and KwargsForCausalLM in older Transformers versions

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>

* Update cross_entropy_loss.py

* Update _utils.py

* Update _utils.py

---------

Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
Co-authored-by: Edd <68678137+Erland366@users.noreply.github.com>
 2024-10-24 12:17:21 -07:00
Daniel Han
0096e5b07f Update _utils.py 2024-10-24 12:17:09 -07:00
Daniel Han
7308bb8299 Update _utils.py 2024-10-24 12:14:14 -07:00
Daniel Han
5961c34a71 Update cross_entropy_loss.py 2024-10-24 12:11:38 -07:00
Edd
faf27477aa
 		fix/transformers-unpack (#1180) 
* Fix DPO, ORPO (#1177)

* Fix TRL

* Update mistral.py

* Patch processing_class

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Installation guide (#1165)

* chore: update chat_templates.py (#1166)

orginal -> original

* Disable Flex Attention

* Update tokenizer_utils.py

* Update _utils.py

* n_items

* Update cross_entropy_loss.py

* Fix DPO, ORPO

* Update _utils.py

---------

Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>

* Add warning for missing Unpack and KwargsForCausalLM in older Transformers versions

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
 2024-10-24 12:10:52 -07:00
Daniel Han
f195ee1e65 Update _utils.py 2024-10-24 01:11:20 -07:00
Daniel Han
4f1c474d4a
 		Fix DPO, ORPO (#1177) 
* Fix TRL

* Update mistral.py

* Patch processing_class

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Installation guide (#1165)

* chore: update chat_templates.py (#1166)

orginal -> original

* Disable Flex Attention

* Update tokenizer_utils.py

* Update _utils.py

* n_items

* Update cross_entropy_loss.py

* Fix DPO, ORPO

* Update _utils.py

---------

Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
 2024-10-24 00:36:37 -07:00
Daniel Han
1c063b4c98 Update _utils.py 2024-10-24 00:25:28 -07:00
Daniel Han
2b858a5b46 Merge branch 'main' into nightly 2024-10-24 00:24:27 -07:00
Daniel Han
4ff247ab18 Fix DPO, ORPO 2024-10-24 00:17:26 -07:00
Daniel Han
e561366365 Update cross_entropy_loss.py 2024-10-23 22:18:24 -07:00
Daniel Han
b821f20b36 n_items 2024-10-23 22:13:45 -07:00
Daniel Han
1e7e0e2368 Update _utils.py 2024-10-23 12:39:58 -07:00
Edd
f6a661f174
 		Fix/patch tokenizer (#1171) 
* fix: correct tokenizer handling in patch_sft_trainer_tokenizer

* Revert "fix: correct tokenizer handling in patch_sft_trainer_tokenizer"

This reverts commit f18ac214cd.

* fix: correct condition for test_text assignment in patch_sft_trainer_tokenizer
 2024-10-23 12:32:33 -07:00
Daniel Han
0e5a507f87
 		Many bug fixes (#1162) 
* Fix TRL

* Update mistral.py

* Patch processing_class

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Update tokenizer_utils.py

* Installation guide (#1165)

* chore: update chat_templates.py (#1166)

orginal -> original

* Disable Flex Attention

* Update tokenizer_utils.py

* Update _utils.py

---------

Co-authored-by: timothelaborie <97834767+timothelaborie@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
 2024-10-23 03:14:57 -07:00
Daniel Han
28e6eeabd8 Update _utils.py 2024-10-23 03:14:48 -07:00
Daniel Han
06f5d75b81 Update tokenizer_utils.py 2024-10-23 03:04:22 -07:00
Daniel Han
28958397ee Disable Flex Attention 2024-10-23 02:58:40 -07:00
Ikko Eltociear Ashimine
30bf33957d
 		chore: update chat_templates.py (#1166) 
orginal -> original
 2024-10-23 00:59:02 -07:00
timothelaborie
aabb5ff54b
 		Installation guide (#1165) 2024-10-23 00:55:26 -07:00
Daniel Han
b3e00335c1 Update tokenizer_utils.py 2024-10-22 01:28:38 -07:00
Daniel Han
c0f0fc987d Update tokenizer_utils.py 2024-10-22 01:22:13 -07:00
Daniel Han
101389d728 Update tokenizer_utils.py 2024-10-22 01:09:20 -07:00
Daniel Han
aa2b20763e Update tokenizer_utils.py 2024-10-22 01:05:41 -07:00
Daniel Han
4f7c527ae0 Update tokenizer_utils.py 2024-10-22 00:57:36 -07:00
Daniel Han
ef842120d2 Update tokenizer_utils.py 2024-10-22 00:55:47 -07:00
Daniel Han
106f213f40 Patch processing_class 2024-10-22 00:53:38 -07:00
Daniel Han
f4ae585405 Update mistral.py 2024-10-22 00:29:30 -07:00