ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
3,059 commits 391 branches 58 tags 304 MiB
Commit graph

3,059 commits

This branch
This branch
All branches
Author SHA1 Message Date
Dan Saunders
89b042f23b Auto-enable padding-free SFT (#3672) 
* implement (sdpa, xformers, fa2) sample packing

* attention dispatching

* ddp working OOTB with CLI

* packed SWA and softcap support

* enable batch flattening

* LGPL license headers

* mask packed sequence boundaries

* auto-enable sample packing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add explicit toggle for sample packing

* Add explicit toggle for sample packing

* Update __init__.py

* Update unsloth/kernels/rope_embedding.py

* Update unsloth/kernels/rope_embedding.py

* remove grad output clones; restore deleted FastLanguageModel arg

* fix

* restore rope embedding clones

* xformers mask cache

* implement (sdpa, xformers, fa2) sample packing

* attention dispatching

* ddp working OOTB with CLI

* packed SWA and softcap support

* enable batch flattening

* LGPL license headers

* mask packed sequence boundaries

* auto-enable sample packing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add explicit toggle for sample packing

* Add explicit toggle for sample packing

* Update __init__.py

* Update unsloth/kernels/rope_embedding.py

* Update unsloth/kernels/rope_embedding.py

* remove grad output clones; restore deleted FastLanguageModel arg

* fix

* restore rope embedding clones

* xformers mask cache

* add back accidental deletion

* Update unsloth/kernels/rope_embedding.py

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix merge conflicts

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add **kwargs

* add back clobbered

* Update rope_embedding.py

* Update rope_embedding.py

* simplify trl warnings filter

* docstring

* nit

* bugfix

* add padding-free seqlen metadata

* auto-enable padding free

* gemma2 disable

* Apply suggestion from @danielhanchen

* Update trainer.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update trainer.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
 2025-12-10 03:07:29 -08:00
Dan Saunders
538d2c0719 SFT sample packing (#3566) 
* implement (sdpa, xformers, fa2) sample packing

* attention dispatching

* ddp working OOTB with CLI

* packed SWA and softcap support

* enable batch flattening

* LGPL license headers

* mask packed sequence boundaries

* auto-enable sample packing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add explicit toggle for sample packing

* Add explicit toggle for sample packing

* Update __init__.py

* Update unsloth/kernels/rope_embedding.py

* Update unsloth/kernels/rope_embedding.py

* remove grad output clones; restore deleted FastLanguageModel arg

* fix

* restore rope embedding clones

* xformers mask cache

* implement (sdpa, xformers, fa2) sample packing

* attention dispatching

* ddp working OOTB with CLI

* packed SWA and softcap support

* enable batch flattening

* LGPL license headers

* mask packed sequence boundaries

* auto-enable sample packing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add explicit toggle for sample packing

* Add explicit toggle for sample packing

* Update __init__.py

* Update unsloth/kernels/rope_embedding.py

* Update unsloth/kernels/rope_embedding.py

* remove grad output clones; restore deleted FastLanguageModel arg

* fix

* restore rope embedding clones

* xformers mask cache

* add back accidental deletion

* Update unsloth/kernels/rope_embedding.py

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix merge conflicts

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add **kwargs

* add back clobbered

* Update rope_embedding.py

* Update rope_embedding.py

* simplify trl warnings filter

* docstring

* nit

* bugfix

* Apply suggestion from @danielhanchen

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update unsloth/trainer.py

* Update unsloth/trainer.py

* Update unsloth/trainer.py

* Update unsloth/trainer.py

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
 2025-12-09 17:36:45 -08:00
Daniel Han
1b70a5f15e Update _utils.py (#3698) 2025-12-09 03:31:20 -08:00
Datta Nimmaturi
c8c8f168bb [Fix] [TRL] load_lora for multi line llm.chat/generate (#3696) 
* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Remove reload_weights rpc call from grpo trainer

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Use regex instead of static string

* patch openenv reload_weights call

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Better handle sleep and wakeup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Reset indentation

* Handle multi line self.llm.chat better

* Use logger

* re-indent

* Stricter regex to replace wildcard

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
 2025-12-09 03:30:23 -08:00
Datta Nimmaturi
b414e43b74 Remove reload_weights rpc call from grpo trainer (#3673) 
* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Remove reload_weights rpc call from grpo trainer

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Use regex instead of static string

* patch openenv reload_weights call

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Better handle sleep and wakeup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Reset indentation

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
 2025-12-08 23:36:22 -08:00
pre-commit-ci[bot]
cde46b2054 [pre-commit.ci] pre-commit autoupdate (#3694) 
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.14.7 → v0.14.8](https://github.com/astral-sh/ruff-pre-commit/compare/v0.14.7...v0.14.8)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-12-08 19:44:56 -08:00
Daniel Han
609357d8aa Versioning 2025-12-08 04:19:10 -08:00
Daniel Han
28c058d1b8 Update pyproject.toml 2025-12-08 04:13:45 -08:00
Daniel Han
8447c6d9bc Versioning 2025-12-08 04:06:01 -08:00
Noah Kirschmann
8618876411 Update transformers version constraint in pyproject.toml (#3689) 
* Update transformers version constraint in pyproject.toml

The latest transformers version just fixes the local training.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update transformers version constraint in pyproject.toml

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-12-08 03:27:18 -08:00
Daniel Han
16ea28daad Add **kwargs 2025-12-08 03:24:51 -08:00
Daniel Han
c8d0134b7d Update rl.py 2025-12-08 02:23:43 -08:00
Daniel Han
e0b36af542 Update vision.py 2025-12-07 23:09:13 -08:00
Daniel Han
f687d7afc2 Update _utils.py 2025-12-07 16:52:59 -08:00
Daniel Han
66e07ede3b Xformers fix 2025-12-07 16:40:51 -08:00
Michael Han
907264d077 Update README.md 2025-12-04 08:21:20 -08:00
Daniel Han
6db8cee887 Update README.md 2025-12-02 04:08:54 -08:00
Daniel Han
4f30c9104f Update README.md 2025-12-02 03:52:50 -08:00
pre-commit-ci[bot]
24adac2296 [pre-commit.ci] pre-commit autoupdate (#3666) 
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.14.6 → v0.14.7](https://github.com/astral-sh/ruff-pre-commit/compare/v0.14.6...v0.14.7)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-12-01 17:45:13 -08:00
Daniel Han
53e6bd1568 Update _utils.py 2025-12-01 08:01:05 -08:00
Daniel Han
d33a5b909c Update rl.py 2025-12-01 08:00:08 -08:00
Daniel Han
a00ec814e5 Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" 
This reverts commit 3beade641d.
 2025-12-01 07:24:58 -08:00
pre-commit-ci[bot]
3beade641d [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2025-12-01 15:24:34 +00:00
Daniel Han
0613fafefe Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" 
This reverts commit 727da805d9.
 2025-12-01 07:24:21 -08:00
pre-commit-ci[bot]
727da805d9 [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2025-12-01 15:23:44 +00:00
Daniel Han
1bf2af5165 Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" 
This reverts commit 3722c95e42.
 2025-12-01 07:23:31 -08:00
pre-commit-ci[bot]
3722c95e42 [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2025-12-01 15:20:22 +00:00
Daniel Han
9ab2d23233 Update unsloth/models/rl.py 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-12-01 07:20:00 -08:00
Daniel Han
e38209de83 Update unsloth/models/rl.py 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-12-01 07:19:50 -08:00
Daniel Han
89fc908d85 Update qwen3_moe.py 2025-12-01 07:19:07 -08:00
Datta Nimmaturi
2bc0765a2d Vllm guided decoding (#3663) 
* vllm sampling params fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* do not patch base_trainer

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* seperate vllm fixes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fixup deletion

* Fix indentation

* revert to old style

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-12-01 07:11:28 -08:00
Daniel Han
d15b9b8acb Verisoning 2025-12-01 07:09:17 -08:00
Daniel Han
b80f2761ac Update rl.py 2025-12-01 06:23:23 -08:00
Daniel Han
5f0c9cff5e Revert "[FIX] Vllm guided decoding params (#3662)" 
This reverts commit e4d566e159.
 2025-12-01 05:43:45 -08:00
Datta Nimmaturi
e4d566e159 [FIX] Vllm guided decoding params (#3662) 
* vllm sampling params fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* do not patch base_trainer

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* seperate vllm fixes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Apply suggestion from @danielhanchen

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks"

This reverts commit fe334c4dffda0c799295a69b3fc5f4c071d83176.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks"

This reverts commit ae94be7d7e0a4780820adfbcde3549aec0ec49cf.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks"

This reverts commit cde2dde170d35bce2b50211eeaf4c9788f33734c.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
 2025-12-01 05:42:37 -08:00
Daniel Han
64bd3e552f Merge branch 'main' into nightly 2025-12-01 04:21:27 -08:00
Santosh Bhavani
6abd9afe99 Fix: Pass gradient_checkpointing parameter to model.for_training() calls (#3659) 2025-12-01 04:18:41 -08:00
Daniel Han
08a399092f Update vision.py 2025-12-01 01:21:26 -08:00
Daniel Han
a40a7b03fb Typos 2025-12-01 00:01:07 -08:00
Daniel Han
3ef6b66f39 Update qwen3_moe.py 2025-11-30 23:37:32 -08:00
Daniel Han
f691b2616e Update vision.py 2025-11-30 21:32:07 -08:00
VED
5d0523c1ca set defualt [128, 128] insted of none (#3658) 
Co-authored-by: Ved <ved.work2024@gmail.com>
 2025-11-30 17:00:31 -08:00
Daniel Han
ad3221a7fd Update rl.py 2025-11-30 04:40:03 -08:00
Daniel Han
a40189fa4d Merge branch 'main' into nightly 2025-11-30 04:39:55 -08:00
DoubleMathew
69f7dfdb90 make unsloth_tiled_mlp a from_pretrained arg (#3655) 
* make unsloth_tiled_mlp a from_pretrained arg

* adjust patching logic
 2025-11-29 22:47:51 -08:00
Bhuvan Prakash
27ae5c335c Fix: prevent load_in_fp8 kwarg from reaching Qwen3MoeForCausalLM constructor (Fix #3649) (#3654) 
* Fix: remove load_in_fp8 from kwargs to prevent Qwen3Moe init TypeError (Fix #3649)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-11-29 20:18:11 -08:00
gitpullpull
d8ea261924 Fix broken link for Advanced pip install instructions (#3652) 2025-11-29 15:33:48 -08:00
Michael Han
f730ad0b3e Update README.md 2025-11-29 08:01:00 -08:00
DoubleMathew
f3803cdee0 fix rope_theta -> rope_parameters['rope_theta'] (#3651) 2025-11-29 06:44:26 -08:00
Michael Han
2769e27566 Update README.md 2025-11-27 20:52:27 -08:00