ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
3,062 commits 391 branches 58 tags 304 MiB
Commit graph

3,062 commits

This branch
This branch
All branches
Author SHA1 Message Date
Daniel Han
ff78e784fe Update import_fixes.py 2025-12-10 04:05:14 -08:00
Daniel Han
d7480d310d Update _utils.py 2025-12-10 03:48:01 -08:00
Datta Nimmaturi
3f92868da4
 		[FIX] fbgemm version check (#3704) 
* fbgemm version check

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* safer version check

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add check for torchvision-torch compatibility

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* refactor package check logic

* Remove logs and enforce torch

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-12-10 03:46:30 -08:00
Dan Saunders
35606da943
 		Auto-enable padding-free SFT (#3672) 
* implement (sdpa, xformers, fa2) sample packing

* attention dispatching

* ddp working OOTB with CLI

* packed SWA and softcap support

* enable batch flattening

* LGPL license headers

* mask packed sequence boundaries

* auto-enable sample packing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add explicit toggle for sample packing

* Add explicit toggle for sample packing

* Update __init__.py

* Update unsloth/kernels/rope_embedding.py

* Update unsloth/kernels/rope_embedding.py

* remove grad output clones; restore deleted FastLanguageModel arg

* fix

* restore rope embedding clones

* xformers mask cache

* implement (sdpa, xformers, fa2) sample packing

* attention dispatching

* ddp working OOTB with CLI

* packed SWA and softcap support

* enable batch flattening

* LGPL license headers

* mask packed sequence boundaries

* auto-enable sample packing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add explicit toggle for sample packing

* Add explicit toggle for sample packing

* Update __init__.py

* Update unsloth/kernels/rope_embedding.py

* Update unsloth/kernels/rope_embedding.py

* remove grad output clones; restore deleted FastLanguageModel arg

* fix

* restore rope embedding clones

* xformers mask cache

* add back accidental deletion

* Update unsloth/kernels/rope_embedding.py

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix merge conflicts

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add **kwargs

* add back clobbered

* Update rope_embedding.py

* Update rope_embedding.py

* simplify trl warnings filter

* docstring

* nit

* bugfix

* add padding-free seqlen metadata

* auto-enable padding free

* gemma2 disable

* Apply suggestion from @danielhanchen

* Update trainer.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update trainer.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
 2025-12-10 03:07:29 -08:00
Dan Saunders
50325e0be7
 		SFT sample packing (#3566) 
* implement (sdpa, xformers, fa2) sample packing

* attention dispatching

* ddp working OOTB with CLI

* packed SWA and softcap support

* enable batch flattening

* LGPL license headers

* mask packed sequence boundaries

* auto-enable sample packing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add explicit toggle for sample packing

* Add explicit toggle for sample packing

* Update __init__.py

* Update unsloth/kernels/rope_embedding.py

* Update unsloth/kernels/rope_embedding.py

* remove grad output clones; restore deleted FastLanguageModel arg

* fix

* restore rope embedding clones

* xformers mask cache

* implement (sdpa, xformers, fa2) sample packing

* attention dispatching

* ddp working OOTB with CLI

* packed SWA and softcap support

* enable batch flattening

* LGPL license headers

* mask packed sequence boundaries

* auto-enable sample packing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add explicit toggle for sample packing

* Add explicit toggle for sample packing

* Update __init__.py

* Update unsloth/kernels/rope_embedding.py

* Update unsloth/kernels/rope_embedding.py

* remove grad output clones; restore deleted FastLanguageModel arg

* fix

* restore rope embedding clones

* xformers mask cache

* add back accidental deletion

* Update unsloth/kernels/rope_embedding.py

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix merge conflicts

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add **kwargs

* add back clobbered

* Update rope_embedding.py

* Update rope_embedding.py

* simplify trl warnings filter

* docstring

* nit

* bugfix

* Apply suggestion from @danielhanchen

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update unsloth/trainer.py

* Update unsloth/trainer.py

* Update unsloth/trainer.py

* Update unsloth/trainer.py

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
 2025-12-09 17:36:45 -08:00
Daniel Han
984bed300f
 		Update _utils.py (#3698) 2025-12-09 03:31:20 -08:00
Datta Nimmaturi
d85e359c45
 		[Fix] [TRL] load_lora for multi line llm.chat/generate (#3696) 
* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Remove reload_weights rpc call from grpo trainer

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Use regex instead of static string

* patch openenv reload_weights call

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Better handle sleep and wakeup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Reset indentation

* Handle multi line self.llm.chat better

* Use logger

* re-indent

* Stricter regex to replace wildcard

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
 2025-12-09 03:30:23 -08:00
Datta Nimmaturi
6324476a76
 		Remove reload_weights rpc call from grpo trainer (#3673) 
* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Remove reload_weights rpc call from grpo trainer

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Use regex instead of static string

* patch openenv reload_weights call

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Better handle sleep and wakeup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Reset indentation

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
 2025-12-08 23:36:22 -08:00
pre-commit-ci[bot]
bf88591be0
 		[pre-commit.ci] pre-commit autoupdate (#3694) 
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.14.7 → v0.14.8](https://github.com/astral-sh/ruff-pre-commit/compare/v0.14.7...v0.14.8)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-12-08 19:44:56 -08:00
Daniel Han
521e20127d Versioning 2025-12-08 04:19:10 -08:00
Daniel Han
b5fd87be65 Update pyproject.toml 2025-12-08 04:13:45 -08:00
Daniel Han
c6934a820f Versioning 2025-12-08 04:06:01 -08:00
Noah Kirschmann
ec77d664cf
 		Update transformers version constraint in pyproject.toml (#3689) 
* Update transformers version constraint in pyproject.toml

The latest transformers version just fixes the local training.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update transformers version constraint in pyproject.toml

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-12-08 03:27:18 -08:00
Daniel Han
db46ca8bbb Add **kwargs 2025-12-08 03:24:51 -08:00
Daniel Han
d1f990fe3c Update rl.py 2025-12-08 02:23:43 -08:00
Daniel Han
3f0e7dd653 Update vision.py 2025-12-07 23:09:13 -08:00
Daniel Han
790bb34ad3 Update _utils.py 2025-12-07 16:52:59 -08:00
Daniel Han
6afaac69dd Xformers fix 2025-12-07 16:40:51 -08:00
Michael Han
d1e312dcdc
 		Update README.md 2025-12-04 08:21:20 -08:00
Daniel Han
6789c279d5 Update README.md 2025-12-02 04:08:54 -08:00
Daniel Han
dbf419399c Update README.md 2025-12-02 03:52:50 -08:00
pre-commit-ci[bot]
96cc050114
 		[pre-commit.ci] pre-commit autoupdate (#3666) 
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.14.6 → v0.14.7](https://github.com/astral-sh/ruff-pre-commit/compare/v0.14.6...v0.14.7)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-12-01 17:45:13 -08:00
Daniel Han
fb9ea50c30 Update _utils.py 2025-12-01 08:01:05 -08:00
Daniel Han
68ffcb7439 Update rl.py 2025-12-01 08:00:08 -08:00
Daniel Han
1679ddee9b Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" 
This reverts commit c427be9f49.
 2025-12-01 07:24:58 -08:00
pre-commit-ci[bot]
c427be9f49 [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2025-12-01 15:24:34 +00:00
Daniel Han
77bb017d5e Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" 
This reverts commit 1146769456.
 2025-12-01 07:24:21 -08:00
pre-commit-ci[bot]
1146769456 [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2025-12-01 15:23:44 +00:00
Daniel Han
d5ab983168 Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" 
This reverts commit 9bf82bb34c.
 2025-12-01 07:23:31 -08:00
pre-commit-ci[bot]
9bf82bb34c [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2025-12-01 15:20:22 +00:00
Daniel Han
68c70d129f
 		Update unsloth/models/rl.py 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-12-01 07:20:00 -08:00
Daniel Han
a9a3db349a
 		Update unsloth/models/rl.py 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-12-01 07:19:50 -08:00
Daniel Han
d71186a1a2 Update qwen3_moe.py 2025-12-01 07:19:07 -08:00
Datta Nimmaturi
4b124da504
 		Vllm guided decoding (#3663) 
* vllm sampling params fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* do not patch base_trainer

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* seperate vllm fixes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fixup deletion

* Fix indentation

* revert to old style

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-12-01 07:11:28 -08:00
Daniel Han
6e8de57f02 Verisoning 2025-12-01 07:09:17 -08:00
Daniel Han
d7c6fc1000 Update rl.py 2025-12-01 06:23:23 -08:00
Daniel Han
3d0be20ef6 Revert "[FIX] Vllm guided decoding params (#3662)" 
This reverts commit 6c47dc57f0.
 2025-12-01 05:43:45 -08:00
Datta Nimmaturi
6c47dc57f0
 		[FIX] Vllm guided decoding params (#3662) 
* vllm sampling params fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* do not patch base_trainer

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* seperate vllm fixes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Apply suggestion from @danielhanchen

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks"

This reverts commit fbb98c5c5c.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks"

This reverts commit c64d5b475e.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks"

This reverts commit c156545515.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
 2025-12-01 05:42:37 -08:00
Daniel Han
afc6ac86ed Merge branch 'main' into nightly 2025-12-01 04:21:27 -08:00
Santosh Bhavani
e64c53a2de
 		Fix: Pass gradient_checkpointing parameter to model.for_training() calls (#3659) 2025-12-01 04:18:41 -08:00
Daniel Han
ee4980f682 Update vision.py 2025-12-01 01:21:26 -08:00
Daniel Han
029f33d4a6 Typos 2025-12-01 00:01:07 -08:00
Daniel Han
2b37b9e938 Update qwen3_moe.py 2025-11-30 23:37:32 -08:00
Daniel Han
f7790f0e9c Update vision.py 2025-11-30 21:32:07 -08:00
VED
349a81f96f
 		set defualt [128, 128] insted of none (#3658) 
Co-authored-by: Ved <ved.work2024@gmail.com>
 2025-11-30 17:00:31 -08:00
Daniel Han
5aa7c17dca Update rl.py 2025-11-30 04:40:03 -08:00
Daniel Han
6234e903cc Merge branch 'main' into nightly 2025-11-30 04:39:55 -08:00
DoubleMathew
0fb14e6a76
 		make unsloth_tiled_mlp a from_pretrained arg (#3655) 
* make unsloth_tiled_mlp a from_pretrained arg

* adjust patching logic
 2025-11-29 22:47:51 -08:00
Bhuvan Prakash
3c658a6e1e
 		Fix: prevent load_in_fp8 kwarg from reaching Qwen3MoeForCausalLM constructor (Fix #3649) (#3654) 
* Fix: remove load_in_fp8 from kwargs to prevent Qwen3Moe init TypeError (Fix #3649)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-11-29 20:18:11 -08:00
gitpullpull
05c6f91c55
 		Fix broken link for Advanced pip install instructions (#3652) 2025-11-29 15:33:48 -08:00