Auto-enable padding-free SFT (#3672)
* implement (sdpa, xformers, fa2) sample packing
* attention dispatching
* ddp working OOTB with CLI
* packed SWA and softcap support
* enable batch flattening
* LGPL license headers
* mask packed sequence boundaries
* auto-enable sample packing
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add explicit toggle for sample packing
* Add explicit toggle for sample packing
* Update __init__.py
* Update unsloth/kernels/rope_embedding.py
* Update unsloth/kernels/rope_embedding.py
* remove grad output clones; restore deleted FastLanguageModel arg
* fix
* restore rope embedding clones
* xformers mask cache
* implement (sdpa, xformers, fa2) sample packing
* attention dispatching
* ddp working OOTB with CLI
* packed SWA and softcap support
* enable batch flattening
* LGPL license headers
* mask packed sequence boundaries
* auto-enable sample packing
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add explicit toggle for sample packing
* Add explicit toggle for sample packing
* Update __init__.py
* Update unsloth/kernels/rope_embedding.py
* Update unsloth/kernels/rope_embedding.py
* remove grad output clones; restore deleted FastLanguageModel arg
* fix
* restore rope embedding clones
* xformers mask cache
* add back accidental deletion
* Update unsloth/kernels/rope_embedding.py
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix merge conflicts
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add **kwargs
* add back clobbered
* Update rope_embedding.py
* Update rope_embedding.py
* simplify trl warnings filter
* docstring
* nit
* bugfix
* add padding-free seqlen metadata
* auto-enable padding free
* gemma2 disable
* Apply suggestion from @danielhanchen
* Update trainer.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update trainer.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
89b042f23b