ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
2,788 commits 391 branches 58 tags 304 MiB
Commit graph

2,788 commits

This branch
This branch
All branches
Author SHA1 Message Date
DoubleMathew
034db11215 simplify uns inference (#3291) 2025-09-08 21:07:02 -07:00
andrewor14
fa93e36312 Add support for QAT full fine-tuning (#3238) 
**Summary:** Following https://github.com/unslothai/unsloth/pull/2976,
which adds support for QAT + LoRA, this PR adds support for QAT
during full fine-tuning. See the [torchao QAT README](https://github.com/pytorch/ao/blob/main/torchao/quantization/qat/README.md)
for more details.

Current QAT schemes supported are:
```
fp8-int4, targeting the torch.ops.fbgemm.f8i4bf16_shuffled kernel
fp8-fp8, targeting the torch.ops.fbgemm.f8f8bf16_rowwise kernel
```

**Test Plan:** https://gist.github.com/andrewor14/048b5c1bd01b7fa23c53913856a8ef9f

Full fine-tuning Llama3.1-8B with and without QAT on `yahma/alpaca-cleaned` for 1 epoch:
- Batch size = 16 (no grad accum)
- Learning rate = 4e-5
- Quantization scheme = fp8-int4

Wikitext perplexity:
- QAT improved perplexity by 19.2% compared to regular fine-tuning
- QAT's int4 quantized model even outperformed the bf16 baseline
- Regular int4 quantized model (without QAT) was significantly worse than the bf16 baseline

```
==> unsloth_model_full_baseline_output/eval_float.log <==
|        |       |none  |     0|word_perplexity|↓  |9.8446|±  |   N/A|

==> unsloth_model_full_baseline_output/eval_quantized.log <==
|        |       |none  |     0|word_perplexity|↓  |11.4595|±  |   N/A|

==> unsloth_model_full_qat_fp8-int4_output/eval_quantized.log <==
|        |       |none  |     0|word_perplexity|↓  |9.2336|±  |   N/A|
```

Fibonacci test:
- Both bf16 baseline and int4 quantized models correctly identified 13 as the next number
- QAT quantized model was more succinct in its response
- No substantial differences here

```
### Instruction:
Continue the fibonnaci sequence.

### Input:
1, 1, 2, 3, 5, 8

==> unsloth_model_full_baseline_output/eval_float.log <==
### Response:
The next number in the Fibonacci sequence is 13.<|end_of_text|>

==> unsloth_model_full_baseline_output/eval_quantized.log <==
### Response:
The next number in the Fibonacci sequence is 13.<|end_of_text|>

==> unsloth_model_full_qat_fp8-int4_output/eval_quantized.log <==
### Response:
13<|end_of_text|>
```
 2025-09-08 15:07:50 -07:00
DoubleMathew
c9c068fa0b GptAttention turn training off during inference (#3289) 2025-09-08 13:47:32 -07:00
Daniel Han
6e237fac7f Versioning 2025-09-08 06:06:04 -07:00
Daniel Han
8a0de46a71 Update __init__.py 2025-09-08 04:57:04 -07:00
Daniel Han
63b2e8fc35 Update __init__.py 2025-09-08 02:02:11 -07:00
Roland Tannous
2011859430 Add TorchAO quantization tests with FP16 models and serialization workarounds (#3269) 
* Add TorchAO quantization tests with FP16 models and serialization workarounds

* remove unrelated files

* cleaned submission
 2025-09-04 17:22:07 -07:00
DoubleMathew
b969975ba5 llama vision inference fix (#3270) 
* llama vision inference fix

* fix via can_compile_fullgraph instead
 2025-09-04 16:06:49 -07:00
Datta Nimmaturi
2c2662b51c Filter executor not sleeping log (#3268) 2025-09-04 22:05:42 +05:30
Daniel Han
5c1b0ae9dd Bug fixes (#3266) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

* Update vision.py

* Update vision.py

* Update vision.py

* Upcast norms

* Update loader.py

* Update vision.py

* Upcast layernorms

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Update rl.py

* Update pyproject.toml

* Update rl.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update _utils.py

* Update __init__.py

* Torch 2.8

* Update rl_replacements.py

* Update loader.py

* UNSLOTH_ENABLE_CCE

* Fix

* Update loader.py

* Update loader.py

* Update __init__.py

* Update __init__.py

* Update __init__.py

* Update __init__.py

* Import fixes

* Update loader.py

* Fix aimv2 issue

* Update loader.py

* Update import_fixes.py

* Update import_fixes.py

* Update loader.py

* Update loader.py

* Update loader.py

* Upgrade

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update vision.py

* Update vision.py

* custom_datatype

* recheck

* Float16

* Update vision.py

* Update vision.py

* Update vision.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Bug fix

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

* torch_dtype

* Update rl.py

* Fix CE Loss

* Versioning

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-09-04 05:17:59 -07:00
DoubleMathew
490027f988 disable _is_vlm (#3265) 2025-09-04 04:52:35 -07:00
Daniel Han
721eee6a80 Update rl.py 2025-09-04 03:25:40 -07:00
Roland Tannous
0135d126df fixed save_pretrained_torchao and associated tests (#3264) 2025-09-03 20:24:12 -07:00
Daniel Han
f42f0d2116 Update import_fixes.py 2025-09-03 20:17:36 -07:00
Daniel Han
4a52d0f78e Update import_fixes.py 2025-09-03 20:12:42 -07:00
Daniel Han
f1f0036a92 Move logging 2025-09-03 20:07:27 -07:00
Daniel Han
7094b4843a Update _utils.py 2025-09-03 20:00:21 -07:00
Daniel Han
fa3575920c Update pyproject.toml 2025-09-03 19:55:10 -07:00
Daniel Han
33ed154e81 Update llama.py 2025-09-03 19:11:53 -07:00
Jerry Zhang
969c6a0bd8 Support saving locally in model.save_pretrained_torchao (#3263) 
Summary:
Previously the test was not ran correctly and the save to local path is not tested
this PR added support for that and tries to test properly

Note: `python tests/saving/test_unsloth_save.py` doesn't run test

Test Plan:
pytest tests/saving/test_unsloth_save.py -k test_save_torchao

Reviewers:

Subscribers:

Tasks:

Tags:
 2025-09-03 17:51:33 -07:00
Daniel Han
15f3ce1372 Update save.py 2025-09-03 15:19:02 -07:00
Lei Zhenyuan
781c890c65 [Intel] make intel device support ROPE (#3164) 
* make intel device pass

* abstract torch device stream
 2025-09-03 04:39:57 -07:00
stevenxdavis
56dd244340 Fix incorrect function call in test_qwen3_grpo.py (#3212) 
* Update test_qwen3_grpo.py to correct function call

This test file uses the incorrect name for the function, which is gradient_checkpointing_disable(), not disable_gradient_checkpointing(). 

I copied the line from test_llama32_sft.py - I'm not sure if this actually is required, just wanted it consistent for when other people like me test this and have no clue what they're doing when it throws an exception.

* Update blackwell/test_qwen3_grpo.py

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
 2025-09-03 04:39:12 -07:00
Defi Wimar
8920d2eed2 chore: Fix Typos (#3246) 2025-09-03 04:38:41 -07:00
Tim Paine
3f6ac1ce25 Remove old version constraint in dependency list (#3237) 
xref: https://github.com/unslothai/unsloth-zoo/pull/258
 2025-09-01 02:29:58 -07:00
pluesclues
2b88f93bce Update mistral.py, showed flag to not call cut cross entropy (#3233) 
* Update mistral.py, showed flag to not call cut cross entropy

* Update mistral.py, made it so if its not equal to zero

* Update unsloth/models/mistral.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
 2025-08-29 01:32:21 -07:00
Roland Tannous
711ec4a3ac tests for mxfp4 and quantized models merge fix unsloth zoo pr 254 (#3223) 2025-08-29 01:30:48 -07:00
Daniel Han
25b21f4899 GPT OSS Bug fixes (#3231) 
* Update rl.py

* Update rl.py

* Update rl.py

* GPT OSS float32

* Update vision.py

* Update loader.py

* Update loader.py
 2025-08-28 09:39:46 -07:00
Daniel Han
4058d7861a Merge branch 'main' of https://github.com/unslothai/unsloth 2025-08-28 03:19:16 -07:00
Daniel Han
01500fdcbb Versioning 2025-08-28 03:19:14 -07:00
Michael Han
a10e9d6d49 Merge pull request #3224 from DefiWimar7/typos 
chore: Fix Typos

Thank you @DefiWimar7
 2025-08-28 02:46:27 -07:00
DoubleMathew
1c08e89cc7 Handle transformers move to dtype from torch_dtype (#3225) 2025-08-28 02:43:41 -07:00
DefiWimar7
8c39cb45e4 chore: Fix Typos 2025-08-28 10:44:28 +08:00
DoubleMathew
ceff1b43b3 Fix gemma-3n (#3219) 
* place gemma-3n handling inside gemma-3 conditional

* cleanup
 2025-08-26 19:46:27 -07:00
Jerry Zhang
f3ab8c21af Support model.save_pretrained_torchao (#3111) 
Summary:
Allow users merge the LoRA weights and then do a post training quantization with torchao

Usage:

```
from torchao.quantization import Int8DynamicActivationInt8WeightConfig
torchao_config = Int8DynamicActivationInt8WeightConfig()
model.save_pretrained_torchao(
    save_path,
    tokenizer=tokenizer,
    torchao_config=torchao_config,
)
```

Test Plan:
python tests/saving/test_unsloth_save.py

Reviewers:

Subscribers:

Tasks:

Tags:
 2025-08-26 04:53:39 -07:00
Lei Zhenyuan
ac78311261 fix is casual for qwen3 (#3213) 2025-08-26 04:45:20 -07:00
Daniel Han
f35077388d Update vision.py 2025-08-22 04:02:59 -07:00
Daniel Han
a33ff972c1 Update loader.py 2025-08-22 04:02:19 -07:00
DoubleMathew
651970094d adallow float32 dtype in FastLanguageModel (#3204) 2025-08-21 16:54:39 -07:00
Daniel Han
2525052e4f Bug fixes (#3195) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

* Update vision.py

* Update vision.py

* Update vision.py

* Upcast norms

* Update loader.py

* Update vision.py

* Upcast layernorms

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Update rl.py

* Update pyproject.toml

* Update rl.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update _utils.py

* Update __init__.py

* Torch 2.8

* Update rl_replacements.py

* Update loader.py

* UNSLOTH_ENABLE_CCE

* Fix

* Update loader.py

* Update loader.py

* Update __init__.py

* Update __init__.py

* Update __init__.py

* Update __init__.py

* Import fixes

* Update loader.py

* Fix aimv2 issue

* Update loader.py

* Update import_fixes.py

* Update import_fixes.py

* Update loader.py

* Update loader.py

* Update loader.py

* Upgrade

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-20 07:39:43 -07:00
Daniel Han
dfb936743d Update _utils.py 2025-08-19 23:43:58 -07:00
Michael Han
6a9f1ada59 Merge pull request #3187 from parth2510/fix-transformers-typo-extras 
Fix extras transformers typo in pyproject.toml
 2025-08-19 15:06:11 -07:00
parth2510
3e9ef8024c Fix extras transformers typo in pyproject.toml 2025-08-19 19:55:29 +05:30
Daniel Han
308f1b422b Update pyproject.toml 2025-08-19 05:20:19 -07:00
Daniel Han
6fc745c731 Protobuf issue 2025-08-19 05:19:41 -07:00
Daniel Han
17a1e13b8b Update rl.py 2025-08-19 05:04:17 -07:00
Daniel Han
7bf39fcef2 Update pyproject.toml 2025-08-19 03:24:02 -07:00
Daniel Han
5a8c81c4f9 Update _auto_install.py 2025-08-19 03:20:37 -07:00
Daniel Han
089a0056e2 Torch 2.8 (#3186) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

* Update vision.py

* Update vision.py

* Update vision.py

* Upcast norms

* Update loader.py

* Update vision.py

* Upcast layernorms

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Update rl.py

* Update pyproject.toml

* Update rl.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update _utils.py

* Update __init__.py

* Torch 2.8

* Update rl_replacements.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-19 03:16:49 -07:00
Daniel Han
10f68527d8 Bug fixes (#3180) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

* Update vision.py

* Update vision.py

* Update vision.py

* Upcast norms

* Update loader.py

* Update vision.py

* Upcast layernorms

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Update rl.py

* Update pyproject.toml

* Update rl.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-18 06:12:49 -07:00