ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
2,756 commits 391 branches 58 tags 304 MiB
Commit graph

2,756 commits

This branch
This branch
All branches
Author SHA1 Message Date
DoubleMathew
e504300a69
 		Handle transformers move to dtype from torch_dtype (#3225) 2025-08-28 02:43:41 -07:00
DoubleMathew
264911916c
 		Fix gemma-3n (#3219) 
* place gemma-3n handling inside gemma-3 conditional

* cleanup
 2025-08-26 19:46:27 -07:00
Jerry Zhang
6108750244
 		Support model.save_pretrained_torchao (#3111) 
Summary:
Allow users merge the LoRA weights and then do a post training quantization with torchao

Usage:

```
from torchao.quantization import Int8DynamicActivationInt8WeightConfig
torchao_config = Int8DynamicActivationInt8WeightConfig()
model.save_pretrained_torchao(
    save_path,
    tokenizer=tokenizer,
    torchao_config=torchao_config,
)
```

Test Plan:
python tests/saving/test_unsloth_save.py

Reviewers:

Subscribers:

Tasks:

Tags:
 2025-08-26 04:53:39 -07:00
Lei Zhenyuan
e45ddb55a7
 		fix is casual for qwen3 (#3213) 2025-08-26 04:45:20 -07:00
Daniel Han
bcb1bcfd6a Update vision.py 2025-08-22 04:02:59 -07:00
Daniel Han
11f58ba904 Update loader.py 2025-08-22 04:02:19 -07:00
DoubleMathew
afe8f41236
 		adallow float32 dtype in FastLanguageModel (#3204) 2025-08-21 16:54:39 -07:00
Daniel Han
26601f9d42
 		Bug fixes (#3195) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

* Update vision.py

* Update vision.py

* Update vision.py

* Upcast norms

* Update loader.py

* Update vision.py

* Upcast layernorms

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Update rl.py

* Update pyproject.toml

* Update rl.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update _utils.py

* Update __init__.py

* Torch 2.8

* Update rl_replacements.py

* Update loader.py

* UNSLOTH_ENABLE_CCE

* Fix

* Update loader.py

* Update loader.py

* Update __init__.py

* Update __init__.py

* Update __init__.py

* Update __init__.py

* Import fixes

* Update loader.py

* Fix aimv2 issue

* Update loader.py

* Update import_fixes.py

* Update import_fixes.py

* Update loader.py

* Update loader.py

* Update loader.py

* Upgrade

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-20 07:39:43 -07:00
Daniel Han
7aa42709fe Update _utils.py 2025-08-19 23:43:58 -07:00
Michael Han
0873189260
 		Merge pull request #3187 from parth2510/fix-transformers-typo-extras 
Fix extras transformers typo in pyproject.toml
 2025-08-19 15:06:11 -07:00
parth2510
7548d2b66f Fix extras transformers typo in pyproject.toml 2025-08-19 19:55:29 +05:30
Daniel Han
133d94010b Update pyproject.toml 2025-08-19 05:20:19 -07:00
Daniel Han
9b456040ca Protobuf issue 2025-08-19 05:19:41 -07:00
Daniel Han
d8f4aab219 Update rl.py 2025-08-19 05:04:17 -07:00
Daniel Han
6a7c07aca0 Update pyproject.toml 2025-08-19 03:24:02 -07:00
Daniel Han
afee065f2e Update _auto_install.py 2025-08-19 03:20:37 -07:00
Daniel Han
d053845133
 		Torch 2.8 (#3186) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

* Update vision.py

* Update vision.py

* Update vision.py

* Upcast norms

* Update loader.py

* Update vision.py

* Upcast layernorms

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Update rl.py

* Update pyproject.toml

* Update rl.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update _utils.py

* Update __init__.py

* Torch 2.8

* Update rl_replacements.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-19 03:16:49 -07:00
Daniel Han
9c8735ae4a
 		Bug fixes (#3180) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

* Update vision.py

* Update vision.py

* Update vision.py

* Upcast norms

* Update loader.py

* Update vision.py

* Upcast layernorms

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Update rl.py

* Update pyproject.toml

* Update rl.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-18 06:12:49 -07:00
andrewor14
b23fe78b7e
 		Add support for QAT + LoRA (#2976) 
**Summary:** Quantization-aware training (QAT) helps mitigate
quantization degradation by simulating quantization numerics
in high precision during training (fake quantization). This PR
combines QAT with LoRA by applying torchao's QAT support to
the peft model.

See the following for more details:

- torchao QAT: https://github.com/pytorch/ao/blob/main/torchao/quantization/qat/README.md
- torchtune QAT + LoRA: https://dev-discuss.pytorch.org/t/speeding-up-qat-by-1-89x-with-lora/2700

Current QAT schemes supported are:

```
fp8-fp8, targeting the torch.ops.fbgemm.f8i4bf16_shuffled kernel
fp8-int4, targeting the torch.ops.fbgemm.f8f8bf16_rowwise kernel
```

**Test Plan:**

```
from unsloth import FastLanguageModel

lora_rank = 32

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Qwen3-4B-Base",
    max_seq_length = 2048,
    load_in_4bit = False,
    fast_inference = False,
    max_lora_rank = lora_rank,
)

model = FastLanguageModel.get_peft_model(
    model,
    r = lora_rank,
    target_modules = [
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_alpha = lora_rank*2,
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
    qat_scheme = "fp8-fp8",
)

lora.Linear(
  (base_layer): FakeQuantizedLinear(
    in_features=2560, out_features=4096, bias=False
    (activation_fake_quantizer): FakeQuantizer(Float8FakeQuantizeConfig(dtype=torch.float8_e4m3fn, granularity=PerRow(), hp_value_lb=None, hp_value_ub=None))
    (weight_fake_quantizer): FakeQuantizer(Float8FakeQuantizeConfig(dtype=torch.float8_e4m3fn, granularity=PerRow(), hp_value_lb=None, hp_value_ub=None))
  )
  ...
)
```
 2025-08-18 05:56:35 -07:00
Ball
44e98f7ea2
 		fix original_push_to_hub fallback (#3115) 
Co-authored-by: root <root@LAPTOP-VEI2ITL9.localdomain>
 2025-08-18 05:54:24 -07:00
RJ Nowling
0bb786812d
 		Replace back ticks with single quotes (#3157) 
Back ticks attempt to execute program and capture output.  Should be using single quote marks.
 2025-08-18 05:52:30 -07:00
Daniel Han
d2a038d5cb Update _utils.py 2025-08-18 05:28:30 -07:00
Roland Tannous
6facca167d
 		fix save_to_gguf_generic quantization_method type error (#3173) 2025-08-18 04:10:17 -07:00
Roland Tannous
b7fdfefdab
 		Convert generator expression to list to prevent potential bugs if the files variable is used multiple times in future modifications. (#3167) 2025-08-18 04:10:08 -07:00
Daniel Han
146b5f7fac Fix Blackwell 2025-08-18 03:46:39 -07:00
QL
13cb037b36
 		Update install instructions for latest vLLM release (#3175) 
1. Removed the `--extra-index-url https://wheels.vllm.ai/nightly` from the uv install instructions because this causes it to crash; Removing that flag solves the issue and is more stable overall. Tested with RTX 5090 CUDA 12.8 on Linux. 

2. Removed `uv pip install -U triton>=3.3.1` because triton 3.3.1 is already installed with the vllm command.
 2025-08-18 03:39:05 -07:00
Daniel Han
dc26a7a0eb
 		Nightly (#3169) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-15 05:03:38 -07:00
Daniel Han
92ea94e479 Update loader.py 2025-08-15 04:08:21 -07:00
Daniel Han
5fd61cf82d GPT OSS MXFP4 fix 2025-08-15 04:00:00 -07:00
Daniel Han
511841d349 Update tokenizer_utils.py 2025-08-14 19:30:33 -07:00
Daniel Han
0f2fc19076 Encoding UTF-8 2025-08-14 15:24:25 -07:00
Daniel Han
8b41001035
 		Fix GPT OSS (#3154) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-14 05:06:58 -07:00
Daniel Han
920b25b52c Update loader.py 2025-08-13 06:33:39 -07:00
Daniel Han
f18331c9d1
 		Nightly (#3148) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-13 06:12:38 -07:00
Michael Han
7dd4e1dd5d
 		Update README.md 2025-08-09 15:53:29 -07:00
Michael Han
79b46f71b2
 		Merge pull request #3120 from Etherll/patch-8825 
Add Qwen3 4B to mapper.py
 2025-08-08 14:31:31 -07:00
Etherll
8b80a00015
 		Update mapper.py 2025-08-08 23:27:21 +03:00
Michael Han
63199569fa
 		Update README.md 2025-08-08 12:14:38 -07:00
Daniel Han
96541982ad Update pyproject.toml 2025-08-08 11:49:56 -07:00
Daniel Han
c08da646e4 Update _utils.py 2025-08-08 11:48:13 -07:00
Daniel Han
2b78d95701 Update loader.py 2025-08-08 11:44:07 -07:00
Daniel Han
59e9ec06b6 Update loader.py 2025-08-08 11:38:03 -07:00
Daniel Han
c7183fa5c0 Merge branch 'main' into nightly 2025-08-08 10:55:52 -07:00
Daniel Han
164fd8c895 Update loader.py 2025-08-08 10:55:30 -07:00
Daniel Han
a70c8e2bcf Update _utils.py 2025-08-08 09:59:24 -07:00
Daniel Han
65d7912bf7 Update vision.py 2025-08-08 09:35:54 -07:00
Daniel Han
1a1d4f9bc9 Update vision.py 2025-08-08 09:31:38 -07:00
Michael Han
67e4fa58e8
 		Merge pull request #3110 from Etherll/qwen3-chat-template 
Add Qwen3 Instruct / Thinking chat templates
 2025-08-08 09:27:12 -07:00
Daniel Han
59d6457443 Update _utils.py 2025-08-08 09:16:37 -07:00
Daniel Han
f1be7e955f Update vision.py 2025-08-08 09:15:52 -07:00