andrewor14
bb7b2f40fc
Add support for QAT + LoRA ( #2976 )
...
**Summary:** Quantization-aware training (QAT) helps mitigate
quantization degradation by simulating quantization numerics
in high precision during training (fake quantization). This PR
combines QAT with LoRA by applying torchao's QAT support to
the peft model.
See the following for more details:
- torchao QAT: https://github.com/pytorch/ao/blob/main/torchao/quantization/qat/README.md
- torchtune QAT + LoRA: https://dev-discuss.pytorch.org/t/speeding-up-qat-by-1-89x-with-lora/2700
Current QAT schemes supported are:
```
fp8-fp8, targeting the torch.ops.fbgemm.f8i4bf16_shuffled kernel
fp8-int4, targeting the torch.ops.fbgemm.f8f8bf16_rowwise kernel
```
**Test Plan:**
```
from unsloth import FastLanguageModel
lora_rank = 32
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "unsloth/Qwen3-4B-Base",
max_seq_length = 2048,
load_in_4bit = False,
fast_inference = False,
max_lora_rank = lora_rank,
)
model = FastLanguageModel.get_peft_model(
model,
r = lora_rank,
target_modules = [
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
lora_alpha = lora_rank*2,
use_gradient_checkpointing = "unsloth",
random_state = 3407,
qat_scheme = "fp8-fp8",
)
lora.Linear(
(base_layer): FakeQuantizedLinear(
in_features=2560, out_features=4096, bias=False
(activation_fake_quantizer): FakeQuantizer(Float8FakeQuantizeConfig(dtype=torch.float8_e4m3fn, granularity=PerRow(), hp_value_lb=None, hp_value_ub=None))
(weight_fake_quantizer): FakeQuantizer(Float8FakeQuantizeConfig(dtype=torch.float8_e4m3fn, granularity=PerRow(), hp_value_lb=None, hp_value_ub=None))
)
...
)
```
2025-08-18 05:56:35 -07:00
Ball
94c7392c40
fix original_push_to_hub fallback ( #3115 )
...
Co-authored-by: root <root@LAPTOP-VEI2ITL9.localdomain>
2025-08-18 05:54:24 -07:00
RJ Nowling
36c563fffd
Replace back ticks with single quotes ( #3157 )
...
Back ticks attempt to execute program and capture output. Should be using single quote marks.
2025-08-18 05:52:30 -07:00
Daniel Han
176805802f
Update _utils.py
2025-08-18 05:28:30 -07:00
Roland Tannous
208f68f164
fix save_to_gguf_generic quantization_method type error ( #3173 )
2025-08-18 04:10:17 -07:00
Roland Tannous
bacfc57380
Convert generator expression to list to prevent potential bugs if the files variable is used multiple times in future modifications. ( #3167 )
2025-08-18 04:10:08 -07:00
Daniel Han
19b2fa3ac8
Fix Blackwell
2025-08-18 03:46:39 -07:00
QL
ea4b7c2c6b
Update install instructions for latest vLLM release ( #3175 )
...
1. Removed the `--extra-index-url https://wheels.vllm.ai/nightly ` from the uv install instructions because this causes it to crash; Removing that flag solves the issue and is more stable overall. Tested with RTX 5090 CUDA 12.8 on Linux.
2. Removed `uv pip install -U triton>=3.3.1` because triton 3.3.1 is already installed with the vllm command.
2025-08-18 03:39:05 -07:00
Daniel Han
7e1581b929
Nightly ( #3169 )
...
* Fix mamba
* Update loader.py
* Update vision.py
* Update loader.py
* Filter vLLM standby logs (#3131 )
* filter vLLM standby logs
* safeguard standby logger patch
* Update unsloth/models/_utils.py
* Update unsloth/models/_utils.py
* Update unsloth/models/_utils.py
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Update loader.py
* Add scaler
* Update llama.py
* Update _utils.py
* Versioning
* GPT OSS fix
* GPT OSS fix
* Update loader.py
* Update vision.py
* Update vision.py
* Update loader.py
* Update vision.py
* Update vision.py
* Update llama.py
* Update llama.py
* Update llama.py
* Versioning
* Update mapper.py
---------
Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
2025-08-15 05:03:38 -07:00
Daniel Han
b45a977c8d
Update loader.py
2025-08-15 04:08:21 -07:00
Daniel Han
64dff1456b
GPT OSS MXFP4 fix
2025-08-15 04:00:00 -07:00
Daniel Han
bbac3e3de7
Update tokenizer_utils.py
2025-08-14 19:30:33 -07:00
Daniel Han
b11d9cc935
Encoding UTF-8
2025-08-14 15:24:25 -07:00
Daniel Han
abbf1f0a43
Fix GPT OSS ( #3154 )
...
* Fix mamba
* Update loader.py
* Update vision.py
* Update loader.py
* Filter vLLM standby logs (#3131 )
* filter vLLM standby logs
* safeguard standby logger patch
* Update unsloth/models/_utils.py
* Update unsloth/models/_utils.py
* Update unsloth/models/_utils.py
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Update loader.py
* Add scaler
* Update llama.py
* Update _utils.py
* Versioning
* GPT OSS fix
* GPT OSS fix
* Update loader.py
* Update vision.py
* Update vision.py
* Update loader.py
---------
Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
2025-08-14 05:06:58 -07:00
Daniel Han
6b5fb59eb7
Update loader.py
2025-08-13 06:33:39 -07:00
Daniel Han
806bee2433
Nightly ( #3148 )
...
* Fix mamba
* Update loader.py
* Update vision.py
* Update loader.py
* Filter vLLM standby logs (#3131 )
* filter vLLM standby logs
* safeguard standby logger patch
* Update unsloth/models/_utils.py
* Update unsloth/models/_utils.py
* Update unsloth/models/_utils.py
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Update loader.py
* Add scaler
* Update llama.py
* Update _utils.py
* Versioning
---------
Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
2025-08-13 06:12:38 -07:00
Michael Han
413ec45d8b
Update README.md
2025-08-09 15:53:29 -07:00
Michael Han
230061cd48
Merge pull request #3120 from Etherll/patch-8825
...
Add Qwen3 4B to mapper.py
2025-08-08 14:31:31 -07:00
Etherll
b0004c0001
Update mapper.py
2025-08-08 23:27:21 +03:00
Michael Han
2a8ca1ef5a
Update README.md
2025-08-08 12:14:38 -07:00
Daniel Han
9975cf889a
Update pyproject.toml
2025-08-08 11:49:56 -07:00
Daniel Han
424e648005
Update _utils.py
2025-08-08 11:48:13 -07:00
Daniel Han
3fc4e4dff9
Update loader.py
2025-08-08 11:44:07 -07:00
Daniel Han
628bb6c97f
Update loader.py
2025-08-08 11:38:03 -07:00
Daniel Han
20e7c33550
Merge branch 'main' into nightly
2025-08-08 10:55:52 -07:00
Daniel Han
84ff585855
Update loader.py
2025-08-08 10:55:30 -07:00
Daniel Han
b277a09401
Update _utils.py
2025-08-08 09:59:24 -07:00
Daniel Han
5931011d6c
Update vision.py
2025-08-08 09:35:54 -07:00
Daniel Han
34a0f2901f
Update vision.py
2025-08-08 09:31:38 -07:00
Michael Han
22326b7b15
Merge pull request #3110 from Etherll/qwen3-chat-template
...
Add Qwen3 Instruct / Thinking chat templates
2025-08-08 09:27:12 -07:00
Daniel Han
6220a4c700
Update _utils.py
2025-08-08 09:16:37 -07:00
Daniel Han
52c6458fdd
Update vision.py
2025-08-08 09:15:52 -07:00
Daniel Han
8372b318c5
Update chat_templates.py
2025-08-08 08:29:52 -07:00
Daniel Han
34776a6b84
Update vision.py
2025-08-08 03:35:56 -07:00
Daniel Han
f83f3f0efe
Update vision.py
2025-08-07 16:21:25 -07:00
Daniel Han
11b5136d99
Update vision.py
2025-08-07 16:19:31 -07:00
Daniel Han
82c4bd6005
Merge branch 'main' into nightly
2025-08-07 16:19:20 -07:00
Etherll
ab03d44d9b
Update chat_templates.py
...
add qwen3-instruct/thinking-chat-template
2025-08-08 01:01:58 +03:00
Daniel Han
180917f148
Update loader.py
2025-08-07 09:38:49 -07:00
Daniel Han
aa96d65534
Update loader.py
2025-08-07 09:38:24 -07:00
Daniel Han
9889015485
Update mapper.py
2025-08-07 08:15:59 -07:00
Daniel Han
1a8eabebab
Update vision.py
2025-08-07 07:13:40 -07:00
Daniel Han
accb7b1ead
Update vision.py
2025-08-07 07:10:36 -07:00
Daniel Han
5a31edef5f
Update loader.py
2025-08-07 05:18:53 -07:00
Daniel Han
4d89527df6
Update vision.py
2025-08-07 05:03:21 -07:00
Daniel Han
d605b629ec
Update mapper.py
2025-08-07 04:59:19 -07:00
Daniel Han
a1746fc03e
Update pyproject.toml
2025-08-07 03:47:08 -07:00
Daniel Han
b7dcf7e5ed
Update pyproject.toml
2025-08-07 03:43:25 -07:00
Daniel Han
0171432daf
Update pyproject.toml
2025-08-07 03:29:44 -07:00
Daniel Han
382042f3b0
GPT OSS fixes
2025-08-07 02:57:19 -07:00
