ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
2,743 commits 391 branches 58 tags 304 MiB
Commit graph

2,743 commits

This branch
This branch
All branches
Author SHA1 Message Date
Daniel Han
f64c5dd5eb Update rl.py 2025-08-19 05:04:17 -07:00
Daniel Han
3905796b53 Update pyproject.toml 2025-08-19 03:24:02 -07:00
Daniel Han
503d5797f6 Update _auto_install.py 2025-08-19 03:20:37 -07:00
Daniel Han
d4574f1791 Torch 2.8 (#3186) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

* Update vision.py

* Update vision.py

* Update vision.py

* Upcast norms

* Update loader.py

* Update vision.py

* Upcast layernorms

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Update rl.py

* Update pyproject.toml

* Update rl.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update _utils.py

* Update __init__.py

* Torch 2.8

* Update rl_replacements.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-19 03:16:49 -07:00
Daniel Han
609fe97cf2 Bug fixes (#3180) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

* Update vision.py

* Update vision.py

* Update vision.py

* Upcast norms

* Update loader.py

* Update vision.py

* Upcast layernorms

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Update rl.py

* Update pyproject.toml

* Update rl.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-18 06:12:49 -07:00
andrewor14
a14cf9a732 Add support for QAT + LoRA (#2976) 
**Summary:** Quantization-aware training (QAT) helps mitigate
quantization degradation by simulating quantization numerics
in high precision during training (fake quantization). This PR
combines QAT with LoRA by applying torchao's QAT support to
the peft model.

See the following for more details:

- torchao QAT: https://github.com/pytorch/ao/blob/main/torchao/quantization/qat/README.md
- torchtune QAT + LoRA: https://dev-discuss.pytorch.org/t/speeding-up-qat-by-1-89x-with-lora/2700

Current QAT schemes supported are:

```
fp8-fp8, targeting the torch.ops.fbgemm.f8i4bf16_shuffled kernel
fp8-int4, targeting the torch.ops.fbgemm.f8f8bf16_rowwise kernel
```

**Test Plan:**

```
from unsloth import FastLanguageModel

lora_rank = 32

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Qwen3-4B-Base",
    max_seq_length = 2048,
    load_in_4bit = False,
    fast_inference = False,
    max_lora_rank = lora_rank,
)

model = FastLanguageModel.get_peft_model(
    model,
    r = lora_rank,
    target_modules = [
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_alpha = lora_rank*2,
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
    qat_scheme = "fp8-fp8",
)

lora.Linear(
  (base_layer): FakeQuantizedLinear(
    in_features=2560, out_features=4096, bias=False
    (activation_fake_quantizer): FakeQuantizer(Float8FakeQuantizeConfig(dtype=torch.float8_e4m3fn, granularity=PerRow(), hp_value_lb=None, hp_value_ub=None))
    (weight_fake_quantizer): FakeQuantizer(Float8FakeQuantizeConfig(dtype=torch.float8_e4m3fn, granularity=PerRow(), hp_value_lb=None, hp_value_ub=None))
  )
  ...
)
```
 2025-08-18 05:56:35 -07:00
Ball
d633a4f4c1 fix original_push_to_hub fallback (#3115) 
Co-authored-by: root <root@LAPTOP-VEI2ITL9.localdomain>
 2025-08-18 05:54:24 -07:00
RJ Nowling
ac48a607d1 Replace back ticks with single quotes (#3157) 
Back ticks attempt to execute program and capture output.  Should be using single quote marks.
 2025-08-18 05:52:30 -07:00
Daniel Han
16e8d65992 Update _utils.py 2025-08-18 05:28:30 -07:00
Roland Tannous
8673300d87 fix save_to_gguf_generic quantization_method type error (#3173) 2025-08-18 04:10:17 -07:00
Roland Tannous
8d62d91c04 Convert generator expression to list to prevent potential bugs if the files variable is used multiple times in future modifications. (#3167) 2025-08-18 04:10:08 -07:00
Daniel Han
134ecb0e13 Fix Blackwell 2025-08-18 03:46:39 -07:00
QL
89397047f4 Update install instructions for latest vLLM release (#3175) 
1. Removed the `--extra-index-url https://wheels.vllm.ai/nightly` from the uv install instructions because this causes it to crash; Removing that flag solves the issue and is more stable overall. Tested with RTX 5090 CUDA 12.8 on Linux. 

2. Removed `uv pip install -U triton>=3.3.1` because triton 3.3.1 is already installed with the vllm command.
 2025-08-18 03:39:05 -07:00
Daniel Han
06f9d1cbc7 Nightly (#3169) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-15 05:03:38 -07:00
Daniel Han
bd7f9f2930 Update loader.py 2025-08-15 04:08:21 -07:00
Daniel Han
d376e62038 GPT OSS MXFP4 fix 2025-08-15 04:00:00 -07:00
Daniel Han
935ef43616 Update tokenizer_utils.py 2025-08-14 19:30:33 -07:00
Daniel Han
56828b5431 Encoding UTF-8 2025-08-14 15:24:25 -07:00
Daniel Han
86a2725cb6 Fix GPT OSS (#3154) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-14 05:06:58 -07:00
Daniel Han
b914f252d3 Update loader.py 2025-08-13 06:33:39 -07:00
Daniel Han
699e46e92e Nightly (#3148) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-13 06:12:38 -07:00
Michael Han
f77676849f Update README.md 2025-08-09 15:53:29 -07:00
Michael Han
eef34cea33 Merge pull request #3120 from Etherll/patch-8825 
Add Qwen3 4B to mapper.py
 2025-08-08 14:31:31 -07:00
Etherll
ada42954a7 Update mapper.py 2025-08-08 23:27:21 +03:00
Michael Han
c2b49c7eaa Update README.md 2025-08-08 12:14:38 -07:00
Daniel Han
8996f5abf6 Update pyproject.toml 2025-08-08 11:49:56 -07:00
Daniel Han
39a27a7800 Update _utils.py 2025-08-08 11:48:13 -07:00
Daniel Han
44617ea268 Update loader.py 2025-08-08 11:44:07 -07:00
Daniel Han
1ef7c9507d Update loader.py 2025-08-08 11:38:03 -07:00
Daniel Han
f536204f01 Merge branch 'main' into nightly 2025-08-08 10:55:52 -07:00
Daniel Han
9c40f9a8cc Update loader.py 2025-08-08 10:55:30 -07:00
Daniel Han
1b3ce449de Update _utils.py 2025-08-08 09:59:24 -07:00
Daniel Han
aa33f84b84 Update vision.py 2025-08-08 09:35:54 -07:00
Daniel Han
416b7768e7 Update vision.py 2025-08-08 09:31:38 -07:00
Michael Han
3f1f73291d Merge pull request #3110 from Etherll/qwen3-chat-template 
Add Qwen3 Instruct / Thinking chat templates
 2025-08-08 09:27:12 -07:00
Daniel Han
64d8a7fa3a Update _utils.py 2025-08-08 09:16:37 -07:00
Daniel Han
f7e1eb7457 Update vision.py 2025-08-08 09:15:52 -07:00
Daniel Han
d9e781088d Update chat_templates.py August-2025 2025-08-08 08:29:52 -07:00
Daniel Han
b1f125eb2a Update vision.py 2025-08-08 03:35:56 -07:00
Daniel Han
8a109a6c9f Update vision.py 2025-08-07 16:21:25 -07:00
Daniel Han
23776d1441 Update vision.py 2025-08-07 16:19:31 -07:00
Daniel Han
5c2fa825f8 Merge branch 'main' into nightly 2025-08-07 16:19:20 -07:00
Etherll
78089066d3 Update chat_templates.py 
add qwen3-instruct/thinking-chat-template
 2025-08-08 01:01:58 +03:00
Daniel Han
5d6d730f88 Update loader.py 2025-08-07 09:38:49 -07:00
Daniel Han
f6bb080c06 Update loader.py 2025-08-07 09:38:24 -07:00
Daniel Han
a913f051bf Update mapper.py 2025-08-07 08:15:59 -07:00
Daniel Han
073aefb725 Update vision.py 2025-08-07 07:13:40 -07:00
Daniel Han
972d1bb659 Update vision.py 2025-08-07 07:10:36 -07:00
Daniel Han
3574799134 Update loader.py 2025-08-07 05:18:53 -07:00
Daniel Han
5eebb206ae Update vision.py 2025-08-07 05:03:21 -07:00