ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
2,760 commits 391 branches 58 tags 304 MiB
Commit graph

2,760 commits

This branch
This branch
All branches
Author SHA1 Message Date
Daniel Han
4058d7861a Merge branch 'main' of https://github.com/unslothai/unsloth 2025-08-28 03:19:16 -07:00
Daniel Han
01500fdcbb Versioning 2025-08-28 03:19:14 -07:00
Michael Han
a10e9d6d49 Merge pull request #3224 from DefiWimar7/typos 
chore: Fix Typos

Thank you @DefiWimar7
 2025-08-28 02:46:27 -07:00
DoubleMathew
1c08e89cc7 Handle transformers move to dtype from torch_dtype (#3225) 2025-08-28 02:43:41 -07:00
DefiWimar7
8c39cb45e4 chore: Fix Typos 2025-08-28 10:44:28 +08:00
DoubleMathew
ceff1b43b3 Fix gemma-3n (#3219) 
* place gemma-3n handling inside gemma-3 conditional

* cleanup
 2025-08-26 19:46:27 -07:00
Jerry Zhang
f3ab8c21af Support model.save_pretrained_torchao (#3111) 
Summary:
Allow users merge the LoRA weights and then do a post training quantization with torchao

Usage:

```
from torchao.quantization import Int8DynamicActivationInt8WeightConfig
torchao_config = Int8DynamicActivationInt8WeightConfig()
model.save_pretrained_torchao(
    save_path,
    tokenizer=tokenizer,
    torchao_config=torchao_config,
)
```

Test Plan:
python tests/saving/test_unsloth_save.py

Reviewers:

Subscribers:

Tasks:

Tags:
 2025-08-26 04:53:39 -07:00
Lei Zhenyuan
ac78311261 fix is casual for qwen3 (#3213) 2025-08-26 04:45:20 -07:00
Daniel Han
f35077388d Update vision.py 2025-08-22 04:02:59 -07:00
Daniel Han
a33ff972c1 Update loader.py 2025-08-22 04:02:19 -07:00
DoubleMathew
651970094d adallow float32 dtype in FastLanguageModel (#3204) 2025-08-21 16:54:39 -07:00
Daniel Han
2525052e4f Bug fixes (#3195) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

* Update vision.py

* Update vision.py

* Update vision.py

* Upcast norms

* Update loader.py

* Update vision.py

* Upcast layernorms

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Update rl.py

* Update pyproject.toml

* Update rl.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update _utils.py

* Update __init__.py

* Torch 2.8

* Update rl_replacements.py

* Update loader.py

* UNSLOTH_ENABLE_CCE

* Fix

* Update loader.py

* Update loader.py

* Update __init__.py

* Update __init__.py

* Update __init__.py

* Update __init__.py

* Import fixes

* Update loader.py

* Fix aimv2 issue

* Update loader.py

* Update import_fixes.py

* Update import_fixes.py

* Update loader.py

* Update loader.py

* Update loader.py

* Upgrade

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-20 07:39:43 -07:00
Daniel Han
dfb936743d Update _utils.py 2025-08-19 23:43:58 -07:00
Michael Han
6a9f1ada59 Merge pull request #3187 from parth2510/fix-transformers-typo-extras 
Fix extras transformers typo in pyproject.toml
 2025-08-19 15:06:11 -07:00
parth2510
3e9ef8024c Fix extras transformers typo in pyproject.toml 2025-08-19 19:55:29 +05:30
Daniel Han
308f1b422b Update pyproject.toml 2025-08-19 05:20:19 -07:00
Daniel Han
6fc745c731 Protobuf issue 2025-08-19 05:19:41 -07:00
Daniel Han
17a1e13b8b Update rl.py 2025-08-19 05:04:17 -07:00
Daniel Han
7bf39fcef2 Update pyproject.toml 2025-08-19 03:24:02 -07:00
Daniel Han
5a8c81c4f9 Update _auto_install.py 2025-08-19 03:20:37 -07:00
Daniel Han
089a0056e2 Torch 2.8 (#3186) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

* Update vision.py

* Update vision.py

* Update vision.py

* Upcast norms

* Update loader.py

* Update vision.py

* Upcast layernorms

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Update rl.py

* Update pyproject.toml

* Update rl.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update _utils.py

* Update __init__.py

* Torch 2.8

* Update rl_replacements.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-19 03:16:49 -07:00
Daniel Han
10f68527d8 Bug fixes (#3180) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

* Update vision.py

* Update vision.py

* Update vision.py

* Upcast norms

* Update loader.py

* Update vision.py

* Upcast layernorms

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Update rl.py

* Update pyproject.toml

* Update rl.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-18 06:12:49 -07:00
andrewor14
bb7b2f40fc Add support for QAT + LoRA (#2976) 
**Summary:** Quantization-aware training (QAT) helps mitigate
quantization degradation by simulating quantization numerics
in high precision during training (fake quantization). This PR
combines QAT with LoRA by applying torchao's QAT support to
the peft model.

See the following for more details:

- torchao QAT: https://github.com/pytorch/ao/blob/main/torchao/quantization/qat/README.md
- torchtune QAT + LoRA: https://dev-discuss.pytorch.org/t/speeding-up-qat-by-1-89x-with-lora/2700

Current QAT schemes supported are:

```
fp8-fp8, targeting the torch.ops.fbgemm.f8i4bf16_shuffled kernel
fp8-int4, targeting the torch.ops.fbgemm.f8f8bf16_rowwise kernel
```

**Test Plan:**

```
from unsloth import FastLanguageModel

lora_rank = 32

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Qwen3-4B-Base",
    max_seq_length = 2048,
    load_in_4bit = False,
    fast_inference = False,
    max_lora_rank = lora_rank,
)

model = FastLanguageModel.get_peft_model(
    model,
    r = lora_rank,
    target_modules = [
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_alpha = lora_rank*2,
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
    qat_scheme = "fp8-fp8",
)

lora.Linear(
  (base_layer): FakeQuantizedLinear(
    in_features=2560, out_features=4096, bias=False
    (activation_fake_quantizer): FakeQuantizer(Float8FakeQuantizeConfig(dtype=torch.float8_e4m3fn, granularity=PerRow(), hp_value_lb=None, hp_value_ub=None))
    (weight_fake_quantizer): FakeQuantizer(Float8FakeQuantizeConfig(dtype=torch.float8_e4m3fn, granularity=PerRow(), hp_value_lb=None, hp_value_ub=None))
  )
  ...
)
```
 2025-08-18 05:56:35 -07:00
Ball
94c7392c40 fix original_push_to_hub fallback (#3115) 
Co-authored-by: root <root@LAPTOP-VEI2ITL9.localdomain>
 2025-08-18 05:54:24 -07:00
RJ Nowling
36c563fffd Replace back ticks with single quotes (#3157) 
Back ticks attempt to execute program and capture output.  Should be using single quote marks.
 2025-08-18 05:52:30 -07:00
Daniel Han
176805802f Update _utils.py 2025-08-18 05:28:30 -07:00
Roland Tannous
208f68f164 fix save_to_gguf_generic quantization_method type error (#3173) 2025-08-18 04:10:17 -07:00
Roland Tannous
bacfc57380 Convert generator expression to list to prevent potential bugs if the files variable is used multiple times in future modifications. (#3167) 2025-08-18 04:10:08 -07:00
Daniel Han
19b2fa3ac8 Fix Blackwell 2025-08-18 03:46:39 -07:00
QL
ea4b7c2c6b Update install instructions for latest vLLM release (#3175) 
1. Removed the `--extra-index-url https://wheels.vllm.ai/nightly` from the uv install instructions because this causes it to crash; Removing that flag solves the issue and is more stable overall. Tested with RTX 5090 CUDA 12.8 on Linux. 

2. Removed `uv pip install -U triton>=3.3.1` because triton 3.3.1 is already installed with the vllm command.
 2025-08-18 03:39:05 -07:00
Daniel Han
7e1581b929 Nightly (#3169) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-15 05:03:38 -07:00
Daniel Han
b45a977c8d Update loader.py 2025-08-15 04:08:21 -07:00
Daniel Han
64dff1456b GPT OSS MXFP4 fix 2025-08-15 04:00:00 -07:00
Daniel Han
bbac3e3de7 Update tokenizer_utils.py 2025-08-14 19:30:33 -07:00
Daniel Han
b11d9cc935 Encoding UTF-8 2025-08-14 15:24:25 -07:00
Daniel Han
abbf1f0a43 Fix GPT OSS (#3154) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-14 05:06:58 -07:00
Daniel Han
6b5fb59eb7 Update loader.py 2025-08-13 06:33:39 -07:00
Daniel Han
806bee2433 Nightly (#3148) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-13 06:12:38 -07:00
Michael Han
413ec45d8b Update README.md 2025-08-09 15:53:29 -07:00
Michael Han
230061cd48 Merge pull request #3120 from Etherll/patch-8825 
Add Qwen3 4B to mapper.py
 2025-08-08 14:31:31 -07:00
Etherll
b0004c0001 Update mapper.py 2025-08-08 23:27:21 +03:00
Michael Han
2a8ca1ef5a Update README.md 2025-08-08 12:14:38 -07:00
Daniel Han
9975cf889a Update pyproject.toml 2025-08-08 11:49:56 -07:00
Daniel Han
424e648005 Update _utils.py 2025-08-08 11:48:13 -07:00
Daniel Han
3fc4e4dff9 Update loader.py 2025-08-08 11:44:07 -07:00
Daniel Han
628bb6c97f Update loader.py 2025-08-08 11:38:03 -07:00
Daniel Han
20e7c33550 Merge branch 'main' into nightly 2025-08-08 10:55:52 -07:00
Daniel Han
84ff585855 Update loader.py 2025-08-08 10:55:30 -07:00
Daniel Han
b277a09401 Update _utils.py 2025-08-08 09:59:24 -07:00
Daniel Han
5931011d6c Update vision.py 2025-08-08 09:35:54 -07:00