ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
2,774 commits 391 branches 58 tags 304 MiB
Commit graph

2,774 commits

This branch
This branch
All branches
Author SHA1 Message Date
Daniel Han
48db7e552d Update import_fixes.py 2025-09-03 20:12:42 -07:00
Daniel Han
29d2f5f09b Move logging 2025-09-03 20:07:27 -07:00
Daniel Han
274b0e9ded Update _utils.py 2025-09-03 20:00:21 -07:00
Daniel Han
286c93c045 Update pyproject.toml 2025-09-03 19:55:10 -07:00
Daniel Han
1f7fe4631d Update llama.py 2025-09-03 19:11:53 -07:00
Jerry Zhang
43d1f1fc2d Support saving locally in model.save_pretrained_torchao (#3263) 
Summary:
Previously the test was not ran correctly and the save to local path is not tested
this PR added support for that and tries to test properly

Note: `python tests/saving/test_unsloth_save.py` doesn't run test

Test Plan:
pytest tests/saving/test_unsloth_save.py -k test_save_torchao

Reviewers:

Subscribers:

Tasks:

Tags:
 2025-09-03 17:51:33 -07:00
Daniel Han
49a9b33cdf Update save.py 2025-09-03 15:19:02 -07:00
Lei Zhenyuan
06840d0f00 [Intel] make intel device support ROPE (#3164) 
* make intel device pass

* abstract torch device stream
 2025-09-03 04:39:57 -07:00
stevenxdavis
fb33feba79 Fix incorrect function call in test_qwen3_grpo.py (#3212) 
* Update test_qwen3_grpo.py to correct function call

This test file uses the incorrect name for the function, which is gradient_checkpointing_disable(), not disable_gradient_checkpointing(). 

I copied the line from test_llama32_sft.py - I'm not sure if this actually is required, just wanted it consistent for when other people like me test this and have no clue what they're doing when it throws an exception.

* Update blackwell/test_qwen3_grpo.py

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
 2025-09-03 04:39:12 -07:00
Defi Wimar
37499c97b0 chore: Fix Typos (#3246) 2025-09-03 04:38:41 -07:00
Tim Paine
4c4e9d678b Remove old version constraint in dependency list (#3237) 
xref: https://github.com/unslothai/unsloth-zoo/pull/258
 2025-09-01 02:29:58 -07:00
pluesclues
fb4ae0134a Update mistral.py, showed flag to not call cut cross entropy (#3233) 
* Update mistral.py, showed flag to not call cut cross entropy

* Update mistral.py, made it so if its not equal to zero

* Update unsloth/models/mistral.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
 2025-08-29 01:32:21 -07:00
Roland Tannous
3ee93ccd43 tests for mxfp4 and quantized models merge fix unsloth zoo pr 254 (#3223) 2025-08-29 01:30:48 -07:00
Daniel Han
bf66b8e4e4 GPT OSS Bug fixes (#3231) 
* Update rl.py

* Update rl.py

* Update rl.py

* GPT OSS float32

* Update vision.py

* Update loader.py

* Update loader.py
 2025-08-28 09:39:46 -07:00
Daniel Han
eded24dad4 Merge branch 'main' of https://github.com/unslothai/unsloth August-2025-v2 2025-08-28 03:19:16 -07:00
Daniel Han
427d597759 Versioning 2025-08-28 03:19:14 -07:00
Michael Han
20e6469959 Merge pull request #3224 from DefiWimar7/typos 
chore: Fix Typos

Thank you @DefiWimar7
 2025-08-28 02:46:27 -07:00
DoubleMathew
603f52e1ff Handle transformers move to dtype from torch_dtype (#3225) 2025-08-28 02:43:41 -07:00
DefiWimar7
16dea671cd chore: Fix Typos 2025-08-28 10:44:28 +08:00
DoubleMathew
41951cafb7 Fix gemma-3n (#3219) 
* place gemma-3n handling inside gemma-3 conditional

* cleanup
 2025-08-26 19:46:27 -07:00
Jerry Zhang
034b637176 Support model.save_pretrained_torchao (#3111) 
Summary:
Allow users merge the LoRA weights and then do a post training quantization with torchao

Usage:

```
from torchao.quantization import Int8DynamicActivationInt8WeightConfig
torchao_config = Int8DynamicActivationInt8WeightConfig()
model.save_pretrained_torchao(
    save_path,
    tokenizer=tokenizer,
    torchao_config=torchao_config,
)
```

Test Plan:
python tests/saving/test_unsloth_save.py

Reviewers:

Subscribers:

Tasks:

Tags:
 2025-08-26 04:53:39 -07:00
Lei Zhenyuan
27caf9ed50 fix is casual for qwen3 (#3213) 2025-08-26 04:45:20 -07:00
Daniel Han
d4c589f384 Update vision.py 2025-08-22 04:02:59 -07:00
Daniel Han
096e9aaddc Update loader.py 2025-08-22 04:02:19 -07:00
DoubleMathew
cdf63907e6 adallow float32 dtype in FastLanguageModel (#3204) 2025-08-21 16:54:39 -07:00
Daniel Han
0b42a72e44 Bug fixes (#3195) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

* Update vision.py

* Update vision.py

* Update vision.py

* Upcast norms

* Update loader.py

* Update vision.py

* Upcast layernorms

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Update rl.py

* Update pyproject.toml

* Update rl.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update _utils.py

* Update __init__.py

* Torch 2.8

* Update rl_replacements.py

* Update loader.py

* UNSLOTH_ENABLE_CCE

* Fix

* Update loader.py

* Update loader.py

* Update __init__.py

* Update __init__.py

* Update __init__.py

* Update __init__.py

* Import fixes

* Update loader.py

* Fix aimv2 issue

* Update loader.py

* Update import_fixes.py

* Update import_fixes.py

* Update loader.py

* Update loader.py

* Update loader.py

* Upgrade

* Update loader.py

* Update loader.py

* Update loader.py

* Update loader.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-20 07:39:43 -07:00
Daniel Han
74f4a5fff4 Update _utils.py 2025-08-19 23:43:58 -07:00
Michael Han
f2a3bc3369 Merge pull request #3187 from parth2510/fix-transformers-typo-extras 
Fix extras transformers typo in pyproject.toml
 2025-08-19 15:06:11 -07:00
parth2510
f58374014e Fix extras transformers typo in pyproject.toml 2025-08-19 19:55:29 +05:30
Daniel Han
ffae287698 Update pyproject.toml 2025-08-19 05:20:19 -07:00
Daniel Han
0e1ab9430e Protobuf issue 2025-08-19 05:19:41 -07:00
Daniel Han
f64c5dd5eb Update rl.py 2025-08-19 05:04:17 -07:00
Daniel Han
3905796b53 Update pyproject.toml 2025-08-19 03:24:02 -07:00
Daniel Han
503d5797f6 Update _auto_install.py 2025-08-19 03:20:37 -07:00
Daniel Han
d4574f1791 Torch 2.8 (#3186) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

* Update vision.py

* Update vision.py

* Update vision.py

* Upcast norms

* Update loader.py

* Update vision.py

* Upcast layernorms

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Update rl.py

* Update pyproject.toml

* Update rl.py

* Update rl_replacements.py

* Update rl.py

* Update rl.py

* Update rl.py

* Update _utils.py

* Update __init__.py

* Torch 2.8

* Update rl_replacements.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-19 03:16:49 -07:00
Daniel Han
609fe97cf2 Bug fixes (#3180) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

* Update vision.py

* Update vision.py

* Update vision.py

* Upcast norms

* Update loader.py

* Update vision.py

* Upcast layernorms

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update llama.py

* Update save.py

* Update rl.py

* Update pyproject.toml

* Update rl.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-18 06:12:49 -07:00
andrewor14
a14cf9a732 Add support for QAT + LoRA (#2976) 
**Summary:** Quantization-aware training (QAT) helps mitigate
quantization degradation by simulating quantization numerics
in high precision during training (fake quantization). This PR
combines QAT with LoRA by applying torchao's QAT support to
the peft model.

See the following for more details:

- torchao QAT: https://github.com/pytorch/ao/blob/main/torchao/quantization/qat/README.md
- torchtune QAT + LoRA: https://dev-discuss.pytorch.org/t/speeding-up-qat-by-1-89x-with-lora/2700

Current QAT schemes supported are:

```
fp8-fp8, targeting the torch.ops.fbgemm.f8i4bf16_shuffled kernel
fp8-int4, targeting the torch.ops.fbgemm.f8f8bf16_rowwise kernel
```

**Test Plan:**

```
from unsloth import FastLanguageModel

lora_rank = 32

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Qwen3-4B-Base",
    max_seq_length = 2048,
    load_in_4bit = False,
    fast_inference = False,
    max_lora_rank = lora_rank,
)

model = FastLanguageModel.get_peft_model(
    model,
    r = lora_rank,
    target_modules = [
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_alpha = lora_rank*2,
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
    qat_scheme = "fp8-fp8",
)

lora.Linear(
  (base_layer): FakeQuantizedLinear(
    in_features=2560, out_features=4096, bias=False
    (activation_fake_quantizer): FakeQuantizer(Float8FakeQuantizeConfig(dtype=torch.float8_e4m3fn, granularity=PerRow(), hp_value_lb=None, hp_value_ub=None))
    (weight_fake_quantizer): FakeQuantizer(Float8FakeQuantizeConfig(dtype=torch.float8_e4m3fn, granularity=PerRow(), hp_value_lb=None, hp_value_ub=None))
  )
  ...
)
```
 2025-08-18 05:56:35 -07:00
Ball
d633a4f4c1 fix original_push_to_hub fallback (#3115) 
Co-authored-by: root <root@LAPTOP-VEI2ITL9.localdomain>
 2025-08-18 05:54:24 -07:00
RJ Nowling
ac48a607d1 Replace back ticks with single quotes (#3157) 
Back ticks attempt to execute program and capture output.  Should be using single quote marks.
 2025-08-18 05:52:30 -07:00
Daniel Han
16e8d65992 Update _utils.py 2025-08-18 05:28:30 -07:00
Roland Tannous
8673300d87 fix save_to_gguf_generic quantization_method type error (#3173) 2025-08-18 04:10:17 -07:00
Roland Tannous
8d62d91c04 Convert generator expression to list to prevent potential bugs if the files variable is used multiple times in future modifications. (#3167) 2025-08-18 04:10:08 -07:00
Daniel Han
134ecb0e13 Fix Blackwell 2025-08-18 03:46:39 -07:00
QL
89397047f4 Update install instructions for latest vLLM release (#3175) 
1. Removed the `--extra-index-url https://wheels.vllm.ai/nightly` from the uv install instructions because this causes it to crash; Removing that flag solves the issue and is more stable overall. Tested with RTX 5090 CUDA 12.8 on Linux. 

2. Removed `uv pip install -U triton>=3.3.1` because triton 3.3.1 is already installed with the vllm command.
 2025-08-18 03:39:05 -07:00
Daniel Han
06f9d1cbc7 Nightly (#3169) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

* Update vision.py

* Update vision.py

* Update llama.py

* Update llama.py

* Update llama.py

* Versioning

* Update mapper.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-15 05:03:38 -07:00
Daniel Han
bd7f9f2930 Update loader.py 2025-08-15 04:08:21 -07:00
Daniel Han
d376e62038 GPT OSS MXFP4 fix 2025-08-15 04:00:00 -07:00
Daniel Han
935ef43616 Update tokenizer_utils.py 2025-08-14 19:30:33 -07:00
Daniel Han
56828b5431 Encoding UTF-8 2025-08-14 15:24:25 -07:00
Daniel Han
86a2725cb6 Fix GPT OSS (#3154) 
* Fix mamba

* Update loader.py

* Update vision.py

* Update loader.py

* Filter vLLM standby logs (#3131)

* filter vLLM standby logs

* safeguard standby logger patch

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

* Update unsloth/models/_utils.py

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>

* Update loader.py

* Add scaler

* Update llama.py

* Update _utils.py

* Versioning

* GPT OSS fix

* GPT OSS fix

* Update loader.py

* Update vision.py

* Update vision.py

* Update loader.py

---------

Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
 2025-08-14 05:06:58 -07:00