GGUF saving (#2017)
* Update rl_replacements.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * fix an import error (#1767) * fix an import error * Delete .gitignore * Update loader.py * Update save.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * SamplingParams * Convert mask to float (#1762) * [Windows Support] Add latest `xformers` wheels to pyproject.toml (#1753) * Add latest xformers * Add a couple of lines to docs * vLLMSamplingParams * Update __init__.py * default num_chunks == -1 * Versioning * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update _utils.py * Update rl_replacements.py * Update rl_replacements.py * Update pyproject.toml * Update pyproject.toml * Export Model to ollama.com (#1648) * Ollama Export Model to ollama.com Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> * Check for model_name Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> * subprocess use instead of requests | added check for ollama server Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> * create_ollama_model Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> * create_ollama_model | fix Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> * Push to Ollama Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> --------- Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> * Update cross_entropy_loss.py * torch_cuda_device * Update utils.py * Update utils.py * Update utils.py * device * device * Update loader.py * Update llama.py * Update README.md * Update llama.py * Update llama.py * Update _utils.py * Update utils.py * Update utils.py * Update utils.py * Update utils.py * Update utils.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update utils.py * Update utils.py * Update utils.py * Update utils.py * __version__ * Update rl.py * Bug fixes * Bug fixes * Update llama.py * Update _utils.py * _wrap_fast_inference * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update _utils.py * SFT dataset prepare * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update llama.py * Update llama.py * Update utils.py * bug fix * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update __init__.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update rl.py * Update rl.py * Update rl.py * Update _utils.py * Update __init__.py * Update _utils.py * Version * versioning * Update _utils.py * Update llama.py * Update llama.py * Bug fixes * FastModel * __doc__ * Update vision.py * Update loader.py * Update loader.py * Update loader.py * version * move use_modelscope to _utils (#1938) * move use_modelscope to _utils * Update _utils.py * Update loader.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Don't use revision when loading model_config and is_peft=True (#1949) * More syntax warnings (#1944) * move use_modelscope to _utils * fix * Update _utils.py * Update loader.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Update loader.py * Full finetuning and other fixes * UNSLOTH_ENABLE_FULL_FINETUNING * Update loader.py * Update loader.py * Update loader.py * Update vision.py * Update vision.py * full finetuning * Update loader.py * Update loader.py * Update loader.py * Update _utils.py * max_seq_length * Update rl.py * Update rl.py * Update rl.py * Update pyproject.toml * AutoModelForImageTextToText * Update mapper.py * Update pyproject.toml * Update _utils.py * Update _utils.py * Update _utils.py * Batch samples * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update _utils.py * Update loader.py * Update vision.py * Update loader.py * Update vision.py * Update vision.py * Update vision.py * Update mapper.py * Update vision.py * Temporary patches * Update loader.py * model names * Gemma 3 chat template * Bug fixes * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update llama.py * Update llama.py * Update rl.py * Update chat_templates.py * Update chat_templates.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Update vision.py * Revert * Update _utils.py * forced precision * Autocast * Update vision.py * Update vision.py * Update rl.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update rl.py * vLLM fixes * constexpr * Update vision.py * Update vision.py * Update vision.py * Update rl.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update save.py * New models * Triton windows update (#1976) * Update pyproject.toml * Update README.md * Update RMS LayerNorm implementation, and list compr. change in chat templates (#1974) * Update RMS LayerNorm implementation with optimizations and testing suite * perf: optimize list comprehension in get_ollama_eos_tokens * Update Zoo * Update llama.py * Update llama.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update rl_replacements.py * Update vision.py * grpo fix * Update rl_replacements.py * Update vision.py * Update rl_replacements.py * Update vision.py * Update mapper.py * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Update save.py * Update save.py * Update save.py --------- Signed-off-by: Jyotin Goel <b22ai063@iitj.ac.in> Co-authored-by: Nino Risteski <95188570+NinoRisteski@users.noreply.github.com> Co-authored-by: Edd <68678137+Erland366@users.noreply.github.com> Co-authored-by: Ben <6579034+versipellis@users.noreply.github.com> Co-authored-by: Jyotin Goel <120490013+gjyotin305@users.noreply.github.com> Co-authored-by: Kareem <81531392+KareemMusleh@users.noreply.github.com> Co-authored-by: Wilson Wu <140025193+wiwu2390@users.noreply.github.com> Co-authored-by: Akshay Behl <126911424+Captain-T2004@users.noreply.github.com>
This commit is contained in:
Daniel Han
GitHub
parent
3410744e88
commit
b8aaf550a7
2 changed files with 53 additions and 5 deletions
|
|
@ -485,7 +485,7 @@ class FastBaseModel:
|
|||
full_finetuning = os.environ.get("UNSLOTH_ENABLE_FULL_FINETUNING", "0") == "1"
|
||||
|
||||
float32_mixed_precision = True
|
||||
if _get_dtype(model.config.torch_dtype) == torch.bfloat16:
|
||||
if _get_dtype(model.config.torch_dtype) == torch.bfloat16 and full_finetuning:
|
||||
# Use bfloat16 precision for full finetuning
|
||||
float32_mixed_precision = False
|
||||
|
||||
|
|
|
|||
|
|
@ -2218,12 +2218,60 @@ pass
|
|||
|
||||
|
||||
from .models.loader_utils import get_model_name
|
||||
from unsloth_zoo.saving_utils import merge_and_overwrite_lora
|
||||
from unsloth_zoo.saving_utils import (
|
||||
merge_and_overwrite_lora,
|
||||
prepare_saving,
|
||||
)
|
||||
from unsloth_zoo.llama_cpp import (
|
||||
install_llama_cpp,
|
||||
convert_to_gguf,
|
||||
convert_to_gguf as _convert_to_gguf,
|
||||
)
|
||||
|
||||
@torch.inference_mode
|
||||
def save_to_gguf_generic(
|
||||
model,
|
||||
save_directory,
|
||||
quantization_type = "Q8_0",
|
||||
repo_id = None,
|
||||
token = None,
|
||||
):
|
||||
if token is None and repo_id is not None: token = get_token()
|
||||
if repo_id is not None and token is None:
|
||||
raise RuntimeError("Unsloth: Please specify a token for uploading!")
|
||||
|
||||
if not os.path.exists(os.path.join("llama.cpp", "unsloth_convert_hf_to_gguf.py")):
|
||||
install_llama_cpp(just_clone_repo = True)
|
||||
pass
|
||||
|
||||
metadata = _convert_to_gguf(
|
||||
save_directory,
|
||||
print_output = True,
|
||||
quantization_type = quantization_type,
|
||||
)
|
||||
if repo_id is not None:
|
||||
prepare_saving(
|
||||
model,
|
||||
repo_id,
|
||||
push_to_hub = True,
|
||||
max_shard_size = "50GB",
|
||||
private = True,
|
||||
token = token,
|
||||
)
|
||||
|
||||
from huggingface_hub import HfApi
|
||||
api = HfApi(token = token)
|
||||
api.upload_folder(
|
||||
folder_path = save_directory,
|
||||
repo_id = repo_id,
|
||||
repo_type = "model",
|
||||
allow_patterns = ["*.gguf"],
|
||||
private = True,
|
||||
)
|
||||
pass
|
||||
return metadata
|
||||
pass
|
||||
|
||||
|
||||
@torch.inference_mode
|
||||
def unsloth_generic_save(
|
||||
model,
|
||||
|
|
@ -2467,8 +2515,8 @@ def patch_saving_functions(model, vision = False):
|
|||
# Vision only 1 option
|
||||
model.push_to_hub_merged = types.MethodType(unsloth_generic_push_to_hub_merged, model)
|
||||
model.save_pretrained_merged = types.MethodType(unsloth_generic_save_pretrained_merged, model)
|
||||
model.push_to_hub_gguf = types.MethodType(not_implemented_save, model)
|
||||
model.save_pretrained_gguf = types.MethodType(not_implemented_save, model)
|
||||
model.push_to_hub_gguf = types.MethodType(save_to_gguf_generic, model)
|
||||
model.save_pretrained_gguf = types.MethodType(save_to_gguf_generic, model)
|
||||
pass
|
||||
return model
|
||||
pass
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue