fix: single-pass GGUF export for directly convertible outtypes in save.py (#7090)
* Single-pass GGUF export for direct outtypes + parallel multi-quant save_to_gguf defaulted first_conversion to model_dtype before the block that picks the optimal base conversion, leaving that block dead since it landed (#3356). Every default export (fast_quantized -> q8_0) therefore ran two passes: convert HF -> 16-bit GGUF, then llama-quantize -> q8_0, writing a 2x-size intermediate that the cleanup step deletes again. - Route single-output exports whose type convert_hf_to_gguf.py emits directly (f32/f16/bf16/q8_0) through one conversion pass with no 16-bit intermediate. Measured on Qwen2.5-0.5B-Instruct (8-core CPU): bytes written 1525 MB -> 531 MB (2.9x less), peak extra disk 994 MB -> 0, wall time neutral on local NVMe (14.8s vs 15.4s). The dequantized q8_0 tensors are bit-identical to the two-pass output (max diff 0 over all 290 tensors, same quant-type table). On disk-capped runtimes (Kaggle 20 GB, Colab) the removed intermediate is the difference between an export that fits and one that dies - see the Kaggle error text this file already carries. imatrix runs keep the two-pass route since only llama-quantize can apply one; explicit first_conversion is still honored. - Run independent llama-quantize passes two at a time when several quant methods are requested (thread budget split between workers, outputs byte-identical, order preserved). Measured 1.38x wall-clock on q4_k_m+q5_k_m+q6_k. Sequential under UNSLOTH_ENABLE_LOGGING=1 to keep subprocess logs readable; kill switch UNSLOTH_PARALLEL_GGUF_QUANTS=0. Duplicate methods now quantize once. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard parallel GGUF quant on Kaggle and make multi-quant failures atomic Each llama-quantize pass loads the whole model into RAM, so running two at once on Kaggle can OOM a host that succeeded sequentially; skip the parallel path there. On a failed multi-quant export, stop launching queued passes and remove orphaned quant outputs so a failure leaves no partial GGUFs behind, keeping the 16-bit base for retry. Also accept 0/false/no/off/empty for UNSLOTH_PARALLEL_GGUF_QUANTS so a well-meant 'false' actually disables parallelism, and add tests/saving/test_gguf_single_pass_export.py to the CI saving bucket so the new tests run. * Preserve pre-existing outputs for canceled quant passes on failure The parallel cleanup unlinked every requested output name, so a failed rerun could delete a valid model.<METHOD>.gguf left by an earlier successful export for a method whose pass was canceled and never ran this session. Skip canceled futures and only remove outputs from passes that actually executed. * Gate parallel GGUF quant on available memory and preserve prior outputs Skip the two-worker path when RAM cannot hold two full-model quantizations at once (and on Colab as well as Kaggle), so a multi-quant export that fit sequentially no longer OOMs. On failure, remove only outputs this run newly created, tracked against a pre-launch snapshot, so a rerun into an existing _gguf directory never deletes a valid artifact from an earlier export. --------- Co-authored-by: djs <dschroers2@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
This commit is contained in:
parent
d8094335b7
commit
8cfd1a2173
3 changed files with 390 additions and 98 deletions
2
.github/workflows/consolidated-tests-ci.yml
vendored
2
.github/workflows/consolidated-tests-ci.yml
vendored
|
|
@ -272,6 +272,7 @@ jobs:
|
|||
tests/saving/test_export_api_surface.py \
|
||||
tests/saving/test_export_dispatch.py \
|
||||
tests/saving/test_imatrix_export.py \
|
||||
tests/saving/test_gguf_single_pass_export.py \
|
||||
tests/utils/test_attention_masks.py \
|
||||
tests/utils/test_trunc_normal_patch.py \
|
||||
tests/python/test_fast_language_model_text_only.py
|
||||
|
|
@ -361,6 +362,7 @@ jobs:
|
|||
tests/saving/test_export_api_surface.py \
|
||||
tests/saving/test_export_dispatch.py \
|
||||
tests/saving/test_imatrix_export.py \
|
||||
tests/saving/test_gguf_single_pass_export.py \
|
||||
tests/utils/test_attention_masks.py \
|
||||
tests/utils/test_trunc_normal_patch.py \
|
||||
tests/python/test_fast_language_model_text_only.py \
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue