b98b1b64f6
Core changes that were missing from the original PR:
1. unsloth/models/llama.py:
- Add _ensure_cache_is_dynamic() to convert tuple/list KV caches
to DynamicCache for transformers v5 compatibility
- Add _slice_position_ids() to handle position_ids slicing when
input_ids is trimmed to uncached tokens
- Fix unsloth_fast_generate: skip setting cache_implementation
when user provides past_key_values (avoids ValueError on
transformers >= 4.57)
- Fix _fast_prepare_inputs_for_generation: when past_key_values
covers fewer tokens than input_ids, keep only the uncached
portion instead of always slicing to last token
- Fix CausalLM_fast_forward: add input_ids.shape[1] == 1 guard
so multi-token prefill with external KV cache falls through
to the regular model forward path
2. unsloth/models/mistral.py:
- Same input_ids.shape[1] == 1 guard and multi-token fallback
as CausalLM_fast_forward
3. tests/test_past_kv_models.py:
- Extract model loading into _load_model() with proper SkipTest
handling (only skips on loading errors, not on test failures)
- Remove blanket try/except that was masking real failures
- Remove exact output match assertion (4-bit quantization with
different computation paths can produce slightly different but
equally valid outputs)
4. tests/test_past_kv_utils.py:
- Fix comment to accurately describe the inlined functions
Tested: Llama-3.2-1B-Instruct, Qwen3-0.6B, gemma-2-2b-it
All 4 integration tests pass, 14 unit tests pass, benchmark runs,
and standard training (Gemma2, Llama) is not regressed.
|
||
|---|---|---|
| .. | ||
| qlora | ||
| saving | ||
| utils | ||
| __init__.py | ||
| test_get_model_name.py | ||
| test_model_registry.py | ||
| test_past_kv_models.py | ||
| test_past_kv_utils.py | ||
| test_raw_text.py | ||