* Fix llama3 RoPE scaling dropped on transformers v5
transformers v5 loads on meta then blanks non-persistent buffers, so
_fix_rope_inv_freq rebuilds inv_freq after load. It recomputed a vanilla
inv_freq and applied _apply_inv_freq_scaling, a no-op on the base
LlamaRotaryEmbedding used by the config/llama3 path, so inv_freq ended up
divided by 1 instead of the config factor (8 for Llama 3.1, 32 for Llama
3.2). This corrupts long-range positions and inflates long-context loss
about 3-5x. transformers 4.x was unaffected.
Route __init__ and the v5 repair through one _unsloth_recompute_inv_freq
so they cannot diverge, and stash the config on the rotary module so the
repair can rebuild the same scaled value.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add test for llama3 RoPE scaling under the transformers v5 repair
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update RoPE drift guard for the recompute refactor and guard the v5 repair
The drift guard's AST tripwire asserted the config-scaling call lived in the
if config is not None branch of LlamaRotaryEmbedding.__init__. The fix moved
that into _unsloth_recompute_inv_freq, so follow it there (with a fallback to
the old inline branch) and add a guard that loader._fix_rope_inv_freq rebuilds
inv_freq through the same helper. Also add a CPU functional check of the helper
and drop the redundant standalone test.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2fada48ef5