ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Resize rope embeddings for long sequence training (#3586)

Browse source
This commit is contained in:
DoubleMathew 2025-11-11 20:11:31 -06:00 • committed by GitHub
commit 0dbff7bade
1 changed files with 4 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

4
unsloth/models/llama.py
View file

@ -1051,6 +1051,10 @@ def LlamaModel_fast_forward(
# Also, transformers 4.45.0 supports granite but with the attention refactor (it always had the refactor)
# unsloth's check for granite too has "version >= 4.45.0 (rightly so)".
# so let granite always use the attention refactor implementation.
self.rotary_emb.extend_rope_embedding(
hidden_states, self.config.max_position_embeddings
)
position_embeddings = self.rotary_emb.get_cached(
self.config.max_position_embeddings, hidden_states.device.index
)