Commit graph

1 commit

Author SHA1 Message Date
Anmol Mishra
4e9d772d36
fix: preserve tokenizer eos token on merged saves (#5451)
Preserves the source tokenizer's eos_token in tokenizer_config.json after merged saves so runtimes such as vLLM read the correct stop token. Centralized inside the patched tokenizer save_pretrained so all save paths (merged_16bit, GGUF, torchao, push_to_hub) benefit, with filename_prefix support.

Fixes #5386
2026-05-17 06:44:23 -07:00