* Fix broken manual response-template markers in Studio's fallback table
Six template families in TEMPLATE_TO_RESPONSES_MAPPER shipped markers that
never match what their chat templates actually render, so the manual
train_on_completions path masked every assistant token and the run died on
the all-labels-masked safety net:
- mistral, llama: '[INST] ' / ' [/INST]' - the surrounding spaces fold into
the neighbouring tokens ('[INST]'/'[/INST]' are single special tokens in
Mistral v0.3, SentencePiece pieces in Llama-2), so the padded strings
never match. Now '[INST]' / '[/INST]'.
- starling: trailing space after 'GPT4 Correct Assistant:' folds into the
next content token. Now no trailing space.
- glm: '[gMASK]<sop>' renders once at text start, never before later user
turns, and '<think>' is generation scaffolding rendered as a lone
'</think>' on non-final turns. Now '<|user|>' / '<|assistant|>'.
- qwen3-thinking: '<think>' is stripped from non-final assistant turns
(Qwen3-Thinking-2507) and never rendered by QwQ. Now the bare assistant
header, matching the other qwen entries.
- zephyr: role tags are plain text and SentencePiece tokenizes them
differently at text start than after '</s>' + newline mid-conversation;
the markers need the leading newline anchor. Now '\n<|user|>\n' /
'\n<|assistant|>\n'.
Validated token-level on each family's representative tokenizer with a
two-turn fixture plus system message: user and system content fully masked,
every assistant turn trained, and the final EOS label never -100. The
fixed mistral, llama, starling and glm markers produce labels identical to
zoo auto-detection; qwen3-thinking differs only in one turn-separator
newline token. All 22 unchanged entries produce byte-identical labels to
before this change.
Adds tests/test_response_template_markers.py pinning the fixed and key
unchanged marker literals (dependency-free) plus token-level masking checks
that skip when tokenizers or unsloth_zoo are unavailable offline.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close tokenizer config handle and read it as UTF-8
Chat templates in tokenizer_config.json are rarely ASCII-only, so the
default locale codec could fail the GLM fallback loader on Windows.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
* Anchor the llama marker on <s> and harden the marker test
On transformers 5.x llama-2 tokenizes [INST] after <s> as a bare left
bracket while the standalone encoding gives the space-prefixed piece, so
the unanchored marker missed every turn boundary and later user turns
leaked into training; 4.57 masked this. Anchoring on <s>[INST] matches
both tokenizations, verified token-level under 4.57.6 and 5.5.0.
The test now unwraps the BatchEncoding that apply_chat_template returns
on 5.x before indexing, and the latent trailing spaces in the unreachable
unsloth and vicuna entries are dropped for table consistency.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
275bad1f64