* Reduce and tighten comments and docstrings in tests
Shorten verbose comments and docstrings across the test suite without
changing any test logic. Remove narration that restates the next line,
collapse long module and test docstrings to a single line, and drop banner
separators. Keep regression context (issue and PR references, run ids),
skip reasons, mocking and timing rationale, license headers, lint and type
directives, and commented-out code.
Comments and docstrings only: an AST signature check confirms no code,
assertions, or string literals changed, and the suite byte-compiles cleanly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.
* Fix ORPO text tokenization with processors
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard ORPO tokenizer rewrite anchor
* Resolve processor pad_token_id and preserve preference data collators for ORPO
Two follow-ups so the text-only ORPO + VL processor path works end to end on
top of the build_tokenized_answer and tokenize_row rewrites:
1. Add orpo_trainer_processor_pad_token to rewrite processing_class.pad_token_id
in ORPOTrainer.__init__ to fall back to processing_class.tokenizer.pad_token_id
when the processor itself has no pad_token_id (Qwen3-VL, Gemma-3, etc.).
Without this, DPODataCollatorWithPadding(pad_token_id=processing_class.pad_token_id)
raises AttributeError before training starts.
2. Stop the outer UnslothORPOTrainer.__init__ collator-swap from clobbering
DPODataCollatorWithPadding when the tokenizer is a processor without .pad.
The swap to TransformersDataCollatorForLanguageModeling is now only applied
to LM-style collators, so ORPO/DPO/CPO/KTO keep their own prompt/chosen/
rejected handling. Otherwise the collator can't pad ORPO rows and raises
"You should supply an encoding ... that includes input_ids" at train time.
Verified with Qwen3-VL-2B-Instruct ORPO + text-only data (training completes
to max_steps, no AttributeError, no collator error) and Llama-3.2-1B-Instruct
ORPO (losses and grad-norms bit-exact identical to main, so the change is a
true no-op for plain text tokenizers).
Extends tests/python/test_orpo_processor_text_tokenizer.py with three new
unit tests covering the pad_token_id rewriter.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
