* tests: read checked-in files as UTF-8 instead of the platform default
Path.read_text() with no encoding uses locale.getpreferredencoding(), which
is UTF-8 on the Linux runners and cp1252 on a stock Windows install. Nine
module-level reads of checked-in source files were relying on that default.
studio/backend/routes/inference.py carries the DeepSeek tool-call token
regexes, so it holds U+FF5C and U+2581. Under cp1252 that read raised
UnicodeDecodeError on byte 0x81 at position 97806, and because the reads run
at import time it took test_cancel_atomicity.py and test_cancel_id_wiring.py
out at collection, not as failures. Green on CI, permanently broken for a
Windows contributor running the suite locally.
Adds a guard: at module scope there is no tmp_path fixture, so a bare
read_text()/write_text()/open() there is always touching a checked-in file.
That makes the rule mechanical enough to enforce with no allowlist, while
staying quiet about temp-dir I/O inside test bodies where the platform
default is harmless.
The repo already spells this correctly in 464 other places; this only stops
the stragglers coming back.
* tests: cover import-time helper reads and keep the guard py3.9-safe
Follows up on the Codex review:
- add `from __future__ import annotations`, since `str | None` in
`_offender` is evaluated at import on Python 3.9 and pyproject declares
requires-python ">=3.9,<3.15".
- widen the guard from module scope to import time. Class bodies and the
bodies of module-level helpers called from an executing statement run
during collection too, so `CODE = _extract_mixed_precision_code()` was
the same hazard as an inline read. `if __name__ == "__main__":` blocks
are skipped: pytest never executes them.
- scan studio/backend/tests/ as well as tests/. Both trees are collected
on Windows by separate CI jobs, and the offender that started this,
test_tool_xml_strip.py reading routes/inference.py, lives there.
Widening it surfaced seven more import-time reads of checked-in sources;
all now name utf-8.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden the import-time encoding guard for PR #7438
Close the detector gaps raised in review, all of which I reproduced against
the actual AST before changing anything.
False negatives (the guard let a real hazard through):
- _is_main_guard ignored the comparison operator, so if __name__ != "__main__"
counted as script-only even though its body runs at import.
- The else arm of a main guard was discarded with the rest of the If node.
- Decorators and argument defaults on a module-level def were skipped with the
body, though both are evaluated when the def executes.
- Path.open() in text mode was invisible; only builtin open() was matched.
- encoding = None and encoding = "locale" both re-select the platform default,
but the keyword merely being present counted as pinned.
False positives (the guard would have blocked a compliant contributor):
- A non-literal mode fell through to the "r" default, so open(p, mode) was
flagged even when mode is "rb", where adding encoding= is a ValueError and
there is no edit that satisfies the rule.
- Same for open(*args) and a **kwargs splat, which hide the mode and can hide
an encoding.
- Lambda bodies and comprehension elements were walked even though neither runs
at definition.
Verified: still reports the same 22 offenders on unpatched main, green on this
branch and on the tree merged with latest main (557 files), and an adversarial
corpus of 33 cases now scores zero false positives and zero false negatives.
Also corrected two docstring claims: neither collecting job runs on Windows,
and the read is governed by locale.getencoding().
* Walk eager comprehensions and treat io.open as the builtin
Two regressions from the previous commit, both reproduced against the AST
before changing anything.
Lumping list, set and dict comprehensions in with generator expressions was
wrong. Only a genexp is lazy; the other three run their element expression,
their filters and their nested iterators immediately, so
CONTENTS = [p.read_text() for p in PATHS] at module scope is an import-time
read the guard was silently missing. Comprehensions are now walked in full and
only the genexp keeps the outermost-iterable-only treatment.
io was also in the not-a-path-opener list, but io.open is the builtin, with the
same mode position and the same platform default. io.open(CHECKED_IN_FILE) is
exactly the hazard this guard exists for, so it is matched now, with binary
modes and a pinned encoding still exempt. tarfile.open and fitz.open stay
exempt since neither has an encoding to name.
Verified: 13 targeted cases covering all five eager comprehension forms and
io.open in text, binary and pinned shapes all classify correctly; still 22
offenders on unpatched main; green on this branch and on the tree merged with
latest main.
* Close three more walker gaps in the import-time guard
All three reproduced against the AST first.
A generator expression handed straight to a call is consumed there, so
DATA = "".join(p.read_text() for p in paths) runs its element at import. Only
an unconsumed genexp bound to a name stays lazy, so the walker now follows the
consumed ones in full and keeps the outermost-iterable-only treatment for the
rest.
if "__main__" == __name__ is an equivalent and accepted spelling of the main
guard, but requiring __name__ on the left meant its body was treated as
import-time code. That is a false positive on a block pytest never runs, so
both operand orders are recognised now.
The helper table was built from module-level defs only, so a def in a class
body invoked while the class is constructed was never followed, contradicting
the walker's stated coverage of class bodies. Helpers are now collected from
the module body and from class bodies at any nesting.
Verified: 15 targeted cases including all three fixes and the earlier ones
still classify correctly; still 22 offenders on unpatched main; green on this
branch and on the tree merged with latest main.
* Handle positional read_text encodings, lazy generators and nested helpers
* Guard reads reached from test bodies, unbound Path calls and __file__ paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Follow derived paths, skip lazy generator helpers, cover compressed openers
* Guard the CLI tests, helper parameters and unbound Path arguments
* Discover test roots and follow literal, in-place and tuple-derived paths
* Identify module openers by import, unwrap starred paths, pin subprocess snippets
* Resolve import origins, seed helper locals, follow named generators and parametrize
* Scope imports lexically, list tracked test files, bind unpacked names
* Resolve aliased openers, keyword-only params, destructured targets, next()
* Pin the encoding on subprocess snippets, workflow lint and CLI output for PR #7438
* Harden the CLI encoding guard against detached streams for PR #7438
* Tighten the encoding guard's path and scope analysis for PR #7438
* Resolve path provenance more precisely and keep POSIX stream encodings for PR #7438
* Resolve qualified path classes and scope conditional imports for PR #7438
* Scope CLI stream setup to the entry point and align two encoding pairs for PR #7438
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.
* Fix DPO trainer multi process hang
* Fix datacollator error
* further dpo vision changes
* cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden DPO vision row processing and source rewrites
- dpo_trainer_vision_signature_columns: also match TRL 0.22.x layout
(image_sizes followed by ref_chosen_logps), so vision keys are not
stripped via remove_unused_columns on the originally-affected version.
- dpo_trainer_concatenated_inputs: fall back to inserting after the
image_sizes block when no token_type_ids anchor follows it.
- Apply the same vision model_kwargs forwarding rewrite to
_compute_loss_liger via dpo_trainer_compute_loss_liger so the Liger DPO
path does not drop pixel_position_ids/image_position_ids/
mm_token_type_ids when args.use_liger_loss is true.
- dpo_trainer_vision_process_row:
- guard chosen/rejected EOS append with tokenizer.eos_token_id is not None
- use features.get("images") and features.get("prompt") to match the
existing get on line 164 and avoid KeyError on rows without those keys
- drop the torch.is_tensor gate so list-form pixel_position_ids/
image_position_ids returned without return_tensors are still aliased
- skip the loop entry for image_position_ids when it was already
promoted to pixel_position_ids, so the output dict no longer carries
both keys with identical data
- dpo_trainer_data_collator_vision_keys: switch from pad_sequence to
trl.trainer.utils.pad with padding_side='left' (matches the DPO
collator's prompt left-pad) and padding_value=-1 for *_position_ids
keys (sentinel for padded patches), 0 otherwise. Skip the key when not
every example carries it. Falls back to pad_sequence if trl.pad is
unavailable or the tensor rank is too high.
- dpo_trainer_prepare_dataset: keep TRL's writer_batch_size=10 when
popping num_proc; removing it defaults to 1000 and reintroduces the
vision OOM risk that writer_batch_size=10 was set to avoid.
* DPO vision row: keep upstream-facing keys and fix patch padding
- dpo_trainer_vision_process_row: no longer aliases image_position_ids
to pixel_position_ids. Each upstream-emitted vision key is forwarded
under its own name. Gemma4 ForConditionalGeneration.forward accepts
image_position_ids directly and renames it to pixel_position_ids only
at the vision-tower call site, so aliasing in the row helper hid the
kwarg the model actually consumes.
- dpo_trainer_vision_process_row: extract pixel_values via "in"
membership instead of unconditional indexing. With the missing-images
path returning [] to the processor, modern processors no longer emit
a pixel_values key, and the previous indexing raised KeyError.
- dpo_trainer_data_collator_vision_keys: pick padding_side per key
family. *_position_ids tensors are patch-aligned to pixel_values
(TRL's DataCollatorForPreference right-pads pixel_values), so pad
them right with the -1 sentinel; mm_token_type_ids is token-aligned
to prompt_input_ids (left-padded by TRL), so pad it left with 0.
* DPO vision: handle multi-image prompts and arbitrary-rank collator pad
- dpo_trainer_vision_process_row: when a prompt is missing vision
placeholders, insert one placeholder per missing image instead of
always inserting a single token. Multi-image rows now satisfy the
processor's token-vs-image count check rather than under-inserting
and tripping the placeholder/feature mismatch.
- dpo_trainer_data_collator_vision_keys: drop the dim()<=2 gate around
trl.trainer.utils.pad. trl.pad handles arbitrary rank correctly,
while the previous fallback to torch.nn.utils.rnn.pad_sequence
raised RuntimeError on rank-3 patch-position tensors with mismatched
non-leading dimensions. The pad_sequence path remains as a degraded
fallback only when trl.pad is unavailable or raises.
* DPO vision row: support scalar images and align prompt-aligned aux ids
- dpo_trainer_vision_process_row: type-aware normalization of the
features['images'] column instead of a truthiness/len check that
raised on single image objects (PIL.Image has no __len__) and on
numpy ndarrays (truthiness ambiguous). Lists/tuples count as their
length, scalar image objects count as one, None counts as zero, and
the original value is forwarded to the processor.
- dpo_trainer_vision_process_row: when max_prompt_length truncates
prompt_input_ids, also slice token_type_ids and mm_token_type_ids
by the same [-max_prompt_length:] suffix. Those keys are 1:1 token
aligned to prompt_input_ids (Gemma 4 vision attention keys off
mm_token_type_ids per modular_gemma4.py), so leaving them at the
original length silently misaligned the multimodal mask.
* DPO vision row: stop synthesizing vision-token placeholders
Pass features['prompt'] and features['images'] straight to the
processor without inserting any extra placeholder tokens. The previous
helper used processing_class.image_token, which is the right prompt
placeholder for Gemma 4 but the wrong one for Gemma 3 (whose prompt
placeholder is boi_token while image_token is the inner expansion
target). Synthesizing that token also broke multi-image rows: text
ended up with N placeholders while the row helper only forwarded the
first image's pixel_values via the standard [0] indexing that mirrors
upstream TRL process_row, so token vs image-feature counts diverged.
Removing the synthesis matches stock TRL behavior; users provide the
correct placeholders for their processor in the prompt.
* Add tests for DPO vision row processor passthrough
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Leo Borcherding