* tests: read checked-in files as UTF-8 instead of the platform default Path.read_text() with no encoding uses locale.getpreferredencoding(), which is UTF-8 on the Linux runners and cp1252 on a stock Windows install. Nine module-level reads of checked-in source files were relying on that default. studio/backend/routes/inference.py carries the DeepSeek tool-call token regexes, so it holds U+FF5C and U+2581. Under cp1252 that read raised UnicodeDecodeError on byte 0x81 at position 97806, and because the reads run at import time it took test_cancel_atomicity.py and test_cancel_id_wiring.py out at collection, not as failures. Green on CI, permanently broken for a Windows contributor running the suite locally. Adds a guard: at module scope there is no tmp_path fixture, so a bare read_text()/write_text()/open() there is always touching a checked-in file. That makes the rule mechanical enough to enforce with no allowlist, while staying quiet about temp-dir I/O inside test bodies where the platform default is harmless. The repo already spells this correctly in 464 other places; this only stops the stragglers coming back. * tests: cover import-time helper reads and keep the guard py3.9-safe Follows up on the Codex review: - add `from __future__ import annotations`, since `str | None` in `_offender` is evaluated at import on Python 3.9 and pyproject declares requires-python ">=3.9,<3.15". - widen the guard from module scope to import time. Class bodies and the bodies of module-level helpers called from an executing statement run during collection too, so `CODE = _extract_mixed_precision_code()` was the same hazard as an inline read. `if __name__ == "__main__":` blocks are skipped: pytest never executes them. - scan studio/backend/tests/ as well as tests/. Both trees are collected on Windows by separate CI jobs, and the offender that started this, test_tool_xml_strip.py reading routes/inference.py, lives there. Widening it surfaced seven more import-time reads of checked-in sources; all now name utf-8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden the import-time encoding guard for PR #7438 Close the detector gaps raised in review, all of which I reproduced against the actual AST before changing anything. False negatives (the guard let a real hazard through): - _is_main_guard ignored the comparison operator, so if __name__ != "__main__" counted as script-only even though its body runs at import. - The else arm of a main guard was discarded with the rest of the If node. - Decorators and argument defaults on a module-level def were skipped with the body, though both are evaluated when the def executes. - Path.open() in text mode was invisible; only builtin open() was matched. - encoding = None and encoding = "locale" both re-select the platform default, but the keyword merely being present counted as pinned. False positives (the guard would have blocked a compliant contributor): - A non-literal mode fell through to the "r" default, so open(p, mode) was flagged even when mode is "rb", where adding encoding= is a ValueError and there is no edit that satisfies the rule. - Same for open(*args) and a **kwargs splat, which hide the mode and can hide an encoding. - Lambda bodies and comprehension elements were walked even though neither runs at definition. Verified: still reports the same 22 offenders on unpatched main, green on this branch and on the tree merged with latest main (557 files), and an adversarial corpus of 33 cases now scores zero false positives and zero false negatives. Also corrected two docstring claims: neither collecting job runs on Windows, and the read is governed by locale.getencoding(). * Walk eager comprehensions and treat io.open as the builtin Two regressions from the previous commit, both reproduced against the AST before changing anything. Lumping list, set and dict comprehensions in with generator expressions was wrong. Only a genexp is lazy; the other three run their element expression, their filters and their nested iterators immediately, so CONTENTS = [p.read_text() for p in PATHS] at module scope is an import-time read the guard was silently missing. Comprehensions are now walked in full and only the genexp keeps the outermost-iterable-only treatment. io was also in the not-a-path-opener list, but io.open is the builtin, with the same mode position and the same platform default. io.open(CHECKED_IN_FILE) is exactly the hazard this guard exists for, so it is matched now, with binary modes and a pinned encoding still exempt. tarfile.open and fitz.open stay exempt since neither has an encoding to name. Verified: 13 targeted cases covering all five eager comprehension forms and io.open in text, binary and pinned shapes all classify correctly; still 22 offenders on unpatched main; green on this branch and on the tree merged with latest main. * Close three more walker gaps in the import-time guard All three reproduced against the AST first. A generator expression handed straight to a call is consumed there, so DATA = "".join(p.read_text() for p in paths) runs its element at import. Only an unconsumed genexp bound to a name stays lazy, so the walker now follows the consumed ones in full and keeps the outermost-iterable-only treatment for the rest. if "__main__" == __name__ is an equivalent and accepted spelling of the main guard, but requiring __name__ on the left meant its body was treated as import-time code. That is a false positive on a block pytest never runs, so both operand orders are recognised now. The helper table was built from module-level defs only, so a def in a class body invoked while the class is constructed was never followed, contradicting the walker's stated coverage of class bodies. Helpers are now collected from the module body and from class bodies at any nesting. Verified: 15 targeted cases including all three fixes and the earlier ones still classify correctly; still 22 offenders on unpatched main; green on this branch and on the tree merged with latest main. * Handle positional read_text encodings, lazy generators and nested helpers * Guard reads reached from test bodies, unbound Path calls and __file__ paths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow derived paths, skip lazy generator helpers, cover compressed openers * Guard the CLI tests, helper parameters and unbound Path arguments * Discover test roots and follow literal, in-place and tuple-derived paths * Identify module openers by import, unwrap starred paths, pin subprocess snippets * Resolve import origins, seed helper locals, follow named generators and parametrize * Scope imports lexically, list tracked test files, bind unpacked names * Resolve aliased openers, keyword-only params, destructured targets, next() * Pin the encoding on subprocess snippets, workflow lint and CLI output for PR #7438 * Harden the CLI encoding guard against detached streams for PR #7438 * Tighten the encoding guard's path and scope analysis for PR #7438 * Resolve path provenance more precisely and keep POSIX stream encodings for PR #7438 * Resolve qualified path classes and scope conditional imports for PR #7438 * Scope CLI stream setup to the entry point and align two encoding pairs for PR #7438 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>
199 lines
8.2 KiB
Python
199 lines
8.2 KiB
Python
"""RTL bidi contract on chat composers: all three need dir="auto", and the IME
|
|
smoke must drop the dead STUDIO_OLD_PW env var."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import re
|
|
from pathlib import Path
|
|
|
|
REPO = Path(__file__).resolve().parents[2]
|
|
THREAD_TSX = REPO / "studio/frontend/src/components/assistant-ui/thread.tsx"
|
|
SHARED_TSX = REPO / "studio/frontend/src/features/chat/shared-composer.tsx"
|
|
WORKFLOW_YML = REPO / ".github/workflows/studio-ui-smoke.yml"
|
|
IME_PY = REPO / "tests/studio/playwright_chat_ime_i18n.py"
|
|
|
|
|
|
def _block_around(
|
|
src: str,
|
|
anchor: str,
|
|
radius: int = 600,
|
|
) -> str:
|
|
idx = src.find(anchor)
|
|
assert idx != -1, f"anchor {anchor!r} not found"
|
|
return src[max(idx - radius, 0) : idx + radius]
|
|
|
|
|
|
def test_main_composer_has_dir_auto():
|
|
# PR #5784 turned the attribute into a JSX conditional; anchor on the inner
|
|
# "Message input" literal, which survives both spellings.
|
|
block = _block_around(THREAD_TSX.read_text(encoding = "utf-8"), '"Message input"')
|
|
assert 'dir="auto"' in block, 'main composer is missing dir="auto"'
|
|
|
|
|
|
def test_edit_composer_has_dir_auto():
|
|
block = _block_around(THREAD_TSX.read_text(encoding = "utf-8"), "aui-edit-composer-input")
|
|
assert 'dir="auto"' in block, 'edit composer is missing dir="auto"'
|
|
|
|
|
|
def test_compare_composer_has_dir_auto():
|
|
block = _block_around(SHARED_TSX.read_text(encoding = "utf-8"), "Send to both models")
|
|
assert 'dir="auto"' in block, 'compare composer is missing dir="auto"'
|
|
|
|
|
|
def test_ime_workflow_step_does_not_set_studio_old_pw():
|
|
yml = WORKFLOW_YML.read_text(encoding = "utf-8")
|
|
drive_idx = yml.find("Drive IME + multilingual paste regression")
|
|
assert drive_idx != -1, "IME drive step not found in workflow"
|
|
next_step_idx = yml.find("- name:", drive_idx + 1)
|
|
drive_block = yml[drive_idx : next_step_idx if next_step_idx != -1 else None]
|
|
assert (
|
|
"STUDIO_OLD_PW" not in drive_block
|
|
), "IME drive step still passes dead STUDIO_OLD_PW env var"
|
|
assert "STUDIO_NEW_PW" in drive_block, "IME drive step missing STUDIO_NEW_PW"
|
|
|
|
|
|
def test_ime_pass_password_step_does_not_export_old_pw():
|
|
yml = WORKFLOW_YML.read_text(encoding = "utf-8")
|
|
pass_idx = yml.find("Pass bootstrap pw for IME / i18n test")
|
|
assert pass_idx != -1, "IME password setup step not found"
|
|
next_step_idx = yml.find("- name:", pass_idx + 1)
|
|
pass_block = yml[pass_idx : next_step_idx if next_step_idx != -1 else None]
|
|
assert (
|
|
"STUDIO_IME_OLD_PW" not in pass_block
|
|
), "IME password setup still exports dead STUDIO_IME_OLD_PW"
|
|
assert "STUDIO_IME_NEW_PW" in pass_block
|
|
|
|
|
|
def test_ime_playwright_script_does_not_read_studio_old_pw():
|
|
src = IME_PY.read_text(encoding = "utf-8")
|
|
code_only = re.sub(r'""".*?"""', "", src, flags = re.DOTALL)
|
|
assert (
|
|
"STUDIO_OLD_PW" not in code_only
|
|
), "IME Playwright script still references dead STUDIO_OLD_PW env var"
|
|
assert 'os.environ["STUDIO_NEW_PW"]' in code_only
|
|
|
|
|
|
def test_main_composer_has_stuck_compositionend_watchdog():
|
|
"""Issue #5546: WSL Chrome never emits compositionend after IME commit, so the
|
|
composer needs a watchdog releasing the composing flag or Send stays disabled."""
|
|
src = THREAD_TSX.read_text(encoding = "utf-8")
|
|
assert (
|
|
"IME_STUCK_TIMEOUT_MS" in src
|
|
), "main composer is missing the stuck-compositionend watchdog (issue #5546)"
|
|
assert "onCompositionUpdate" in src, (
|
|
"main composer is missing onCompositionUpdate wiring; the "
|
|
"watchdog only resets while the IME is actively emitting events"
|
|
)
|
|
|
|
|
|
def test_compare_composer_has_stuck_compositionend_watchdog():
|
|
src = SHARED_TSX.read_text(encoding = "utf-8")
|
|
assert (
|
|
"IME_STUCK_TIMEOUT_MS" in src
|
|
), "compare composer is missing the stuck-compositionend watchdog (issue #5546)"
|
|
assert "onCompositionUpdate" in src, "compare composer is missing onCompositionUpdate wiring"
|
|
|
|
|
|
def test_main_composer_keydown_repins_composing_during_ime():
|
|
"""Issue #5546: the keydown IME gate must re-pin composingRef so a follow-up
|
|
Enter does not submit preedit text after the watchdog clears it."""
|
|
src = THREAD_TSX.read_text(encoding = "utf-8")
|
|
assert "onKeyDown" in src, "main composer is missing onKeyDown IME gate"
|
|
assert "e.nativeEvent.isComposing" in src and "keyCode === 229" in src, (
|
|
"main composer keydown gate must check both nativeEvent.isComposing "
|
|
"and the IME keyCode 229 sentinel"
|
|
)
|
|
|
|
|
|
def test_compare_composer_keydown_repins_composing_during_ime():
|
|
"""Compare composer onKeyDown re-pins composingRef on IME keypress so a
|
|
follow-up click-Send during the watchdog window does not slip preedit text."""
|
|
src = SHARED_TSX.read_text(encoding = "utf-8")
|
|
assert "composingRef.current = true" in src, (
|
|
"compare composer keydown gate must re-pin composingRef when the "
|
|
"browser still considers the IME active"
|
|
)
|
|
|
|
|
|
def _extract_block(
|
|
src: str,
|
|
anchor: str,
|
|
opener: str = "(",
|
|
closer: str = ")",
|
|
) -> str:
|
|
"""Source within the first balanced opener/closer after `anchor`, scoping
|
|
assertions to one handler."""
|
|
start = src.find(anchor)
|
|
assert start != -1, f"anchor {anchor!r} not found"
|
|
open_idx = src.find(opener, start)
|
|
assert open_idx != -1, f"opener {opener!r} after {anchor!r} not found"
|
|
depth = 0
|
|
for i in range(open_idx, len(src)):
|
|
c = src[i]
|
|
if c == opener:
|
|
depth += 1
|
|
elif c == closer:
|
|
depth -= 1
|
|
if depth == 0:
|
|
return src[start : i + 1]
|
|
raise AssertionError(f"unbalanced {opener!r}/{closer!r} after {anchor!r}")
|
|
|
|
|
|
def test_main_composer_keydown_rearms_watchdog():
|
|
"""After keydown re-pins composingRef the watchdog must re-arm, else the
|
|
WSL+Chrome no-compositionend path locks Send after any IME keypress (#5546)."""
|
|
src = THREAD_TSX.read_text(encoding = "utf-8")
|
|
block = _extract_block(src, "const onKeyDown = useCallback")
|
|
assert "refreshStuckTimer" in block, (
|
|
"main composer keydown gate must call refreshStuckTimer after "
|
|
"re-pinning composingRef so the watchdog runs again on the "
|
|
"stuck-compositionend path"
|
|
)
|
|
assert "clearStuckTimer();" not in block.replace("clearStuckTimer\n", "").replace(
|
|
"clearStuckTimer,", ""
|
|
), (
|
|
"main composer keydown gate must not leave the watchdog only "
|
|
"cleared; that would regress the stuck-compositionend path"
|
|
)
|
|
|
|
|
|
def test_compare_composer_keydown_rearms_watchdog():
|
|
"""Same re-arm contract for the compare-mode composer."""
|
|
src = SHARED_TSX.read_text(encoding = "utf-8")
|
|
block = _extract_block(src, "function onKeyDown", opener = "{", closer = "}")
|
|
assert (
|
|
"refreshStuckImeTimer" in block
|
|
), "compare composer keydown gate must call refreshStuckImeTimer after re-pinning composingRef"
|
|
|
|
|
|
def _assert_enter_guard_before_immediate_recovery(block: str, refresh_call: str) -> None:
|
|
enter_idx = block.find('e.key === "Enter"')
|
|
recovery_idx = block.find("setCompositionState(false)")
|
|
assert enter_idx != -1, "keydown handler is missing an Enter guard"
|
|
assert recovery_idx != -1, "keydown handler is missing immediate recovery"
|
|
assert enter_idx < recovery_idx, (
|
|
"stuck-composition recovery must guard Enter before clearing "
|
|
"composingRef; candidate-confirming Enter must not submit"
|
|
)
|
|
guard_block = block[enter_idx:recovery_idx]
|
|
assert (
|
|
"preventDefault()" in guard_block
|
|
), "Enter while composingRef is stuck must prevent the same key from falling through to submit"
|
|
assert (
|
|
refresh_call in guard_block
|
|
), "Enter while composingRef is stuck must keep the watchdog armed"
|
|
assert (
|
|
"return;" in guard_block
|
|
), "Enter while composingRef is stuck must not reach immediate recovery"
|
|
|
|
|
|
def test_main_composer_stuck_enter_does_not_clear_before_submit():
|
|
src = THREAD_TSX.read_text(encoding = "utf-8")
|
|
block = _extract_block(src, "const onKeyDown = useCallback")
|
|
_assert_enter_guard_before_immediate_recovery(block, "refreshStuckTimer")
|
|
|
|
|
|
def test_compare_composer_stuck_enter_does_not_clear_before_submit():
|
|
src = SHARED_TSX.read_text(encoding = "utf-8")
|
|
block = _extract_block(src, "function onKeyDown", opener = "{", closer = "}")
|
|
_assert_enter_guard_before_immediate_recovery(block, "refreshStuckImeTimer")
|