CI(ui): STUDIO_UI_STRICT mode + theme cycle fix + Recents thread-match assertion

The existing UI test was passing too easily: every "if button.count() == 0:
log WARN" branch silently degraded into a green run. Three places this
hid real bugs:

  1. The theme toggle for-loop bailed after cycle 1 because the Radix
     Account-menu's data-state="open" lingered through the view-transition
     and the next acct.click() hit the still-open dropdown. The test
     went green observing only one polarity.
  2. The regenerate button branch silently skipped when the assistant
     action bar didn't render (every CI run so far -- the locator was
     wrong, but no one noticed because it was a soft skip).
  3. The Recents click accepted ANY non-nav sidebar entry, so a freshly
     deleted thread or an unrelated entry would still pass.

Fixes:

  - Add STUDIO_UI_STRICT=1 env (default on in CI via workflow,
    default off locally). When on, every soft "if not visible: log
    WARN" branch hard-fails. The strict-skip pattern is centralised
    in a soft_fail() helper so the local-vs-CI split is one knob.
  - Theme toggle: wait for [role="menu"] to detach between cycles
    (the dropdown stay-open was the cycle-2 bail), assert the loop
    actually ran 3 times.
  - Model picker search: capture popover text after typing "qwen" vs
    "llama"; the two snapshots must DIFFER, proving the typeahead
    actually filters (a regression that rendered the picker but
    ignored input would silently pass before).
  - Recents click: after navigating to the clicked thread, the
    rendered turns must include at least one of our sent prompts
    ("hello", "world", "tree", "1+1", etc.) -- proves we landed on
    OUR thread, not a leftover from a previous run.
  - Use [data-tour="chat-model-selector"] as the primary selector
    for the model picker -- the guided-tour anchor is at least as
    stable as anything else in the codebase (the tour breaks if it
    moves), and there's no separate data-testid system to maintain.
This commit is contained in:
Daniel Han 2026-05-07 04:00:35 +00:00
commit 0e9888645f
2 changed files with 122 additions and 24 deletions

View file

@ -154,6 +154,10 @@ jobs:
# against a freshly-installed Studio (BASE_URL=...; STUDIO_OLD_PW=
# $(cat ~/.unsloth/studio/auth/.bootstrap_password); python ...).
PW_ART_DIR: logs/playwright
# Strict mode: in CI a missing button / nav / dialog must
# FAIL the test. Locally the test still runs against partial
# Studio installs without STUDIO_UI_STRICT.
STUDIO_UI_STRICT: '1'
run: |
mkdir -p logs/playwright
python tests/studio/playwright_chat_ui.py