15 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Daniel Han
|
8848a310df |
Studio: clean-room compact RAG (knowledge bases, hybrid search, fast indexing) (#5910)
Adds a self-contained RAG stack to Studio: knowledge bases with chunked indexing, hybrid (dense + lexical) retrieval, and an automatic first-pass context inject into chat. Embeddings run through a local llama-server GGUF backend (default unsloth/bge-small-en-v1.5-GGUF) with a sentence-transformers fallback. The chat tool loop gains a search_knowledge_base tool, a per-turn re-search cap, and source citation, layered on top of the shared ToolLoopController. |
||
|
Nilay
|
9a907a8acb |
Studio: add remote MCP server support (#5750)
* added remote MCP server support * trim * added tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * increased timeout * disabling MCP chat toggle * Fix MCP OpenAI function-name validation + cancel propagation for PR #5750 OpenAI requires function.name to match ^[a-zA-Z0-9_-]{1,64}$ before streaming starts. The existing 64-char length check is necessary but not sufficient: MCP servers can return tool names containing '.', '/', spaces, etc. that would 400 the whole chat request. Validate the composed mcp__<server_id>__<tool> name against the regex, skip + warn on miss, and drop duplicate tool names from the same server (which would also 400 the request as "duplicates"). Also propagate the agentic-loop cancel_event into MCP tool execution so a /cancel POST during a long-running MCP call (e.g. GitHub MCP search across a large repo) actually interrupts the in-flight HTTP call instead of waiting out the 300 s timeout. The watcher polls the threading.Event at 50 ms cadence inside the asyncio loop (matches routes/inference.py's existing cancel-watcher cadence) and races against the call task with asyncio.wait FIRST_COMPLETED. Tests added: - test_mcp_specs_skip_invalid_openai_function_names: drops bad chars - test_mcp_specs_skip_empty_tool_name - test_mcp_specs_drops_duplicate_names - test_call_tool_sync_respects_pre_set_cancel_event Also fix test_desktop_auth.py's router stub that listed every existing router but missed mcp_servers_router, so importing main.py fails after this PR adds it to routes/__init__.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * PR #5750 round 2: OAuth cleanup on delete/url-change + mcp_enabled standalone Round 2 of cross-platform validation surfaced two more P1 findings: 1. OAuth tokens never get cleared. fastmcp keys tokens by MCP URL, not by server row, and delete / URL change / use_oauth toggle only updated the SQLite row. Re-registering the same URL would silently reuse the old account's credentials. Adds clear_oauth_tokens_async() in mcp_client.py and calls it from the delete + put route handlers when the row had use_oauth=True and either the URL changes or OAuth is turned off. 2. mcp_enabled=true was ignored unless the caller also sent enable_tools=true. The frontend always sends both together so the UI path was fine, but a direct API caller sending only mcp_enabled would silently get no MCP tools, which contradicts the field's documented "append tools from every enabled MCP server" behavior. Loosens the use_tools gate in both the GGUF and safetensors paths so mcp_enabled opens the tool loop on its own; when the caller did not also opt into built-ins, the built-in list starts empty. Tests added: - test_clear_oauth_tokens_async_no_op_safe - test_delete_server_calls_oauth_cleanup_when_oauth_was_on - test_delete_server_skips_oauth_cleanup_when_oauth_off - test_update_server_clears_oauth_on_url_change - test_update_server_clears_oauth_when_oauth_disabled 26 backend MCP tests pass; full studio/backend suite 1710 passed locally. Cross-platform CI (Linux, macOS, Windows) green on staging fork. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * PR #5750 round 3: reject null bool updates + /test surfaces 400 Round 3 of cross-platform validation: 1. PUT /api/mcp/servers/<id> would 500 with TypeError when the body explicitly set is_enabled or use_oauth to null. Pydantic accepts None for an Optional[bool] and _changes_from_payload then passed None into mcp_servers_db.update_server, which int(None)d. Reject explicit null at the validation layer with 400 instead. 2. POST /api/mcp/servers/test caught HTTPException under "except Exception", so an invalid URL came back as HTTP 200 with {"ok": false, "error": "400: ..."} instead of a real 400. The create + update paths return 400 for the same input. Move validation outside the transport try/except so it surfaces 400. Tests added: - test_changes_from_payload_rejects_null_is_enabled - test_changes_from_payload_rejects_null_use_oauth - test_test_endpoint_surfaces_url_validation_as_400 * PR #5750 round 4: hyphenated MCP tool names + empty-tool-list gate Round 4 surfaces two more interaction bugs between the new MCP path and existing safetensors tool plumbing: 1. OpenAI accepts ^[a-zA-Z0-9_-]{1,64}$ for function.name, and round 1 widened the MCP regex to that set, so MCP tools can now be advertised as `mcp__srv__list-issues`. But the XML tool-call parser in tool_call_parser.py used `\w+` (no hyphen), so the model could call the tool but Studio could not parse the call. Same in routes/inference.py's `_TOOL_XML_RE` stripper, which would leave hyphenated tool-call XML in the visible content. Both regexes now use `[\w-]+`. 2. safetensors_agentic treats `tools=[]` as "allow all" (documented contract, exercised by test_empty_tools_list_does_not_enforce_allowlist). When a caller sends `enable_tools=true` + `enabled_tools=[]` + `mcp_enabled=true` and MCP discovery returns 0, the resolved tool list is genuinely empty and built-in tools (web_search / python / terminal) could execute via the model's emitted call. Fix at the route gate instead of breaking the documented contract: set `use_tools=False` when the resolved list is empty, in both GGUF and safetensors paths. Existing callers who omit `enabled_tools` still get ALL_TOOLS and are unaffected. Tests added (32 total): - test_tool_xml_parser_handles_hyphenated_function_names - test_tool_xml_strip_handles_hyphenated_function_names - test_safetensors_agentic_empty_allowlist_still_means_allow_all (documents the contract round 4 preserved) 1716 passed locally; cross-platform CI on staging fork still green. * PR #5750 round 5: GGUF allow-list + CLI policy + hyphenated params + cancel race Round 5 of parallel-reviewer aggregation surfaced six additional findings; five are real and fixed here: 1. Hyphenated MCP parameter names (`<parameter=issue-number>`) were dropped by the XML parser's `\w+` regex. Extended to `[\w-]+` in both core/inference/tool_call_parser.py and core/tool_healing.py. The latter is GGUF's own copy of the parser/strip patterns and was missed by round 4. 2. core/tool_healing.py's `strip_tool_call_markup` still used `<function=\w+>` so hyphenated MCP tool-call XML leaked into the GGUF visible content even after round 4 fixed the shared parser. 3+4. `mcp_enabled` re-opened the tool loop even when the operator passed `unsloth run --disable-tools` (CLI policy False). Round 2's `(_tools_on or payload.mcp_enabled)` gate ignored the raw process policy. Now reads `state.tool_policy.get_tool_policy()` and gates mcp_enabled on `_cli_policy is not False`. Applied to both GGUF and safetensors paths. 5. GGUF's agentic loop called `execute_tool(tool_name, ...)` without checking the model-emitted name against the per-request tool list, while the safetensors loop already enforces this. Added the same allow-list check so a model that hallucinates a filtered MCP name or a built-in the caller opted out of returns "not enabled" instead of executing. Bonus P2 fixes: - `call_tool_sync` now checks `cancel_event.is_set()` BEFORE creating the call task, so a pre-set cancellation does not open the HTTP transport. - `clear_oauth_tokens_async` moved the OAuth import + construction inside the protected try block; a fastmcp.client.auth load error used to escape and 500 the delete / update route. NOT fixed (verified false or out of scope): - finding #10 "structured_content vs structuredContent": fastmcp's CallToolResult dataclass uses snake_case (verified live against structured-only tool result; fields are `dict_keys(['content', 'structured_content', 'meta', 'data', 'is_error'])`). - finding #11 "asyncio.run from running loop": call_tool_sync is invoked from `asyncio.to_thread` worker threads which have no event loop; asyncio.run() is safe there. Tests added (37 total): hyphenated param names, tool_healing strip, GGUF allow-list gate, cancel pre-set short-circuit, OAuth cleanup constructor-error swallowing. 1721 passed locally, no regressions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com> |
||
|
Lee Jackson
|
61ed4cac51 |
Studio: persist chat history in backend storage (#5272)
* feat: Persist chat history in backend storage * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address chat tombstone batching review * fix: update desktop auth routes stub * chat db settings storage * chat db settings routes * chat db settings client * chat db settings store * chat db settings wiring * chat db history storage * chat db settings migration * chat db settings fallback * chat db container metadata * chat db legacy migration fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * chat ci auth background reads * chat auth storage fixes * chat migration final fixes * chat export batch message lookup * chat history review fixes * chat prune sync fix * chat settings hydration retry * gate settings persistence * Scope chat-history rows by subject; fix hijack, clear-confirm, hydrate race Backend storage and routes: - chat_threads / chat_messages / chat_settings carry a NOT NULL subject column with composite PRIMARY KEY (id, subject). Two authenticated identities can no longer see or wipe each other's data. - Pre-existing rows on an existing studio.db migrate under sentinel subject __legacy_unscoped__ via rename + rebuild + copy; single-user installs see no behavior change. - ON CONFLICT(id, subject) DO UPDATE ... WHERE chat_messages.thread_id = excluded.thread_id refuses cross-thread re-parenting via upsert. upsert_chat_message + sync_chat_messages now raise ChatMessageThreadMismatch which the routes map to HTTP 409. - replace_thread_messages rejects body messages whose threadId does not match the URL thread (HTTP 400) instead of silently rewriting them. - DELETE /api/chat requires ?confirm=true, returns row count, logs the subject and count. - upsert_chat_settings_merge does read + deep-merge + write inside a single BEGIN IMMEDIATE so concurrent writers no longer drop each other's updates. The route delegates to this helper. - New POST /api/chat/messages:batch returns {thread_id -> messages[]} for many threads in one HTTP call. Subject-scoped. Unknown ids return empty lists instead of 404 so the sidebar/search caller can rebuild atomically. Frontend: - chat-runtime-store: hydrate-failure catch sets settingsHydrated:true so a transient backend blip no longer permanently disables persistence. setParams bumps inferenceParamMutationVersions unconditionally so a slow hydration response cannot clobber a pre-hydrate user edit. saveSettingsPatch replaces the serial chain with a debounced pendingPatch + deep merge; flush on beforeunload. - chat-history-storage: clearStoredChats returns ClearStoredChatsResult distinguishing backend / legacy / both outcomes. listStoredChatThreadsWithMessages uses the batched fetch (one HTTP call) instead of Promise.all per-thread; legacy Dexie fallback only fires when the batch result is empty. - chat-api: batchListChatMessages with graceful 404 / 405 fallback to per-thread listChatMessages for older servers. - chat-thread-tombstones: store {id, deletedAt} tuples with 90-day GC and a 5000-entry cap so localStorage stays bounded. Back-compat reads pre-fix plain strings. Adds removeChatThreadTombstones (rollback) and clearAllChatThreadTombstones (post-legacy-purge clean-up). - use-chat-sidebar-items: deleteChatItem tombstones synchronously BEFORE the backend round-trip and rolls back on failure (restores pre-PR optimistic UX). 300 ms trailing debounce on CHAT_HISTORY_UPDATED_EVENT plus requestSeq guard so stream-time event bursts produce at most one fetch per quiet window. Tests: - studio/backend/tests/pr5272_sim/ adds 64 regression tests covering schema migration from pre-fix shape, subject scoping, cross-thread hijack, bulk-replace mismatch, clear-confirm, concurrent settings, unicode + 2MB content + SQL-injection-safe binding, chunking boundary at 900 and 901 ids, batched endpoint (multi-subject + 1200 ids + per-thread order), and grep contracts for the frontend patches. test_chat_history_storage.py updated to pass subject. Verified locally on Linux + macOS + Windows GitHub Actions runners (staging fork): 64 pass + 2 from the PR's own backend test on all three OSes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop subject scoping and clear-confirm gate (Studio is single-user) Per maintainer feedback: subject scoping, cross-thread message hijack guard, and DELETE /api/chat ?confirm=true gate are unnecessary because Studio is intentionally single-user (the client already shows a confirm dialog before clear-all). This commit reverts those backend changes and keeps only the non-multi-user pieces from the earlier fix commit: - studio_db.py: restored to pre-fix shape; adds upsert_chat_settings_merge which does atomic read + deep-merge + write under BEGIN IMMEDIATE so two concurrent slider drags cannot drop one another's updates. - routes/chat_history.py: restored; put_settings now calls the atomic merge instead of doing the read-merge-write across three separate connections. Adds POST /api/chat/messages:batch to collapse the sidebar/search rebuild from N round-trips to 1. - frontend/api/chat-api.ts: align batchListChatMessages request and response keys with the backend (threadIds / messagesByThreadId). - tests/test_chat_history_storage.py: add atomic-merge concurrency test, deep-merge nested-key test, and 901-id chunking-boundary test. - Drop the pr5272_sim test directory (those tests covered the reverted subject-scoping/hijack/confirm behavior). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix sidebar delete crash, keepalive on settings beforeunload flush, search rebuild race Two correctness bugs and one perf race surfaced by a fresh code review of the prior fix commit: - chat-api.ts: notifyChatHistoryUpdated was declared as a non-exported function, but use-chat-sidebar-items.ts imports it. The import would fail tsc with TS2305 and at runtime the optimistic-delete and delete-failure rollback paths would both throw. - chat-runtime-store.ts + chat-settings-api.ts + chat-settings-storage.ts: the beforeunload settings flush is now actually keepalive. Without it the browser cancels the in-flight PUT on tab close, so the last slider drag is silently dropped (which is exactly the case the debounce+beforeunload combination was meant to protect against). - use-chat-search-index.ts: rebuilds now coalesce with a 300ms trailing debounce and discard out-of-order responses via a requestSeq guard. Matches the sibling pattern in use-chat-sidebar-items.ts so two rapid CHAT_HISTORY_UPDATED_EVENTs (run-start + run-end save during a turn) cannot land with stale data winning. - chat-thread-tombstones.ts: drop dead clearAllChatThreadTombstones with no call sites; Dexie is never wiped so the function has no use. * fix(studio): protect chat persistence writes * fix(studio): align chat history clear semantics * fix(studio): show partial chat clear feedback * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): preserve chat persistence fallbacks * fix(studio): harden chat thread persistence checks * Preserve chat message timestamps * Gate chat stream on history save * Make chat thread backfill best effort * Avoid chat message 404 probe * Tighten chat legacy fallbacks * chat: server-side ledger so legacy Dexie import is recoverable The boolean localStorage sentinel (unsloth_chat_legacy_imported_to_studio_db) made importLegacyChatsIfNeeded non-recoverable: deleting studio.db while the browser keeps the flag silently hides every legacy Dexie thread from the sidebar (verified by the 3-GPU validation probe; matches the third review comment on PR #5272). Same trap fires for browser-profile sync to a fresh machine and any other path that wipes studio.db while keeping IndexedDB. Source of truth moves into studio.db itself via a new chat_legacy_import_log table keyed by legacy thread id. The ledger disappears together with studio.db, so the next launch re-runs the import from whatever Dexie still holds. localStorage stays as a per-session perf hint only. Performance, all bounded by the three new fast-paths before any backend work: A) localStorage hint says "imported earlier in this session" -- 0 network, ~0 ms. Covers the warm sidebar mount. B) indexedDB.databases() reports no "unsloth-chat" DB -- 0 network, ~1 ms. Covers every new user who never had the old browser-only Studio (the common case after launch). C) db.threads.count() + db.messages.count() are both 0 -- 0 network, ~5 ms. Covers returning users who migrated long ago and Dexie was never repopulated. Only when all three miss does the code talk to the backend (GET /api/chat/import-ledger -> diff vs Dexie -> existing import path -> POST /api/chat/import-ledger to record what was just imported). Per-thread tracking is enough because Dexie is read-only after this PR; a thread's message set does not grow. Backend deployments that predate the import-ledger routes are handled transparently: the client treats 404/405 as an empty ledger and re-runs the (idempotent via UPSERT) import on next launch. Changes: - storage/studio_db.py: new chat_legacy_import_log table (WITHOUT ROWID, PK on legacy_thread_id) + list_chat_legacy_import_log() + record_chat_legacy_import_log() (idempotent batch UPSERT). - routes/chat_history.py: GET + POST /api/chat/import-ledger with the obvious request/response models. - frontend api/chat-api.ts: listChatImportLedger() (returns a Set for O(1) diff) + recordChatImportLedger(), both with 404/405 fallback. - frontend utils/chat-history-storage.ts: importLegacyChatsIfNeeded gains three fast-paths, ledger fetch on the slow path, and writes the ledger after a successful import. The localStorage helper is unchanged on the surface; it just stops being authoritative. - tests: 5 new test_legacy_import_log_* cases (empty default, record + list round-trip, idempotency, input dedup, empty/null ignore). All 9 pre-existing tests still pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the legacy-import recovery actually recoverable The previous commit added a server-side ledger to make Dexie -> studio.db import recoverable after a studio.db wipe, but the localStorage perf hint still short-circuited the import gate before the ledger was ever consulted. After a wipe, the hint stayed "true" and the bulk re-import never ran -- the ledger sat empty and only the per-thread lazy materialize-on-continue path restored data. Changes: - Remove the localStorage short-circuit from importLegacyChatsIfNeeded so the ledger is checked on every fresh tab. legacyChatImportPromise keeps the per-session cache; the hint now only matters for the listing paths. - Batch the slow path: one db.messages.where().anyOf().toArray() and one batchListChatMessages() instead of 2N round-trips. At 1k threads this drops a multi-second blocking import to a single request pair. - recordChatImportLedger returns {accepted, inserted, supported}. The localStorage hint is only flipped when supported is true, so old backends (404 / 405 / 501) no longer permanently poison recovery. - Ledger backfill: threads already present in chat_threads but missing from the ledger now get added too, so old-FE-then-new-FE deployments don't redo the diff every launch. - Backend response field renamed recorded -> {accepted, inserted}. accepted is the deduped non-empty input count; inserted is the rows actually new (via INSERT ... RETURNING). Bounded by Field(max_length= 10_000) on the request payload. - Storage helpers renamed: chat_legacy_import_log -> chat_legacy_imports, record_* -> upsert_* to match the existing noun/verb conventions. - DEXIE_DB_NAME exported from db.ts; duplicate constant in chat-history-storage.ts removed. - 3 new route-level tests for /api/chat/import-ledger covering the round-trip, the (accepted, inserted) split, and the 10k payload cap. All 18 chat-history tests pass. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: shine1i <wasimysdev@gmail.com> Co-authored-by: danielhanchen <michaelhan2050@gmail.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> |
||
|
Roland Tannous
|
9a0d6f80cb |
studio: API external provider support for chat (OpenAI, Mistral, Gemini, Cohere, Anthropic, OpenRouter, DeepSeek, custom providers) (#4706)
* studio: add external provider support for chat inference Adds the ability to connect to OpenAI, Mistral, Google, Cohere, Together, Fireworks, and Perplexity from the Studio chat interface. - Provider configs stored in SQLite (no API keys persisted) - RSA-2048 key pair generated at startup for client-side key encryption - httpx proxy client streams SSE responses in OpenAI-compatible format - New /api/providers routes: registry, CRUD, test, models - /v1/chat/completions routes to external provider when provider fields present - Integration test suite covering CRUD, connection, model listing, and inference - Frontend spec doc with full API contract * remove frontend spec doc from branch * fix auth fixture: handle forced password change on fresh install * fix tests: default port 8000, allow 400 for no-model-loaded * fix: update Cohere models to current (command-r retired Sept 2025) * feat: add OpenRouter as 8th provider * feat: add native Anthropic provider with Messages API translation * fix: correct Anthropic base URL and drop top_p (conflicts with temperature) * feat: add DeepSeek provider (deepseek-chat, deepseek-reasoner) * feat: rename google -> gemini, refresh model list to 2.5 series * feat: remove together, fireworks, perplexity providers * feat: multimodal image support for external providers - Add _build_external_messages() that preserves image_url parts for vision-capable providers instead of stripping them - Update _proxy_to_external_provider() to use new helper - Translate image_url content parts to Anthropic native image format in _stream_anthropic() - Add TestVisionInference pytest class (1x1 PNG smoke test) * test: use sloth photo URL for vision test, add Anthropic remote URL support * fix: update Mistral model to mistral-small-2506 * update mistral default model to mistral-large-2512 * fix gemini vision test: download image as base64 data URI instead of remote URL * add gemini-3-flash-preview as default gemini model * fix gemini truncated reply (max_tokens 16->64) and suppress GeneratorExit on client disconnect * increase vision test max_tokens to 215 * fix GeneratorExit: aclose stream generator before closing httpx client * fix httpcore GeneratorExit: explicitly aclose aiter_lines before response closes * fix duplicate [DONE] and suppress httpcore RuntimeError on Python 3.13 asyncgen cleanup * fix: call response.aclose() before lines_gen.aclose() to prevent httpcore RuntimeError on Python 3.13 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Potential fix for code scanning alert no. 36: Clear-text logging of sensitive information Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * review: add comments for manual iteration rationale, mask password in test print, clarify Anthropic URL/models support * perf: use shared module-level httpx client for connection pooling across requests * studio: add API provider UI and integrate wiring (#4737) * feat: expose external models in selector and chat settings * feat(chat): wire external providers to backend + RSA key flow - Fetch registry/configs; create/update/delete saved providers - Encrypt API keys (Web Crypto RSA-OAEP) for test/models/chat - External model selection + chat payload (provider_id/type, external_model, encrypted key, optional base URL) - Local storage for keys + provider list; small UX/copy and guardrails * add missing providers-api.ts file by Imagineer99 * fix: address PR review comments — system prompt visibility, retry loop, test logging * feat(studio): encrypt external provider API keys at rest in localStorage API keys for external providers (OpenAI, Mistral, etc.) were stored as plaintext in localStorage, vulnerable to browser extensions and XSS. Add password-derived AES-256-GCM encryption: on login the user's password is used via PBKDF2 (100k iterations, SHA-256) to derive an in-memory encryption key. API keys are encrypted before writing to localStorage and decrypted on read. The derived key is never persisted — cleared on logout, re-derived on next login. Legacy plaintext keys are transparently migrated on first access. Password changes re-encrypt all stored keys. No backend changes required — the existing RSA-OAEP transit encryption is unaffected. * fix: cast PBKDF2 salt to BufferSource for strict TypeScript lib types * fix: persist session password in sessionStorage to survive page refreshes * feat(studio): preserve image parts in external provider chat requests toOpenAIMessage() now returns multimodal content arrays (OpenAI vision format) when messages contain images, instead of always flattening to plain text. This enables vision-capable external providers (OpenAI, Gemini, Anthropic, etc.) to receive user images. The backend already handles image_url content parts in _build_external_messages(). * studio: fix external models selectable in chat-only mode (#4779) * fix: external models selectable in chat-only mode * fix: model selector tabs default to active model kind * Studio: API external provider registry + curated catalogs (HF/OpenRouter) and chat UX (#4787) * fix: external models selectable in chat-only mode * fix: model selector tabs default to active model kind * feat(studio): expand provider registry, curated catalogs, and chat UX - Add Hugging Face, Kimi, Qwen; remove Cohere; reorder registry - model_list_mode curated for HF/OpenRouter; lightweight /models check - API returns default models for curated providers; expose model_list_mode - Frontend: provider logos in model picker, providerType on external models - Chat providers dialog: curated vs remote flows, motion polish - Thread: LayoutGroup + composer motion alignment with app easing * fix(studio): disable Anthropic tool-calling flag and preselect curated defaults * feat(studio): add external provider logos and ApiProviderLogo helper * Studio: Polish API Providers dialog (#4899) * fix: lower verbage in API providers page * fix: fix(studio): tune API Providers dialog width with rem-based responsive caps * feat: add custom provider support (#4902) * fix: replace crypto.subtle with node-forge for HTTP compatibility crypto.subtle is only available in secure contexts (HTTPS/localhost), which breaks provider API key encryption when Studio is accessed over plain HTTP on remote GPU VMs. Switch to node-forge for RSA-OAEP and AES-256-GCM operations — same algorithms, works on any origin. * fix: store provider API keys as plaintext in localStorage Drop AES-256-GCM at-rest encryption for provider API keys. The session-password-derived encryption broke on auto-login via refresh token (password never captured), causing keys to silently vanish. API keys are still RSA-encrypted in transit via node-forge. At-rest encryption in localStorage added no real security since the decryption key also had to live client-side. Removes crypto-storage.ts, session password plumbing, and reEncryptAllKeys. * fix: use max_completion_tokens for OpenAI provider Newer OpenAI models (gpt-4o, gpt-5.x) reject the max_tokens param and require max_completion_tokens instead. Other providers still use max_tokens. * fix: skip empty assistant messages in external provider requests Some providers (Mistral) reject assistant messages with empty content. Filter them out when building the message list for external providers. * Update model-selector.tsx * Update model-selector.tsx * Update model-selector.tsx * Update chat-adapter.ts * Update chat-adapter.ts * Update chat-page.tsx * Update chat-settings-sheet.tsx * Update chat-settings-sheet.tsx * Update chat-settings-sheet.tsx * Update chat-providers-dialog.tsx * feat: polish providers settings form UI * style: polish provider row icon sizing and alignment * style: stabilize provider layout * style: add provider API key visibility toggle * fix: add provider render on empty list * studio/frontend: sync package-lock.json with package.json npm ci was failing because node-forge and @types/node-forge were declared in package.json but missing from the lockfile. Ran npm install to regenerate. * studio/backend: fix backend CI failures for providers router - test_desktop_auth: include providers_router in the routes stub so studio.backend.main imports cleanly under the monkeypatched module - test_providers_api: skip the whole module when STUDIO_TEST_PASSWORD is unset (it is an integration test against a live Studio server, same shape as the already-ignored test_studio_api.py) * studio/chat: drive ChatSettingsPanel from a per-provider capability map Replace the binary isExternalModel toggle in the sampling section with a provider-aware capability map. Each external provider type advertises which of top_k / min_p / repetition_penalty / presence_penalty its chat-completions API actually accepts, so the panel only renders the knobs that map onto the active provider's request body. Anthropic now exposes top_k; DeepSeek hides presence_penalty (deprecated in their docs); OpenRouter and custom providers continue to show every knob (OpenRouter drops unsupported server-side, custom assumes OpenAI-compat or a permissive vLLM/Ollama backend). Local models are unaffected — null capabilities means 'show everything'. chat-adapter.ts now forwards top_k / presence_penalty to the external proxy only when the active provider's capabilities permit it, so the request body matches what the UI shows. * studio/backend: forward top_k to Anthropic; filter OpenAI model list Two paired changes so the frontend capability map has matching backend behaviour: 1. ExternalProviderClient.stream_chat_completion now accepts top_k and forwards it to the Anthropic Messages body. OpenAI-compat providers (which all reject unknown sampling params) still receive only the fields they document. The proxy route in routes/inference.py passes payload.top_k through, so a UI request with top_k actually reaches Anthropic instead of being silently dropped at the boundary. 2. PROVIDER_REGISTRY['openai'] gains a model_id_allowlist regex that scopes the /models picker to current-gen ids (gpt-5.5 / gpt-5.4 / gpt-5.3 / gpt-4.5 / o3 families). The remote /v1/models listing otherwise returns dozens of historical snapshots, fine-tunes and non-chat models (embeddings, TTS, image, moderation) that we never want in the chat UI. default_models is refreshed to match. * studio/chat: relax presence_penalty to optional on OpenAIChatCompletionsRequest Followup to 1fbf445a — chat-adapter now omits presence_penalty for providers that do not accept it (Anthropic / DeepSeek), but the request type still required it as a non-optional number, breaking tsc. The backend pydantic model already defaults presence_penalty to 0, so making it optional client-side matches reality. * studio/backend: route OpenAI traffic through /v1/responses OpenAI's new flagship models (gpt-5.x) return 404 'This is not a chat model' on /v1/chat/completions and are only reachable via /v1/responses. Add a dedicated _stream_openai_responses path in ExternalProviderClient that: - Translates outbound messages into the Responses shape: system messages are folded into the top-level 'instructions' field, user/assistant messages become {role, content} items with input_text / input_image content parts (data URLs and https URLs both pass through). - Drops presence_penalty / top_k / frequency_penalty, none of which the Responses contract accepts. - Translates inbound SSE events back into OpenAI Chat Completions chunks so the frontend keeps a single SSE shape: response.output_text.delta -> delta chunk with content response.completed -> chunk with finish_reason='stop' response.incomplete -> chunk with finish_reason='length' response.failed / error -> propagated error SSE line Stream terminates with data: [DONE] (Responses emits this verbatim). stream_chat_completion dispatches all provider_type='openai' calls to this path; other OpenAI-compatible providers (mistral, gemini, etc.) continue to use /v1/chat/completions. Frontend provider-capabilities map updated to hide presence_penalty for OpenAI in the chat settings panel, matching the new request contract. Includes unit coverage in tests/test_openai_responses_translation.py exercising the request body translation, image-part rewriting, and SSE-to-chat-completions translation via httpx.MockTransport. * studio/chat: clamp external max_tokens to 32k to stay within provider caps The chat settings slider already capped maxTokens at 32768 for external models, but a value persisted from a prior local-model session (where the cap can be 128k+) was sent verbatim to the provider — Claude Opus returns 'max_tokens: 131072 > 128000' on requests like that, and other providers have stricter limits still. Expose EXTERNAL_MAX_OUTPUT_TOKENS from provider-capabilities (32k) and use it both for the slider max and as the clamp inside chat-adapter's external-request body. 32k sits below the tightest declared output limit across the providers we ship and well above what a typical chat reply needs; the local-model path is unaffected. * studio: drop temperature/top_p for OpenAI reasoning models gpt-5.x / o3 / gpt-4.5 are reasoning-class models served via /v1/responses, and reject temperature and top_p with 'Unsupported parameter' 400s. The OpenAI registry allowlist already scopes the picker to those families, so neither knob ever applies on this branch. - external_provider._stream_openai_responses no longer puts temperature or top_p in the request body (kept on the method signature for API symmetry with the other stream methods). - ProviderCapabilities gains temperature/topP flags; OpenAI sets both to false. ChatSettingsPanel hides the sliders for OpenAI so the user does not see inert controls. - chat-adapter omits temperature/top_p from the external request body when the active provider does not advertise them. - OpenAIChatCompletionsRequest type marks both as optional, matching the new chat-adapter shape. - test_responses_request_body_uses_input_and_instructions: assertions flipped to confirm temperature / top_p are absent from the body. * studio: stop forwarding top_k to Anthropic Claude 4.x (Opus / Sonnet / Haiku 4.x) returns 400 'top_k is deprecated for this model' on any request that includes top_k. It was always optional on the older 3.x line, so dropping it unconditionally for every Anthropic call is the simplest path — no per-model gate to maintain. - external_provider._stream_anthropic no longer adds top_k to the Messages body (kept on the method signature for API symmetry). - provider-capabilities sets anthropic.topK = false so the chat settings panel hides the Top K slider for Anthropic providers and chat-adapter does not send top_k in the external request. * studio: gate Anthropic top_k drop to Claude 4.7 only Previous commit ( |
||
|
Daniel Han
|
eb8b0dee2e |
Studio: make stop button actually stop generation (#5069)
* Studio: make stop button actually stop generation The UI stop button routes through assistant-ui's cancelRun, which aborts the frontend fetch. Four issues combined to let llama-server keep decoding long after the user clicked stop: 1. request.is_disconnected() does not fire reliably behind proxies (e.g. Colab) that don't propagate fetch aborts. 2. llama-server defaults n_predict to n_ctx when max_tokens is not sent, so a cancelled request keeps producing tokens up to 262144. 3. The httpx.Client pool keeps TCP keep-alive, so even a cleanly closed stream reuses the same connection and llama-server's liveness poll never sees a disconnect. 4. No explicit backend route to cancel - every cancel path relied on is_disconnected. Changes: - Add POST /api/inference/cancel keyed by session_id/completion_id, with a registry populated for the lifetime of each streaming response. - Have the frontend (chat-adapter.ts) POST /inference/cancel on AbortController abort, alongside the existing fetch teardown. - Send max_tokens=4096 + t_max_predict_ms=120000 as defaults on every outbound chat completion to llama-server; honoured by user overrides. - Disable httpx keep-alive on the streaming client so connection close reaches llama-server and its 1s liveness check fires. No behaviour changes for non-streaming paths or for existing callers that already pass max_tokens/session_id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: harden stop-button cancel path and scope cancel route - Require at least one identifier for /api/inference/cancel so a missing thread id cannot silently cancel every in-flight generation. - Scope /cancel to a dedicated studio_router so it is not exposed under the /v1 OpenAI-compat prefix as a surprise endpoint. - Store a set of cancel events per key in _CANCEL_REGISTRY so concurrent requests on the same session_id do not overwrite each other, and deduplicate in _cancel_by_keys so the cancelled count reflects unique requests. - Always send session_id with chat completions (not only when tools are enabled) so non-tool GGUF streams register under it and are reachable from /cancel. - Register the non-GGUF stream_chunks path in the cancel registry too, so transformers-based stop-button works behind proxies that swallow fetch aborts. - Only apply the 2-minute t_max_predict_ms wall-clock cap when the caller did not pass max_tokens, so legitimate long generations on slow CPU/macOS/Windows supported installs are not silently truncated. - Remove the abort listener on normal stream completion so reused AbortSignals cannot fire a spurious cancel POST after the fact. * studio: close cancel-race and stale-cancel gaps in stop path - Register the cancel tracker before returning StreamingResponse so a stop POST that arrives during prefill / warmup / proxy buffering finds an entry in _CANCEL_REGISTRY. Cleanup now runs via a Starlette BackgroundTask instead of a finally inside the async generator body. - Add a per-run cancel_id on the frontend (crypto.randomUUID) and in ChatCompletionRequest so /api/inference/cancel matches one specific generation. Removes the stale-cancel bug where pressing stop then starting a new run in the same thread would cancel the retry. - Apply t_max_predict_ms unconditionally in all three llama-server payload builders (previously gated on max_tokens=None, which made it dead code for UI callers that always send params.maxTokens). Raise the default to 10 minutes so slow CPU / macOS / Windows installs are not cut off mid-generation. - Make _cancel_by_keys refuse empty input (return 0) so a future internal caller can not accidentally mass-cancel every in-flight request. - Accept cancel_id (primary), session_id, and completion_id on the /api/inference/cancel route. Unify the three streaming sites on the same _cancel_keys / _tracker variable names. - Annotate _CANCEL_REGISTRY as dict[str, set[threading.Event]]. * Add review tests for PR #5069 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: harden stop-button cancel semantics and wall-clock cap - Make /inference/cancel match cancel_id EXCLUSIVELY when supplied. Previously the handler iterated ('cancel_id','session_id','completion_id') and unioned matches, so a stale cancel POST carrying {cancel_id:old, session_id:thr} would still cancel a later run on the same thread via the shared session_id. cancel_id is now a per-run exclusive key; session_id / completion_id are only used as fallbacks when cancel_id is absent. - Close the early-cancel race. If /inference/cancel lands before the streaming handler reaches _TrackedCancel.__enter__() (stop clicked during prefill / warmup / proxy buffering), the cancel was silently dropped. Stash unmatched cancel_ids in _PENDING_CANCELS with a 30 s TTL; _TrackedCancel.__enter__() now replays any matching pending cancel by set()-ing the event immediately after registration. - Make t_max_predict_ms = _DEFAULT_T_MAX_PREDICT_MS conditional on max_tokens is None at all three llama-server payload sites. The cap is a safety net for callers who leave max_tokens unset (otherwise llama-server defaults n_predict to n_ctx, up to 262144). Callers who set an explicit max_tokens are already self-limiting and must not be silently truncated at 10 minutes on slow CPU / macOS / Windows legitimate long generations. - Guard each StreamingResponse return with try/except BaseException so _tracker.__exit__ runs even if StreamingResponse construction or any preceding statement raises between _tracker.__enter__() and the BackgroundTask attachment. Prevents a registry leak on that narrow window. * studio: close TOCTOU race and restore wall-clock backstop on UI path - Close TOCTOU race in the pending-cancel mechanism. The previous fix split cancel_inference's (cancel_by_keys + remember_pending_cancel) and _TrackedCancel.__enter__'s (register + consume_pending) into four separate lock acquisitions. Under contention a cancel POST could acquire-then-release the lock, find the registry empty, and stash ONLY AFTER __enter__ had already registered and consumed an empty pending map -- silently dropping the cancel. Both call sites now do their work inside a single _CANCEL_LOCK critical section, via the new atomic helper _cancel_by_cancel_id_or_stash() and an inlined consume-pending step in __enter__. Reproduced the race under forced interleaving pre-fix; 0/2000 drops post-fix under parallel stress. - Apply t_max_predict_ms UNCONDITIONALLY at all three llama-server payload sites. The previous iteration gated the cap on `max_tokens is None`, which turned out to be dead code on the primary Studio UI path: chat-adapter.ts sets maxTokens=loadResp.context_length after every model load, so every chat request carries an explicit max_tokens and the wall-clock safety net never fired. The cap's original purpose is to bound stuck decodes regardless of the token budget; it must always apply. - Raise _DEFAULT_T_MAX_PREDICT_MS from 10 minutes to 1 hour. 10 minutes was too aggressive for legitimate slow-CPU chat responses (a 4096-token reply at 2 tok/s takes ~34 min); 1 hour accommodates that and still catches genuine zombie decodes. - Prune _PENDING_CANCELS inside _cancel_by_keys as well, so stashed entries expire proportionally to overall cancel traffic rather than only to cancel_id-specific POSTs. * studio: trim verbose comments and docstrings in cancel path * studio/llama_cpp: drop upstream PR hashes from benchmark comment * Add review tests for Studio stop button * Consolidate review tests for Studio stop button * Align cancel-route test with exclusive cancel_id semantics * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: move cancel cleanup to generator finally; drop dead helper - Move _tracker.__exit__ from Starlette BackgroundTask into each streaming generator's finally block. Starlette skips the background callback when stream_response raises (OSError / ClientDisconnect), which leaked _CANCEL_REGISTRY entries on abrupt disconnect. - Check cancel_event.is_set() at the top of each GGUF while loop so a pending-replay cancel falls through to final_chunk + [DONE] instead of propagating GeneratorExit out of _stream_with_retry. - Remove unused _remember_pending_cancel; _cancel_by_cancel_id_or_stash superseded it. * Add review tests for Studio stop-button * studio: wire audio-input stream into cancel registry - Register cancel_event with _TrackedCancel on the audio-input streaming path so POST /api/inference/cancel can stop whisper / audio-input GGUF runs. Previously the registry stayed empty on this branch, so the stop button returned {"cancelled":0} and the decode ran to completion. - Apply the same finally-based cleanup and pre-iteration cancel-event check used on the other three streaming paths. - Update the _CANCEL_REGISTRY block comment to list cancel_id as the primary key (was stale "session_id preferred"). * Consolidate review tests for Studio stop-button cancel flow - Merge the 6 behavioral tests from test_stream_cleanup_on_disconnect.py (finally cleanup on normal/exception/aclose, pre-set cancel_event pattern, and its regressions) into test_stream_cancel_registration_timing.py, which is the PR's existing file covering the same area. - Extend structural invariants to include audio_input_stream alongside the three GGUF / Unsloth streaming generators: no _tracker.__enter__ inside the async gen body, cleanup via try/finally, no background= on StreamingResponse. - Delete test_stream_cleanup_on_disconnect.py (now empty). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: make cancel-via-POST interrupt Unsloth and audio-input streams Close two remaining gaps in the stop-button cancellation wiring: - stream_chunks (Unsloth path): add a top-of-loop cancel_event check and call backend.reset_generation_state() so cancel POSTs flush GPU state and close the SSE cleanly instead of relying on request.is_disconnected (which does not fire through proxies like Colab's). - audio_input_stream: run the synchronous audio_input_generate() via asyncio.to_thread so blocking whisper chunks do not freeze the event loop, matching the pattern already used by the GGUF streaming paths. * Add review tests for Studio stop-button cancel flow * Consolidate review tests for Studio stop-button cancel flow - Delete standalone test_cancel_registry.py at repo root: tests duplicated test_cancel_atomicity.py / test_cancel_id_wiring.py and re-implemented registry primitives inline (scaffolding). - Extend tests/studio/test_stream_cancel_registration_timing.py with regression guards for the iter-1 cancel-loop fixes: structural: each streaming generator checks cancel_event in its loop; audio_input_stream offloads next() via asyncio.to_thread; stream_chunks cancel branch calls reset_generation_state(). runtime: Unsloth loop breaks on external cancel and resets state; audio loop stays responsive under blocking next(); both loops emit zero tokens on pre-set cancel (replay path). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: extend stop-path to passthrough streams; tighten wall-clock cap - Lower _DEFAULT_T_MAX_PREDICT_MS from 1 hour to 10 minutes so the wall-clock backstop actually bounds runaway decodes when cancel signaling fails. - Wire _TrackedCancel and cancel_event.is_set() into _openai_passthrough_stream and _anthropic_passthrough_stream and disable httpx keepalive so stop requests from /v1 and /v1/messages tool-calling clients reach llama-server. - Apply t_max_predict_ms to the tool-passthrough request body so the backstop covers passthrough paths as well. - Symmetric pre-registration stash for session_id/completion_id cancels (_cancel_by_keys_or_stash) so early cancels by those keys replay on later registration like cancel_id. - Drop dead except BaseException guards around StreamingResponse() at four streaming sites; cleanup lives in the generator's finally. * studio: harden cancel registry against ghost-cancel and leak paths - Revert the session_id/completion_id stash in the fallback cancel helper. session_id is thread-scoped and reused across runs, so stashing it on an unmatched POST would fire cancel_event for the user's next unrelated request via _TrackedCancel.__enter__. cancel_id remains the only per-run unique key that gets stashed. - Default max_tokens to _DEFAULT_MAX_TOKENS in the tool-passthrough body. Mirror the direct GGUF path so OpenAI/Anthropic passthrough callers who omit max_tokens get the same zombie-decode cap instead of relying on the wall-clock backstop alone. - Wrap _openai_passthrough_stream setup with an outer try/except BaseException. The inner except httpx.RequestError does not catch asyncio.CancelledError at await client.send, which would otherwise leave _tracker registered in _CANCEL_REGISTRY indefinitely. - Frontend stop POST uses plain fetch + manual Authorization header instead of authFetch. A 401 on the cancel POST no longer refreshes tokens or redirects the user to the login page mid-stop. * Add review tests for Studio stop-button cancel flow * studio: trim comments on stop-button review changes Collapse multi-paragraph rationale blocks on the cancel registry, _openai_passthrough_stream, and the frontend onAbortCancel handler into one-line explanations of why the non-obvious behaviour exists. Drop authFetch import that became unused when the cancel POST switched to plain fetch. * Consolidate review tests for Studio stop-button cancel flow Move review-added tests out of test_cancel_dispatch_edges.py into the existing PR test files that already cover the same areas: - backend registry fan-out / exclusivity / idempotency / falsy-keys edge cases moved into tests/studio/test_cancel_atomicity.py - frontend plain-fetch (not authFetch) + manual Authorization header moved into tests/studio/test_cancel_id_wiring.py Delete the now-empty test_cancel_dispatch_edges.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop default-capping responses at 4096 tokens (follow-up to #5069) (#5174) * Studio: stop default-capping responses at 4096 tokens Follow-up to #5069. The 4096 default introduced for runaway-decode defense silently truncates any caller that omits max_tokens. The Studio chat UI sets params.maxTokens = loadResp.context_length after a GGUF load, so it's fine, but every other consumer is not: - OpenAI-API direct callers (/v1/chat/completions, /v1/responses, /v1/messages, /v1/completions) where the OpenAI default is effectively unlimited per response. langchain, llama-index, raw curl, and the openai SDK all rely on that. - Reasoning models. Qwen3 / gpt-oss reasoning traces routinely exceed 4096 tokens before the model emits a single visible content token. The user sees the trace cut off mid-thought. - Long-form generation ("write a chapter", "produce a full SVG"). Reproduced on this branch: gemma-4-E2B-it-GGUF Q8_0, prompt asking for a 10000-word story, no max_tokens in the request: finish_reason: stop (misleading -- should be 'length') content_chars: 19772 content_tail: ...'a comforting, yet immense, pressure.\n\n*"' Body ended mid-sentence on a stray opening quote, right at the 4096 token mark. After this patch the same request returns 38357 chars ending with '...held in a perfect, dynamic equilibrium.' -- a natural stop, not a truncation. Implementation: rename the constant to _DEFAULT_MAX_TOKENS_FLOOR and set it to 32768. Each call site now uses the model's effective context length when known, falling back to the floor: default_cap = self._effective_context_length or _DEFAULT_MAX_TOKENS_FLOOR The 10-minute t_max_predict_ms wall-clock backstop from #5069 is preserved as the second line of defense. Plumbed _build_passthrough_payload + _build_openai_passthrough_body through the routes layer so the Anthropic and OpenAI passthrough paths also respect the model's context length. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Studio: cancel passthrough streams during llama-server prefill + route through apiUrl for Tauri Three reviewer-flagged correctness gaps in the stop-button mechanism. 1) `_openai_passthrough_stream` could not honor cancel during prefill. The cancel check ran inside the `async for raw_line in lines_iter` body, so a cancel POST that arrived before llama-server emitted the first SSE line was unobservable until prefill completed. With a long prompt under proxy/Colab conditions -- the exact target scenario for this PR -- that left the model decoding for a long time after the user clicked Stop. Add an asyncio watcher task that closes `resp` as soon as `cancel_event` is set, raising in `aiter_lines` so the generator can exit. The watcher polls a threading.Event because the cancel registry is keyed by threading.Event for the synchronous /cancel handler. 2) `_anthropic_passthrough_stream` had the same blocking-prefill pattern. Same fix. 3) The frontend's stop-button cancel POST used a bare relative `fetch("/api/inference/cancel", ...)`, which targets the webview origin in Tauri production builds (where the backend is at `http://127.0.0.1:8888`). Route through the existing `apiUrl()` helper from `lib/api-base.ts` to match every other Studio call. Browser/dev builds get the empty base, so behavior is unchanged there. Verified via temp/pr_simulation/sim_5069_prefill_cancel.py: cancel during prefill terminates within ~250ms on both passthrough paths (was 145s+ on the Anthropic path before this change), and the standard non-passthrough chat path still cancels with no regression. * Studio: log cancel-body parse errors instead of silently swallowing Reviewer-flagged defensive logging gap. The bare `except Exception: pass` in `cancel_inference` would mask malformed payloads that hint at a buggy client or a transport issue. Log at debug so future investigation isn't left guessing whether `body={}` came from a missing body or a parse failure. Behavior is unchanged: an unparseable body still falls through to the empty-dict path and the cancel call returns `{"cancelled": 0}`. * Studio: Anthropic passthrough cancel parity with OpenAI passthrough Two reviewer-flagged consistency gaps in the cancel surface for /v1/messages. 1) Anthropic passthrough did not register cancel_id, so a per-run cancel POST (the cleanest Studio-style cancel path) silently missed when the route hit `_anthropic_passthrough_stream`. The OpenAI passthrough has registered (cancel_id, session_id, completion_id) since this PR was first opened; mirror that here. Also add `cancel_id` to `AnthropicMessagesRequest` so the route handler can plumb it through. 2) The cancel handler's fallback key list checked only completion_id and session_id, never message_id. Anthropic clients that send their native `id` (returned in the SSE message_start event) for cancel had no way to hit the registry. Add message_id to the fallback list. Verified via temp/pr_simulation/sim_5069_prefill_cancel.py: P2 now cancels by cancel_id in 137ms (was hanging pre-fix), and the new P2b case cancels by message_id in 77ms. P1 (OpenAI) and P3 (standard chat) still pass with no regression. --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> |
||
|
Wasim Yousef Said
|
208862218d |
feat(studio): training history persistence and past runs viewer (#4501)
* feat(db): add SQLite storage layer for training history * feat(api): add training history endpoints and response models * feat(training): integrate DB persistence into training event loop * feat(ui): add training history views and card grid * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): address review issues in training history persistence - Strip hf_token/wandb_token from config before SQLite storage - Add UUID suffix to job_id for collision resistance - Use isfinite() for 0.0 metric handling throughout - Respect _should_stop in error event finalization - Run schema DDL once per process, not per connection - Close connection on schema init failure - Guard cleanup_orphaned_runs at startup - Cap _metric_buffer at 500 entries - Make FLUSH_THRESHOLD a class constant - Map 'running' to 'training' phase in historical view - Derive LR/GradNorm from history arrays in historical view - Fix nested button with div[role=button] in history cards - Guard String(value) against null/undefined in config popover - Clear selectedHistoryRunId on auto tab switch * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): address round-2 review findings across training backend and frontend Backend (training.py): - Move state mutation after proc.start() so a failed spawn does not wedge the backend with is_training=True - Create DB run row eagerly after proc.start() so runs appear in history during model loading, not after first metric event - Rewrite _flush_metrics_to_db() with snapshot-before-insert pattern to preserve metrics arriving during the write and retain buffer on failure - Guard eval_loss with float() coercion and math.isfinite(), matching the existing grad_norm guard - Increase pump thread join timeout from 3s to 8s to cover SQLite's default 5s lock timeout Frontend (studio-page.tsx): - Fix history navigation: check isTrainingRunning instead of showTrainingView in onSelectRun so completed runs are not misrouted - Replace activeTab state + auto-switch useEffect with derived tab to eliminate react-hooks/set-state-in-effect lint violation Frontend (historical-training-view.tsx): - Add explicit "running" branch to message ternary so running runs no longer fall through to "Training errored" - Derive loading from detail/error state and move cleanup to effect return to eliminate react-hooks/set-state-in-effect lint violation Frontend (progress-section.tsx): - Derive stopRequested from isTrainingRunning && stopRequestedLocal to eliminate react-hooks/set-state-in-effect lint violation and remove unused useEffect import * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): resolve 3 remaining bugs from round-2 review 1. Stuck on Current Run tab [12/20]: Only force "current-run" tab when isTrainingRunning is true, not when stale completed-run data exists. After training ends, users can freely navigate to Configure. 2. Incomplete metric sanitization [7/20]: Apply float() coercion and isfinite() guards to loss and learning_rate, matching the existing pattern used by grad_norm and eval_loss. Prevents TypeError from string values and NaN leaks into history arrays. 3. Stop button state leak across runs [10/20]: Add key={runtime.jobId} to ProgressSection so React remounts it when a new run starts, resetting stopRequestedLocal state. * fix(studio): deduplicate loss/lr sanitization in training event handler Reuse _safe_loss/_safe_lr from the progress update block instead of re-sanitizing the same raw event values for metric history. * fix(studio): restore loss > 0 guard to prevent eval steps injecting 0.0 into metric histories Round-2/3 fixes relaxed the history append guard from `loss > 0` to `loss is not None`, which let eval-only log events (where loss defaults to 0.0) append fake zeros into loss_history and lr_history. Restore the `loss > 0` check to match the worker's own has_train_loss gate. The float() coercion and isfinite() sanitization from round-3 remain intact. * fix(studio): resolve training history bugs — nullable loss/lr, tab nav, sparkline * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> |
||
|
Roland Tannous
|
a2baf80511 | Update license headers | ||
|
Roland Tannous
|
d882678fe4 | Add AGPL-3.0 SPDX headers to all source files | ||
|
Shine1i
|
43c66da783 | merge nightly | ||
|
Shine1i
|
85653237ea | feat: add Data Recipe core functionality with job manager, API routes, and validation services | ||
|
sshah229
|
40bfe42974 | added the pydantic models and routes for export | ||
|
sshah229
|
50ff5626f1 | refactored the code for username/password and added pydantic models and routes for the same | ||
|
Roland Tannous
|
75bb6c08a5 | Add datasets check-format endpoint | ||
|
Roland Tannous
|
b4ec0389f0 | refactor/inference-api-routes-part-1 | ||
|
Roland Tannous
|
544d6944d1 | root studio folder |
Renamed from backend/routes/__init__.py (Browse further)