12 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Roland Tannous
|
9a0d6f80cb |
studio: API external provider support for chat (OpenAI, Mistral, Gemini, Cohere, Anthropic, OpenRouter, DeepSeek, custom providers) (#4706)
* studio: add external provider support for chat inference Adds the ability to connect to OpenAI, Mistral, Google, Cohere, Together, Fireworks, and Perplexity from the Studio chat interface. - Provider configs stored in SQLite (no API keys persisted) - RSA-2048 key pair generated at startup for client-side key encryption - httpx proxy client streams SSE responses in OpenAI-compatible format - New /api/providers routes: registry, CRUD, test, models - /v1/chat/completions routes to external provider when provider fields present - Integration test suite covering CRUD, connection, model listing, and inference - Frontend spec doc with full API contract * remove frontend spec doc from branch * fix auth fixture: handle forced password change on fresh install * fix tests: default port 8000, allow 400 for no-model-loaded * fix: update Cohere models to current (command-r retired Sept 2025) * feat: add OpenRouter as 8th provider * feat: add native Anthropic provider with Messages API translation * fix: correct Anthropic base URL and drop top_p (conflicts with temperature) * feat: add DeepSeek provider (deepseek-chat, deepseek-reasoner) * feat: rename google -> gemini, refresh model list to 2.5 series * feat: remove together, fireworks, perplexity providers * feat: multimodal image support for external providers - Add _build_external_messages() that preserves image_url parts for vision-capable providers instead of stripping them - Update _proxy_to_external_provider() to use new helper - Translate image_url content parts to Anthropic native image format in _stream_anthropic() - Add TestVisionInference pytest class (1x1 PNG smoke test) * test: use sloth photo URL for vision test, add Anthropic remote URL support * fix: update Mistral model to mistral-small-2506 * update mistral default model to mistral-large-2512 * fix gemini vision test: download image as base64 data URI instead of remote URL * add gemini-3-flash-preview as default gemini model * fix gemini truncated reply (max_tokens 16->64) and suppress GeneratorExit on client disconnect * increase vision test max_tokens to 215 * fix GeneratorExit: aclose stream generator before closing httpx client * fix httpcore GeneratorExit: explicitly aclose aiter_lines before response closes * fix duplicate [DONE] and suppress httpcore RuntimeError on Python 3.13 asyncgen cleanup * fix: call response.aclose() before lines_gen.aclose() to prevent httpcore RuntimeError on Python 3.13 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Potential fix for code scanning alert no. 36: Clear-text logging of sensitive information Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * review: add comments for manual iteration rationale, mask password in test print, clarify Anthropic URL/models support * perf: use shared module-level httpx client for connection pooling across requests * studio: add API provider UI and integrate wiring (#4737) * feat: expose external models in selector and chat settings * feat(chat): wire external providers to backend + RSA key flow - Fetch registry/configs; create/update/delete saved providers - Encrypt API keys (Web Crypto RSA-OAEP) for test/models/chat - External model selection + chat payload (provider_id/type, external_model, encrypted key, optional base URL) - Local storage for keys + provider list; small UX/copy and guardrails * add missing providers-api.ts file by Imagineer99 * fix: address PR review comments — system prompt visibility, retry loop, test logging * feat(studio): encrypt external provider API keys at rest in localStorage API keys for external providers (OpenAI, Mistral, etc.) were stored as plaintext in localStorage, vulnerable to browser extensions and XSS. Add password-derived AES-256-GCM encryption: on login the user's password is used via PBKDF2 (100k iterations, SHA-256) to derive an in-memory encryption key. API keys are encrypted before writing to localStorage and decrypted on read. The derived key is never persisted — cleared on logout, re-derived on next login. Legacy plaintext keys are transparently migrated on first access. Password changes re-encrypt all stored keys. No backend changes required — the existing RSA-OAEP transit encryption is unaffected. * fix: cast PBKDF2 salt to BufferSource for strict TypeScript lib types * fix: persist session password in sessionStorage to survive page refreshes * feat(studio): preserve image parts in external provider chat requests toOpenAIMessage() now returns multimodal content arrays (OpenAI vision format) when messages contain images, instead of always flattening to plain text. This enables vision-capable external providers (OpenAI, Gemini, Anthropic, etc.) to receive user images. The backend already handles image_url content parts in _build_external_messages(). * studio: fix external models selectable in chat-only mode (#4779) * fix: external models selectable in chat-only mode * fix: model selector tabs default to active model kind * Studio: API external provider registry + curated catalogs (HF/OpenRouter) and chat UX (#4787) * fix: external models selectable in chat-only mode * fix: model selector tabs default to active model kind * feat(studio): expand provider registry, curated catalogs, and chat UX - Add Hugging Face, Kimi, Qwen; remove Cohere; reorder registry - model_list_mode curated for HF/OpenRouter; lightweight /models check - API returns default models for curated providers; expose model_list_mode - Frontend: provider logos in model picker, providerType on external models - Chat providers dialog: curated vs remote flows, motion polish - Thread: LayoutGroup + composer motion alignment with app easing * fix(studio): disable Anthropic tool-calling flag and preselect curated defaults * feat(studio): add external provider logos and ApiProviderLogo helper * Studio: Polish API Providers dialog (#4899) * fix: lower verbage in API providers page * fix: fix(studio): tune API Providers dialog width with rem-based responsive caps * feat: add custom provider support (#4902) * fix: replace crypto.subtle with node-forge for HTTP compatibility crypto.subtle is only available in secure contexts (HTTPS/localhost), which breaks provider API key encryption when Studio is accessed over plain HTTP on remote GPU VMs. Switch to node-forge for RSA-OAEP and AES-256-GCM operations — same algorithms, works on any origin. * fix: store provider API keys as plaintext in localStorage Drop AES-256-GCM at-rest encryption for provider API keys. The session-password-derived encryption broke on auto-login via refresh token (password never captured), causing keys to silently vanish. API keys are still RSA-encrypted in transit via node-forge. At-rest encryption in localStorage added no real security since the decryption key also had to live client-side. Removes crypto-storage.ts, session password plumbing, and reEncryptAllKeys. * fix: use max_completion_tokens for OpenAI provider Newer OpenAI models (gpt-4o, gpt-5.x) reject the max_tokens param and require max_completion_tokens instead. Other providers still use max_tokens. * fix: skip empty assistant messages in external provider requests Some providers (Mistral) reject assistant messages with empty content. Filter them out when building the message list for external providers. * Update model-selector.tsx * Update model-selector.tsx * Update model-selector.tsx * Update chat-adapter.ts * Update chat-adapter.ts * Update chat-page.tsx * Update chat-settings-sheet.tsx * Update chat-settings-sheet.tsx * Update chat-settings-sheet.tsx * Update chat-providers-dialog.tsx * feat: polish providers settings form UI * style: polish provider row icon sizing and alignment * style: stabilize provider layout * style: add provider API key visibility toggle * fix: add provider render on empty list * studio/frontend: sync package-lock.json with package.json npm ci was failing because node-forge and @types/node-forge were declared in package.json but missing from the lockfile. Ran npm install to regenerate. * studio/backend: fix backend CI failures for providers router - test_desktop_auth: include providers_router in the routes stub so studio.backend.main imports cleanly under the monkeypatched module - test_providers_api: skip the whole module when STUDIO_TEST_PASSWORD is unset (it is an integration test against a live Studio server, same shape as the already-ignored test_studio_api.py) * studio/chat: drive ChatSettingsPanel from a per-provider capability map Replace the binary isExternalModel toggle in the sampling section with a provider-aware capability map. Each external provider type advertises which of top_k / min_p / repetition_penalty / presence_penalty its chat-completions API actually accepts, so the panel only renders the knobs that map onto the active provider's request body. Anthropic now exposes top_k; DeepSeek hides presence_penalty (deprecated in their docs); OpenRouter and custom providers continue to show every knob (OpenRouter drops unsupported server-side, custom assumes OpenAI-compat or a permissive vLLM/Ollama backend). Local models are unaffected — null capabilities means 'show everything'. chat-adapter.ts now forwards top_k / presence_penalty to the external proxy only when the active provider's capabilities permit it, so the request body matches what the UI shows. * studio/backend: forward top_k to Anthropic; filter OpenAI model list Two paired changes so the frontend capability map has matching backend behaviour: 1. ExternalProviderClient.stream_chat_completion now accepts top_k and forwards it to the Anthropic Messages body. OpenAI-compat providers (which all reject unknown sampling params) still receive only the fields they document. The proxy route in routes/inference.py passes payload.top_k through, so a UI request with top_k actually reaches Anthropic instead of being silently dropped at the boundary. 2. PROVIDER_REGISTRY['openai'] gains a model_id_allowlist regex that scopes the /models picker to current-gen ids (gpt-5.5 / gpt-5.4 / gpt-5.3 / gpt-4.5 / o3 families). The remote /v1/models listing otherwise returns dozens of historical snapshots, fine-tunes and non-chat models (embeddings, TTS, image, moderation) that we never want in the chat UI. default_models is refreshed to match. * studio/chat: relax presence_penalty to optional on OpenAIChatCompletionsRequest Followup to 1fbf445a — chat-adapter now omits presence_penalty for providers that do not accept it (Anthropic / DeepSeek), but the request type still required it as a non-optional number, breaking tsc. The backend pydantic model already defaults presence_penalty to 0, so making it optional client-side matches reality. * studio/backend: route OpenAI traffic through /v1/responses OpenAI's new flagship models (gpt-5.x) return 404 'This is not a chat model' on /v1/chat/completions and are only reachable via /v1/responses. Add a dedicated _stream_openai_responses path in ExternalProviderClient that: - Translates outbound messages into the Responses shape: system messages are folded into the top-level 'instructions' field, user/assistant messages become {role, content} items with input_text / input_image content parts (data URLs and https URLs both pass through). - Drops presence_penalty / top_k / frequency_penalty, none of which the Responses contract accepts. - Translates inbound SSE events back into OpenAI Chat Completions chunks so the frontend keeps a single SSE shape: response.output_text.delta -> delta chunk with content response.completed -> chunk with finish_reason='stop' response.incomplete -> chunk with finish_reason='length' response.failed / error -> propagated error SSE line Stream terminates with data: [DONE] (Responses emits this verbatim). stream_chat_completion dispatches all provider_type='openai' calls to this path; other OpenAI-compatible providers (mistral, gemini, etc.) continue to use /v1/chat/completions. Frontend provider-capabilities map updated to hide presence_penalty for OpenAI in the chat settings panel, matching the new request contract. Includes unit coverage in tests/test_openai_responses_translation.py exercising the request body translation, image-part rewriting, and SSE-to-chat-completions translation via httpx.MockTransport. * studio/chat: clamp external max_tokens to 32k to stay within provider caps The chat settings slider already capped maxTokens at 32768 for external models, but a value persisted from a prior local-model session (where the cap can be 128k+) was sent verbatim to the provider — Claude Opus returns 'max_tokens: 131072 > 128000' on requests like that, and other providers have stricter limits still. Expose EXTERNAL_MAX_OUTPUT_TOKENS from provider-capabilities (32k) and use it both for the slider max and as the clamp inside chat-adapter's external-request body. 32k sits below the tightest declared output limit across the providers we ship and well above what a typical chat reply needs; the local-model path is unaffected. * studio: drop temperature/top_p for OpenAI reasoning models gpt-5.x / o3 / gpt-4.5 are reasoning-class models served via /v1/responses, and reject temperature and top_p with 'Unsupported parameter' 400s. The OpenAI registry allowlist already scopes the picker to those families, so neither knob ever applies on this branch. - external_provider._stream_openai_responses no longer puts temperature or top_p in the request body (kept on the method signature for API symmetry with the other stream methods). - ProviderCapabilities gains temperature/topP flags; OpenAI sets both to false. ChatSettingsPanel hides the sliders for OpenAI so the user does not see inert controls. - chat-adapter omits temperature/top_p from the external request body when the active provider does not advertise them. - OpenAIChatCompletionsRequest type marks both as optional, matching the new chat-adapter shape. - test_responses_request_body_uses_input_and_instructions: assertions flipped to confirm temperature / top_p are absent from the body. * studio: stop forwarding top_k to Anthropic Claude 4.x (Opus / Sonnet / Haiku 4.x) returns 400 'top_k is deprecated for this model' on any request that includes top_k. It was always optional on the older 3.x line, so dropping it unconditionally for every Anthropic call is the simplest path — no per-model gate to maintain. - external_provider._stream_anthropic no longer adds top_k to the Messages body (kept on the method signature for API symmetry). - provider-capabilities sets anthropic.topK = false so the chat settings panel hides the Top K slider for Anthropic providers and chat-adapter does not send top_k in the external request. * studio: gate Anthropic top_k drop to Claude 4.7 only Previous commit ( |
||
|
Daniel Han
|
eb8b0dee2e |
Studio: make stop button actually stop generation (#5069)
* Studio: make stop button actually stop generation The UI stop button routes through assistant-ui's cancelRun, which aborts the frontend fetch. Four issues combined to let llama-server keep decoding long after the user clicked stop: 1. request.is_disconnected() does not fire reliably behind proxies (e.g. Colab) that don't propagate fetch aborts. 2. llama-server defaults n_predict to n_ctx when max_tokens is not sent, so a cancelled request keeps producing tokens up to 262144. 3. The httpx.Client pool keeps TCP keep-alive, so even a cleanly closed stream reuses the same connection and llama-server's liveness poll never sees a disconnect. 4. No explicit backend route to cancel - every cancel path relied on is_disconnected. Changes: - Add POST /api/inference/cancel keyed by session_id/completion_id, with a registry populated for the lifetime of each streaming response. - Have the frontend (chat-adapter.ts) POST /inference/cancel on AbortController abort, alongside the existing fetch teardown. - Send max_tokens=4096 + t_max_predict_ms=120000 as defaults on every outbound chat completion to llama-server; honoured by user overrides. - Disable httpx keep-alive on the streaming client so connection close reaches llama-server and its 1s liveness check fires. No behaviour changes for non-streaming paths or for existing callers that already pass max_tokens/session_id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: harden stop-button cancel path and scope cancel route - Require at least one identifier for /api/inference/cancel so a missing thread id cannot silently cancel every in-flight generation. - Scope /cancel to a dedicated studio_router so it is not exposed under the /v1 OpenAI-compat prefix as a surprise endpoint. - Store a set of cancel events per key in _CANCEL_REGISTRY so concurrent requests on the same session_id do not overwrite each other, and deduplicate in _cancel_by_keys so the cancelled count reflects unique requests. - Always send session_id with chat completions (not only when tools are enabled) so non-tool GGUF streams register under it and are reachable from /cancel. - Register the non-GGUF stream_chunks path in the cancel registry too, so transformers-based stop-button works behind proxies that swallow fetch aborts. - Only apply the 2-minute t_max_predict_ms wall-clock cap when the caller did not pass max_tokens, so legitimate long generations on slow CPU/macOS/Windows supported installs are not silently truncated. - Remove the abort listener on normal stream completion so reused AbortSignals cannot fire a spurious cancel POST after the fact. * studio: close cancel-race and stale-cancel gaps in stop path - Register the cancel tracker before returning StreamingResponse so a stop POST that arrives during prefill / warmup / proxy buffering finds an entry in _CANCEL_REGISTRY. Cleanup now runs via a Starlette BackgroundTask instead of a finally inside the async generator body. - Add a per-run cancel_id on the frontend (crypto.randomUUID) and in ChatCompletionRequest so /api/inference/cancel matches one specific generation. Removes the stale-cancel bug where pressing stop then starting a new run in the same thread would cancel the retry. - Apply t_max_predict_ms unconditionally in all three llama-server payload builders (previously gated on max_tokens=None, which made it dead code for UI callers that always send params.maxTokens). Raise the default to 10 minutes so slow CPU / macOS / Windows installs are not cut off mid-generation. - Make _cancel_by_keys refuse empty input (return 0) so a future internal caller can not accidentally mass-cancel every in-flight request. - Accept cancel_id (primary), session_id, and completion_id on the /api/inference/cancel route. Unify the three streaming sites on the same _cancel_keys / _tracker variable names. - Annotate _CANCEL_REGISTRY as dict[str, set[threading.Event]]. * Add review tests for PR #5069 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: harden stop-button cancel semantics and wall-clock cap - Make /inference/cancel match cancel_id EXCLUSIVELY when supplied. Previously the handler iterated ('cancel_id','session_id','completion_id') and unioned matches, so a stale cancel POST carrying {cancel_id:old, session_id:thr} would still cancel a later run on the same thread via the shared session_id. cancel_id is now a per-run exclusive key; session_id / completion_id are only used as fallbacks when cancel_id is absent. - Close the early-cancel race. If /inference/cancel lands before the streaming handler reaches _TrackedCancel.__enter__() (stop clicked during prefill / warmup / proxy buffering), the cancel was silently dropped. Stash unmatched cancel_ids in _PENDING_CANCELS with a 30 s TTL; _TrackedCancel.__enter__() now replays any matching pending cancel by set()-ing the event immediately after registration. - Make t_max_predict_ms = _DEFAULT_T_MAX_PREDICT_MS conditional on max_tokens is None at all three llama-server payload sites. The cap is a safety net for callers who leave max_tokens unset (otherwise llama-server defaults n_predict to n_ctx, up to 262144). Callers who set an explicit max_tokens are already self-limiting and must not be silently truncated at 10 minutes on slow CPU / macOS / Windows legitimate long generations. - Guard each StreamingResponse return with try/except BaseException so _tracker.__exit__ runs even if StreamingResponse construction or any preceding statement raises between _tracker.__enter__() and the BackgroundTask attachment. Prevents a registry leak on that narrow window. * studio: close TOCTOU race and restore wall-clock backstop on UI path - Close TOCTOU race in the pending-cancel mechanism. The previous fix split cancel_inference's (cancel_by_keys + remember_pending_cancel) and _TrackedCancel.__enter__'s (register + consume_pending) into four separate lock acquisitions. Under contention a cancel POST could acquire-then-release the lock, find the registry empty, and stash ONLY AFTER __enter__ had already registered and consumed an empty pending map -- silently dropping the cancel. Both call sites now do their work inside a single _CANCEL_LOCK critical section, via the new atomic helper _cancel_by_cancel_id_or_stash() and an inlined consume-pending step in __enter__. Reproduced the race under forced interleaving pre-fix; 0/2000 drops post-fix under parallel stress. - Apply t_max_predict_ms UNCONDITIONALLY at all three llama-server payload sites. The previous iteration gated the cap on `max_tokens is None`, which turned out to be dead code on the primary Studio UI path: chat-adapter.ts sets maxTokens=loadResp.context_length after every model load, so every chat request carries an explicit max_tokens and the wall-clock safety net never fired. The cap's original purpose is to bound stuck decodes regardless of the token budget; it must always apply. - Raise _DEFAULT_T_MAX_PREDICT_MS from 10 minutes to 1 hour. 10 minutes was too aggressive for legitimate slow-CPU chat responses (a 4096-token reply at 2 tok/s takes ~34 min); 1 hour accommodates that and still catches genuine zombie decodes. - Prune _PENDING_CANCELS inside _cancel_by_keys as well, so stashed entries expire proportionally to overall cancel traffic rather than only to cancel_id-specific POSTs. * studio: trim verbose comments and docstrings in cancel path * studio/llama_cpp: drop upstream PR hashes from benchmark comment * Add review tests for Studio stop button * Consolidate review tests for Studio stop button * Align cancel-route test with exclusive cancel_id semantics * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: move cancel cleanup to generator finally; drop dead helper - Move _tracker.__exit__ from Starlette BackgroundTask into each streaming generator's finally block. Starlette skips the background callback when stream_response raises (OSError / ClientDisconnect), which leaked _CANCEL_REGISTRY entries on abrupt disconnect. - Check cancel_event.is_set() at the top of each GGUF while loop so a pending-replay cancel falls through to final_chunk + [DONE] instead of propagating GeneratorExit out of _stream_with_retry. - Remove unused _remember_pending_cancel; _cancel_by_cancel_id_or_stash superseded it. * Add review tests for Studio stop-button * studio: wire audio-input stream into cancel registry - Register cancel_event with _TrackedCancel on the audio-input streaming path so POST /api/inference/cancel can stop whisper / audio-input GGUF runs. Previously the registry stayed empty on this branch, so the stop button returned {"cancelled":0} and the decode ran to completion. - Apply the same finally-based cleanup and pre-iteration cancel-event check used on the other three streaming paths. - Update the _CANCEL_REGISTRY block comment to list cancel_id as the primary key (was stale "session_id preferred"). * Consolidate review tests for Studio stop-button cancel flow - Merge the 6 behavioral tests from test_stream_cleanup_on_disconnect.py (finally cleanup on normal/exception/aclose, pre-set cancel_event pattern, and its regressions) into test_stream_cancel_registration_timing.py, which is the PR's existing file covering the same area. - Extend structural invariants to include audio_input_stream alongside the three GGUF / Unsloth streaming generators: no _tracker.__enter__ inside the async gen body, cleanup via try/finally, no background= on StreamingResponse. - Delete test_stream_cleanup_on_disconnect.py (now empty). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: make cancel-via-POST interrupt Unsloth and audio-input streams Close two remaining gaps in the stop-button cancellation wiring: - stream_chunks (Unsloth path): add a top-of-loop cancel_event check and call backend.reset_generation_state() so cancel POSTs flush GPU state and close the SSE cleanly instead of relying on request.is_disconnected (which does not fire through proxies like Colab's). - audio_input_stream: run the synchronous audio_input_generate() via asyncio.to_thread so blocking whisper chunks do not freeze the event loop, matching the pattern already used by the GGUF streaming paths. * Add review tests for Studio stop-button cancel flow * Consolidate review tests for Studio stop-button cancel flow - Delete standalone test_cancel_registry.py at repo root: tests duplicated test_cancel_atomicity.py / test_cancel_id_wiring.py and re-implemented registry primitives inline (scaffolding). - Extend tests/studio/test_stream_cancel_registration_timing.py with regression guards for the iter-1 cancel-loop fixes: structural: each streaming generator checks cancel_event in its loop; audio_input_stream offloads next() via asyncio.to_thread; stream_chunks cancel branch calls reset_generation_state(). runtime: Unsloth loop breaks on external cancel and resets state; audio loop stays responsive under blocking next(); both loops emit zero tokens on pre-set cancel (replay path). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: extend stop-path to passthrough streams; tighten wall-clock cap - Lower _DEFAULT_T_MAX_PREDICT_MS from 1 hour to 10 minutes so the wall-clock backstop actually bounds runaway decodes when cancel signaling fails. - Wire _TrackedCancel and cancel_event.is_set() into _openai_passthrough_stream and _anthropic_passthrough_stream and disable httpx keepalive so stop requests from /v1 and /v1/messages tool-calling clients reach llama-server. - Apply t_max_predict_ms to the tool-passthrough request body so the backstop covers passthrough paths as well. - Symmetric pre-registration stash for session_id/completion_id cancels (_cancel_by_keys_or_stash) so early cancels by those keys replay on later registration like cancel_id. - Drop dead except BaseException guards around StreamingResponse() at four streaming sites; cleanup lives in the generator's finally. * studio: harden cancel registry against ghost-cancel and leak paths - Revert the session_id/completion_id stash in the fallback cancel helper. session_id is thread-scoped and reused across runs, so stashing it on an unmatched POST would fire cancel_event for the user's next unrelated request via _TrackedCancel.__enter__. cancel_id remains the only per-run unique key that gets stashed. - Default max_tokens to _DEFAULT_MAX_TOKENS in the tool-passthrough body. Mirror the direct GGUF path so OpenAI/Anthropic passthrough callers who omit max_tokens get the same zombie-decode cap instead of relying on the wall-clock backstop alone. - Wrap _openai_passthrough_stream setup with an outer try/except BaseException. The inner except httpx.RequestError does not catch asyncio.CancelledError at await client.send, which would otherwise leave _tracker registered in _CANCEL_REGISTRY indefinitely. - Frontend stop POST uses plain fetch + manual Authorization header instead of authFetch. A 401 on the cancel POST no longer refreshes tokens or redirects the user to the login page mid-stop. * Add review tests for Studio stop-button cancel flow * studio: trim comments on stop-button review changes Collapse multi-paragraph rationale blocks on the cancel registry, _openai_passthrough_stream, and the frontend onAbortCancel handler into one-line explanations of why the non-obvious behaviour exists. Drop authFetch import that became unused when the cancel POST switched to plain fetch. * Consolidate review tests for Studio stop-button cancel flow Move review-added tests out of test_cancel_dispatch_edges.py into the existing PR test files that already cover the same areas: - backend registry fan-out / exclusivity / idempotency / falsy-keys edge cases moved into tests/studio/test_cancel_atomicity.py - frontend plain-fetch (not authFetch) + manual Authorization header moved into tests/studio/test_cancel_id_wiring.py Delete the now-empty test_cancel_dispatch_edges.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop default-capping responses at 4096 tokens (follow-up to #5069) (#5174) * Studio: stop default-capping responses at 4096 tokens Follow-up to #5069. The 4096 default introduced for runaway-decode defense silently truncates any caller that omits max_tokens. The Studio chat UI sets params.maxTokens = loadResp.context_length after a GGUF load, so it's fine, but every other consumer is not: - OpenAI-API direct callers (/v1/chat/completions, /v1/responses, /v1/messages, /v1/completions) where the OpenAI default is effectively unlimited per response. langchain, llama-index, raw curl, and the openai SDK all rely on that. - Reasoning models. Qwen3 / gpt-oss reasoning traces routinely exceed 4096 tokens before the model emits a single visible content token. The user sees the trace cut off mid-thought. - Long-form generation ("write a chapter", "produce a full SVG"). Reproduced on this branch: gemma-4-E2B-it-GGUF Q8_0, prompt asking for a 10000-word story, no max_tokens in the request: finish_reason: stop (misleading -- should be 'length') content_chars: 19772 content_tail: ...'a comforting, yet immense, pressure.\n\n*"' Body ended mid-sentence on a stray opening quote, right at the 4096 token mark. After this patch the same request returns 38357 chars ending with '...held in a perfect, dynamic equilibrium.' -- a natural stop, not a truncation. Implementation: rename the constant to _DEFAULT_MAX_TOKENS_FLOOR and set it to 32768. Each call site now uses the model's effective context length when known, falling back to the floor: default_cap = self._effective_context_length or _DEFAULT_MAX_TOKENS_FLOOR The 10-minute t_max_predict_ms wall-clock backstop from #5069 is preserved as the second line of defense. Plumbed _build_passthrough_payload + _build_openai_passthrough_body through the routes layer so the Anthropic and OpenAI passthrough paths also respect the model's context length. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Studio: cancel passthrough streams during llama-server prefill + route through apiUrl for Tauri Three reviewer-flagged correctness gaps in the stop-button mechanism. 1) `_openai_passthrough_stream` could not honor cancel during prefill. The cancel check ran inside the `async for raw_line in lines_iter` body, so a cancel POST that arrived before llama-server emitted the first SSE line was unobservable until prefill completed. With a long prompt under proxy/Colab conditions -- the exact target scenario for this PR -- that left the model decoding for a long time after the user clicked Stop. Add an asyncio watcher task that closes `resp` as soon as `cancel_event` is set, raising in `aiter_lines` so the generator can exit. The watcher polls a threading.Event because the cancel registry is keyed by threading.Event for the synchronous /cancel handler. 2) `_anthropic_passthrough_stream` had the same blocking-prefill pattern. Same fix. 3) The frontend's stop-button cancel POST used a bare relative `fetch("/api/inference/cancel", ...)`, which targets the webview origin in Tauri production builds (where the backend is at `http://127.0.0.1:8888`). Route through the existing `apiUrl()` helper from `lib/api-base.ts` to match every other Studio call. Browser/dev builds get the empty base, so behavior is unchanged there. Verified via temp/pr_simulation/sim_5069_prefill_cancel.py: cancel during prefill terminates within ~250ms on both passthrough paths (was 145s+ on the Anthropic path before this change), and the standard non-passthrough chat path still cancels with no regression. * Studio: log cancel-body parse errors instead of silently swallowing Reviewer-flagged defensive logging gap. The bare `except Exception: pass` in `cancel_inference` would mask malformed payloads that hint at a buggy client or a transport issue. Log at debug so future investigation isn't left guessing whether `body={}` came from a missing body or a parse failure. Behavior is unchanged: an unparseable body still falls through to the empty-dict path and the cancel call returns `{"cancelled": 0}`. * Studio: Anthropic passthrough cancel parity with OpenAI passthrough Two reviewer-flagged consistency gaps in the cancel surface for /v1/messages. 1) Anthropic passthrough did not register cancel_id, so a per-run cancel POST (the cleanest Studio-style cancel path) silently missed when the route hit `_anthropic_passthrough_stream`. The OpenAI passthrough has registered (cancel_id, session_id, completion_id) since this PR was first opened; mirror that here. Also add `cancel_id` to `AnthropicMessagesRequest` so the route handler can plumb it through. 2) The cancel handler's fallback key list checked only completion_id and session_id, never message_id. Anthropic clients that send their native `id` (returned in the SSE message_start event) for cancel had no way to hit the registry. Add message_id to the fallback list. Verified via temp/pr_simulation/sim_5069_prefill_cancel.py: P2 now cancels by cancel_id in 137ms (was hanging pre-fix), and the new P2b case cancels by message_id in 77ms. P1 (OpenAI) and P3 (standard chat) still pass with no regression. --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> |
||
|
Wasim Yousef Said
|
208862218d |
feat(studio): training history persistence and past runs viewer (#4501)
* feat(db): add SQLite storage layer for training history * feat(api): add training history endpoints and response models * feat(training): integrate DB persistence into training event loop * feat(ui): add training history views and card grid * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): address review issues in training history persistence - Strip hf_token/wandb_token from config before SQLite storage - Add UUID suffix to job_id for collision resistance - Use isfinite() for 0.0 metric handling throughout - Respect _should_stop in error event finalization - Run schema DDL once per process, not per connection - Close connection on schema init failure - Guard cleanup_orphaned_runs at startup - Cap _metric_buffer at 500 entries - Make FLUSH_THRESHOLD a class constant - Map 'running' to 'training' phase in historical view - Derive LR/GradNorm from history arrays in historical view - Fix nested button with div[role=button] in history cards - Guard String(value) against null/undefined in config popover - Clear selectedHistoryRunId on auto tab switch * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): address round-2 review findings across training backend and frontend Backend (training.py): - Move state mutation after proc.start() so a failed spawn does not wedge the backend with is_training=True - Create DB run row eagerly after proc.start() so runs appear in history during model loading, not after first metric event - Rewrite _flush_metrics_to_db() with snapshot-before-insert pattern to preserve metrics arriving during the write and retain buffer on failure - Guard eval_loss with float() coercion and math.isfinite(), matching the existing grad_norm guard - Increase pump thread join timeout from 3s to 8s to cover SQLite's default 5s lock timeout Frontend (studio-page.tsx): - Fix history navigation: check isTrainingRunning instead of showTrainingView in onSelectRun so completed runs are not misrouted - Replace activeTab state + auto-switch useEffect with derived tab to eliminate react-hooks/set-state-in-effect lint violation Frontend (historical-training-view.tsx): - Add explicit "running" branch to message ternary so running runs no longer fall through to "Training errored" - Derive loading from detail/error state and move cleanup to effect return to eliminate react-hooks/set-state-in-effect lint violation Frontend (progress-section.tsx): - Derive stopRequested from isTrainingRunning && stopRequestedLocal to eliminate react-hooks/set-state-in-effect lint violation and remove unused useEffect import * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): resolve 3 remaining bugs from round-2 review 1. Stuck on Current Run tab [12/20]: Only force "current-run" tab when isTrainingRunning is true, not when stale completed-run data exists. After training ends, users can freely navigate to Configure. 2. Incomplete metric sanitization [7/20]: Apply float() coercion and isfinite() guards to loss and learning_rate, matching the existing pattern used by grad_norm and eval_loss. Prevents TypeError from string values and NaN leaks into history arrays. 3. Stop button state leak across runs [10/20]: Add key={runtime.jobId} to ProgressSection so React remounts it when a new run starts, resetting stopRequestedLocal state. * fix(studio): deduplicate loss/lr sanitization in training event handler Reuse _safe_loss/_safe_lr from the progress update block instead of re-sanitizing the same raw event values for metric history. * fix(studio): restore loss > 0 guard to prevent eval steps injecting 0.0 into metric histories Round-2/3 fixes relaxed the history append guard from `loss > 0` to `loss is not None`, which let eval-only log events (where loss defaults to 0.0) append fake zeros into loss_history and lr_history. Restore the `loss > 0` check to match the worker's own has_train_loss gate. The float() coercion and isfinite() sanitization from round-3 remain intact. * fix(studio): resolve training history bugs — nullable loss/lr, tab nav, sparkline * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> |
||
|
Roland Tannous
|
a2baf80511 | Update license headers | ||
|
Roland Tannous
|
d882678fe4 | Add AGPL-3.0 SPDX headers to all source files | ||
|
Shine1i
|
43c66da783 | merge nightly | ||
|
Shine1i
|
85653237ea | feat: add Data Recipe core functionality with job manager, API routes, and validation services | ||
|
sshah229
|
40bfe42974 | added the pydantic models and routes for export | ||
|
sshah229
|
50ff5626f1 | refactored the code for username/password and added pydantic models and routes for the same | ||
|
Roland Tannous
|
75bb6c08a5 | Add datasets check-format endpoint | ||
|
Roland Tannous
|
b4ec0389f0 | refactor/inference-api-routes-part-1 | ||
|
Roland Tannous
|
544d6944d1 | root studio folder |
Renamed from backend/routes/__init__.py (Browse further)