From 27b6d553fe9dcd5a55d6e9e005825c7842e7c014 Mon Sep 17 00:00:00 2001 From: Eyera Date: Tue, 21 Jul 2026 07:53:22 +0200 Subject: [PATCH] Feat/model picker per model config v2 (#7207) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * refactor(studio): move chat model picker into features/model-picker Relocate model-selector + its support files from components/assistant-ui into a self-contained features/model-picker feature (own barrel), mirroring the modular Hub layout. Pure move + import repoint; no behaviour change. * feat(model-picker): add per-model config persistence layer Superset PerModelConfig (customContextLength, kvCacheDtype, speculativeType, specDraftNMax, tensorParallel, chatTemplateOverride, trustRemoteCode) persisted to localStorage (unsloth_model_configs) with schema versioning + LRU budget. KV-dtype and speculative value sets match main's sidebar (no q4_0/ngram-simple). Reuses features/hub/lib/model-identity for normalization; adds storage-key layer and applyPerModelConfigToRuntime (sets tensorParallel, which the old PR omitted). * feat(picker): modular backend for chat-template validate + default fetch New studio/backend/picker package (schemas/service/routes) mounted at /api/picker: - POST /api/picker/validate-chat-template (Jinja syntax validation, no false positives) - GET /api/picker/chat-template/{model_name} (default template from tokenizer_config.json, reusing get_cache_path/resolve_cached_repo_id_case; graceful null, no model-code exec) Frontend api/templates.ts client + hooks/use-model-defaults lazy cache. No backend changes to the existing inference load route (per-model load fields already supported). * feat(model-picker): bind picker on-device list to shared hub inventory Picker now sources cached + local models from useHubInventory (the Hub's shared store) via a thin adapter, replacing its own /api/models/* fetchers + module caches. Hub, download manager, and picker now share one source of truth, so completed downloads reflect in the picker automatically. Partial/live-download rows are filtered from the cached lists (unchanged rendering). Local naming/search preserved via additive LocalInventoryRow modelId/displayName. Variant expander, scan-folder management, recommended-fit, search, external providers untouched. Known minor: cached 'Downloaded date' sort tiebreak degrades to alphabetical (hub cached rows carry no mtime); default 'recent' (load-time) sort preserved. * feat(model-picker): per-model config step inside the picker Picking a (non-external) model now opens an in-picker config view built from main's current load controls (context length, KV cache dtype, speculative decoding, draft tokens, tensor parallel) plus a chat-template editor backed by the picker validate/default endpoints. 'Remember for this model' persists the config per model+variant; Run forwards the config to the existing load flow via meta.config. External models bypass the step. Two-view orchestration lives in model-selector (single interception point); pickers.tsx call sites untouched. trustRemoteCode dropped from PerModelConfig to preserve main's per-load consent. * feat(chat): apply/persist per-model config through the load flow handleCheckpointChange threads meta.config into the selection; stageOrLoad and the autoload/Hub-run paths now apply the picker config (explicit pick or saved remembered config) via applyPerModelConfigToRuntime before staging/loading, with keepSpeculative set so a remembered speculative mode survives the model switch. Replaces the old remembered-load-settings seeding (resolveInitialConfig now the single source). SelectedModelInput carries config. * refactor(chat): remove per-model load config from the right sidebar The load knobs (context, KV cache, speculative, draft tokens, tensor parallel) and the chat-template editor now live only in the picker config step. The sheet's Model section keeps the staged Load/Cancel flow (config is applied at pick time); sampling params, system prompt, and RAG are unchanged. Deletes the superseded remembered-load-settings module + the store's applyRememberedLoadSettings action, removes the now-dead sheet state/imports, and points the settings reset at unsloth_model_configs. Delete-cleanup deferred (stale config is LRU-capped). * fix(model-picker): remove leftover sidebar-staging cogwheel + empty Model section The downloaded-variant gear (ModelLoadSettingsAction) staged a model straight into the right-sidebar Run-settings flow -- the old 'configure before load' path now fully replaced by the in-picker config step. Removed the gear + its component. Also gate the sheet's 'Model' section to staged picks only (pendingSelection): after the load-knob strip its content is staged-only, so it was rendering an empty section header whenever a model was merely loaded. * chore(chat): remove dead per-model-config setters + modelControlsDisabled After the load-config UI moved into the picker, the store's per-model setters (setKvCacheDtype/setSpeculativeType/setSpecDraftNMax/setTensorParallel/ setCustomContextLength/setChatTemplateOverride) had zero callers (applyPerModelConfigToRuntime writes via setState), and the sheet's modelControlsDisabled was unreferenced. Verified dead across the whole tree. * fix(chat): config-step Load actually loads (ignore Load-on-selection) Root cause: with Settings > Chat > 'Load on selection' turned OFF, the config step's load went down the deferred-staging path -- opening the right sidebar with ' is staged, not loaded yet / Choose Load model'. The in-picker config step IS the deliberate load action, so its Load now loads immediately (or downloads + auto-loads when not cached) regardless of the toggle. Renamed the button 'Run model' -> 'Load model' to match. Native/dropped picks still honor the toggle. * refactor(chat,hub): retire 'Load on selection' — config step is the only load flow The in-picker config step (and the Hub Run button) now fully supersede the old stage-to-sidebar flow, so the Load-on-selection toggle is removed everywhere: - chat stageOrLoad: every pick loads immediately, or downloads + auto-loads when not cached (the previous default behaviour, now universal). - hub Run: drops the stage branch; downloaded GGUFs load directly with their saved per-model config (no collision with the chat config step — both end at selectModel). - store: removed loadOnSelection field/setter/key/default; Settings>Chat toggle and its settings-reset entry removed. - staged sidebar section is now a download-progress view (auto-loads on completion). No manual staging remains; stageModel is used only for background auto-load downloads. * feat(model-picker): default chat template from GGUF + thread variant through config flow Read the embedded tokenizer.chat_template from GGUF files (read_gguf_chat_template in gguf_metadata) and use it as the per-model default. Plumb gguf_variant through the picker service, /api/picker/chat-template route, frontend templates API, and use-model-defaults so the right variant's template is fetched. Also refine the picker config-page/model-selector wiring, drop the dead ggufNativeContextLength runtime path, and add the per-model-config storage keys to the settings prefs export. * feat(model-picker): read safetensors chat template + hide editor where it has no effect Resolve the default chat template for safetensors models: prefer the modern chat_template.jinja, fall back to the tokenizer_config.json chat_template field, then chat_template.json (multimodal processor), then the GGUF embedded template. Applied to local dirs, the HF cache snapshot scan, and the HF remote fetch. Hide the chat-template editor in the picker for safetensors models — the override is only applied at load by the GGUF/llama.cpp backend, so editing it on safetensors currently has no effect. GGUF keeps the editor. Nothing removed; the dialog stays for when the safetensors apply path is wired up in a later branch. * fix(model-picker): set legacy-migration flag only after the write succeeds Set unsloth_model_configs_migrated only once writeMap confirms the migrated map persisted, so a quota/storage failure no longer marks migration done and silently drops the user's pre-existing remembered settings — the next load retries. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * MVP model picker fixes * MVP picker config fix * MVP safetensors config * MVP max seq config * MVP max seq fix * Fix static max tokens cap ignoring model context * Fix picker GGUF scan parity * fix(studio): harden model picker config loading Apply remembered per-model configs consistently from picker and Hub loads, keep default configs from overriding standing speculative settings, add config access for direct local GGUF files, and support saving or forgetting active model settings without a reload. * Fix model picker config flow * Fix model picker config loads * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Avoid recursive per-model config migration reads * Apply the displayed context length when loading a GGUF * Fix template validation, cached template lookup, and failed load rollback - Validate chat templates with the loopcontrols extension so templates that use break or continue tags pass the picker validator, matching the inference renderer that already accepts them. - Read the default chat template from the newest cache snapshot rather than an arbitrary iterdir order, so an older cached revision no longer prefills a stale template. - Capture the runtime per-model config before a load and reapply it when the load fails, so a failed switch leaves the active model context, KV cache, template, and speculative settings as they were. * Make chat template view only for safetensors models Custom chat template overrides are applied at inference only for GGUF models, which pass the template to llama-server. The safetensors backend renders with the model built-in template and ignores the override, so editing it would save a value that never loads. For safetensors the config page now opens the template as a read-only preview with a note that editing is not available yet. This can become editable once inference support for custom safetensors templates lands in main. * Fix model picker config edge cases - Restore prior runtime config when a load no-ops for the active model - Cap the picker validator request body via the protected prefixes - Keep the GGUF context slider max above the loaded context - Fetch subfolder chat templates for uncached Hub repos - Show the compare side config when reopening the picker * Keep saved GGUF context above the fallback ceiling * Show the model config in the run settings sidebar * Fix model config sidebar reset and context slider - Stack the remember toggle and action buttons in the sidebar - Reset the config to defaults instead of the loaded values - Fetch the native context so the slider max is not the loaded value * Fix model picker config and download regressions - Run picker chat template routes off the event loop - Depth and root guard local template directory scans - Restore download manager flow for uncached hub picks - Apply per model context length on reload - Import model picker symbols from the feature barrel * Fix model picker config and cached download sorting - Restore load settings when a Hub run is rejected mid load - Reuse one NumericValueInput instead of a duplicate copy - Fix double decode of the model name in the template route - Remove the unused reset-to-loaded settings action - Fix cached model download sorting * Fix model picker per-model config edge cases Honor a saved or typed max seq length above the model's native context so RoPE extended values are no longer clamped and silently overwritten. Allow typing past native while the slider keeps native as a soft ceiling. Guard the fetch success paths in use-model-defaults against an aborted signal, and refetch when the HF token changes. Hash the chat template content in the sidebar remount key instead of its length. Enable reset for a GGUF whose native context is unknown, and floor the context slider max so it can never fall below the min. * Fix GGUF context auto-fit and gated model config token Stop forcing a 32768 context when a GGUF native context is unknown so the backend auto-fits to VRAM again, while still honoring an explicit context edit. Send the HF token as a query param so gated safetensors models resolve their max position embeddings. Derive model default state during render to drop the set-state-in-effect calls. * Fix native GGUF context ceiling and guard picker template reads Restore the native context store field so the sidebar slider keeps the full ceiling for drag and drop GGUFs. Limit local chat template reads to the browse allowlist, skip malformed repo ids, and drop unused model picker exports. * Fix model picker lint boundaries * Fix model picker review findings Chat template editor never seeded its draft. Radix only calls onOpenChange from internal events, so the seed in the nextOpen branch was dead and a model with a saved override opened empty. Saving then cleared the override. Drop the dead branch, treat draft as an untouched sentinel, and reset it on every close. Uncached Hub picks could auto load a model after the user left the chat. Main detached the staged pick on route exit and on chat context change. Carry the context key on the pending pick and skip the load when it no longer matches. Also clear configTarget when the picker closes, restore the onUpdated ref so variant rows stop resubscribing on every parent render, skip the LRU write when the entry is already most recent, import NumericValueInput relatively, and drop the unused ModelUpdateAction barrel export. * Preserve GGUF context on active reload * Fix model picker per-model config regressions - Stop reloading the already loaded model on re-pick - Hide infra models from the chat picker - Detect vision support on cached GGUF repos - Honor saved maxSeqLength on auto load - Restore default chat template for local GGUFs - Warn on save failure and revert config on cancel - Refetch picker inventory on open - Persist read only per model config safely * Fix stale model auto load * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix model picker numeric input sizing and constraints Size value inputs to their content so long context lengths are not clipped, restrict them to numeric characters, and stop the speculative decoding label from truncating in the sidebar. * Fix picker CI tests and harden chat template resolution for PR #6647 - tests: point the descender guard at the moved model-selector.tsx path - tests: exclude the disabled Reload model button from the regenerate locator so .first targets the real Regenerate - picker/service.py: reject symlinked template/gguf leaves that resolve outside the browse allowlist (HF cache reads unchanged) - compare mode: resolve each pane's own remembered chat template instead of inheriting the other pane's from the store * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Protect future-schema per-model configs from deletion for PR #6647 savePerModelConfig already refuses to overwrite a stored config whose schema version is newer than this client understands, but deletePerModelConfig did not. Unchecking Remember on an older client therefore silently destroyed a newer client's saved config. Apply the same guard on delete and surface the blocked case through the existing saveFailed toast. * Protect future-schema per-model configs from quota eviction for PR #6647 The save and delete guards already refuse to touch a stored config whose schema version is newer than this client understands, but the quota-eviction path did not, so a full store on an older client could still evict a newer client's config. Skip future-schema entries when evicting and fail the save if the budget cannot be met without them. * Fix GGUF context persistence, compare context, and rollback settings for PR #6647 Persist a GGUF context override from the user's intent instead of collapsing it against the loaded context, which reintroduced the context-reset (f4838782cb reverted the native-baseline fix). model-config-page now collapses the saved value against native, and use-chat-model-runtime and chat-adapter retain the requested context on load so re-saving another setting keeps the override; a null request stays null so a VRAM auto-fit never becomes a stored override. shared-composer: a compare pane with no explicit GGUF context now loads at native (0) like single-view, not the session maxSeqLength that silently shrank the shown context. use-chat-model-runtime: restore the previous model's KV cache dtype and chat template on a failed-load rollback so it runs as it was, not with backend defaults. * Preserve native path token when reloading the active model for PR #6647 handleReloadActiveModel rebuilt the selection without the store's activeNativePathToken, so reloading a file-picked GGUF after a settings change validated the display label as a repo/path and failed. Thread the active native token through the reload selection so native-loaded models reopen correctly. * Make picker template validation resilient and accept HF generation tags for PR #6647 Import Jinja lazily inside validate_chat_template so a backend without the optional jinja2 package (GGUF-only installs) still starts instead of raising ModuleNotFoundError at import time. Register a no-op extension for the Transformers {% generation %} assistant-mask tag so pasting a valid HF chat template validates, matching the renderer, rather than being rejected as an unknown tag. * Honor remembered compare config and parse processor chat_template.json for PR #6647 * Fix failed-load rollback context and processor template map fallback for PR #6647 * Restore speculative decoding config on failed-switch rollback When a model switch fails after the previous model was unloaded, the rollback reload restored tensor_parallel, KV cache dtype and the chat template override, but omitted speculative_type and spec_draft_n_max and cleared their loaded shadows to null. The previous model therefore came back running at backend defaults (speculation off) while the UI still showed it enabled, and the status resync confirmed the off state. Resend the previous model's speculative settings in the rollback load and keep the store's active and loaded speculative fields in sync with them. * Reset max sequence length when a model has no saved config applyPerModelConfigToRuntime reset every per-model field except maxSeqLength, which it only wrote when the incoming config had one. maxSeqLength is the sole field carried on store.params, so selecting a model with no remembered config left the previous model's value in place and later loaded the new model at that leaked length. Fall back to the standing default so an unremembered model loads at its own default. * Surface a message when a variant update cannot start startManagedUpdate handled the conflict and error start outcomes but let busy fall through as if the update began, so the confirm dialog closed with no job created and the cached variant stayed stale. Show an info message when the repo is busy with a sibling transfer so the click is not silently dropped. * Keep per-model speculative choices out of the global default A staged load with a per-model or one-off config sets keepSpeculative, which already skips reading the global speculative preference. The matching save still ran unconditionally, so the model-specific choice was written to the global unsloth_chat_speculative_type and a later model with no saved config started from it instead of Auto. Skip saveSpeculativeType when keepSpeculative so the per-model choice stays isolated. * Seed non-active model settings from the app default max length The Run settings page captured initialMaxSeqLength from the loaded model's runtime params and fell back to it for a model with no saved config. Opening settings for a different, unloaded model and clicking Load then sent the active model's context (for example 64k) instead of the 4096 default, risking validation failures or OOMs. Seed the default for non-active models and keep the runtime value only for the active one. * Prefer sidecar tokenizer chat template over the GGUF copy for variants _chat_template_from_dir returned the embedded GGUF template first when a variant was selected, reversing the tokenizer-first precedence of the no-variant path. A model whose chat_template.jinja or tokenizer_config.json supersedes a stale embedded template then got the wrong template on variant selection. Keep tokenizer files first regardless of variant; the variant only picks which GGUF is the fallback. Adds regression tests for both the tokenizer-wins and gguf-fallback cases. * Keep per-model speculative choices load-local in autoload and compare The interactive load path treats a per-model speculative choice as load-local and skips writing it to the global default. Autoload and generalized compare still called saveSpeculativeType unconditionally, so a remembered off or ngram setting leaked into unsloth_chat_speculative_type and later models with no saved config inherited it. Persist the global preference only when the value came from the global settings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: record the compare pane's loaded context in runtime state so the active model's settings and any reload or save use it, not the previous context * Studio: notify the user when a Hub autoload can't start because another download for the model is already running, instead of silently dropping it * Studio: drop the merge's orphaned staged-model store helpers and unused alert imports The main merge left isPendingGguf and pendingSelectionMatches referencing the removed PendingModelSelection type, and the alert-dialog/alert imports unused after the permission-mode dropdown replaced the bypass dialog, so tsc -b failed. * Studio: cache a null default chat template so the viewer stops re-fetching it A model with no sidecar or embedded template resolves to a terminal null, but that result was never cached, so reopening the template viewer re-ran the backend and Hugging Face lookup every time. * Studio: detect direct-file GGUFs in run settings so Max Tokens uses their context A GGUF loaded from a local file or custom folder has no variant label, so the run-settings panel treated it as non-GGUF and clamped Max Tokens to the session max_seq_length instead of the loaded GGUF context. Detect it via the reported GGUF context and the .gguf checkpoint suffix, matching the chat page. * Studio: prompt to re-select a local model file when its lease expired before reload A file-picked GGUF is reachable only through a native path token that the desktop host prunes after a TTL. Reloading reused that token blindly, so a reload long after the initial load failed with an opaque error. Track the token's expiry and, when it has passed, ask the user to re-select the file instead of attempting a doomed reload. * Fix descender-clipping test to tolerate sidebar layout utilities The sidebar account-block div carries layout utilities (min-w-0, flex-1) between 'flex' and 'flex-col', so the descender-clipping guard's regex, which required 'flex' immediately followed by 'flex-col', no longer matched and the test failed to locate the account-block div. Generalize the prefix to allow intervening flex utilities while still capturing the leading-* class before the collapsible visibility utility and asserting leading-tight, so the guard against clipped glyph descenders is fully preserved. * Harden picker chat-template resolution Enforce the 64 KiB chat-template contract at the validate endpoint's request model so a direct caller cannot submit a template far larger than the frontend allows (MaxBodyMiddleware only bounds the whole request body, not this field); oversized templates now return a clean 422. Apply sidecar-over-GGUF template precedence globally across cached snapshots instead of per snapshot. A repo with multiple cached revisions previously returned the first snapshot's template, so a newer GGUF-only revision could win over an older revision's maintained chat_template.jinja sidecar, which contradicted the documented intent that sidecars supersede the embedded copy. * Guard per-model config against future-schema and lossy migration Two forward-compatibility gaps in the versioned per-model config store: - The load/apply path returned and normalized a stored record without checking its schema version, so a record written by a newer client was reinterpreted under the current schema and applied to a live model load, even though save, delete and eviction all refuse to touch future-schema records. Reject future-schema records on load too. - The one-time legacy migration enforced the storage budget without protecting the entries it had just migrated and set the completion flag unconditionally. When storage was already full of future-schema records (which are unevictable by an older client), the migrated entries were the only evictable ones and could be dropped while migration was still marked complete. Protect the migrated keys during eviction and only mark migration complete when they survive, so it retries once space frees up. * Discard chat-template validation results after the dialog closes Server-side template validation is async, but closing or cancelling the editor did not abort it, so a late-arriving valid response still called onSave and applied a template the user had already dismissed. Track a validation token that is bumped on close and ignore any validation result whose token is stale. * Record native lease expiry when loading a picked GGUF from the chip The pending-native-model chip loaded via stageOrLoad directly, bypassing loadNativeModelIntent, so activeNativePathExpiresAtMs was never recorded for a chip-loaded file. A later reload then either skipped the lease-expiry guard entirely (expiry left null) or compared against a previously loaded file's stale expiry, so reload could reuse an already-pruned token or wrongly block a still-valid one. Route the chip through loadNativeModelIntent, which builds the same selection and records the expiry. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Prefer sidecar template for a directly selected local GGUF file A direct .gguf file path read its embedded chat template without checking the parent directory for a maintained sidecar (chat_template.jinja / tokenizer_config.json), while directory and variant selections already prefer the sidecar. That let the config editor preview or save a stale embedded template for the same model depending on how it was selected. Check the parent directory sidecars first, then fall back to the embedded copy, and cover both paths with tests. * Resolve cached chat template per revision, newest first The earlier change searched every cached snapshot for a sidecar before considering any snapshot's embedded GGUF template, which let an obsolete sidecar from an older revision override the newest revision's template. Restore per-snapshot resolution (newest first): a revision's sidecar still supersedes its own embedded GGUF copy, but a newer revision is no longer overridden by an older revision's sidecar. * Preserve autoload transport conflicts and surface background busy downloads - When a Hub autoload hits a transport conflict, keep pendingHubAutoLoad bound instead of clearing it. Clearing it re-keyed the download surface and its cleanup cancelled the conflict the toast tells the user to resolve, so the Hub resume affordance was gone the moment it appeared. Return early on conflict, mirroring the started branch, so resolving it from the Hub still auto-loads on completion. - The background-download branch handled started and conflict but silently dropped a busy outcome, leaving the user with no feedback when a peer variant of the same repo was already downloading. Surface the same busy toast the autoload path uses. * Fix context length, GGUF template, fetch state and lease expiry bugs Keep explicit context length values instead of collapsing to null at native. The collapse made the slider jump back at the native maximum and made Reload load the previous context instead of the chosen one. Prefer the first split when resolving a GGUF without a variant. Later splits carry no chat template metadata, so picking the largest file could return no template for a sharded model. Clear stale fetch state when template and metadata lookups retry, so a previous terminal error is not shown while a new fetch is running. Record native path lease expiry together with the token when a load commits. The expiry was written by only one load path and even when the load did not start, so a reload could be blocked with an expired file message for a still valid token. * fix(model-picker): resolve review findings across config, inventory, and templates - Apply remembered per-model config in the training-compare chat handoff so a prior model's customContextLength no longer leaks into the next load - Match GGUF variant labels with the inventory extractor too, so cached no-quant-token files resolve their default chat template - Show "Auto" instead of a fabricated 32768 when native context is unknown - Reuse the identical staged auto-load object on same-pick so a re-pick during download pre-flight no longer disarms auto-load via "busy" - Union supports_vision when deduping cross-cache inventory rows - Serve hidden-model needles from a new GET /api/hub/hidden-models endpoint and merge them client-side, covering runtime-configured RAG embedders - Clamp GET chat templates to MAX_CHAT_TEMPLATE_BYTES (route + jinja sidecar), matching the validate endpoint's contract - Lower-clamp stored customContextLength to shared CONTEXT_LENGTH_MIN - Wipe unsloth_chat_load_on_selection in Settings "Reset all" - Drop stale pendingHasContext comment describing deleted staging machinery * Fix stale defaults cache, token in query string and rounded up context ceiling Refresh cached chat template and max position data when a model update completes. Send the HF token for model config requests in the dedicated header instead of the URL. Snap the native sequence length ceiling down to the nearest step so the slider cannot exceed the declared maximum. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix compare pane reverting active checkpoint on non-GGUF load Re-read runtime params after setCheckpoint so the fresh checkpoint is kept instead of being overwritten by the pre-setCheckpoint snapshot. * Send the HF token via header for the vision and embedding checks checkVisionModel and checkEmbeddingModel still passed the HuggingFace token as a ?hf_token= query parameter, so it landed in server access logs, proxy logs, and browser history. Move them to the X-Unsloth-HF-Token header like getModelConfig already does, and accept the header on the check-vision and check-embedding routes with the existing query parameter kept as a fallback for older clients. * Cap the chat template on the model load path The load endpoint accepted an unbounded chat_template_override, so a direct caller could hand llama.cpp an arbitrarily large Jinja template even though the frontend, the validate endpoint, and the read paths all enforce the 64 KiB limit. Reuse MAX_CHAT_TEMPLATE_BYTES in the LoadRequest validator, rejecting oversized templates with a fast character-count check before the exact UTF-8 byte check. * Protect existing per-model configs during legacy migration When the one-time legacy import pushes the store over budget, eviction now protects the entries the user already has and drops only the just-migrated legacy entries, so importing old load settings can never discard a newer per-model config. * Reset clears the context override instead of pinning the native value Reset wrote the discovered native context into customContextLength for GGUF models, but isDefaultConfig treats any non-null customContextLength as an explicit pin, so Reset with Remember enabled persisted a fixed context and future loads stopped using the native auto context. Reset now restores the full default (customContextLength null); the native value is still shown through the existing display fallback. * Bound chat-template sidecar reads to a size limit The chat_template.json, tokenizer_config.json, and Hub-downloaded sidecar readers decoded and json-parsed the whole file before the extracted template hit the 64 KiB response cap, so an oversized metadata file could exhaust memory. Read them through a bounded reader (4 MiB envelope) that returns None when the file is larger, matching the existing chat_template.jinja size guard. Adds tests for oversized tokenizer_config.json and chat_template.json. * Keep the native-path token and lease expiry in sync Rollback after a failed reload restored the previous token but left the failed load's expiry in the store, so a later reload could be falsely blocked as expired (token A paired with load B's lease). Restore the previous lease alongside the token, and clear the expiry wherever the token is cleared on a non-GGUF transition, so the two never diverge. * Clear the native file lease on compare-pane loads * Studio: add regression tests for the model-picker per-model-config Guard the specific regressions that reverted the predecessor change: - backend pytest (studio/backend/tests/test_model_picker_regression.py): infra-model hiding, HF token via header with query fallback, and the chat-template byte caps. - source contracts (tests/studio/test_model_picker_contracts.py): the token stays out of the URL, the context ceiling is floored, the native lease is cleared on compare-load and restored on rollback, the default caches key on the inventory version, and the hidden needles stay present. - Playwright E2E (tests/studio/playwright_model_config.py) wired into studio-ui-smoke.yml on port 18898: Context Length persists across a reload, Reset clears the stored override, and infra models are absent from the picker. - optional GPU-gated inference smoke (tests/studio/test_gpu_inference_smoke.py) that auto-skips on GPU-less CI and stays short on a GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: model pinning, row menus, hub inference settings, and inventory filters Pinning - Add a pinned models store (localStorage) with repo and per-quant pins - Pinned section in the model selector's On Device list and the hub inventory, with newest pins first so Pin to top lands on top - Deleting a repo drops its pins Row menus - Replace loose row icons with a shared 3-dots menu (pin, reveal in file manager, copy identifier, copy path, delete) on picker rows, hub quant rows, the hub run bar, and on-device inventory rows - Menus only render for models actually on disk; platform-aware reveal labels - Backend: cached-model-path and reveal-cached-model endpoints resolving managed HF-cache repos only Hub inference settings - Gear in the GGUF run bar opens an Inference settings dialog reusing the chat page's controls: model config (context length, KV cache, speculative decoding, chat template), system prompt, reasoning, sampling, tools and retrieval Inventory - Model-type filter (text, vision, embedding, STT, TTS, diffusion) beside the sort pill, both with a sort icon, capped widths and truncation so the On device heading never wraps - Unsloth-owned repos without an upstream provider logo fall back to the Unsloth mascot avatar - Discover / On Device tabs widened; hub search bar narrowed to match * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: revert the Unsloth mascot avatar fallback Unsloth-owned repos without an upstream provider match go back to the colored-initial tile, and unslothai is no longer a relabeled owner. * Studio: run-bar options on single models, and aligned type/capability filters - Give single-model (non-GGUF) run bars the same 3-dots options menu and settings gear as GGUF, at repo level - Drop Pin to top from the run-bar menus; pinning stays in the On Device list - Add an Image to text (diffusion) capability with detection, and surface it in both the hub Discover capability filter and the On Device type filter - Align the On Device type filter with the Discover capability options and share the same detection so both dropdowns match * Studio: apply hub inference config on reload, eject action, and run-bar polish - Fix inference settings not applying: the hub dialog now writes the config to the runtime before reload, matching the chat page (selectModel reads runtime state, not the selection) - Order the settings gear before the 3-dots menu in the run bars - Replace the loaded-model run-bar action (New Chat) with Eject, wired through the inspector to the hub's ejectModel - Truncate the results heading so a long search query clips instead of overlapping the header pills in split view - Use a plain magnifying-glass icon for the no-results empty state * Studio: fix GPU settings loss, load guards, pins, filters, and cached paths Reloading a model from the chat sidebar or the hub gear dialog rebuilt the per-model config without the GPU memory fields, so manual GPU layers, MoE placement, and the GPU pick were reset on every reload and could be saved over a remembered config. The active config now comes from a shared useActiveModelConfig hook that carries the GPU fields for GGUF models, and the sidebar remount signature tracks them through a shared gpuFieldsSignature helper. The in-flight load guard lived in a ref inside each useChatModelRuntime instance, so the chat page, hub page, and gear dialog could not see each other's loads. A load started from the gear dialog left the hub page free to eject the model mid-reload or start a second concurrent load. The runtime store now records the loading pick, selectModel checks it across instances, and ejectModel refuses with a toast while any load is in flight. The cached-model-path endpoint matched GGUF files by basename and excluded only mmproj, so Copy path and Reveal could return an MTP drafter for a quant and returned 404 for directory layouts like BF16/model-00001.gguf. Variant files are now resolved from snapshot-relative paths with the same drafter, mmproj, and big-endian exclusions as the load path, shared through a new _main_variant_gguf_label helper. Hub and picker fixes: - rename the diffusion capability label from "Image to text" to "Image generation", since it detects image generators - validate pinned quants through the cached variant listing, keep the last verified set while revalidating, and drop deleted quants immediately - pass a measured scroll margin to the on-device virtual list so rows past the overscan stay visible below the pinned block - keep the delete menu for stopped partial safetensors downloads - give the inventory type filter a reset in Clear filters, a truthful empty state with a Show all types action, and hide it on the datasets view - order picker pinned rows by pin recency, include pinned matches in the empty-state check, and sync pins across browser tabs - count only the visible rows in the On device list header Tests: contract checks for each fix in test_model_picker_contracts.py and a backend test for the variant label selection. * Studio: reveal cached models in Windows Explorer under WSL The reveal endpoint only branched on macOS, Windows, and generic Linux. Under WSL the Linux branch spawned xdg-open, which is missing on a stock distro without a Linux desktop, so the request failed with a 500 and the UI showed a failed to open file manager error. WSL is now detected with the existing helper and the path is converted with wslpath before opening explorer.exe, selecting the file the same way native Windows does. Directories open directly. When interop is unavailable the old xdg-open fallback still runs. The macOS, native Windows, and native Linux branches are unchanged, and the Tauri app is covered since its hub reveal calls this same local endpoint. Tests: platform guards for the WSL reveal, the interop fallback, and the unchanged native Linux behavior in tests/studio/test_reveal_file_manager.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Adjust model picker row spacing and cogwheel hover consistency * Studio: exact hidden model ids and newest revision cached paths A custom RAG embedder repo was published to the frontend as a basename substring needle, so a generic name like org/model could hide unrelated models in the pickers. The hidden-models endpoint now sends full repo ids that are matched exactly. Copy path and Reveal picked a GGUF variant from an arbitrary cache revision when the same file existed in more than one. The newest revision now wins, matching the whole repo lookup. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix model picker GPU config, metadata, and cache selection Load each compare model with its saved GPU memory mode, GPU layers, CPU MoE layers, and selected GPU IDs. Reconcile saved GPU IDs with the current hardware. Include the active native GGUF path token in metadata checks. Search all Hugging Face cache roots when resolving cached models and select the largest visible cache entry. Remove obsolete barrel exports and the staging-only GPU memory helper. * Studio: hide hub inference settings gear for now The cogwheel in the hub download cards is out of scope for this PR. The dialog component stays in place and a TODO marks where the button returns in a future PR. * Refresh hidden model matchers * Fix GGUF detection, compare context pin, and picker delete staleness Treat any pick with a GGUF variant as GGUF in selectModel so the first load after downloading an uncached quant validates and sizes with the right GPU settings instead of unloading the current model on a wrong preflight. Variant picks now also set isGguf on their selection meta. Stop compare panes from inheriting the active model's context pin when their own saved config says Auto. Null context in a remembered config now means no pin, matching how the pane settings are shown. Route picker deletes through the hub inventory client, which invalidates the HF cache scan and the variants cache. The legacy delete route left the scan cache warm, so deleted models reappeared in the picker until the TTL expired. Removed the now unused legacy delete client and updated the contract test to match. * Studio: fix stale GGUF load-marker ordering test The load-in-flight marker still precedes the hub-download guard and the unload, but the llama_extra_args inheritance that used to sit between the marker and the guard now runs ahead of the GGUF branch, so it is no longer a landmark inside the sliced source. Drop it from the ordering assertion and keep the marker -> guard -> unload invariant. * Studio: fix per-model config edge cases in compare loads and saved defaults - chat-settings-sheet: gate the MTP fallback note and context/VRAM warning on the broader isGguf (variant, loaded gguf context, or .gguf suffix) instead of isLoadedGguf, so direct-file and custom-folder GGUF loads still surface those diagnostics. - shared-composer: a compare pane's context now comes from its own config only (a saved pin, else null for Auto/native). It no longer inherits the active model's shared snapshot, which resolveFitMaxSeqLength treated as an explicit pin and could load a pane at another model's context (VRAM/OOM), matching the single-model load path. - model-config-page: when an auto-fit GGUF is saved with fixed GPU layers (Manual) and Remember, pin the displayed fitted context so a later fresh load keeps the placement instead of sending native/0 and recreating the OOM. - per-model-config: treat Auto GPU memory mode and Auto/default speculative type as follow-global defaults; do not persist them as per-model overrides so later global preference changes keep applying. * Studio: gate vision capability on GGUF projectors and bound remote template downloads - cache_inventory: only mark a cached repo vision-capable when it holds an actual GGUF mmproj projector, not any file whose name merely contains "mmproj" (e.g. mmproj_config.json), matching the runtime's GGUF-only projector detection. - picker/service: pre-check the remote file size before downloading an uncached repo's chat template / tokenizer config, so a maliciously large sidecar is skipped instead of fetched and retained in full, mirroring the size gate the local-file path already applies. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add source-contract guards for the per-model-config edge-case fixes Guard the four per-model-config fixes against silent regression in CI: - local GGUF diagnostics gate on the broad isGguf, not the variant-only isLoadedGguf - fixed-layer GGUF saves pin the displayed context - Auto GPU mode and Auto/default speculative are not persisted as per-model overrides - a compare pane's context comes from its own config, not the active model's snapshot * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: clear manual GPU knobs on Default and resolve local embedders before repo-id - model-config-page: switching GPU Memory back to Default now clears the Manual-only knobs (gpuLayers/nCpuMoe/selectedGpuIds); otherwise a remembered config kept stale pins that a later load re-applied when the global GPU preference was Manual, despite the page showing Default. - routes/models hidden_model_matchers: resolve an existing local path before the repo-id regex, mirroring is_hidden_model, so a local embedder shaped like "models/embedder" is hidden by exact path instead of leaking as a chat model. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add _is_mtp_drafter to the model_config stub in the export-paths test routes/models.py imports _is_mtp_drafter from utils.models.model_config at module load, but the lightweight stub in test_export_absolute_paths.py did not provide it, so loading the module under the stub raised ImportError on Backend CI. Add the stub. * Studio: read a picked GGUF's chat template through the native path lease The picker chat-template GET has no native-path-lease plumbing, so a desktop-picked (drag-drop) GGUF could not show its default chat template in Run Settings until the model was loaded: the endpoint only receives the display label, not the leased file path. Read the embedded template through the existing lease-aware /api/inference/validate probe instead. A new include_chat_template flag resolves the granted canonical path and returns the GGUF's own embedded template, never a sibling sidecar (the grant authorizes just that one file); it skips the training guard like include_context_length and is bounded by MAX_CHAT_TEMPLATE_BYTES. The frontend fetch mints a one-shot validate-model lease when a native token is present and keeps the plain GET path for HF and allowlisted local models. Adds backend and source-contract regression tests. * Studio: call worker.direct_wheel_url in the ROCm wheel-url test The ROCm Mamba/SSM test referenced worker.py's private _direct_wheel_url, but the worker imports the wheel helper under its public name direct_wheel_url (utils.wheel_utils). When the worker module loads (its imports resolve in CI), worker_mod._direct_wheel_url raised AttributeError; the test only masked it by skipping when the worker could not be imported. Call the name that actually exists so the assertion runs; it still returns None for an empty cuda_major (ROCm). * Studio: reset max sequence length to the app default, not the loaded value For a non-GGUF active model, the per-model config seeds maxSeqLength from the loaded runtime value so the panel opens showing the running context. Reset set config.maxSeqLength to null, but the null fallback resolved back to that captured runtime value, so the field kept showing the old custom length and the config saved/reloaded it again. A remembered or active max-length override therefore could not be cleared from Run settings. Fall the null/default case back to the app default (clamped to the model's native ceiling) instead of the active runtime snapshot, so Reset actually clears the override. The initial view is unaffected: an active model's config.maxSeqLength is already non-null, so it still shows the loaded value. Adds a source-contract regression guard. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: persist default max length, refresh deleted quants, hide non-chat locals Three follow-up fixes from review of the per-model-config picker: - Max sequence length: the persisted per-model record now keeps config's maxSeqLength (null after Reset) so isDefaultConfig can clear a remembered override; the resolved app-default is substituted only into the load request, never the saved record. Previously Reset saved the concrete default and left the model pinned/remembered. - GGUF variant expander: deleting a downloaded quant from a repo that still has other cached quants now bumps the expander refresh key, so the removed quant stops showing as downloaded and clickable (which would try to reload the deleted file) until the repo is collapsed and reopened. - Local picker rows: require capabilities.canChat before listing a local models-folder / LM Studio row. A weightless folder (only config.json) is classified non-chat, and toLocalModelInfo drops capabilities, so selecting such a row would try to load a path the inventory already marked non-chat. Adds source-contract regression guards for all three. * Fix compare-pane and Reset context defaults in model picker Two related per-model-config default regressions: - A non-GGUF compare pane with no saved maxSeqLength fell back to the active model's shared runtime snapshot, so comparing a saved 128K model against an unconfigured pane loaded the latter at 128K and could OOM. It now falls back to the shared app default (DEFAULT_MAX_SEQ_LENGTH), the same fallback the single-model config path uses. - contextAtDefault treated an explicit customContextLength equal to the native ceiling as a default, which wedged the Reset button disabled for a deliberate pin-to-native. It now counts as default only when there is no override at all. DEFAULT_MAX_SEQ_LENGTH becomes a single exported constant in per-model-config.ts so the single-model config and the compare path share one source of truth. Adds source-contract guards for both fixes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Skip over-cap remote Jinja templates so the tokenizer template wins The remote chat-template resolver bounded raw chat_template.jinja downloads only by MAX_TEMPLATE_METADATA_BYTES (4 MiB), then returned the first non-empty Jinja unconditionally. The picker route drops any template larger than MAX_CHAT_TEMPLATE_BYTES (64 KiB), so an uncached repo whose chat_template.jinja sits between 64 KiB and 4 MiB returned no template at all, even when a valid smaller tokenizer_config.json template existed. The local path already skips oversized .jinja files and falls through. Gate the extracted Jinja on MAX_CHAT_TEMPLATE_BYTES and continue searching when it exceeds the cap, matching _chat_template_from_jinja_file. The 4 MiB download bound stays for JSON files that merely embed a small template. Adds a regression test that a big Jinja plus a valid tokenizer config resolves to the tokenizer template. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard legacy per-model-config migration idempotency The v1->v2 localStorage migration (unsloth_load_settings -> unsloth_model_configs) runs on every store read, so it must migrate exactly once and never re-run, duplicate, or clobber a newer per-model config on a reload or restart. That was covered only by a manual proof, so add durable guards: - Source-contract test pinning the three idempotency layers (the in-memory legacyMigrationChecked guard, the persistent unsloth_model_configs_migrated flag set in every terminal branch, and the non-overwriting Object.hasOwn merge-skip) plus the readMap invocation. Reddens if any layer is dropped. - Playwright model-config E2E: promote the legacy-migration step to a gating check (soft_fail, which gates under the CI STUDIO_UI_STRICT=1) that the migrated value is preserved and the flag is set, then reload again with a fresh legacy seed present and assert the stored key set is unchanged, so a second reload cannot re-migrate, duplicate, or clobber. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Note the migration E2E now gates idempotency under STUDIO_UI_STRICT * Tighten model-picker per-model-config code comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: danielhanchen Co-authored-by: shimmyshimmer Co-authored-by: Unsloth Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> --- .github/workflows/studio-ui-smoke.yml | 50 + studio/backend/hub/routes/inventory.py | 11 + studio/backend/hub/schemas/inventory.py | 7 + .../hub/services/models/cache_inventory.py | 84 +- studio/backend/main.py | 3 + studio/backend/models/inference.py | 24 +- studio/backend/picker/__init__.py | 2 + studio/backend/picker/routes/__init__.py | 6 + studio/backend/picker/routes/templates.py | 45 + studio/backend/picker/schemas.py | 32 + studio/backend/picker/service.py | 426 +++ studio/backend/routes/inference.py | 44 +- studio/backend/routes/models.py | 249 +- .../tests/test_chat_load_during_training.py | 74 + .../tests/test_export_absolute_paths.py | 1 + .../tests/test_model_picker_regression.py | 232 ++ .../tests/test_model_update_robustness.py | 61 + studio/backend/tests/test_picker_service.py | 266 ++ studio/backend/utils/models/gguf_metadata.py | 83 +- studio/frontend/src/app/routes/__root.tsx | 7 - .../frontend/src/components/app-sidebar.tsx | 22 - .../remembered-load-settings.ts | 79 - .../src/features/chat/api/chat-adapter.ts | 99 +- .../src/features/chat/api/chat-api.ts | 25 +- .../frontend/src/features/chat/chat-page.tsx | 513 +-- .../src/features/chat/chat-settings-sheet.tsx | 1348 +------ .../chat/hooks/use-chat-model-runtime.ts | 195 +- .../hooks/use-staged-model-preparation.ts | 169 - studio/frontend/src/features/chat/index.ts | 23 +- .../lib/apply-inference-status-to-store.ts | 14 +- .../src/features/chat/shared-composer.tsx | 198 +- .../chat/stores/chat-runtime-store.ts | 323 +- .../frontend/src/features/chat/types/api.ts | 3 + .../export/components/export-run-panel.tsx | 71 +- .../features/hub/catalog/catalog-states.tsx | 4 + .../hub/catalog/dataset-download-section.tsx | 6 +- .../features/hub/catalog/download-section.tsx | 6 +- .../hub/catalog/gguf-download-card.tsx | 349 +- .../features/hub/catalog/hub-option-menu.tsx | 17 +- .../hub/catalog/local-dataset-card.tsx | 6 +- .../hub/catalog/local-on-device-card.tsx | 119 +- .../features/hub/catalog/model-inspector.tsx | 16 +- .../hub/catalog/models-catalog-lists.tsx | 200 +- .../hub/catalog/models-catalog-rows.tsx | 134 +- .../features/hub/catalog/models-catalog.tsx | 4 + .../src/features/hub/catalog/models-table.tsx | 59 +- .../features/hub/catalog/models-toolbar.tsx | 188 +- .../hub/catalog/on-device-folders-dialog.tsx | 33 +- .../features/hub/catalog/path-info-button.tsx | 155 +- .../hub/catalog/safetensors-download-card.tsx | 85 +- .../hub/catalog/sampling-settings-dialog.tsx | 435 +++ .../src/features/hub/catalog/shared.tsx | 20 +- .../download-manager-controller.ts | 23 - .../features/hub/download-manager/index.ts | 1 - studio/frontend/src/features/hub/hub-page.tsx | 242 +- studio/frontend/src/features/hub/index.ts | 49 +- .../src/features/hub/inventory/api.ts | 2 + .../src/features/hub/inventory/types.ts | 3 + .../hub/inventory/use-device-inventory.ts | 4 + .../src/features/hub/inventory/view-models.ts | 9 + .../src/features/hub/lib/hidden-models.ts | 69 +- .../features/hub/lib/model-capabilities.ts | 28 +- .../src/features/hub/lib/model-type-filter.ts | 52 + .../src/features/hub/lib/view-models.ts | 7 +- .../model-picker/api/model-metadata.ts | 20 + .../features/model-picker/api/templates.ts | 87 + .../chat-template-editor-dialog.tsx | 191 + .../components/model-config-page.tsx | 991 +++++ .../components}/model-selector.tsx | 153 +- .../model-selector/folder-browser.tsx | 90 +- .../model-selector/model-capabilities.ts | 0 .../model-selector/model-delete-action.tsx | 9 +- .../model-load-settings-action.tsx | 24 +- .../model-selector/model-row-menu.tsx | 289 ++ .../model-selector/model-update-action.tsx | 25 +- .../components}/model-selector/model-usage.ts | 3 +- .../components}/model-selector/pickers.tsx | 3335 ++++++++--------- .../components}/model-selector/pill-tabs.tsx | 3 +- .../model-selector/pinned-models.ts | 19 +- .../model-selector/recommended-fit.ts | 10 +- .../components}/model-selector/row-meta.ts | 0 .../components}/model-selector/source-tabs.ts | 0 .../components}/model-selector/types.ts | 14 + .../components/numeric-value-input.tsx | 113 + .../components/sidebar-model-config.tsx | 91 + .../hooks/use-active-model-config.ts | 77 + .../model-picker/hooks/use-model-defaults.ts | 191 + .../src/features/model-picker/index.ts | 39 + .../inventory/use-chat-picker-inventory.ts | 123 + .../model-config/apply-per-model-config.ts | 127 + .../model-config/model-identity.ts | 69 + .../model-config/per-model-config.ts | 665 ++++ .../src/features/settings/tabs/chat-tab.tsx | 39 - .../features/settings/tabs/general-tab.tsx | 4 +- .../src/features/training/api/models-api.ts | 17 +- .../frontend/src/features/training/index.ts | 4 +- tests/studio/install/test_rocm_support.py | 4 +- tests/studio/playwright_chat_ui.py | 14 +- tests/studio/playwright_model_config.py | 740 ++++ .../test_cached_model_path_selection.py | 241 ++ tests/studio/test_gpu_inference_smoke.py | 65 + tests/studio/test_model_picker_contracts.py | 407 ++ tests/studio/test_reveal_file_manager.py | 128 + .../test_studio_text_descender_clipping.py | 9 +- 104 files changed, 10755 insertions(+), 4789 deletions(-) create mode 100644 studio/backend/picker/__init__.py create mode 100644 studio/backend/picker/routes/__init__.py create mode 100644 studio/backend/picker/routes/templates.py create mode 100644 studio/backend/picker/schemas.py create mode 100644 studio/backend/picker/service.py create mode 100644 studio/backend/tests/test_model_picker_regression.py create mode 100644 studio/backend/tests/test_picker_service.py delete mode 100644 studio/frontend/src/components/assistant-ui/model-selector/remembered-load-settings.ts delete mode 100644 studio/frontend/src/features/chat/hooks/use-staged-model-preparation.ts create mode 100644 studio/frontend/src/features/hub/catalog/sampling-settings-dialog.tsx create mode 100644 studio/frontend/src/features/hub/lib/model-type-filter.ts create mode 100644 studio/frontend/src/features/model-picker/api/model-metadata.ts create mode 100644 studio/frontend/src/features/model-picker/api/templates.ts create mode 100644 studio/frontend/src/features/model-picker/components/chat-template-editor-dialog.tsx create mode 100644 studio/frontend/src/features/model-picker/components/model-config-page.tsx rename studio/frontend/src/{components/assistant-ui => features/model-picker/components}/model-selector.tsx (83%) rename studio/frontend/src/{components/assistant-ui => features/model-picker/components}/model-selector/folder-browser.tsx (84%) rename studio/frontend/src/{components/assistant-ui => features/model-picker/components}/model-selector/model-capabilities.ts (100%) rename studio/frontend/src/{components/assistant-ui => features/model-picker/components}/model-selector/model-delete-action.tsx (90%) rename studio/frontend/src/{components/assistant-ui => features/model-picker/components}/model-selector/model-load-settings-action.tsx (63%) create mode 100644 studio/frontend/src/features/model-picker/components/model-selector/model-row-menu.tsx rename studio/frontend/src/{components/assistant-ui => features/model-picker/components}/model-selector/model-update-action.tsx (82%) rename studio/frontend/src/{components/assistant-ui => features/model-picker/components}/model-selector/model-usage.ts (93%) rename studio/frontend/src/{components/assistant-ui => features/model-picker/components}/model-selector/pickers.tsx (58%) rename studio/frontend/src/{components/assistant-ui => features/model-picker/components}/model-selector/pill-tabs.tsx (98%) rename studio/frontend/src/{components/assistant-ui => features/model-picker/components}/model-selector/pinned-models.ts (78%) rename studio/frontend/src/{components/assistant-ui => features/model-picker/components}/model-selector/recommended-fit.ts (91%) rename studio/frontend/src/{components/assistant-ui => features/model-picker/components}/model-selector/row-meta.ts (100%) rename studio/frontend/src/{components/assistant-ui => features/model-picker/components}/model-selector/source-tabs.ts (100%) rename studio/frontend/src/{components/assistant-ui => features/model-picker/components}/model-selector/types.ts (74%) create mode 100644 studio/frontend/src/features/model-picker/components/numeric-value-input.tsx create mode 100644 studio/frontend/src/features/model-picker/components/sidebar-model-config.tsx create mode 100644 studio/frontend/src/features/model-picker/hooks/use-active-model-config.ts create mode 100644 studio/frontend/src/features/model-picker/hooks/use-model-defaults.ts create mode 100644 studio/frontend/src/features/model-picker/index.ts create mode 100644 studio/frontend/src/features/model-picker/inventory/use-chat-picker-inventory.ts create mode 100644 studio/frontend/src/features/model-picker/model-config/apply-per-model-config.ts create mode 100644 studio/frontend/src/features/model-picker/model-config/model-identity.ts create mode 100644 studio/frontend/src/features/model-picker/model-config/per-model-config.ts create mode 100644 tests/studio/playwright_model_config.py create mode 100644 tests/studio/test_cached_model_path_selection.py create mode 100644 tests/studio/test_gpu_inference_smoke.py create mode 100644 tests/studio/test_model_picker_contracts.py create mode 100644 tests/studio/test_reveal_file_manager.py diff --git a/.github/workflows/studio-ui-smoke.yml b/.github/workflows/studio-ui-smoke.yml index 30280c281e..0ad55ebd6d 100644 --- a/.github/workflows/studio-ui-smoke.yml +++ b/.github/workflows/studio-ui-smoke.yml @@ -237,6 +237,54 @@ jobs: kill "${STUDIO_EXTRA_PID}" 2>/dev/null || true sleep 2 + # Model-picker per-model-config regression (PR #7207 re-land of #6647). + # Fourth Unsloth on its own port; loads the tiny GGUF and drives the + # picker's run-settings surface: Context Length persists across a reload, + # Reset clears the stored override (never pins it), and the infra models + # (RAG embedder + llama.cpp probe) stay hidden from the picker. + - name: Reset auth + boot Unsloth for model-config tests (port 18898) + run: | + unsloth studio reset-password + mkdir -p logs + UNSLOTH_API_ONLY=1 unsloth studio -H 127.0.0.1 -p 18898 \ + > logs/studio_modelcfg.log 2>&1 & + echo "STUDIO_MODELCFG_PID=$!" >> "$GITHUB_ENV" + + - name: Wait for /api/health on 18898 + run: | + for i in $(seq 1 180); do + if curl -fs "http://127.0.0.1:18898/api/health" > /tmp/health4.json; then + jq -e '.status == "healthy"' /tmp/health4.json && break + fi + sleep 1 + done + jq -e '.status == "healthy"' /tmp/health4.json + + - name: Pass bootstrap pw for model-config test + run: | + NEW="CIModelCfg-$(python -c 'import secrets; print(secrets.token_urlsafe(16))')" + echo "::add-mask::$NEW" + echo "STUDIO_MODELCFG_NEW_PW=$NEW" >> "$GITHUB_ENV" + + - name: Drive model-picker per-model-config with Playwright + env: + BASE_URL: http://127.0.0.1:18898 + STUDIO_NEW_PW: ${{ env.STUDIO_MODELCFG_NEW_PW }} + PW_ART_DIR: logs/playwright_modelcfg + STUDIO_UI_STRICT: '1' + GGUF_REPO: ${{ env.GGUF_REPO }} + GGUF_VARIANT: ${{ env.GGUF_VARIANT }} + STUDIO_MODEL_HINT: gemma-3-270m + run: | + mkdir -p logs/playwright_modelcfg + python tests/studio/playwright_model_config.py + + - name: Stop fourth Unsloth + if: always() + run: | + kill "${STUDIO_MODELCFG_PID}" 2>/dev/null || true + sleep 2 + # IME + multilingual paste regression (issue #5318 / PR #5327). # Third Unsloth on its own port so a hang here cannot poison the # earlier UI tests. No GGUF -- the bug surface is the composer. @@ -297,12 +345,14 @@ jobs: path: | logs/studio.log logs/studio_extra.log + logs/studio_modelcfg.log logs/studio_ime.log logs/install.log logs/server-logs/ logs/playwright logs/playwright-permissions-* logs/playwright_extra + logs/playwright_modelcfg logs/playwright_ime logs/studio-permissions-*.log retention-days: 7 diff --git a/studio/backend/hub/routes/inventory.py b/studio/backend/hub/routes/inventory.py index 4b6c179a2b..1ffadf0544 100644 --- a/studio/backend/hub/routes/inventory.py +++ b/studio/backend/hub/routes/inventory.py @@ -28,6 +28,7 @@ from hub.schemas.inventory import ( CachedModelsResponse, DeleteCachedModelResponse, GgufVariantsResponse, + HiddenModelsResponse, LocalModelListResponse, ModelsFolderResponse, RecommendedFoldersResponse, @@ -214,6 +215,16 @@ async def list_cached_models( return await cache_inventory.list_cached_models_response(hf_token) +@router.get("/hidden-models", response_model = HiddenModelsResponse) +async def list_hidden_models(current_subject: str = Depends(get_current_subject)): + import asyncio + + from routes.models import hidden_model_matchers + + needles, exact_ids, exact_paths = await asyncio.to_thread(hidden_model_matchers) + return HiddenModelsResponse(needles = needles, exact_ids = exact_ids, exact_paths = exact_paths) + + @router.delete( "/delete-cached", response_model = DeleteCachedModelResponse, diff --git a/studio/backend/hub/schemas/inventory.py b/studio/backend/hub/schemas/inventory.py index ef95efe2f2..19d6da3e11 100644 --- a/studio/backend/hub/schemas/inventory.py +++ b/studio/backend/hub/schemas/inventory.py @@ -160,6 +160,7 @@ class CachedRepoBase(BaseModel): repo_id: str size_bytes: int = 0 cache_path: Optional[str] = None + last_modified: Optional[float] = None partial: bool = False partial_transport: Optional[str] = None inventory_id: Optional[str] = None @@ -189,6 +190,12 @@ class CachedModelsResponse(BaseModel): cached: List[CachedModelRepo] = Field(default_factory = list) +class HiddenModelsResponse(BaseModel): + needles: List[str] = Field(default_factory = list) + exact_ids: List[str] = Field(default_factory = list) + exact_paths: List[str] = Field(default_factory = list) + + class AddScanFolderRequest(BaseModel): """Request body for adding a custom scan folder.""" diff --git a/studio/backend/hub/services/models/cache_inventory.py b/studio/backend/hub/services/models/cache_inventory.py index 54a25482f2..c1b864bb63 100644 --- a/studio/backend/hub/services/models/cache_inventory.py +++ b/studio/backend/hub/services/models/cache_inventory.py @@ -31,6 +31,7 @@ from hub.services.models.common import ( _is_checkpoint_weight_name, _is_gguf_filename, _is_main_gguf_filename, + _is_mmproj_filename, _is_transformers_safetensors_weight_name, _local_inventory_id, _prefer_complete_larger, @@ -132,6 +133,39 @@ def _repo_has_gguf_files(repo_info) -> bool: return _repo_gguf_size_bytes(repo_info) > 0 +def _blob_mtime(file_obj) -> float: + ts = getattr(file_obj, "blob_last_modified", None) + if isinstance(ts, (int, float)) and ts > 0: + return float(ts) + blob_path = getattr(file_obj, "blob_path", None) + if blob_path: + try: + return float(Path(blob_path).stat().st_mtime) + except OSError: + pass + return 0.0 + + +def _repo_gguf_last_modified(repo_info) -> float: + latest = 0.0 + for revision in repo_info.revisions: + for f in revision.files: + if _is_main_gguf_filename(f.file_name): + latest = max(latest, _blob_mtime(f)) + return latest + + +def _repo_has_mmproj(repo_info) -> bool: + # An mmproj file only makes a repo vision-capable when it is an actual GGUF + # projector; a non-GGUF sidecar (e.g. mmproj_config.json) does not, and the + # runtime's projector detection is GGUF-only. + return any( + _is_gguf_filename(f.file_name) and _is_mmproj_filename(f.file_name) + for revision in repo_info.revisions + for f in revision.files + ) + + def _cached_repo_file_name(file_obj) -> str: file_path = getattr(file_obj, "file_path", None) if file_path: @@ -291,6 +325,7 @@ def _scan_cached_gguf() -> list[dict]: continue key = repo_id.lower() existing = seen_lower.get(key) + last_modified = _repo_gguf_last_modified(repo_info) row = { "repo_id": repo_id, "size_bytes": max(total_size, variant_state_size), @@ -300,6 +335,9 @@ def _scan_cached_gguf() -> list[dict]: # per-variant detail lives on GgufVariantDetail. "partial_transport": None, } + last_modified = max(last_modified, (existing or {}).get("last_modified", 0.0)) + if last_modified > 0: + row["last_modified"] = last_modified row.update( _cache_inventory_fields( repo_id, @@ -308,11 +346,20 @@ def _scan_cached_gguf() -> list[dict]: requires_variant = True, ) ) + if _repo_has_mmproj(repo_info): + row["capabilities"]["supports_vision"] = True # Visible infra variants remain management-only. if is_hidden_infra: row["capabilities"]["can_chat"] = False if _prefer_cache_row(row, existing): + if existing and existing["capabilities"].get("supports_vision"): + row["capabilities"]["supports_vision"] = True seen_lower[key] = row + else: + if last_modified > existing.get("last_modified", 0.0): + existing["last_modified"] = last_modified + if row["capabilities"].get("supports_vision"): + existing["capabilities"]["supports_vision"] = True except Exception as e: repo_label = getattr(repo_info, "repo_id", "") logger.warning(f"Skipping cached GGUF repo {repo_label}: {e}") @@ -340,13 +387,14 @@ class _CachedNonGgufPayload(NamedTuple): size_bytes: int has_runnable_weights: bool model_format: ModelFormat + last_modified: float def _repo_non_gguf_model_payload(repo_info) -> _CachedNonGgufPayload: - all_weight_blobs: dict[str, int] = {} - adapter_blobs: dict[str, int] = {} - safetensors_blobs: dict[str, int] = {} - checkpoint_blobs: dict[str, int] = {} + all_weight_blobs: dict[str, tuple[int, float]] = {} + adapter_blobs: dict[str, tuple[int, float]] = {} + safetensors_blobs: dict[str, tuple[int, float]] = {} + checkpoint_blobs: dict[str, tuple[int, float]] = {} has_config = False has_adapter_config = False has_adapter_weights = False @@ -354,12 +402,15 @@ def _repo_non_gguf_model_payload(repo_info) -> _CachedNonGgufPayload: has_transformers_safetensors = False has_checkpoint = False - def _record_blob(target: dict[str, int], file_obj, rev_id: str, file_name: str) -> None: + def _record_blob( + target: dict[str, tuple[int, float]], file_obj, rev_id: str, file_name: str + ) -> None: blob_path = getattr(file_obj, "blob_path", None) size = int(file_obj.size_on_disk or 0) key = str(blob_path) if blob_path else f"{rev_id}:{file_name}" - target[key] = size - all_weight_blobs[key] = size + value = (size, _blob_mtime(file_obj)) + target[key] = value + all_weight_blobs[key] = value for revision in repo_info.revisions: rev_id = getattr(revision, "commit_hash", None) or str(id(revision)) @@ -403,18 +454,19 @@ def _repo_non_gguf_model_payload(repo_info) -> _CachedNonGgufPayload: or "unknown" ) if model_format == "adapter": - size_bytes = sum(adapter_blobs.values()) + selected_blobs = adapter_blobs elif model_format == "safetensors": - size_bytes = sum(safetensors_blobs.values()) + selected_blobs = safetensors_blobs elif model_format == "checkpoint": - size_bytes = sum(checkpoint_blobs.values()) + selected_blobs = checkpoint_blobs else: - size_bytes = sum(all_weight_blobs.values()) + selected_blobs = all_weight_blobs return _CachedNonGgufPayload( - size_bytes = size_bytes, + size_bytes = sum(size for size, _mtime in selected_blobs.values()), has_runnable_weights = model_format != "unknown", model_format = model_format, + last_modified = max((mtime for _size, mtime in selected_blobs.values()), default = 0.0), ) @@ -544,6 +596,12 @@ def _scan_cached_models() -> list[dict]: ), **_cached_model_local_metadata(repo_path), } + last_modified = max( + payload.last_modified, + (existing or {}).get("last_modified", 0.0), + ) + if last_modified > 0: + row["last_modified"] = last_modified row.update( _cache_inventory_fields( repo_id, @@ -553,6 +611,8 @@ def _scan_cached_models() -> list[dict]: ) if _prefer_cache_row(row, existing): seen_lower[key] = row + elif last_modified > existing.get("last_modified", 0.0): + existing["last_modified"] = last_modified except Exception as e: repo_label = getattr(repo_info, "repo_id", "") logger.warning(f"Skipping cached model repo {repo_label}: {e}") diff --git a/studio/backend/main.py b/studio/backend/main.py index 48675b9539..3f244dc22e 100644 --- a/studio/backend/main.py +++ b/studio/backend/main.py @@ -315,6 +315,7 @@ from hub.routes import ( datasets_router as hub_datasets_router, token_router as hub_token_router, ) +from picker.routes import templates_router as picker_templates_router from hub.schemas.downloads import TransportCapabilities from hub.utils.download_registry import ( get_download_transport_capabilities, @@ -764,6 +765,7 @@ _BODY_PROTECTED_PREFIXES = ( "/v1/completions", "/p/", "/api/inference", + "/api/picker", "/api/data-recipe", "/api/datasets", "/api/hub", @@ -995,6 +997,7 @@ app.include_router(rag_router, prefix = "/api/rag", tags = ["rag"]) app.include_router(training_history_router, prefix = "/api/train", tags = ["training-history"]) app.include_router(hub_inventory_router, prefix = "/api/hub", tags = ["hub"]) app.include_router(hub_datasets_router, prefix = "/api/hub/datasets", tags = ["hub"]) +app.include_router(picker_templates_router, prefix = "/api/picker", tags = ["picker"]) app.include_router(hub_token_router, prefix = "/api/hub", tags = ["hub"]) # Re-wrap client-error responses on the /v1/* surface into OpenAI/Anthropic diff --git a/studio/backend/models/inference.py b/studio/backend/models/inference.py index d51d35189b..580a74dddf 100644 --- a/studio/backend/models/inference.py +++ b/studio/backend/models/inference.py @@ -18,6 +18,8 @@ from pydantic import ( model_validator, ) +from picker.schemas import MAX_CHAT_TEMPLATE_BYTES + class LoadRequest(BaseModel): """Request to load a model for inference""" @@ -54,8 +56,16 @@ class LoadRequest(BaseModel): @field_validator("chat_template_override") @classmethod def normalize_blank_chat_template_override(cls, value: Optional[str]) -> Optional[str]: - if value is not None and value.strip() == "": + if value is None: return None + # Char count is a lower bound on UTF-8 byte length: reject an oversized + # template before spending work encoding it. + if len(value) > MAX_CHAT_TEMPLATE_BYTES: + raise ValueError(f"Chat template exceeds the {MAX_CHAT_TEMPLATE_BYTES}-byte limit.") + if value.strip() == "": + return None + if len(value.encode("utf-8")) > MAX_CHAT_TEMPLATE_BYTES: + raise ValueError(f"Chat template exceeds the {MAX_CHAT_TEMPLATE_BYTES}-byte limit.") return value cache_type_kv: Optional[str] = Field( @@ -206,6 +216,13 @@ class ValidateModelRequest(BaseModel): description = "Also read the native context length from the local GGUF header. " "Opt-in so the normal load preflight doesn't pay for a cache scan it doesn't need.", ) + include_chat_template: bool = Field( + False, + description = "Also read the embedded chat template from the local GGUF header, so a " + "native (picked / drag-drop) file's default template can be shown before it is loaded. " + "Opt-in and, like include_context_length, a metadata-only probe that skips the training " + "guard. Only the leased file's own embedded template is read, never sibling sidecars.", + ) class TransformersUpgradeInfo(BaseModel): @@ -266,6 +283,11 @@ class ValidateModelResponse(BaseModel): description = "MoE expert-layer count (the manual --n-cpu-moe ceiling), read from the GGUF " "header alongside context_length; 0 for dense models, None when not read.", ) + chat_template: Optional[str] = Field( + None, + description = "Embedded GGUF chat template, read from the header when include_chat_template " + "is set (native lease-backed picks); None for non-GGUF, over-cap, or not-read templates.", + ) # Additive fields; the consuming consent dialog ships in a follow-up frontend PR. requires_transformers_upgrade: bool = Field( False, diff --git a/studio/backend/picker/__init__.py b/studio/backend/picker/__init__.py new file mode 100644 index 0000000000..32014236c6 --- /dev/null +++ b/studio/backend/picker/__init__.py @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: AGPL-3.0-only +# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0 diff --git a/studio/backend/picker/routes/__init__.py b/studio/backend/picker/routes/__init__.py new file mode 100644 index 0000000000..c0e988c8bb --- /dev/null +++ b/studio/backend/picker/routes/__init__.py @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: AGPL-3.0-only +# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0 + +from .templates import router as templates_router + +__all__ = ["templates_router"] diff --git a/studio/backend/picker/routes/templates.py b/studio/backend/picker/routes/templates.py new file mode 100644 index 0000000000..02b8bf7184 --- /dev/null +++ b/studio/backend/picker/routes/templates.py @@ -0,0 +1,45 @@ +# SPDX-License-Identifier: AGPL-3.0-only +# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0 + +from __future__ import annotations + +import asyncio +from typing import Optional + +from fastapi import APIRouter, Body, Depends, Query + +from auth.authentication import get_current_subject +from hub.dependencies import get_hf_token + +from ..schemas import ( + MAX_CHAT_TEMPLATE_BYTES, + ModelTemplateResponse, + ValidateChatTemplateRequest, + ValidateChatTemplateResponse, +) +from ..service import read_default_chat_template, validate_chat_template + +router = APIRouter() + + +@router.post("/validate-chat-template", response_model = ValidateChatTemplateResponse) +async def validate_chat_template_route( + body: ValidateChatTemplateRequest = Body(...), + current_subject: str = Depends(get_current_subject), +) -> ValidateChatTemplateResponse: + return await asyncio.to_thread(validate_chat_template, body.template) + + +@router.get("/chat-template/{model_name:path}", response_model = ModelTemplateResponse) +async def get_default_chat_template_route( + model_name: str, + gguf_variant: Optional[str] = Query(None), + hf_token: Optional[str] = Depends(get_hf_token), + current_subject: str = Depends(get_current_subject), +) -> ModelTemplateResponse: + template = await asyncio.to_thread( + read_default_chat_template, model_name, hf_token, gguf_variant + ) + if template is not None and len(template.encode("utf-8")) > MAX_CHAT_TEMPLATE_BYTES: + template = None + return ModelTemplateResponse(model_name = model_name, chat_template = template) diff --git a/studio/backend/picker/schemas.py b/studio/backend/picker/schemas.py new file mode 100644 index 0000000000..b4f956188f --- /dev/null +++ b/studio/backend/picker/schemas.py @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: AGPL-3.0-only +# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0 + +from typing import Optional + +from pydantic import BaseModel, Field, field_validator + +# Mirror the frontend's 64 KiB chat-template contract (per-model-config.ts) at +# the API boundary so a direct caller cannot make Jinja parse an oversized +# template. MaxBodyMiddleware only caps the whole request body, not this field. +MAX_CHAT_TEMPLATE_BYTES = 65_536 + + +class ValidateChatTemplateRequest(BaseModel): + template: str = Field(default = "") + + @field_validator("template") + @classmethod + def _enforce_template_size(cls, value: str) -> str: + if len(value.encode("utf-8")) > MAX_CHAT_TEMPLATE_BYTES: + raise ValueError(f"Chat template exceeds the {MAX_CHAT_TEMPLATE_BYTES}-byte limit.") + return value + + +class ValidateChatTemplateResponse(BaseModel): + valid: bool + error: Optional[str] = None + + +class ModelTemplateResponse(BaseModel): + model_name: str + chat_template: Optional[str] = None diff --git a/studio/backend/picker/service.py b/studio/backend/picker/service.py new file mode 100644 index 0000000000..13065b2920 --- /dev/null +++ b/studio/backend/picker/service.py @@ -0,0 +1,426 @@ +# SPDX-License-Identifier: AGPL-3.0-only +# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0 + +from __future__ import annotations + +import json +import logging +import os +import re +from pathlib import Path +from typing import Optional + +from hub.services.models.folder_browser import ( + _build_browse_allowlist, + _is_path_inside_allowlist, +) +from hub.utils.gguf import extract_quant_label, iter_hf_cache_snapshots +from utils.models.gguf_metadata import read_gguf_chat_template +from utils.models.model_config import ( + _extract_quant_label, + _is_big_endian_gguf_path, + _is_mmproj, + _is_mtp_drafter, +) +from utils.paths.path_utils import ( + is_local_path, + normalize_path, + resolve_cached_repo_id_case, +) + +from .schemas import MAX_CHAT_TEMPLATE_BYTES, ValidateChatTemplateResponse + +logger = logging.getLogger(__name__) + +_VALID_REPO_ID = re.compile(r"^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$") + + +def _is_valid_repo_id(repo_id: str) -> bool: + return bool(_VALID_REPO_ID.fullmatch(repo_id)) + + +_TOKENIZER_CONFIG_PATHS = ("tokenizer_config.json", "LLM/tokenizer_config.json") +_JINJA_TEMPLATE_PATHS = ("chat_template.jinja", "LLM/chat_template.jinja") +_PROCESSOR_TEMPLATE_PATHS = ("chat_template.json", "LLM/chat_template.json") + +# Cap sidecar reads so a malformed or hostile metadata file cannot exhaust memory +# before its template is size-checked. The JSON envelope may exceed a bare template +# (it carries other tokenizer metadata); the extracted template is still bounded by +# MAX_CHAT_TEMPLATE_BYTES downstream. +MAX_TEMPLATE_METADATA_BYTES = 4 * 1024 * 1024 + + +def _read_bounded_text(path: Path, limit: int) -> Optional[str]: + """Read at most `limit` bytes of UTF-8 text; None if larger or unreadable.""" + try: + with path.open("rb") as f: + data = f.read(limit + 1) + except OSError: + return None + if len(data) > limit: + return None + try: + return data.decode("utf-8") + except UnicodeError: + return None + + +def _leaf_inside_allowlist(path: Path, allow_roots: Optional[list[Path]]) -> bool: + # Block symlinked children from escaping the validated directory (realpath-checked). + # None = trusted caller (HF cache / remote download). + return allow_roots is None or _is_path_inside_allowlist(path, allow_roots) + + +def validate_chat_template(template: str) -> ValidateChatTemplateResponse: + text = (template or "").strip() + if not text: + return ValidateChatTemplateResponse(valid = True, error = None) + # Import Jinja lazily: optional at runtime (e.g. GGUF-only installs), so a + # missing dependency must not crash API startup. + try: + from jinja2 import TemplateError + from jinja2.ext import Extension + from jinja2.sandbox import ImmutableSandboxedEnvironment + except ImportError: + return ValidateChatTemplateResponse(valid = True, error = None) + + class _GenerationTag(Extension): + # Accept Transformers' {% generation %} assistant-mask tag so a pasted HF + # chat template validates (we only parse it). + tags = {"generation"} + + def parse(self, parser): + next(parser.stream) + return parser.parse_statements(["name:endgeneration"], drop_needle = True) + + try: + env = ImmutableSandboxedEnvironment( + trim_blocks = True, + lstrip_blocks = True, + extensions = ["jinja2.ext.loopcontrols", _GenerationTag], + ) + env.parse(text) + return ValidateChatTemplateResponse(valid = True, error = None) + except TemplateError as exc: + message = getattr(exc, "message", None) or str(exc) + lineno = getattr(exc, "lineno", None) + if lineno: + message = f"Line {lineno}: {message}" + return ValidateChatTemplateResponse(valid = False, error = message) + except Exception as exc: + return ValidateChatTemplateResponse(valid = False, error = str(exc)) + + +def _chat_template_from_tokenizer_config(config: dict) -> Optional[str]: + if not isinstance(config, dict): + return None + raw = config.get("chat_template") + if isinstance(raw, str) and raw.strip(): + return raw + if isinstance(raw, list): + fallback: Optional[str] = None + for entry in raw: + if not isinstance(entry, dict): + continue + template = entry.get("template") + if not isinstance(template, str): + continue + if entry.get("name") == "default": + return template + if fallback is None: + fallback = template + return fallback + return None + + +def _chat_template_from_jinja_file( + dir_path: Path, allow_roots: Optional[list[Path]] = None +) -> Optional[str]: + for rel in _JINJA_TEMPLATE_PATHS: + template_file = dir_path / rel + if not template_file.exists() or not _leaf_inside_allowlist(template_file, allow_roots): + continue + try: + if template_file.stat().st_size > MAX_CHAT_TEMPLATE_BYTES: + continue + template = template_file.read_text(encoding = "utf-8") + except Exception: + continue + if template.strip(): + return template + return None + + +def _chat_template_from_processor_payload(payload: object) -> Optional[str]: + # processor chat_template.json may be the template string itself or a + # {name: template} map, not only a tokenizer_config-shaped object. + if isinstance(payload, str): + return payload if payload.strip() else None + template = _chat_template_from_tokenizer_config(payload) # type: ignore[arg-type] + if template: + return template + if isinstance(payload, dict): + # Named-template map: prefer "default", else the first non-empty entry + # (mirrors the tokenizer-config list fallback). + default = payload.get("default") + if isinstance(default, str) and default.strip(): + return default + for value in payload.values(): + if isinstance(value, str) and value.strip(): + return value + return None + + +def _chat_template_from_processor_json( + dir_path: Path, allow_roots: Optional[list[Path]] = None +) -> Optional[str]: + for rel in _PROCESSOR_TEMPLATE_PATHS: + config_file = dir_path / rel + if not config_file.exists() or not _leaf_inside_allowlist(config_file, allow_roots): + continue + raw = _read_bounded_text(config_file, MAX_TEMPLATE_METADATA_BYTES) + if raw is None: + continue + try: + payload = json.loads(raw) + except Exception: + continue + template = _chat_template_from_processor_payload(payload) + if template: + return template + return None + + +def _chat_template_from_tokenizer_dir( + dir_path: Path, allow_roots: Optional[list[Path]] = None +) -> Optional[str]: + jinja = _chat_template_from_jinja_file(dir_path, allow_roots) + if jinja: + return jinja + for rel in _TOKENIZER_CONFIG_PATHS: + config_file = dir_path / rel + if not config_file.exists() or not _leaf_inside_allowlist(config_file, allow_roots): + continue + raw = _read_bounded_text(config_file, MAX_TEMPLATE_METADATA_BYTES) + if raw is None: + continue + try: + config = json.loads(raw) + except Exception: + continue + template = _chat_template_from_tokenizer_config(config) + if template: + return template + return _chat_template_from_processor_json(dir_path, allow_roots) + + +_GGUF_SCAN_MAX_DEPTH = 2 + + +def _iter_ggufs(dir_path: Path) -> list[Path]: + if dir_path == dir_path.parent: + return [] + root = str(dir_path) + found: list[Path] = [] + for current, dirs, files in os.walk(root, followlinks = False): + rel = os.path.relpath(current, root) + depth = 0 if rel == os.curdir else rel.count(os.sep) + 1 + if depth >= _GGUF_SCAN_MAX_DEPTH: + dirs[:] = [] + for name in files: + if not name.lower().endswith(".gguf") or _is_mmproj(name): + continue + path = Path(current) / name + try: + rel = path.relative_to(dir_path).as_posix() + except ValueError: + rel = name + quant = _extract_quant_label(rel) + if _is_mtp_drafter(rel) or _is_big_endian_gguf_path(rel, quant): + continue + found.append(path) + return found + + +def _variant_matches(relative_path: str, needle: str) -> bool: + quant = _extract_quant_label(relative_path).lower() + if quant == needle: + return True + if extract_quant_label(relative_path).lower() == needle: + return True + prefix = f"{needle}-" + if not quant.startswith(prefix): + return False + suffix = quant[len(prefix) :] + if not suffix.endswith("bpw"): + return False + value = suffix[:-3] + return bool(value) and value.replace(".", "", 1).isdigit() + + +_GGUF_SPLIT_INDEX_RE = re.compile(r"-(\d{3,})-of-\d{3,}$", re.IGNORECASE) + + +def _is_nonfirst_gguf_split(path: Path) -> bool: + match = _GGUF_SPLIT_INDEX_RE.search(path.stem) + return match is not None and int(match.group(1)) != 1 + + +def _find_gguf_in_dir(dir_path: Path, gguf_variant: Optional[str]) -> Optional[Path]: + try: + ggufs = sorted(_iter_ggufs(dir_path)) + except OSError: + return None + if not ggufs: + return None + needle = (gguf_variant or "").strip().lower() + if needle: + for path in ggufs: + try: + relative = path.relative_to(dir_path).as_posix() + except ValueError: + relative = path.name + if _variant_matches(relative, needle): + return path + return None + candidates = [path for path in ggufs if not _is_nonfirst_gguf_split(path)] or ggufs + try: + return max(candidates, key = lambda path: path.stat().st_size) + except OSError: + return candidates[0] + + +def _chat_template_from_dir( + dir_path: Path, + gguf_variant: Optional[str] = None, + allow_roots: Optional[list[Path]] = None, +) -> Optional[str]: + def from_gguf() -> Optional[str]: + gguf = _find_gguf_in_dir(dir_path, gguf_variant) + if gguf is None or not _leaf_inside_allowlist(gguf, allow_roots): + return None + return read_gguf_chat_template(str(gguf)) + + # Sidecar tokenizer files (chat_template.jinja / tokenizer_config.json) are the + # author's maintained template and supersede the GGUF's possibly-stale embedded + # copy. The variant only picks the GGUF fallback, so tokenizer-first precedence + # holds whether or not a variant is given. + return _chat_template_from_tokenizer_dir(dir_path, allow_roots) or from_gguf() + + +def read_default_chat_template( + model_name: str, + hf_token: Optional[str] = None, + gguf_variant: Optional[str] = None, +) -> Optional[str]: + if not isinstance(model_name, str) or not model_name.strip(): + return None + name = model_name.strip() + + if is_local_path(name): + try: + target = Path(normalize_path(name)).expanduser() + allow_roots = _build_browse_allowlist() + if not _is_path_inside_allowlist(target, allow_roots): + logger.debug("Refused chat template read outside allowed folders: %s", name) + return None + if name.lower().endswith(".gguf"): + # Prefer a maintained sidecar next to the file over the GGUF's + # embedded copy (tokenizer-first precedence, as elsewhere). + sidecar = _chat_template_from_tokenizer_dir(target.parent, allow_roots) + if sidecar: + return sidecar + return read_gguf_chat_template(str(target)) + return _chat_template_from_dir(target, gguf_variant, allow_roots) + except Exception as exc: + logger.debug("Could not read local chat template for %s: %s", name, exc) + return None + + if not _is_valid_repo_id(name): + return None + + resolved = resolve_cached_repo_id_case(name) + + try: + # Resolve within each cached revision, newest first. A revision's sidecar + # supersedes its own embedded GGUF copy, but must not override a newer + # revision, so precedence stays per-snapshot rather than global. + for snapshot in iter_hf_cache_snapshots(resolved): + template = _chat_template_from_dir(snapshot, gguf_variant) + if template: + return template + except Exception as exc: + logger.debug("Could not read cached chat template for %s: %s", resolved, exc) + + try: + from huggingface_hub import HfApi, hf_hub_download + + _api = HfApi() + + def _remote_exceeds_cap(rel: str) -> bool: + # Best-effort: skip the download when the remote's advertised size + # exceeds the cap, so a maliciously large sidecar is never fetched. + try: + infos = _api.get_paths_info(resolved, [rel], repo_type = "model", token = hf_token) + except Exception: + return False + for info in infos: + size = getattr(info, "size", None) + if ( + getattr(info, "path", None) == rel + and isinstance(size, int) + and size > MAX_TEMPLATE_METADATA_BYTES + ): + return True + return False + + def _download_text(rel: str) -> Optional[str]: + if _remote_exceeds_cap(rel): + return None + try: + path = hf_hub_download(resolved, rel, token = hf_token) + return _read_bounded_text(Path(path), MAX_TEMPLATE_METADATA_BYTES) + except Exception: + return None + + for rel in _JINJA_TEMPLATE_PATHS: + template = _download_text(rel) + if not template or not template.strip(): + continue + # A raw Jinja sidecar is the whole template, so it must fit the route's + # response cap (the local path skips oversized .jinja too). Download stays + # bounded at MAX_TEMPLATE_METADATA_BYTES so a large JSON embedding a small + # template still extracts below, but an over-cap Jinja is dropped so the + # search falls through to the tokenizer/processor template. + if len(template.encode("utf-8")) > MAX_CHAT_TEMPLATE_BYTES: + continue + return template + + for rel in _TOKENIZER_CONFIG_PATHS: + raw = _download_text(rel) + if not raw: + continue + try: + config = json.loads(raw) + except Exception: + continue + template = _chat_template_from_tokenizer_config(config) + if template: + return template + + for rel in _PROCESSOR_TEMPLATE_PATHS: + raw = _download_text(rel) + if not raw: + continue + try: + payload = json.loads(raw) + except Exception: + continue + template = _chat_template_from_processor_payload(payload) + if template: + return template + + return None + except Exception as exc: + logger.debug("Could not fetch chat template for %s: %s", resolved, exc) + return None diff --git a/studio/backend/routes/inference.py b/studio/backend/routes/inference.py index afd942e9a5..d3e588bb0b 100644 --- a/studio/backend/routes/inference.py +++ b/studio/backend/routes/inference.py @@ -5144,10 +5144,10 @@ async def validate_model( latest_tier_active_for, config.identifier, request.hf_token ): effective_load_in_4bit = False - # A metadata-only probe just reads the GGUF header and allocates no VRAM, - # so it must not be refused by the training guard. Real loads validate - # without include_context_length and /load applies the guard again. - if not request.include_context_length: + # A metadata-only probe reads the GGUF header and allocates no VRAM, so the + # training guard must not refuse it. Real loads omit include_context_length / + # include_chat_template, and /load applies the guard again. + if not (request.include_context_length or request.include_chat_template): # Match /load's inherited llama.cpp extras and parallel slot count so # validation cannot pass a smaller estimate than the subsequent load. effective_extra_args = _resolve_inherited_extra_args( @@ -5189,9 +5189,15 @@ async def validate_model( context_length: Optional[int] = None layer_count: Optional[int] = None moe_layer_count: Optional[int] = None - if request.include_context_length and is_gguf: + chat_template: Optional[str] = None + # Both header probes read the same local GGUF, so resolve it once. + if (request.include_context_length or request.include_chat_template) and is_gguf: from hub.utils.gguf import resolve_local_gguf_path - from utils.models.gguf_metadata import read_gguf_staged_dims + from picker.schemas import MAX_CHAT_TEMPLATE_BYTES + from utils.models.gguf_metadata import ( + read_gguf_chat_template, + read_gguf_staged_dims, + ) # Best-effort: a header-read failure must never fail validation of an # otherwise-valid model (the outer except turns it into a 400). @@ -5207,13 +5213,24 @@ async def validate_model( model_identifier, request.gguf_variant ) if local_gguf: - # Header walk reads tokenizer arrays for dense models (tens of - # ms); keep it off the event loop. - dims = await asyncio.to_thread(read_gguf_staged_dims, local_gguf) - if dims: - context_length = dims["context_length"] - layer_count = dims["layer_count"] - moe_layer_count = dims["moe_layer_count"] + if request.include_context_length: + # Header walk reads tokenizer arrays (tens of ms); keep it + # off the event loop. + dims = await asyncio.to_thread(read_gguf_staged_dims, local_gguf) + if dims: + context_length = dims["context_length"] + layer_count = dims["layer_count"] + moe_layer_count = dims["moe_layer_count"] + if request.include_chat_template: + # Read only the leased GGUF's own embedded template (the copy + # llama.cpp loads), never a sibling sidecar: the native grant + # authorizes just this path, so neighbours would be scope escalation. + raw_template = await asyncio.to_thread(read_gguf_chat_template, local_gguf) + if ( + raw_template is not None + and len(raw_template.encode("utf-8")) <= MAX_CHAT_TEMPLATE_BYTES + ): + chat_template = raw_template except Exception as e: logger.debug("Header probe failed for %s: %s", model_log_label, e) @@ -5232,6 +5249,7 @@ async def validate_model( context_length = context_length, layer_count = layer_count, moe_layer_count = moe_layer_count, + chat_template = chat_template, requires_transformers_upgrade = transformers_upgrade is not None, transformers_upgrade = transformers_upgrade, ) diff --git a/studio/backend/routes/models.py b/studio/backend/routes/models.py index 0806c2f513..a5ce1a72f0 100644 --- a/studio/backend/routes/models.py +++ b/studio/backend/routes/models.py @@ -60,13 +60,52 @@ def _safe_is_dir(path) -> bool: # Shared with the hub inventory scans; keep the private aliases so existing -# importers (core.inference.local_model_resolver, tests) stay valid. +# importers stay valid. ``_HF_REPO_ID_RE`` is the Hub repo id shape ("owner/name"); +# anything else is treated as a local filesystem path. from utils.hidden_models import ( + _HF_REPO_ID_RE, + _existing_resolved_path, _safe_resolve, is_hidden_model as _is_hidden_model, ) +def hidden_model_matchers() -> tuple[list[str], list[str], list[str]]: + """Substring needles, exact repo ids, and exact resolved paths identifying + infra models (the RAG embedder and the llama.cpp install validation probe) + that pickers hide. Served by the ``/api/hub/hidden-models`` endpoint. A + configured HF-repo embedder is published as its exact lowercased repo id + (mirroring ``utils.hidden_models.is_hidden_model``) and a local-path + embedder as its exact resolved path only: a generic basename like "model" + must not substring-hide unrelated chat models.""" + from core.rag import config as rag_config + + needles = [ + # The validation probe's repo and its exact filename. The filename carries + # .gguf so it won't hide unrelated repos like ``user/stories260K-finetune-GGUF``. + "ggml-org/models", + "stories260k.gguf", + ] + exact_ids: list[str] = [] + exact_paths: list[str] = [] + for model in ( + rag_config.effective_embedding_model(), + rag_config.effective_gguf_repo(), + ): + # Resolve an existing local path before the repo-id regex: a local embedder + # shaped like "models/embedder" is an exact path, not a Hub repo id. + existing_path = _existing_resolved_path(model) + if existing_path: + exact_paths.append(existing_path.lower()) + elif _HF_REPO_ID_RE.match(model): + exact_ids.append(model.lower()) + else: + resolved = _safe_resolve(Path(model).expanduser()) + if resolved: + exact_paths.append(resolved.lower()) + return needles, exact_ids, exact_paths + + backend_path = Path(__file__).parent.parent.parent if str(backend_path) not in sys.path: sys.path.insert(0, str(backend_path)) @@ -91,6 +130,7 @@ try: _pick_best_gguf, _extract_quant_label, _is_big_endian_gguf_path, + _is_mtp_drafter, is_audio_input_type, ) from core.inference import get_inference_backend @@ -123,6 +163,7 @@ except ImportError: _pick_best_gguf, _extract_quant_label, _is_big_endian_gguf_path, + _is_mtp_drafter, is_audio_input_type, ) from core.inference import get_inference_backend @@ -803,7 +844,7 @@ def collect_local_models(models_root: Path) -> List[LocalModelInfo]: models = sorted( deduped.values(), - key = lambda item: (item.updated_at or 0), + key = lambda item: item.updated_at or 0, reverse = True, ) return [m for m in models if not _is_hidden_model(m.id, m.model_id, m.path)] @@ -1750,9 +1791,11 @@ def _get_model_size_bytes(model_name: str, hf_token: Optional[str] = None) -> Op async def get_model_config( model_name: str, hf_token: Optional[str] = Query(None), + header_hf_token: Optional[str] = Depends(get_hf_token), current_subject: str = Depends(get_current_subject), ): """Get configuration for a specific model (wraps load_model_defaults).""" + hf_token = _normalize_hf_token(header_hf_token) or _normalize_hf_token(hf_token) try: if not is_local_path(model_name): resolved = resolve_cached_repo_id_case(model_name) @@ -2471,6 +2514,7 @@ async def get_lora_base_model(lora_path: str, current_subject: str = Depends(get async def check_vision_model( model_name: str, hf_token: Optional[str] = Query(None), + header_hf_token: Optional[str] = Depends(get_hf_token), current_subject: str = Depends(get_current_subject), ): """ @@ -2478,6 +2522,7 @@ async def check_vision_model( This endpoint wraps the backend is_vision_model function. """ + hf_token = _normalize_hf_token(header_hf_token) or _normalize_hf_token(hf_token) try: logger.info(f"Checking if vision model: {model_name}") # Authenticate so a gated/private VLM classifies correctly (else 404 -> non-vision). @@ -2503,6 +2548,7 @@ async def check_vision_model( async def check_embedding_model( model_name: str, hf_token: Optional[str] = Query(None), + header_hf_token: Optional[str] = Depends(get_hf_token), current_subject: str = Depends(get_current_subject), ): """ @@ -2510,6 +2556,7 @@ async def check_embedding_model( This endpoint wraps the backend is_embedding_model function. """ + hf_token = _normalize_hf_token(header_hf_token) or _normalize_hf_token(hf_token) try: logger.info(f"Checking if embedding model: {model_name}") is_embedding = is_embedding_model(model_name, hf_token = hf_token) @@ -2573,12 +2620,6 @@ def _resolve_quant_gguf(repo_id: str, quant: str, is_local: bool) -> tuple[Optio Q8_0 weights). Never raises. """ try: - from utils.models.model_config import ( - _extract_quant_label, - _is_big_endian_gguf_path, - _is_mtp_drafter, - ) - if is_local: roots = [Path(repo_id)] else: @@ -2595,25 +2636,19 @@ def _resolve_quant_gguf(repo_id: str, quant: str, is_local: bool) -> tuple[Optio if snaps.is_dir(): roots.extend(s for s in snaps.iterdir() if s.is_dir()) - want = quant.lower().replace("-", "").replace("_", "") + want = _normalized_quant_label(quant) best_total = 0 best_first: Optional[str] = None for root in roots: matches: list[tuple[str, Path]] = [] total = 0 for f in _iter_gguf_paths(root): - if _is_mmproj_filename(f.name): - continue try: rel = f.relative_to(root).as_posix() except ValueError: rel = f.name - if _is_mtp_drafter(rel): - continue - q = _extract_quant_label(rel) - if _is_big_endian_gguf_path(rel, q): - continue - if q.lower().replace("-", "").replace("_", "") != want: + q = _main_variant_gguf_label(rel) + if q is None or _normalized_quant_label(q) != want: continue try: total += f.stat().st_size @@ -3035,6 +3070,22 @@ def _is_main_gguf_filename(name: str) -> bool: return _is_gguf_filename(name) and not _is_mmproj_filename(name) +def _main_variant_gguf_label(rel_path: str) -> Optional[str]: + name = rel_path.rsplit("/", 1)[-1] + if not _is_main_gguf_filename(name): + return None + if _is_mtp_drafter(rel_path): + return None + label = _extract_quant_label(rel_path) + if _is_big_endian_gguf_path(rel_path, label): + return None + return label + + +def _normalized_quant_label(label: str) -> str: + return label.lower().replace("-", "").replace("_", "") + + def _repo_has_mmproj(repo_info) -> bool: """True if the repo ships a GGUF vision adapter (mmproj), so it can take image inputs. Cheap: scans already-listed file names only.""" @@ -3362,6 +3413,170 @@ async def delete_cached_model( ) +def _resolve_cached_model_path(repo_id: str, variant: Optional[str]) -> Path: + """Absolute path of a cached repo (newest snapshot dir) or, with *variant*, + that quant's main GGUF file (first split of a sharded quant). Paths come + from the HF cache scan only, so callers can't probe arbitrary paths.""" + cache_scans = _all_hf_cache_scans() + + matching_repos = [] + for hf_cache in cache_scans: + for repo_info in hf_cache.repos: + if repo_info.repo_type != "model": + continue + if repo_info.repo_id.lower() == repo_id.lower(): + matching_repos.append(repo_info) + if not matching_repos: + raise HTTPException(status_code = 404, detail = "Model not found in cache") + + if variant: + want = _normalized_quant_label(variant) + candidate_revisions = sorted( + (rev for repo_info in matching_repos for rev in repo_info.revisions), + key = lambda rev: getattr(rev, "last_modified", 0) or 0, + reverse = True, + ) + for rev in candidate_revisions: + snapshot = getattr(rev, "snapshot_path", None) + matches = [] + for f in rev.files: + p = Path(f.file_path) + rel = f.file_name + if snapshot: + try: + rel = p.relative_to(snapshot).as_posix() + except ValueError: + pass + label = _main_variant_gguf_label(rel) + if label is None or _normalized_quant_label(label) != want: + continue + if p.exists() or p.is_symlink(): + matches.append((rel, p)) + if matches: + # Path-sorted so a sharded quant deterministically yields its first split. + return sorted(matches, key = lambda m: m[0].lower())[0][1] + raise HTTPException( + status_code = 404, + detail = f"Variant {variant} not found in cache for {repo_id}", + ) + + def repo_size(repo_info) -> int: + gguf_size = _repo_gguf_size_bytes(repo_info) + if gguf_size > 0: + return gguf_size + return sum( + (getattr(f, "size_on_disk", None) or 0) + for rev in repo_info.revisions + for f in rev.files + ) + + def repo_last_modified(repo_info) -> float: + return max( + (getattr(rev, "last_modified", 0) or 0 for rev in repo_info.revisions), + default = 0, + ) + + target_repo = max( + matching_repos, + key = lambda repo_info: (repo_size(repo_info), repo_last_modified(repo_info)), + ) + + # Whole repo: the newest revision's snapshot dir holds the visible files. + revisions = sorted( + (rev for rev in target_repo.revisions if getattr(rev, "snapshot_path", None)), + key = lambda rev: getattr(rev, "last_modified", 0) or 0, + reverse = True, + ) + for rev in revisions: + p = Path(rev.snapshot_path) + if p.exists(): + return p + p = Path(target_repo.repo_path) + if p.exists(): + return p + raise HTTPException(status_code = 404, detail = "Cached model path not found") + + +def _wsl_reveal_in_explorer(path: Path) -> bool: + import subprocess + + from utils.paths.path_utils import _IS_WSL + + if not _IS_WSL: + return False + try: + windows_path = subprocess.run( + ["wslpath", "-w", str(path)], + capture_output = True, + text = True, + check = True, + timeout = 10, + ).stdout.strip() + if not windows_path: + return False + argument = f"/select,{windows_path}" if path.is_file() else windows_path + subprocess.Popen(["explorer.exe", argument]) + return True + except (OSError, subprocess.SubprocessError): + return False + + +def _reveal_in_file_manager(path: Path) -> None: + """Open the OS file manager with *path* selected (best effort per platform).""" + import subprocess + + target = str(path) + if sys.platform == "darwin": + cmd = ["open", "-R", target] if path.is_file() else ["open", target] + subprocess.Popen(cmd) + elif os.name == "nt": + if path.is_file(): + subprocess.Popen(["explorer", f"/select,{target}"]) + else: + os.startfile(target) # noqa: S606 - local user's own file manager + elif not _wsl_reveal_in_explorer(path): + # No cross-desktop "select file" standard on Linux; open the directory. + directory = target if path.is_dir() else str(path.parent) + subprocess.Popen(["xdg-open", directory]) + + +class CachedModelPathResponse(BaseModel): + path: str + is_dir: bool + + +@router.get("/cached-model-path", response_model = CachedModelPathResponse) +async def get_cached_model_path( + repo_id: str = Query(..., description = "HuggingFace repo ID"), + variant: str = Query("", description = "Quantization variant (empty for whole repo)"), + current_subject: str = Depends(get_current_subject), +): + """Absolute on-disk path of a cached repo or one of its GGUF variants.""" + if not _is_valid_repo_id(repo_id): + raise HTTPException(status_code = 400, detail = "Invalid repo_id format") + path = await asyncio.to_thread(_resolve_cached_model_path, repo_id, variant.strip() or None) + return {"path": str(path), "is_dir": path.is_dir()} + + +@router.post("/reveal-cached-model") +async def reveal_cached_model( + repo_id: str = Body(...), + variant: Optional[str] = Body(None), + current_subject: str = Depends(get_current_subject), +): + """Reveal a cached repo (or one GGUF variant's file) in the OS file manager.""" + if not _is_valid_repo_id(repo_id): + raise HTTPException(status_code = 400, detail = "Invalid repo_id format") + variant = (variant or "").strip() or None + path = await asyncio.to_thread(_resolve_cached_model_path, repo_id, variant) + try: + await asyncio.to_thread(_reveal_in_file_manager, path) + except Exception as e: + logger.error(f"Failed to reveal {path}: {e}") + raise HTTPException(status_code = 500, detail = "Failed to open file manager") + return {"status": "ok", "path": str(path)} + + @router.get("/checkpoints", response_model = CheckpointListResponse) async def list_checkpoints( outputs_dir: str = Query( diff --git a/studio/backend/tests/test_chat_load_during_training.py b/studio/backend/tests/test_chat_load_during_training.py index 7daa4224aa..a5fd71b6a0 100644 --- a/studio/backend/tests/test_chat_load_during_training.py +++ b/studio/backend/tests/test_chat_load_during_training.py @@ -801,6 +801,80 @@ class TestValidateRefusesDuringTraining(unittest.TestCase): asyncio.run(self.route.validate_model(request, current_subject = "u")) self.assertEqual(guard_called, []) + def _validate_gguf_template( + self, + *, + template, + canonical_path = "/picked/model.gguf", + ): + # Drive validate_model for a native lease-backed GGUF template probe and + # capture what the embedded-template reader was called with. + from models.inference import ValidateModelRequest + + request = ValidateModelRequest( + model_path = "model.gguf", + gguf_variant = "Q4_K_M", + native_path_lease = "signed-lease", + include_chat_template = True, + ) + cfg = SimpleNamespace( + identifier = canonical_path, + display_name = "model.gguf", + is_gguf = True, + is_lora = False, + is_vision = False, + gguf_file = canonical_path, + path = None, + base_model = None, + ) + import utils.models.gguf_metadata as gguf_meta + + seen = {} + + def _fake_read(path): + seen["path"] = path + return template + + guard_called = [] + with ( + patch.object( + self.route, + "_resolve_model_identifier_for_request", + return_value = (canonical_path, "model.gguf", True), + ), + patch.object(self.route.ModelConfig, "from_identifier", return_value = cfg), + patch.object(self.route, "load_inference_config", return_value = {}), + patch.object(gguf_meta, "read_gguf_chat_template", _fake_read), + patch.object( + self.route, + "_guard_chat_load_against_training", + lambda *a, **kw: guard_called.append(True), + ), + ): + resp = asyncio.run(self.route.validate_model(request, current_subject = "u")) + return resp, seen, guard_called + + def test_include_chat_template_reads_leased_gguf_embedded_template(self): + # The picker chat-template GET has no lease plumbing, so a native picked + # GGUF surfaces its default template through this lease-aware probe: the + # embedded template is read from the granted canonical path and returned. + resp, seen, _ = self._validate_gguf_template(template = "{{ messages }}") + self.assertEqual(resp.chat_template, "{{ messages }}") + # Read strictly the leased file's own embedded template, never a sibling + # sidecar: the grant authorizes just this one path. + self.assertEqual(seen["path"], "/picked/model.gguf") + + def test_include_chat_template_skips_training_guard(self): + # A template-only probe allocates no VRAM, so like include_context_length + # it must not be refused by the training guard. + _, _, guard_called = self._validate_gguf_template(template = "{{ messages }}") + self.assertEqual(guard_called, []) + + def test_include_chat_template_over_cap_is_dropped(self): + from picker.schemas import MAX_CHAT_TEMPLATE_BYTES + resp, _, _ = self._validate_gguf_template(template = "a" * (MAX_CHAT_TEMPLATE_BYTES + 1)) + self.assertIsNone(resp.chat_template) + # ── _estimate_gguf_required_gb (sizes the same weights the loader loads) ────── diff --git a/studio/backend/tests/test_export_absolute_paths.py b/studio/backend/tests/test_export_absolute_paths.py index 761ea08e3f..5097f9f53a 100644 --- a/studio/backend/tests/test_export_absolute_paths.py +++ b/studio/backend/tests/test_export_absolute_paths.py @@ -158,6 +158,7 @@ def _install_lightweight_backend_stubs(monkeypatch): utils_model_config._pick_best_gguf = lambda variants: variants[0] if variants else None utils_model_config._extract_quant_label = lambda value: value utils_model_config._is_big_endian_gguf_path = lambda *args, **kwargs: False + utils_model_config._is_mtp_drafter = lambda *args, **kwargs: False utils_model_config.is_audio_input_type = lambda *args, **kwargs: None monkeypatch.setitem( sys.modules, diff --git a/studio/backend/tests/test_model_picker_regression.py b/studio/backend/tests/test_model_picker_regression.py new file mode 100644 index 0000000000..f38a4d0b8d --- /dev/null +++ b/studio/backend/tests/test_model_picker_regression.py @@ -0,0 +1,232 @@ +# SPDX-License-Identifier: AGPL-3.0-only +# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0 + +"""Regression guards for the model-picker per-model-config feature (the set of +bugs that got the predecessor PR reverted). Pure-function / validation checks +only, so they run on CPU in the backend pytest job with no model download. + +Covers, at the backend layer: + - infra-model hiding: the RAG embedder (bge-small-en-v1.5) and the llama.cpp + install-validation probe (ggml-org/models / stories260K) stay hidden, while + normal chat repos are not hidden; + - the HF token is honored from the dedicated header with the query string as a + fallback, never the other way around; + - the chat-template byte caps reject oversized overrides (both the char-count + fast path and the UTF-8 byte path) and the sidecar reader is size-bounded. +""" + +from __future__ import annotations + +import sys +import types + +import pytest + +# Keep this test runnable without the optional structlog dependency (mirrors +# tests/test_cached_gguf_routes.py), since importing routes.models pulls it in. +if "structlog" not in sys.modules: + + class _DummyLogger: + def __getattr__(self, _name): + return lambda *args, **kwargs: None + + sys.modules["structlog"] = types.SimpleNamespace( + BoundLogger = _DummyLogger, + get_logger = lambda *args, **kwargs: _DummyLogger(), + ) + +import routes.models as models_route +from core.rag import config as rag_config +from hub.dependencies import get_hf_token +from models.inference import LoadRequest +from picker.schemas import MAX_CHAT_TEMPLATE_BYTES +from picker.service import _read_bounded_text +from utils.hidden_models import is_hidden_model + + +@pytest.fixture(autouse = True) +def _pin_default_embedder(monkeypatch): + """Pin the effective embedder to Studio's static default so hiding is + deterministic and cannot depend on ambient RAG config / env.""" + default = "unsloth/bge-small-en-v1.5" + monkeypatch.setattr(rag_config, "EMBEDDING_MODEL", default, raising = False) + monkeypatch.setattr(rag_config, "effective_embedding_model", lambda: default) + monkeypatch.setattr(rag_config, "effective_gguf_repo", lambda: default) + monkeypatch.setattr(rag_config, "default_gguf_repo", lambda: default) + + +# --------------------------------------------------------------------------- # +# Infra-model hiding (the "infra models resurfaced in the picker" regression) # +# --------------------------------------------------------------------------- # + + +@pytest.mark.parametrize( + "value", + [ + "ggml-org/models", # the probe repo id + "unsloth/bge-small-en-v1.5", # the RAG embedder repo + "unsloth/bge-small-en-v1.5-GGUF", # its GGUF companion + "/root/.cache/huggingface/hub/x/stories260K.gguf", # probe on disk + "/root/.cache/x/Stories260K.GGUF", # case-insensitive + r"C:\\models\\stories260K.gguf", # windows-style path + "/opt/models/bge-small-en-v1.5", # embedder basename folder + "/opt/models/bge-small-en-v1.5-Q8_0.gguf", # suffixed local weight + ], +) +def test_infra_models_are_hidden(value): + assert is_hidden_model(value) is True + + +@pytest.mark.parametrize( + "value", + [ + "unsloth/gemma-3-270m-it-GGUF", # a normal small chat GGUF + "unsloth/Qwen3-0.6B", # a normal non-GGUF chat model + "user/stories260K-finetune-GGUF", # repo id merely contains "stories260k" + "user/model-chat", # generic repo must not be hidden + "meta-llama/Llama-3.1-8B-Instruct", + ], +) +def test_normal_models_are_not_hidden(value): + assert is_hidden_model(value) is False + + +def test_is_hidden_model_ignores_empty_values(): + assert is_hidden_model(None) is False + assert is_hidden_model("") is False + assert is_hidden_model(None, "", "unsloth/gemma-3-270m-it-GGUF") is False + + +def test_hidden_model_matchers_expose_probe_needles(): + needles, exact_ids, _exact_paths = models_route.hidden_model_matchers() + lowered = [n.lower() for n in needles] + assert "ggml-org/models" in lowered + assert "stories260k.gguf" in lowered + # The configured embedder is exposed as an exact repo id, never as a + # basename needle that would substring-hide unrelated chat models. + assert "bge-small-en-v1.5" not in lowered + assert "unsloth/bge-small-en-v1.5" in exact_ids + + +def test_hidden_model_matchers_custom_repo_publishes_exact_ids(monkeypatch): + monkeypatch.setattr(rag_config, "effective_embedding_model", lambda: "org/model") + monkeypatch.setattr(rag_config, "effective_gguf_repo", lambda: "org/model-GGUF") + needles, exact_ids, exact_paths = models_route.hidden_model_matchers() + assert needles == ["ggml-org/models", "stories260k.gguf"] + assert "org/model" in exact_ids + assert "org/model-gguf" in exact_ids + assert exact_paths == [] + + +def test_hidden_model_matchers_local_owner_name_path_is_exact_path(monkeypatch, tmp_path): + # A local embedder shaped like owner/name that exists on disk must be an + # exact resolved path, not a Hub repo id (mirroring is_hidden_model), so the + # local row stays hidden instead of showing as a chat model. + (tmp_path / "models" / "embedder").mkdir(parents = True) + monkeypatch.chdir(tmp_path) + monkeypatch.setattr(rag_config, "effective_embedding_model", lambda: "models/embedder") + monkeypatch.setattr(rag_config, "effective_gguf_repo", lambda: "ggml-org/models") + _needles, exact_ids, exact_paths = models_route.hidden_model_matchers() + resolved = str((tmp_path / "models" / "embedder").resolve()).lower() + assert resolved in exact_paths + assert "models/embedder" not in exact_ids + + +# --------------------------------------------------------------------------- # +# HF token via header, query string only as a fallback (the token-leak fix) # +# --------------------------------------------------------------------------- # + + +def test_get_hf_token_strips_and_returns(): + assert get_hf_token(" hf_abc ") == "hf_abc" + + +@pytest.mark.parametrize("value", [None, "", " ", "\n\t"]) +def test_get_hf_token_blank_is_none(value): + assert get_hf_token(value) is None + + +@pytest.mark.parametrize( + "value,expected", + [(" hf_x ", "hf_x"), ("", None), (" ", None), (None, None), (1234, None)], +) +def test_normalize_hf_token(value, expected): + assert models_route._normalize_hf_token(value) == expected + + +def test_header_token_wins_over_query(): + header, query = "hf_header", "hf_query" + resolved = models_route._normalize_hf_token(header) or models_route._normalize_hf_token(query) + assert resolved == "hf_header" + + +def test_query_token_is_fallback_when_header_absent(): + resolved = models_route._normalize_hf_token(None) or models_route._normalize_hf_token( + "hf_query" + ) + assert resolved == "hf_query" + + +# --------------------------------------------------------------------------- # +# Chat-template byte caps (the unbounded-template hardening) # +# --------------------------------------------------------------------------- # + + +def _load_request(**overrides): + data = {"model_path": "unsloth/test-model-GGUF", "gguf_variant": "Q4_K_M"} + data.update(overrides) + return LoadRequest.model_validate(data) + + +def test_blank_chat_template_override_normalizes_to_none(): + assert _load_request(chat_template_override = " \n\t").chat_template_override is None + + +def test_nonblank_chat_template_override_preserved_verbatim(): + template = " {{ messages }} " + assert _load_request(chat_template_override = template).chat_template_override == template + + +def test_chat_template_at_byte_limit_is_accepted(): + template = "a" * MAX_CHAT_TEMPLATE_BYTES # exactly the limit, 1 byte/char + assert ( + len(_load_request(chat_template_override = template).chat_template_override) + == MAX_CHAT_TEMPLATE_BYTES + ) + + +def test_chat_template_over_char_limit_is_rejected(): + with pytest.raises(Exception): # pydantic ValidationError wrapping ValueError + _load_request(chat_template_override = "a" * (MAX_CHAT_TEMPLATE_BYTES + 1)) + + +def test_chat_template_over_byte_limit_is_rejected(): + # Char count stays under the limit but UTF-8 bytes exceed it (3 bytes/char), + # so only the byte-count branch can catch this. + multibyte = "€" * (MAX_CHAT_TEMPLATE_BYTES // 2) # euro sign, 3 bytes each + assert len(multibyte) <= MAX_CHAT_TEMPLATE_BYTES + assert len(multibyte.encode("utf-8")) > MAX_CHAT_TEMPLATE_BYTES + with pytest.raises(Exception): + _load_request(chat_template_override = multibyte) + + +def test_read_bounded_text_reads_within_limit(tmp_path): + p = tmp_path / "t.json" + p.write_text("hello", encoding = "utf-8") + assert _read_bounded_text(p, 16) == "hello" + + +def test_read_bounded_text_rejects_over_limit(tmp_path): + p = tmp_path / "big.json" + p.write_bytes(b"x" * 100) + assert _read_bounded_text(p, 50) is None + + +def test_read_bounded_text_at_limit_is_read(tmp_path): + p = tmp_path / "exact.json" + p.write_bytes(b"x" * 50) + assert _read_bounded_text(p, 50) == "x" * 50 + + +def test_read_bounded_text_missing_file_is_none(tmp_path): + assert _read_bounded_text(tmp_path / "nope.json", 50) is None diff --git a/studio/backend/tests/test_model_update_robustness.py b/studio/backend/tests/test_model_update_robustness.py index edf55812e2..7d98766616 100644 --- a/studio/backend/tests/test_model_update_robustness.py +++ b/studio/backend/tests/test_model_update_robustness.py @@ -314,6 +314,7 @@ def test_cached_model_scan_keeps_local_safetensors_repo(monkeypatch, tmp_path): file_name = "model.safetensors", size_on_disk = 100, blob_path = str(repo_path / "blobs" / "modelsha"), + blob_last_modified = 3_000.0, ), ] ) @@ -336,6 +337,51 @@ def test_cached_model_scan_keeps_local_safetensors_repo(monkeypatch, tmp_path): assert rows[0]["repo_id"] == "Org/SafeTensorRepo" assert rows[0]["model_format"] == "safetensors" assert rows[0]["size_bytes"] == 100 + assert rows[0]["last_modified"] == 3_000.0 + + +def test_cached_gguf_scan_keeps_download_timestamp(monkeypatch, tmp_path): + repo_path = tmp_path / "models--Org--GgufRepo" + repo = SimpleNamespace( + repo_id = "Org/GgufRepo", + repo_type = "model", + repo_path = repo_path, + revisions = [ + SimpleNamespace( + files = [ + SimpleNamespace( + file_name = "model-Q4_K_M.gguf", + size_on_disk = 100, + blob_path = None, + blob_last_modified = 5_000.0, + ), + ] + ) + ], + ) + monkeypatch.setattr( + CI, + "all_hf_cache_scans", + lambda: [SimpleNamespace(repos = [repo])], + ) + monkeypatch.setattr( + CI.hf_cache_scan, + "is_gguf_repo_partial", + lambda *args, **kwargs: False, + ) + monkeypatch.setattr( + CI, + "_gguf_variant_state_summary", + lambda _repo_id: (False, 0), + ) + + rows = CI._scan_cached_gguf() + + assert len(rows) == 1 + assert rows[0]["repo_id"] == "Org/GgufRepo" + assert rows[0]["model_format"] == "gguf" + assert rows[0]["size_bytes"] == 100 + assert rows[0]["last_modified"] == 5_000.0 # ── hf_hub_download_with_xet_fallback force_download bypass (X2/F2) ─── @@ -636,3 +682,18 @@ def test_reclaim_replaced_gguf_variant_keeps_no_symlink_current_file(monkeypatch assert snap.exists() is True # the current file must survive assert result["removed_snapshots"] == 0 assert result["deleted_blobs"] == 0 + + +def _mmproj_repo(*file_names: str): + return SimpleNamespace( + revisions = [SimpleNamespace(files = [SimpleNamespace(file_name = n) for n in file_names])] + ) + + +def test_repo_has_mmproj_requires_gguf_projector(): + # A non-GGUF sidecar whose name merely contains "mmproj" must NOT mark the + # repo vision-capable; the runtime's projector detection is GGUF-only. + assert CI._repo_has_mmproj(_mmproj_repo("model-Q4_K_M.gguf", "mmproj_config.json")) is False + assert CI._repo_has_mmproj(_mmproj_repo("model-Q4_K_M.gguf", "README-mmproj.md")) is False + # A real GGUF projector still marks the repo vision-capable. + assert CI._repo_has_mmproj(_mmproj_repo("model-Q4_K_M.gguf", "mmproj-F16.gguf")) is True diff --git a/studio/backend/tests/test_picker_service.py b/studio/backend/tests/test_picker_service.py new file mode 100644 index 0000000000..be7ea18f03 --- /dev/null +++ b/studio/backend/tests/test_picker_service.py @@ -0,0 +1,266 @@ +# SPDX-License-Identifier: AGPL-3.0-only +# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0 + +import json +from types import SimpleNamespace + +from picker.service import ( + MAX_TEMPLATE_METADATA_BYTES, + _chat_template_from_dir, + _chat_template_from_processor_json, + _chat_template_from_tokenizer_config, + _chat_template_from_tokenizer_dir, + _find_gguf_in_dir, + _iter_ggufs, + read_default_chat_template, + validate_chat_template, +) + + +def test_iter_ggufs_skips_gguf_companions(tmp_path): + mtp_dir = tmp_path / "MTP" + mtp_dir.mkdir() + main = tmp_path / "model-Q8_0.gguf" + main.write_bytes(b"") + (tmp_path / "mmproj-F16.gguf").write_bytes(b"") + (tmp_path / "mtp-model-Q8_0.gguf").write_bytes(b"") + (mtp_dir / "model-Q8_0-MTP.gguf").write_bytes(b"") + (tmp_path / "model-Q8_0-be.gguf").write_bytes(b"") + + assert _iter_ggufs(tmp_path) == [main] + + +def test_find_gguf_in_dir_matches_quant_label(tmp_path): + mtp_dir = tmp_path / "MTP" + mtp_dir.mkdir() + main = tmp_path / "model-Q8_0.gguf" + main.write_bytes(b"") + (mtp_dir / "model-Q8_0-MTP.gguf").write_bytes(b"") + (tmp_path / "model-Q4_K_M.gguf").write_bytes(b"") + + assert _find_gguf_in_dir(tmp_path, "Q8_0") == main + assert _find_gguf_in_dir(tmp_path, "Q4_K") is None + + +def test_find_gguf_in_dir_without_variant_prefers_largest_model(tmp_path): + smaller = tmp_path / "a-model-Q4_K_M.gguf" + larger = tmp_path / "z-model-Q8_0.gguf" + smaller.write_bytes(b"0") + larger.write_bytes(b"00") + + assert _find_gguf_in_dir(tmp_path, None) == larger + + +def test_find_gguf_in_dir_without_variant_prefers_first_split(tmp_path): + first = tmp_path / "model-Q4_K_M-00001-of-00003.gguf" + second = tmp_path / "model-Q4_K_M-00002-of-00003.gguf" + third = tmp_path / "model-Q4_K_M-00003-of-00003.gguf" + first.write_bytes(b"0") + second.write_bytes(b"000") + third.write_bytes(b"00") + + assert _find_gguf_in_dir(tmp_path, None) == first + + first.unlink() + assert _find_gguf_in_dir(tmp_path, None) == second + + +def test_find_gguf_in_dir_matches_bpw_variant_base_label(tmp_path): + target = tmp_path / "model-IQ4_XS-3.53bpw.gguf" + target.write_bytes(b"") + (tmp_path / "model-Q4_K_M.gguf").write_bytes(b"") + + assert _find_gguf_in_dir(tmp_path, "IQ4_XS") == target + assert _find_gguf_in_dir(tmp_path, "IQ4_XS-3.53bpw") == target + assert _find_gguf_in_dir(tmp_path, "Q4_K") is None + + +def test_validate_chat_template_accepts_valid_and_empty(): + assert validate_chat_template("{{ messages[0].content }}").valid is True + assert validate_chat_template("").valid is True + assert validate_chat_template(" ").valid is True + + +def test_validate_chat_template_reports_syntax_error_with_line(): + result = validate_chat_template("{% if %}{% endif %}") + assert result.valid is False + assert result.error is not None + assert result.error.startswith("Line ") + + +def test_chat_template_from_tokenizer_config_reads_string(): + assert _chat_template_from_tokenizer_config({"chat_template": "HELLO"}) == "HELLO" + assert _chat_template_from_tokenizer_config({"chat_template": " "}) is None + assert _chat_template_from_tokenizer_config({}) is None + + +def test_chat_template_from_tokenizer_config_prefers_named_default(): + config = { + "chat_template": [ + {"name": "tool_use", "template": "TOOL"}, + {"name": "default", "template": "DEFAULT"}, + ] + } + assert _chat_template_from_tokenizer_config(config) == "DEFAULT" + + +def test_chat_template_from_tokenizer_config_falls_back_to_first_entry(): + config = { + "chat_template": [ + {"name": "tool_use", "template": "TOOL"}, + {"name": "other", "template": "OTHER"}, + ] + } + assert _chat_template_from_tokenizer_config(config) == "TOOL" + + +def test_chat_template_from_tokenizer_dir_prefers_jinja_file(tmp_path): + (tmp_path / "chat_template.jinja").write_text("FROM_JINJA", encoding = "utf-8") + (tmp_path / "tokenizer_config.json").write_text( + json.dumps({"chat_template": "FROM_CONFIG"}), encoding = "utf-8" + ) + assert _chat_template_from_tokenizer_dir(tmp_path) == "FROM_JINJA" + + +def test_chat_template_from_tokenizer_dir_reads_tokenizer_config(tmp_path): + (tmp_path / "tokenizer_config.json").write_text( + json.dumps({"chat_template": "FROM_CONFIG"}), encoding = "utf-8" + ) + assert _chat_template_from_tokenizer_dir(tmp_path) == "FROM_CONFIG" + + +def test_chat_template_from_dir_without_variant_prefers_tokenizer(tmp_path): + (tmp_path / "tokenizer_config.json").write_text( + json.dumps({"chat_template": "FROM_CONFIG"}), encoding = "utf-8" + ) + assert _chat_template_from_dir(tmp_path) == "FROM_CONFIG" + + +def test_chat_template_from_dir_with_variant_still_prefers_tokenizer(tmp_path, monkeypatch): + (tmp_path / "tokenizer_config.json").write_text( + json.dumps({"chat_template": "FROM_CONFIG"}), encoding = "utf-8" + ) + (tmp_path / "model-Q4_K_M.gguf").write_bytes(b"") + monkeypatch.setattr("picker.service.read_gguf_chat_template", lambda _path: "FROM_GGUF") + # Selecting a variant must not flip precedence to the embedded GGUF template. + assert _chat_template_from_dir(tmp_path, "Q4_K_M") == "FROM_CONFIG" + + +def test_chat_template_from_dir_with_variant_falls_back_to_gguf(tmp_path, monkeypatch): + (tmp_path / "model-Q4_K_M.gguf").write_bytes(b"") + monkeypatch.setattr("picker.service.read_gguf_chat_template", lambda _path: "FROM_GGUF") + # With no tokenizer sidecar, the embedded GGUF template is still the fallback. + assert _chat_template_from_dir(tmp_path, "Q4_K_M") == "FROM_GGUF" + + +def test_chat_template_from_dir_returns_none_when_absent(tmp_path): + assert _chat_template_from_dir(tmp_path) is None + + +def test_read_default_chat_template_direct_gguf_prefers_sidecar(tmp_path, monkeypatch): + gguf = tmp_path / "model-Q4_K_M.gguf" + gguf.write_bytes(b"") + (tmp_path / "tokenizer_config.json").write_text( + json.dumps({"chat_template": "FROM_CONFIG"}), encoding = "utf-8" + ) + monkeypatch.setattr("picker.service._build_browse_allowlist", lambda: [tmp_path]) + monkeypatch.setattr("picker.service.read_gguf_chat_template", lambda _path: "FROM_GGUF") + # A directly selected .gguf must prefer a maintained sidecar over its embedded copy. + assert read_default_chat_template(str(gguf)) == "FROM_CONFIG" + + +def test_read_default_chat_template_direct_gguf_falls_back_to_embedded(tmp_path, monkeypatch): + gguf = tmp_path / "model-Q4_K_M.gguf" + gguf.write_bytes(b"") + monkeypatch.setattr("picker.service._build_browse_allowlist", lambda: [tmp_path]) + monkeypatch.setattr("picker.service.read_gguf_chat_template", lambda _path: "FROM_GGUF") + # With no sidecar next to the file, the embedded GGUF template is the fallback. + assert read_default_chat_template(str(gguf)) == "FROM_GGUF" + + +def test_tokenizer_config_over_size_limit_is_skipped_not_parsed(tmp_path): + # An oversized tokenizer_config.json must be skipped before json.loads so a + # hostile sidecar cannot exhaust memory. + padding = "x" * (MAX_TEMPLATE_METADATA_BYTES + 1024) + (tmp_path / "tokenizer_config.json").write_text( + json.dumps({"chat_template": "HELLO", "_pad": padding}), encoding = "utf-8" + ) + assert _chat_template_from_tokenizer_dir(tmp_path) is None + + +def test_processor_json_over_size_limit_is_skipped_not_parsed(tmp_path): + padding = "x" * (MAX_TEMPLATE_METADATA_BYTES + 1024) + (tmp_path / "chat_template.json").write_text( + json.dumps({"default": "HELLO", "_pad": padding}), encoding = "utf-8" + ) + assert _chat_template_from_processor_json(tmp_path) is None + + +def test_tokenizer_config_at_size_limit_is_still_read(tmp_path): + # A normal-sized config is unaffected by the bound (regression guard). + (tmp_path / "tokenizer_config.json").write_text( + json.dumps({"chat_template": "FROM_CONFIG"}), encoding = "utf-8" + ) + assert _chat_template_from_tokenizer_dir(tmp_path) == "FROM_CONFIG" + + +def test_remote_template_over_size_limit_is_skipped_before_download(monkeypatch): + # An uncached Hub repo whose template exceeds the cap must be skipped via the + # remote size pre-check, never downloaded. + import huggingface_hub + + monkeypatch.setattr("picker.service.resolve_cached_repo_id_case", lambda name: name) + monkeypatch.setattr("picker.service.iter_hf_cache_snapshots", lambda resolved: []) + + def _fail_download(*args, **kwargs): + raise AssertionError("oversized remote template must not be downloaded") + + def _fake_get_paths_info(self, repo_id, paths, **kwargs): + return [SimpleNamespace(path = p, size = MAX_TEMPLATE_METADATA_BYTES + 1) for p in paths] + + monkeypatch.setattr(huggingface_hub, "hf_hub_download", _fail_download) + monkeypatch.setattr(huggingface_hub.HfApi, "get_paths_info", _fake_get_paths_info) + + assert read_default_chat_template("org/oversized-model") is None + + +def test_remote_oversized_jinja_falls_through_to_tokenizer_template(tmp_path, monkeypatch): + # A raw chat_template.jinja between the response cap (MAX_CHAT_TEMPLATE_BYTES) + # and the download bound (MAX_TEMPLATE_METADATA_BYTES) must not be returned: the + # route drops it, so the remote path must skip the oversized Jinja and fall + # through to the smaller tokenizer_config.json. + import huggingface_hub + from picker.schemas import MAX_CHAT_TEMPLATE_BYTES + + big_jinja = tmp_path / "chat_template.jinja" + big_jinja.write_text("{{ x }}" * (MAX_CHAT_TEMPLATE_BYTES // 4), encoding = "utf-8") + assert MAX_CHAT_TEMPLATE_BYTES < big_jinja.stat().st_size < MAX_TEMPLATE_METADATA_BYTES + tokenizer_config = tmp_path / "tokenizer_config.json" + tokenizer_config.write_text(json.dumps({"chat_template": "SMALL_TEMPLATE"}), encoding = "utf-8") + files = { + "chat_template.jinja": big_jinja, + "tokenizer_config.json": tokenizer_config, + } + + monkeypatch.setattr("picker.service.resolve_cached_repo_id_case", lambda name: name) + monkeypatch.setattr("picker.service.iter_hf_cache_snapshots", lambda resolved: []) + + def _fake_download(repo_id, rel, **kwargs): + target = files.get(rel) + if target is None: + raise FileNotFoundError(rel) + return str(target) + + def _fake_get_paths_info(self, repo_id, paths, **kwargs): + return [ + SimpleNamespace( + path = p, + size = files[p].stat().st_size if p in files else 0, + ) + for p in paths + ] + + monkeypatch.setattr(huggingface_hub, "hf_hub_download", _fake_download) + monkeypatch.setattr(huggingface_hub.HfApi, "get_paths_info", _fake_get_paths_info) + + assert read_default_chat_template("org/big-jinja-model") == "SMALL_TEMPLATE" diff --git a/studio/backend/utils/models/gguf_metadata.py b/studio/backend/utils/models/gguf_metadata.py index 50b3cd3513..749f2c9234 100644 --- a/studio/backend/utils/models/gguf_metadata.py +++ b/studio/backend/utils/models/gguf_metadata.py @@ -50,10 +50,14 @@ _CACHE_MAX_ENTRIES = 4096 # keyed by (file cache key, wanted key). None = key absent / file unreadable. _BOOL_CACHE: Dict[Tuple[_CacheKey, str], Optional[bool]] = {} +_STRING_CACHE: Dict[Tuple[_CacheKey, str], Optional[str]] = {} + # GGUF header dims for the staged/deferred-load UI: context_length, layer_count # (block_count), and moe_layer_count (block_count minus leading dense layers; 0 # if not MoE). One cached pass fills all three so the staged sheet can size every -# slider before the model loads. None = unreadable / not a GGUF. +# slider before the model loads. None = unreadable / not a GGUF. The native +# training context length (``{arch}.context_length``) the UI shows before a model +# loads is read from here via read_gguf_context_length. _DIMS_CACHE: Dict[_CacheKey, Optional[Dict[str, Optional[int]]]] = {} @@ -408,6 +412,83 @@ def _read_gguf_bool(path: str, wanted_key: str) -> Optional[bool]: return result +def _parse_gguf_string(path: str, wanted_key: str) -> Optional[str]: + try: + with open(path, "rb") as f: + head = f.read(24) + if len(head) < 24: + return None + magic, _version, _tcount, kv_count = struct.unpack(" 1 << 20: + break + kbytes = f.read(klen) + if len(kbytes) < klen: + break + key = kbytes.decode("utf-8", "replace") + vt_bytes = f.read(4) + if len(vt_bytes) < 4: + break + vtype = struct.unpack(" 1 << 22: + break + sbytes = f.read(slen) + if len(sbytes) < slen: + break + return sbytes.decode("utf-8", "replace") + if not _skip_gguf_value(f, vtype): + break + except (struct.error, UnicodeDecodeError): + break + except OSError as e: + logger.debug(f"_parse_gguf_string: cannot open {path}: {e}") + return None + except Exception as e: + logger.debug(f"_parse_gguf_string: parse failure on {path}: {e}") + return None + return None + + +def _read_gguf_string(path: str, wanted_key: str) -> Optional[str]: + fkey = _cache_key(path) + if fkey is None: + return None + ckey = (fkey, wanted_key) + with _CACHE_LOCK: + if ckey in _STRING_CACHE: + return _STRING_CACHE[ckey] + result = _parse_gguf_string(path, wanted_key) + with _CACHE_LOCK: + while len(_STRING_CACHE) >= _CACHE_MAX_ENTRIES: + try: + _STRING_CACHE.pop(next(iter(_STRING_CACHE))) + except StopIteration: + break + _STRING_CACHE[ckey] = result + return result + + +def read_gguf_chat_template(path: str) -> Optional[str]: + template = _read_gguf_string(path, "tokenizer.chat_template") + if isinstance(template, str) and template.strip(): + return template + return None + + def read_mmproj_audio_capability(path: str) -> Optional[bool]: """``clip.has_audio_encoder`` from an mmproj GGUF (e.g. Gemma 4's gemma4ua): ``True``/``False`` if present, ``None`` if absent/unreadable. diff --git a/studio/frontend/src/app/routes/__root.tsx b/studio/frontend/src/app/routes/__root.tsx index 57e890dd5a..7137fd6f96 100644 --- a/studio/frontend/src/app/routes/__root.tsx +++ b/studio/frontend/src/app/routes/__root.tsx @@ -196,9 +196,6 @@ function RootLayout() { chatRuntime.setActiveThreadId(null); chatRuntime.setActiveProjectId(null); chatRuntime.setIncognito(false); - // Detach the staging UI but keep any in-flight download running, like Hub. - if (chatRuntime.pendingSelection) - chatRuntime.abandonStagedModel({ keepDownload: true }); void navigate({ to: "/chat", search: { new: crypto.randomUUID() }, @@ -221,10 +218,6 @@ function RootLayout() { chatRuntime.setActiveProjectId(null); chatRuntime.setActiveThreadId(null); chatRuntime.setIncognito(false); - // Leaving chat must not kill an in-flight download: detach the staging UI - // but keep the transfer running in the manager, like a Hub download. - if (chatRuntime.pendingSelection) - chatRuntime.abandonStagedModel({ keepDownload: true }); }, [isChatRoute]); return ( diff --git a/studio/frontend/src/components/app-sidebar.tsx b/studio/frontend/src/components/app-sidebar.tsx index 8eab03133b..293971904f 100644 --- a/studio/frontend/src/components/app-sidebar.tsx +++ b/studio/frontend/src/components/app-sidebar.tsx @@ -1011,28 +1011,6 @@ export function AppSidebar() { - {isPinned ? ( - - - - - - Unpin - - - ) : null} ); } diff --git a/studio/frontend/src/components/assistant-ui/model-selector/remembered-load-settings.ts b/studio/frontend/src/components/assistant-ui/model-selector/remembered-load-settings.ts deleted file mode 100644 index 08492ab480..0000000000 --- a/studio/frontend/src/components/assistant-ui/model-selector/remembered-load-settings.ts +++ /dev/null @@ -1,79 +0,0 @@ -// SPDX-License-Identifier: AGPL-3.0-only -// Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0 - -// Per-model pre-load inference settings, persisted in localStorage so the load -// dialog can offer "Remember settings for ". GGUF picks only: every -// field is a llama.cpp load knob, so all save/restore call sites gate on -// GGUF-ness (a non-GGUF blob would only snapshot leftover standing values). - -const KEY = "unsloth_load_settings"; - -export interface RememberedLoadSettings { - contextLength: number | null; - kvCacheDtype: string | null; - speculativeType: string | null; - specDraftNMax: number | null; - tensorParallel: boolean; - // GPU Memory controls. Optional so an older blob (which lacked them) still - // parses, leaving the live knobs untouched on apply. The mode is kept with the - // manual knobs (gpuLayers/nCpuMoe are ignored outside Manual mode). A null - // selectedGpuIds is meaningful (all GPUs), so it's distinguished from absent. - // The per-GPU split ratio is deliberately NOT remembered: it's positionally - // bound to the exact GPU set/order and unvalidated, so it would mismatch. - gpuMemoryMode?: "auto" | "manual"; - gpuLayers?: number; - nCpuMoe?: number; - selectedGpuIds?: number[] | null; -} - -// Storage key for a pick's remembered settings, scoped per quant (the VRAM-budget -// knobs differ per quant). An HF repo collapses its GGUF variants into one `id`, -// so fold the variant in. Local .gguf paths are already file-specific; native -// drag-drop files key by display label, so same-named files share an entry. -export function rememberedLoadSettingsKey(selection: { - id: string; - ggufVariant?: string | null; -}): string { - return selection.ggufVariant - ? `${selection.id}::${selection.ggufVariant}` - : selection.id; -} - -function readAll(): Record { - try { - return JSON.parse(localStorage.getItem(KEY) ?? "{}"); - } catch { - return {}; - } -} - -function writeAll(all: Record) { - try { - localStorage.setItem(KEY, JSON.stringify(all)); - } catch { - // Ignore quota / unavailable storage. - } -} - -export function loadRememberedLoadSettings( - key: string, -): RememberedLoadSettings | null { - return readAll()[key] ?? null; -} - -export function saveRememberedLoadSettings( - key: string, - settings: RememberedLoadSettings, -) { - const all = readAll(); - all[key] = settings; - writeAll(all); -} - -export function clearRememberedLoadSettings(key: string) { - const all = readAll(); - if (key in all) { - delete all[key]; - writeAll(all); - } -} diff --git a/studio/frontend/src/features/chat/api/chat-adapter.ts b/studio/frontend/src/features/chat/api/chat-adapter.ts index 7083f02288..b0127b5e40 100644 --- a/studio/frontend/src/features/chat/api/chat-adapter.ts +++ b/studio/frontend/src/features/chat/api/chat-adapter.ts @@ -2,10 +2,7 @@ // Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0 import { getAuthToken } from "@/features/auth"; -import { - loadRememberedLoadSettings, - rememberedLoadSettingsKey, -} from "@/components/assistant-ui/model-selector/remembered-load-settings"; +import { resolveInitialConfig } from "@/features/model-picker"; import { projectHasSources } from "@/features/rag/api/rag-api"; import { apiUrl } from "@/lib/api-base"; import { parseParamCountB } from "@/lib/model-size"; @@ -46,7 +43,7 @@ import { type PendingImageEditReference, type RagAutoInject, GPU_LAYERS_AUTO, - loadedGpuMemoryFieldsUnlessStaged, + loadedGpuMemoryFields, reconcilePersistedGpuIds, resolveLoadedSpeculativeSettings, resolveSpeculativeSettingsForLoad, @@ -1533,65 +1530,56 @@ async function autoLoadSmallestModel(): Promise<{ return false; } const currentStore = useChatRuntimeStore.getState(); - // Blobs are saved for GGUF picks only (the sheet gates on it), so don't - // let a legacy non-GGUF blob feed a stale context/spec choice into a - // safetensors auto-load. - const remembered = - candidate.kind === "gguf" - ? loadRememberedLoadSettings( - rememberedLoadSettingsKey({ - id: candidate.id, - ggufVariant: candidate.ggufVariant, - }), - ) - : null; + const { config } = resolveInitialConfig(candidate.id, candidate.ggufVariant); const effectiveMaxSeqLength = resolveLoadMaxSeqLength({ modelId: candidate.id, ggufVariant: candidate.ggufVariant, isGguf: candidate.kind === "gguf", - customContextLength: remembered?.contextLength ?? null, + customContextLength: config.customContextLength, ggufContextLength: null, currentCheckpoint: currentStore.params.checkpoint, activeGgufVariant: currentStore.activeGgufVariant, - maxSeqLength: candidate.maxSeqLength, + maxSeqLength: config.maxSeqLength ?? candidate.maxSeqLength, presetSource: currentStore.activePresetSource, }); - // The GPU knobs are per-model, so read them from the same remembered - // settings that fed effectiveMaxSeqLength -- on a background auto-load the - // live store holds session defaults, not the saved Manual mode / layer pin / - // GPU pick. Absent fields fall back like applyRememberedLoadSettings: the - // mode to the store (a persisted standing preference), the per-model knobs to - // their defaults. The saved GPU pick is reconciled against the GPUs present - // now, like the interactive restore. + // The GPU knobs are per-model, so read them from the same per-model config + // that fed effectiveMaxSeqLength -- on a background auto-load the live store + // holds session defaults, not the saved Manual mode / layer pin / GPU pick. + // Absent fields fall back like the interactive restore: the mode to the store + // (a persisted standing preference), the per-model knobs to their defaults. + // The saved GPU pick is reconciled against the GPUs present now. const effectiveGpuMemoryMode = - remembered?.gpuMemoryMode ?? currentStore.gpuMemoryMode; - const effectiveGpuLayers = remembered?.gpuLayers ?? GPU_LAYERS_AUTO; - const effectiveNCpuMoe = remembered?.nCpuMoe ?? 0; - if (remembered?.selectedGpuIds != null) { + config.gpuMemoryMode ?? currentStore.gpuMemoryMode; + const effectiveGpuLayers = config.gpuLayers ?? GPU_LAYERS_AUTO; + const effectiveNCpuMoe = config.nCpuMoe ?? 0; + if (config.selectedGpuIds != null) { // Warm the device cache first: on a cold cache the reconcile passes the // saved pick through unvalidated, and a stale cross-host pick then fails // the load with the picker hidden. await ensureGpuDeviceCache(); } const effectiveGpuIds = - remembered?.selectedGpuIds !== undefined - ? reconcilePersistedGpuIds(remembered.selectedGpuIds) + config.selectedGpuIds !== undefined + ? reconcilePersistedGpuIds(config.selectedGpuIds) : null; // Under Manual GPU memory + Auto layers, llama.cpp's --fit owns context // sizing, so send 0 (or the pinned length). GGUF-only; a no-op otherwise. - // The context pin is per-model too, so it comes from remembered settings, - // not the live store. + // The context pin is per-model too, so it comes from the saved config, not + // the live store. const fitMaxSeqLength = resolveFitMaxSeqLength( candidate.kind === "gguf", effectiveGpuMemoryMode, effectiveGpuLayers, - remembered?.contextLength ?? null, + config.customContextLength ?? null, effectiveMaxSeqLength, ); const effectiveSpeculativeType = - remembered?.speculativeType ?? specSettings.speculativeType; + config.speculativeType ?? specSettings.speculativeType; const effectiveSpecDraftNMax = - remembered?.specDraftNMax ?? specSettings.specDraftNMax; + config.specDraftNMax ?? specSettings.specDraftNMax; + const effectiveChatTemplateOverride = config.chatTemplateOverride?.trim() + ? config.chatTemplateOverride + : null; if ( !(await canAutoLoad({ model_path: candidate.id, @@ -1621,10 +1609,11 @@ async function autoLoadSmallestModel(): Promise<{ is_lora: false, gguf_variant: candidate.ggufVariant, trust_remote_code: trustRemoteCode, - cache_type_kv: remembered?.kvCacheDtype ?? null, + chat_template_override: effectiveChatTemplateOverride, + cache_type_kv: config.kvCacheDtype, speculative_type: effectiveSpeculativeType, spec_draft_n_max: effectiveSpecDraftNMax, - tensor_parallel: remembered?.tensorParallel ?? false, + tensor_parallel: config.tensorParallel, // GGUF-only: the safetensors fallback loads via HF auto-placement (no // explicit pins). The split ratio is deliberately never remembered // (positionally bound to an exact GPU set), so auto-load leaves llama.cpp's @@ -1638,7 +1627,12 @@ async function autoLoadSmallestModel(): Promise<{ } : {}), }); - saveSpeculativeType(effectiveSpeculativeType); + // Only persist the global preference when the value came from the global + // settings. A per-model config's choice must stay load-local, or autoloading + // a remembered model on startup would rewrite the global default. + if (config.speculativeType == null) { + saveSpeculativeType(effectiveSpeculativeType); + } // Self-gates on is_gguf (skips diffusion), so persists only for a real GGUF load. persistGpuMemoryModeOnLoad(loadResp, effectiveGpuMemoryMode); useChatRuntimeStore @@ -1650,6 +1644,9 @@ async function autoLoadSmallestModel(): Promise<{ ); store.setParams({ ...store.params, + ...(candidate.kind === "gguf" + ? {} + : { maxSeqLength: effectiveMaxSeqLength }), maxTokens: candidate.kind === "gguf" ? loadResp.context_length ?? 131072 @@ -1676,7 +1673,7 @@ async function autoLoadSmallestModel(): Promise<{ const keepCustomCtx = resolveManualAutoCtxPin( effectiveGpuMemoryMode, effectiveGpuLayers, - remembered?.contextLength ?? null, + config.customContextLength ?? null, ); useChatRuntimeStore.setState({ ggufContextLength: loadResp.context_length ?? 131072, @@ -1694,13 +1691,14 @@ async function autoLoadSmallestModel(): Promise<{ loadedKvCacheDtype: loadResp.cache_type_kv ?? null, tensorParallel: loadResp.tensor_parallel ?? false, loadedTensorParallel: loadResp.tensor_parallel ?? false, - ...loadedGpuMemoryFieldsUnlessStaged(loadResp, { - customContextLength: keepCustomCtx, - }), + ...loadedGpuMemoryFields(loadResp), loadedCustomContextLength: keepCustomCtx, defaultChatTemplate: loadResp.chat_template ?? null, - chatTemplateOverride: null, - loadedChatTemplateOverride: null, + chatTemplateOverride: effectiveChatTemplateOverride, + loadedChatTemplateOverride: effectiveChatTemplateOverride, + // Retain the saved requested context so re-saving the config keeps the + // override; null stays null (auto/VRAM-fit). + customContextLength: config.customContextLength, loadedIsMultimodal: isMultimodalResponse(loadResp), loadedIsDiffusion: loadResp.is_diffusion ?? false, ...resolveLoadedSpeculativeSettings(loadResp), @@ -1720,10 +1718,11 @@ async function autoLoadSmallestModel(): Promise<{ loadedTensorParallel: loadResp.tensor_parallel ?? false, // Non-GGUF response: clears any stale GPU baseline a prior manual-GPU // GGUF load left, matching the interactive/status sibling load paths. - ...loadedGpuMemoryFieldsUnlessStaged(loadResp), + ...loadedGpuMemoryFields(loadResp), defaultChatTemplate: loadResp.chat_template ?? null, - chatTemplateOverride: null, - loadedChatTemplateOverride: null, + chatTemplateOverride: effectiveChatTemplateOverride, + loadedChatTemplateOverride: effectiveChatTemplateOverride, + customContextLength: null, ...resolveLoadedSpeculativeSettings(loadResp), loadedIsMultimodal: isMultimodalResponse(loadResp), loadedIsDiffusion: loadResp.is_diffusion ?? false, @@ -1988,7 +1987,7 @@ async function autoLoadSmallestModel(): Promise<{ loadedKvCacheDtype: loadResp.cache_type_kv ?? null, tensorParallel: loadResp.tensor_parallel ?? false, loadedTensorParallel: loadResp.tensor_parallel ?? false, - ...loadedGpuMemoryFieldsUnlessStaged(loadResp), + ...loadedGpuMemoryFields(loadResp), // Drives the GPU Memory controls' diffusion gate; set alongside the // GPU fields on every load path so the gate can't read stale. loadedIsDiffusion: loadResp.is_diffusion ?? false, diff --git a/studio/frontend/src/features/chat/api/chat-api.ts b/studio/frontend/src/features/chat/api/chat-api.ts index 631474c39a..de3e5e370c 100644 --- a/studio/frontend/src/features/chat/api/chat-api.ts +++ b/studio/frontend/src/features/chat/api/chat-api.ts @@ -377,14 +377,33 @@ export async function listCachedModels( return data.cached; } -export async function deleteCachedModel( +export interface CachedModelPath { + path: string; + is_dir: boolean; +} + +/** Absolute on-disk path of a cached repo or one of its GGUF variants. */ +export async function getCachedModelPath( + repoId: string, + variant?: string, +): Promise { + const params = new URLSearchParams({ repo_id: repoId }); + if (variant) params.set("variant", variant); + const response = await authFetch( + `/api/models/cached-model-path?${params.toString()}`, + ); + return parseJsonOrThrow(response); +} + +/** Reveal a cached repo (or one GGUF variant's file) in the OS file manager. */ +export async function revealCachedModel( repoId: string, variant?: string, ): Promise { const payload: Record = { repo_id: repoId }; if (variant) payload.variant = variant; - const response = await authFetch("/api/models/delete-cached", { - method: "DELETE", + const response = await authFetch("/api/models/reveal-cached-model", { + method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(payload), }); diff --git a/studio/frontend/src/features/chat/chat-page.tsx b/studio/frontend/src/features/chat/chat-page.tsx index 217eaf8b6d..ef018445e0 100644 --- a/studio/frontend/src/features/chat/chat-page.tsx +++ b/studio/frontend/src/features/chat/chat-page.tsx @@ -2,16 +2,19 @@ // Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0 import { + applyModelLoadConfigToRuntime, + currentRuntimePerModelConfig, type DeletedModelRef, type ExternalModelOption, type LoraModelOption, type ModelOption, ModelSelector, -} from "@/components/assistant-ui/model-selector"; -import { - loadRememberedLoadSettings, - rememberedLoadSettingsKey, -} from "@/components/assistant-ui/model-selector/remembered-load-settings"; + type ModelSelectorChangeMeta, + type PerModelConfig, + resolveInitialConfig, + SidebarModelConfig, + useActiveModelConfig, +} from "@/features/model-picker"; import { ProjectComposer, Thread } from "@/components/assistant-ui/thread"; import { CopyableErrorChip } from "@/components/ui/copyable-error-chip"; import { @@ -27,10 +30,10 @@ import { } from "@/components/ui/resizable"; import { useSidebar } from "@/components/ui/sidebar"; import { Tooltip, TooltipContent } from "@/components/ui/tooltip"; -import { useLatestRef } from "@/features/hub/hooks/use-latest-ref"; import { DOWNLOAD_KIND, downloadManager, + useRepoDownload, } from "@/features/hub/download-manager"; import { type NativeIntent, @@ -93,7 +96,6 @@ import { renameChatItem, useChatSidebarItems, } from "./hooks/use-chat-sidebar-items"; -import { useStagedModelPreparation } from "./hooks/use-staged-model-preparation"; import { clearTrainingCompareHandoff, getTrainingCompareHandoff, @@ -128,10 +130,8 @@ import { hasGgufSource, isDownloadableHubRepo, loadOptionalBool, - pendingSelectionMatches, useChatRuntimeStore, } from "./stores/chat-runtime-store"; -import type { PendingModelSelection } from "./stores/chat-runtime-store"; import { useChatPreferencesStore } from "./stores/chat-preferences-store"; import { useExternalProvidersStore } from "./stores/external-providers-store"; import { buildChatTourSteps } from "./tour"; @@ -385,6 +385,7 @@ type CompareModelSelection = { id: string; isLora: boolean; ggufVariant?: string; + config?: PerModelConfig; }; function modelMatchesDeleted( @@ -645,6 +646,8 @@ function GeneralCompareHeader({ loraModels, externalModels, value, + selectedConfig, + selectedGgufVariant, onValueChange, onFoldersChange, onModelsChange, @@ -655,9 +658,11 @@ function GeneralCompareHeader({ loraModels: LoraModelOption[]; externalModels: ExternalModelOption[]; value: string; + selectedConfig?: PerModelConfig | null; + selectedGgufVariant?: string | null; onValueChange: ( id: string, - meta: { isLora: boolean; ggufVariant?: string }, + meta: ModelSelectorChangeMeta, ) => void; onFoldersChange?: () => void; onModelsChange?: (deletedModel?: DeletedModelRef) => void; @@ -684,6 +689,8 @@ function GeneralCompareHeader({ loraModels={loraModels} externalModels={externalModels} value={value} + selectedConfig={selectedConfig} + selectedGgufVariant={selectedGgufVariant} onValueChange={onValueChange} onFoldersChange={onFoldersChange} onModelsChange={onModelsChange} @@ -811,11 +818,14 @@ const GeneralCompareContent = memo(function GeneralCompareContent({ loraModels={loraModels} externalModels={externalModels} value={model1.id} + selectedConfig={model1.config} + selectedGgufVariant={model1.ggufVariant} onValueChange={(id, meta) => setModel1({ id, isLora: meta.isLora, ggufVariant: meta.ggufVariant, + config: meta.config, }) } onFoldersChange={onFoldersChange} @@ -838,11 +848,14 @@ const GeneralCompareContent = memo(function GeneralCompareContent({ loraModels={loraModels} externalModels={externalModels} value={model2.id} + selectedConfig={model2.config} + selectedGgufVariant={model2.ggufVariant} onValueChange={(id, meta) => setModel2({ id, isLora: meta.isLora, ggufVariant: meta.ggufVariant, + config: meta.config, }) } onFoldersChange={onFoldersChange} @@ -1236,6 +1249,13 @@ export function validateChatSearch(search: Record): ChatSearch }; } +type PendingHubAutoLoad = { + selection: SelectedModelInput; + contextKey: string; + originCheckpoint: string; + originGgufVariant: string | null; +}; + // `search` comes from RootLayout (not useSearch) so ChatPage stays mounted off-route // (keeping an in-flight generation alive), frozen to the last /chat search. `active` // is false off-route: close body-portaled surfaces and stop route-specific listeners @@ -1248,30 +1268,6 @@ export function ChatPage({ const settingsOpen = useChatRuntimeStore((s) => s.settingsPanelOpen); const setSettingsOpen = useChatRuntimeStore((s) => s.setSettingsPanelOpen); - // Deferred-load staging: downloads a staged GGUF (if needed) and reads its - // header context so the sheet can show the context slider before the load. - // autoLoad picks instead load the cached file as soon as the download ends; - // selectModel is defined below, so the load runs through a ref. - const autoLoadStagedRef = useRef< - ((pending: PendingModelSelection) => void) | null - >(null); - const stagedDownload = useStagedModelPreparation({ - onAutoLoad: (pending) => autoLoadStagedRef.current?.(pending), - }); - // Abandon a staged pick: the store action cancels its in-flight download and - // reverts the edited knobs, so nothing lingers after the user walks away. - const abandonStaged = useCallback(() => { - useChatRuntimeStore.getState().abandonStagedModel(); - }, []); - // Detach a staged pick on navigation without cancelling its download: the - // transfer keeps running in the manager and lands in cache, like Hub. - const detachStaged = useCallback(() => { - useChatRuntimeStore.getState().abandonStagedModel({ keepDownload: true }); - }, []); - // Tracks whether the chat page is still mounted, so a staged-load failure that - // resolves after the user left chat doesn't resurrect the abandoned pick. - const mountedRef = useRef(true); - useEffect(() => () => void (mountedRef.current = false), []); const incognito = useChatRuntimeStore((s) => s.incognito); const setIncognito = useChatRuntimeStore((s) => s.setIncognito); const incognitoLabel = incognito @@ -1363,6 +1359,9 @@ export function ChatPage({ const ggufContextLength = useChatRuntimeStore( (state) => state.ggufContextLength, ); + const ggufNativeContextLength = useChatRuntimeStore( + (state) => state.ggufNativeContextLength, + ); const contextUsage = useChatRuntimeStore((state) => state.contextUsage); const modelsFromStore = useChatRuntimeStore((state) => state.models); const lorasFromStore = useChatRuntimeStore((state) => state.loras); @@ -1440,39 +1439,37 @@ export function ChatPage({ refreshRef.current = refresh; selectModelRef.current = selectModel; }, [refresh, selectModel]); - // Load a cached autoLoad pick once its download finishes. The sheet was never - // opened, so on a load failure just drop the orphaned staged knobs. The knobs - // were already seeded on stage, so keepSpeculative only when a config was - // saved -- otherwise the standing speculative preference should win. - autoLoadStagedRef.current = (pending) => { - // Blobs are saved for GGUF picks only (the sheet gates on it), so don't - // let a legacy non-GGUF blob claim a seeded config here. - const remembered = hasGgufSource(pending) - ? loadRememberedLoadSettings(rememberedLoadSettingsKey(pending)) - : null; - void selectModel({ - ...pending, - isDownloaded: true, - forceReload: true, - keepSpeculative: remembered != null, - throwOnError: true, - }).catch(() => { - const store = useChatRuntimeStore.getState(); - // selectModel only clears pendingSelection on success, so a failed - // auto-load leaves our staged pick (and its edited load knobs) behind. - // Abandon it when it is still the active stage; otherwise just revert the - // settings if the stage was already cleared by something else. - if (pendingSelectionMatches(store.pendingSelection, pending)) { - store.abandonStagedModel(); - } else if (!store.pendingSelection) { - store.resetModelSettingsToLoaded(); - } - }); - }; + const rememberedConfigFor = useCallback( + (selection: { + id: string; + ggufVariant?: string | null; + source?: string; + }) => { + if (selection.source === "external") return null; + const resolved = resolveInitialConfig(selection.id, selection.ggufVariant); + return resolved.remembered ? resolved.config : null; + }, + [], + ); const isExternalModel = useMemo( () => isExternalModelId(inferenceParams.checkpoint), [inferenceParams.checkpoint], ); + const { + checkpoint: runtimeCheckpoint, + isGguf: runtimeModelIsGguf, + config: activeModelConfig, + } = useActiveModelConfig(); + const activeModelIsGguf = + runtimeCheckpoint != null && !isExternalModel && runtimeModelIsGguf; + const activeModelIsLora = useMemo(() => { + const checkpoint = inferenceParams.checkpoint; + if (!checkpoint || isExternalModel) return false; + const model = modelsFromStore.find((entry) => entry.id === checkpoint); + if (model) return model.isLora; + const lora = lorasFromStore.find((entry) => entry.id === checkpoint); + return lora?.exportType === "lora"; + }, [inferenceParams.checkpoint, isExternalModel, modelsFromStore, lorasFromStore]); const reasoningEnabled = useChatRuntimeStore((s) => s.reasoningEnabled); const reasoningStyle = useChatRuntimeStore((s) => s.reasoningStyle); const reasoningEffort = useChatRuntimeStore((s) => s.reasoningEffort); @@ -1783,75 +1780,21 @@ export function ChatPage({ closeArtifactSurface(); }, [activeThreadId, closeArtifactSurface, selectedArtifact, view]); - // Abandon a staged (not-yet-loaded) pick when the chat context actually - // changes — switching threads, leaving single view, or starting a new chat / - // project — so a stale Load button can't resurface in a different context. - // New Chat keeps activeThreadId null and only bumps the `new` search nonce, so - // the key includes the route identity, not just the thread. Mirrors the - // incognito reset pattern. (Route exit is handled in __root.tsx, which runs - // after this unmounts.) Clear only on a real change, never on mount: staging - // from the Hub sets pendingSelection then navigates here, and clearing on - // mount would wipe it. Comparing the previous context (rather than a first-run - // flag) is also safe under StrictMode's double-invoke and component remounts. - const chatContextKey = `${view.mode}|${activeThreadId ?? ""}|${search.new ?? ""}|${search.project ?? ""}`; - const chatContextKeyRef = useLatestRef(chatContextKey); - const prevChatContextRef = useRef(null); - useEffect(() => { - const prev = prevChatContextRef.current; - prevChatContextRef.current = chatContextKey; - if (prev === null || prev === chatContextKey) return; - detachStaged(); - }, [chatContextKey, detachStaged]); - const hasActiveModel = Boolean(inferenceParams.checkpoint); - // Load immediately, or — when "Load on selection" is off — stage the pick so - // its load options can be set first. Shared by the main selector, native - // drag-drop/picker, and the dropped-file chip (the Hub stages via the store). + const chatContextKey = `${view.mode}|${activeThreadId ?? ""}|${search.new ?? ""}|${search.project ?? ""}`; + const [pendingHubAutoLoad, setPendingHubAutoLoad] = + useState(null); const stageOrLoad = useCallback( async (selection: SelectedModelInput) => { const store = useChatRuntimeStore.getState(); - // An un-cached HF repo (GGUF variant or a full non-GGUF snapshot) downloads - // through the manager first (global indicator), then auto-loads. Everything - // else -- cached picks, local/native files, LoRA, external -- loads now. const wantManagerDownload = isDownloadableHubRepo(selection) && !selection.isDownloaded; - if ( - (!hasGgufSource(selection) && !wantManagerDownload) || - (store.loadOnSelection && selection.isDownloaded) - ) { - // Detach any staged pick first so its edited knobs (e.g. a custom - // context length) don't leak into this immediate load -- resolveLoad - // reads customContextLength before checking the target is GGUF. Detach - // (not abandon) keeps its download running. - detachStaged(); - // Load-on-selection skips the sheet, so seed the saved knobs here the - // way the sheet's restore effect would; the switch would otherwise reset - // the remembered speculative choice (keepSpeculative below prevents it). - const remembered = hasGgufSource(selection) - ? loadRememberedLoadSettings(rememberedLoadSettingsKey(selection)) - : null; - if (remembered) store.applyRememberedLoadSettings(remembered); - await selectModel( - remembered ? { ...selection, keepSpeculative: true } : selection, - ); - return; - } - // Loads can't queue behind each other, but a download is independent: if - // the pick needs downloading, start it in the manager so it runs alongside - // the load. Nothing to download (already on device) just waits. if (store.modelLoading) { - // Both an uncached non-GGUF snapshot (wantManagerDownload) and an - // uncached remote GGUF quant download through the manager, so either can - // run in the background while another model loads. wantManagerDownload - // excludes GGUF by design, so the GGUF case is checked separately. const wantBackgroundDownload = wantManagerDownload || (selection.source === "hub" && hasGgufSource(selection) && !selection.isDownloaded); - // The model currently loading already downloads as part of its own load - // (the /load flow fetches before setting the checkpoint), so re-picking - // it must not kick off a second transfer against the same cache. const isLoadingThisPick = !!loadingModel && normalizeModelRef(loadingModel.id) === @@ -1862,11 +1805,6 @@ export function ChatPage({ description: "It's downloading as part of the load in progress.", }); } else if (wantBackgroundDownload) { - // Only claim the download started once a job is actually created. A - // transport conflict records state that is only resolvable from the - // Hub download card, so point the user there instead of showing a - // success toast for a transfer that never began; "busy" and "error" - // already surface their own toasts. const outcome = await downloadManager.requestStart({ kind: DOWNLOAD_KIND.MODEL, repoId: selection.id, @@ -1883,6 +1821,11 @@ export function ChatPage({ description: "An earlier partial download used a different transport. Open the Hub tab to resume or restart it.", }); + } else if (outcome === "busy") { + toast.info("Download already in progress", { + description: + "Another download for this model is still running. Reselect it once that finishes to load it.", + }); } } else { toast.info("Another model is already loading", { @@ -1891,23 +1834,128 @@ export function ChatPage({ } return; } - // Detach the prior staged pick (keeping its download) before rebinding, so - // a second pick downloads alongside the first instead of cancelling it. - detachStaged(); - store.stageModel({ - id: selection.id, - isLora: selection.isLora, - ggufVariant: selection.ggufVariant, - isDownloaded: selection.isDownloaded, - expectedBytes: selection.expectedBytes, - nativePathToken: selection.nativePathToken, - isGguf: selection.isGguf, - isHubRepo: wantManagerDownload || undefined, - autoLoad: store.loadOnSelection, + const wantManagerStage = + wantManagerDownload || + (selection.source === "hub" && + hasGgufSource(selection) && + !selection.isDownloaded); + if (wantManagerStage) { + setPendingHubAutoLoad((current) => + current && + current.selection.id === selection.id && + (current.selection.ggufVariant ?? null) === + (selection.ggufVariant ?? null) && + current.contextKey === chatContextKey && + current.originCheckpoint === store.params.checkpoint && + current.originGgufVariant === store.activeGgufVariant + ? current + : { + selection, + contextKey: chatContextKey, + originCheckpoint: store.params.checkpoint, + originGgufVariant: store.activeGgufVariant, + }, + ); + return; + } + setPendingHubAutoLoad(null); + const previousConfig = currentRuntimePerModelConfig({ + includeMaxSeqLength: true, + }); + const hasAppliedConfig = applyModelLoadConfigToRuntime( + selection.config ?? rememberedConfigFor(selection), + ); + await selectModel({ + ...selection, + ...(hasAppliedConfig ? { keepSpeculative: true } : {}), + previousConfig, }); }, - [detachStaged, selectModel, loadingModel], + [selectModel, loadingModel, rememberedConfigFor, chatContextKey], ); + useRepoDownload({ + kind: DOWNLOAD_KIND.MODEL, + repoId: pendingHubAutoLoad?.selection.id ?? "__hub_autoload_idle__", + activeVariant: pendingHubAutoLoad?.selection.ggufVariant ?? null, + onComplete: (variant) => { + const pending = pendingHubAutoLoad; + if ( + !pending || + (pending.selection.ggufVariant ?? null) !== (variant ?? null) + ) { + return; + } + setPendingHubAutoLoad(null); + const store = useChatRuntimeStore.getState(); + if ( + !active || + pending.contextKey !== chatContextKey || + normalizeModelRef(pending.originCheckpoint) !== + normalizeModelRef(store.params.checkpoint) || + pending.originGgufVariant !== store.activeGgufVariant + ) { + return; + } + void stageOrLoad({ ...pending.selection, isDownloaded: true }); + }, + onError: (variant) => { + if ( + pendingHubAutoLoad && + (pendingHubAutoLoad.selection.ggufVariant ?? null) === (variant ?? null) + ) { + setPendingHubAutoLoad(null); + } + }, + onCancelled: (variant) => { + if ( + pendingHubAutoLoad && + (pendingHubAutoLoad.selection.ggufVariant ?? null) === (variant ?? null) + ) { + setPendingHubAutoLoad(null); + } + }, + }); + useEffect(() => { + const pending = pendingHubAutoLoad; + if (!pending) return; + let active = true; + void (async () => { + const outcome = await downloadManager.requestStart({ + kind: DOWNLOAD_KIND.MODEL, + repoId: pending.selection.id, + variant: pending.selection.ggufVariant ?? null, + expectedBytes: pending.selection.expectedBytes ?? 0, + }); + if (!active) return; + if (outcome === "started") { + toast.info("Downloading model", { + description: "It'll load automatically once the download finishes.", + }); + return; + } + if (outcome === "conflict") { + // Keep pendingHubAutoLoad bound so this surface's cleanup does not wipe + // the conflict just recorded by requestStart (which the toast points the + // user to); resolving it from the Hub completes the download and this + // surface's onComplete auto-loads, mirroring the "started" branch. + toast.info("Resume this download from the Hub", { + description: + "An earlier partial download used a different transport. Open the Hub tab to resume or restart it.", + }); + return; + } + if (outcome === "busy") { + toast.info("Download already in progress", { + description: + "Another download for this model is still running. Reselect it once that finishes to load it.", + }); + } + setPendingHubAutoLoad((current) => (current === pending ? null : current)); + })(); + return () => { + active = false; + }; + }, [pendingHubAutoLoad]); const loadNativeModelIntent = useCallback( async (intent: NativeIntent, loadingDescription: string) => { const label = @@ -1915,6 +1963,7 @@ export function ChatPage({ await stageOrLoad({ id: label, nativePathToken: intent.path.token, + nativePathExpiresAtMs: intent.path.expiresAtMs ?? null, isDownloaded: true, loadingDescription, forceReload: true, @@ -1965,28 +2014,20 @@ export function ChatPage({ const handleCheckpointChange = useCallback( ( value: string, - meta?: { - source?: string; - isLora: boolean; - ggufVariant?: string; - isDownloaded?: boolean; - expectedBytes?: number; - isGguf?: boolean; - }, + meta?: ModelSelectorChangeMeta, ) => { const store = useChatRuntimeStore.getState(); const currentCheckpoint = store.params.checkpoint; const currentVariant = store.activeGgufVariant; - if ( - !value || - (value === currentCheckpoint && - (meta?.ggufVariant ?? null) === (currentVariant ?? null)) - ) + if (!value) return; + setPendingHubAutoLoad(null); + const isSameLoadedModel = + value === currentCheckpoint && + (meta?.ggufVariant ?? null) === (currentVariant ?? null); + if (isSameLoadedModel && !meta?.forceReload) { return; + } if (meta?.source === "external" || isExternalModelId(value)) { - // Switching to an external model abandons any staged local pick: cancel - // its download too (setCheckpoint below only clears the pending + knobs). - abandonStaged(); const selectedExternal = parseExternalModelId(value); const selectedProvider = selectedExternal ? externalProvidersForChat.find( @@ -2087,6 +2128,7 @@ export function ChatPage({ ggufMaxContextLength: null, ggufNativeContextLength: null, activeNativePathToken: null, + activeNativePathExpiresAtMs: null, // Clear previous-model counters, else the relaxed external-provider // render gate shows stale stats until the next completion. contextUsage: null, @@ -2158,19 +2200,18 @@ export function ChatPage({ source: meta?.source, isLora: meta?.isLora, ggufVariant: meta?.ggufVariant, - isDownloaded: meta?.isDownloaded, + isDownloaded: meta?.isDownloaded || isSameLoadedModel, expectedBytes: meta?.expectedBytes, isGguf: meta?.isGguf, + config: meta?.config, + nativePathToken: meta?.nativePathToken, + nativePathExpiresAtMs: meta?.nativePathExpiresAtMs, + forceReload: isSameLoadedModel || undefined, }; - // "Load on selection" off: stage the model and open settings so its - // load knobs (tensor parallel, context length…) can be set, then it - // loads once via the sheet's Load button. The currently loaded model - // stays put until the user commits. await stageOrLoad(selection); })(); }, [ - abandonStaged, activeThreadId, externalProvidersForChat, modelsFromStore, @@ -2178,6 +2219,45 @@ export function ChatPage({ view, ], ); + const handleReloadActiveModel = useCallback( + (config: PerModelConfig) => { + const checkpoint = inferenceParams.checkpoint; + if (!checkpoint) return; + const runtime = useChatRuntimeStore.getState(); + const nativeToken = runtime.activeNativePathToken; + const nativeExpiry = runtime.activeNativePathExpiresAtMs; + // A file-picked GGUF is reachable only via its native path token, which + // the desktop host prunes after a TTL. Reusing an expired token makes the + // reload fail with an opaque error, so prompt the user to re-select the + // file instead. + if (nativeToken && nativeExpiry != null && Date.now() >= nativeExpiry) { + toast.error("This local model file's access has expired.", { + description: "Re-select the model file to reload it.", + }); + return; + } + handleCheckpointChange(checkpoint, { + source: "local", + isLora: activeModelIsLora, + ggufVariant: activeGgufVariant ?? undefined, + // Without the native token the reload validates the display label as a + // repo and fails. + nativePathToken: nativeToken ?? undefined, + nativePathExpiresAtMs: nativeExpiry, + isGguf: activeModelIsGguf, + isDownloaded: true, + config, + forceReload: true, + }); + }, + [ + inferenceParams.checkpoint, + activeGgufVariant, + activeModelIsLora, + activeModelIsGguf, + handleCheckpointChange, + ], + ); const handleEject = useCallback(() => { void (async () => { if (await ejectModel()) { @@ -2446,12 +2526,27 @@ export function ChatPage({ const state = useChatRuntimeStore.getState(); const targetLora = pickBestLoraForBase(state.loras, handoff.baseModel); + const selectWithConfig = async ( + selection: Pick, + ) => { + const previousConfig = currentRuntimePerModelConfig({ + includeMaxSeqLength: true, + }); + const hasAppliedConfig = applyModelLoadConfigToRuntime( + rememberedConfigFor(selection), + ); + await selectModelRef.current({ + ...selection, + ...(hasAppliedConfig ? { keepSpeculative: true } : {}), + previousConfig, + }); + }; if (targetLora) { console.info("[chat-handoff] loading lora", { id: targetLora.id, baseModel: targetLora.baseModel, }); - await selectModelRef.current({ id: targetLora.id, isLora: true }); + await selectWithConfig({ id: targetLora.id, isLora: true }); if (canceled) return; useChatRuntimeStore.getState().setActiveThreadId(null); useChatRuntimeStore.getState().setContextUsage(null); @@ -2468,10 +2563,7 @@ export function ChatPage({ console.info("[chat-handoff] no lora match, loading base", { id: handoff.baseModel, }); - await selectModelRef.current({ - id: handoff.baseModel, - isLora: false, - }); + await selectWithConfig({ id: handoff.baseModel, isLora: false }); if (canceled) return; } else { console.warn("[chat-handoff] no lora/base match found", { @@ -2491,7 +2583,7 @@ export function ChatPage({ return () => { canceled = true; }; - }, [active, navigate]); + }, [active, navigate, rememberedConfigFor]); const tourSteps = useMemo( () => @@ -2580,6 +2672,8 @@ export function ChatPage({ externalModels={externalModels} value={inferenceParams.checkpoint} activeGgufVariant={activeGgufVariant} + activeModelConfig={activeModelConfig} + activeGgufContextLength={ggufContextLength} onValueChange={handleCheckpointChange} onEject={handleEject} onFoldersChange={refreshLocalModels} @@ -2633,7 +2727,12 @@ export function ChatPage({ stageOrLoad(selection)} + onLoad={() => + loadNativeModelIntent( + pendingNativeModelIntent, + "Loading selected local GGUF model.", + ) + } /> ) : null} {loadingModel && loadToastDismissed ? ( @@ -2790,13 +2889,22 @@ export function ChatPage({ open={active && settingsOpen} onOpenChange={(open) => { setSettingsOpen(open); - // Closing the sheet abandons a staged (not-yet-loaded) pick: cancel its - // download and revert the staged knobs so nothing lingers as a dirty - // edit (or a background download) on the loaded model. - if (!open) abandonStaged(); }} params={inferenceParams} onParamsChange={setInferenceParams} + modelConfig={ + view.mode !== "compare" && activeModelConfig && !modelLoading ? ( + + ) : null + } isExternalModel={isExternalModel} providerCapabilities={activeProviderCapabilities} activeExternalProvider={activeExternalProvider} @@ -2808,67 +2916,6 @@ export function ChatPage({ ); }} externalProviderType={activeExternalProviderType} - loadingModel={loadingModel} - onReloadModel={() => { - const state = useChatRuntimeStore.getState(); - if (state.params.checkpoint) { - selectModel({ - id: state.params.checkpoint, - ggufVariant: state.activeGgufVariant ?? undefined, - // A native (drag-drop / picked) GGUF's checkpoint is only a display - // label, so the reload needs its path token to re-mint a lease -- - // else applying the now-exposed GPU/context controls can't resolve - // the file. Null for non-native loads, which reload by id as before. - nativePathToken: state.activeNativePathToken ?? undefined, - forceReload: true, - isDownloaded: true, - loadingDescription: "Reloading with updated chat template.", - }); - } - }} - onLoadPendingModel={() => { - const pending = useChatRuntimeStore.getState().pendingSelection; - if (!pending) return; - const keyAtLoad = chatContextKey; - // forceReload: the staged model isn't loaded yet, so bypass the - // same-checkpoint dedupe. keepSpeculative: honor the speculative mode - // set on the sidebar. - void selectModel({ - ...pending, - forceReload: true, - keepSpeculative: true, - throwOnError: true, - }).catch(() => { - // Recoverable failure (expired token, gated repo, OOM…): the pick is - // cleared only on success, so it normally stays staged with edited - // knobs intact — nothing to restore. - const store = useChatRuntimeStore.getState(); - // Still staged (this pick, or a newer one queued meanwhile): leave it. - if (store.pendingSelection) return; - // Cleared mid-load (sheet closed / switched chats). Re-stage only if - // the staged-load is still wanted: same chat context, sheet still - // open, page still mounted. - const stillWanted = - mountedRef.current && - store.settingsPanelOpen && - chatContextKeyRef.current === keyAtLoad; - if (stillWanted) { - store.setPendingSelection(pending); - } else { - // Abandoned (closed the sheet / switched chats / left chat): drop - // the orphaned staged knob edits so they don't linger as dirty - // settings over the loaded model. - store.resetModelSettingsToLoaded(); - } - }); - }} - stagedDownloadFraction={stagedDownload.progress?.fraction ?? null} - onCancelStagedDownload={() => - stagedDownload.cancelDownload( - useChatRuntimeStore.getState().pendingSelection?.ggufVariant ?? - null, - ) - } /> diff --git a/studio/frontend/src/features/chat/chat-settings-sheet.tsx b/studio/frontend/src/features/chat/chat-settings-sheet.tsx index bd22cc4f55..d4f154882c 100644 --- a/studio/frontend/src/features/chat/chat-settings-sheet.tsx +++ b/studio/frontend/src/features/chat/chat-settings-sheet.tsx @@ -1,19 +1,7 @@ // SPDX-License-Identifier: AGPL-3.0-only // Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0 -import { - Alert, - AlertDescription, - AlertTitle, -} from "@/components/ui/alert"; import { Button } from "@/components/ui/button"; -import { Checkbox } from "@/components/ui/checkbox"; -import { - clearRememberedLoadSettings, - loadRememberedLoadSettings, - rememberedLoadSettingsKey, - saveRememberedLoadSettings, -} from "@/components/assistant-ui/model-selector/remembered-load-settings"; import { Dialog, DialogContent, @@ -29,7 +17,7 @@ import { DropdownMenuSeparator, DropdownMenuTrigger, } from "@/components/ui/dropdown-menu"; -import { Input } from "@/components/ui/input"; +import { InfoHint } from "@/components/ui/info-hint"; import { InputGroup, InputGroupAddon, @@ -50,27 +38,22 @@ import { SheetTitle, } from "@/components/ui/sheet"; import { Slider } from "@/components/ui/slider"; -import { Spinner } from "@/components/ui/spinner"; import { Switch } from "@/components/ui/switch"; import { Textarea } from "@/components/ui/textarea"; -import { InfoHint } from "@/components/ui/info-hint"; import { Tooltip, TooltipContent } from "@/components/ui/tooltip"; -import { useGpuDevices } from "@/hooks/use-gpu-info"; -import { useIsMobile } from "@/hooks/use-mobile"; +import { NumericValueInput, snapToStep } from "@/features/model-picker"; +import { RetrievalSettingsSection } from "@/features/rag"; import { useLlamaUpdateCheck } from "@/hooks/use-llama-update-check"; -import { cn } from "@/lib/utils"; -import { - ArrowTurnBackwardIcon, - Edit03Icon, - LayoutAlignRightIcon, -} from "@hugeicons/core-free-icons"; +import { useIsMobile } from "@/hooks/use-mobile"; import { ChevronDownStandardIcon } from "@/lib/chevron-icons"; +import { toast } from "@/lib/toast"; +import { cn } from "@/lib/utils"; +import { Edit03Icon, LayoutAlignRightIcon } from "@hugeicons/core-free-icons"; import { HugeiconsIcon } from "@hugeicons/react"; import { Braces, ChevronDown, ExternalLink } from "lucide-react"; import { Tooltip as TooltipPrimitive } from "radix-ui"; import { Fragment, type ReactNode } from "react"; import { useCallback, useEffect, useMemo, useRef, useState } from "react"; -import { toast } from "@/lib/toast"; import { OpenAICodeExecSection } from "./components/openai-code-exec-section"; import { PermissionModeDropdown } from "./permission-mode-select"; import { resyncInferenceStatusAfterServerModelChange } from "./hooks/use-chat-model-runtime"; @@ -78,8 +61,8 @@ import { type ExternalProviderConfig, getExternalProviderApiKey, parseExternalModelId, - supportsProviderPromptCaching, supportsProviderPromptCacheTtl, + supportsProviderPromptCaching, } from "./external-providers"; import { BUILTIN_PRESETS, @@ -99,15 +82,7 @@ import { providerSupportsBuiltinCodeExecution, providerSupportsFastMode, } from "./provider-capabilities"; -import { - GPU_LAYERS_AUTO, - distributeByWeight, - isPendingGguf, - pendingSelectionMatches, - rebalanceSplit, - useChatRuntimeStore, -} from "./stores/chat-runtime-store"; -import { RetrievalSettingsSection } from "@/features/rag/components/retrieval-settings-section"; +import { useChatRuntimeStore } from "./stores/chat-runtime-store"; import type { InferenceParams } from "./types/runtime"; export { defaultInferenceParams, type Preset } from "./presets/preset-policy"; @@ -130,7 +105,7 @@ function getPromptVariablesError(raw: string): string | null { return null; } } catch { - return "Use valid JSON, for example { \"env\": \"staging\" }."; + return 'Use valid JSON, for example { "env": "staging" }.'; } return "Variables must be a JSON object."; } @@ -139,112 +114,7 @@ function hasPromptVariableSyntax(prompt: string): boolean { return PROMPT_VARIABLE_PATTERN.test(prompt); } -/** - * Editable numeric value display, shared by every slider value and the Context - * Length input. An that looks like text (shows `displayValue ?? value`, - * so "Off"/"Max" labels render) until focus, when it swaps to the raw number, - * selects it, and accepts free text. Commits on blur/Enter, reverts on Escape. - * Clamping happens on commit so typing intermediate values isn't fought. - */ -function snapToStep( - value: number, - step: number, - min?: number, - max?: number, -): number { - const lo = min ?? Number.NEGATIVE_INFINITY; - const hi = max ?? Number.POSITIVE_INFINITY; - const clamped = Math.min(Math.max(value, lo), hi); - const stepStr = String(step); - const decimals = stepStr.includes(".") ? stepStr.split(".")[1].length : 0; - const base = Number.isFinite(lo) ? lo : 0; - const snapped = base + Math.round((clamped - base) / step) * step; - const reclamped = Math.min(Math.max(snapped, lo), hi); - return Number(reclamped.toFixed(decimals)); -} - -function NumericValueInput({ - value, - min, - max, - step, - onChange, - displayValue, - className, - ariaLabel, - size: sizeAttr, - disabled = false, -}: { - value: number; - min?: number; - max?: number; - step: number; - onChange: (v: number) => void; - displayValue?: string; - className?: string; - ariaLabel?: string; - size?: number; - disabled?: boolean; -}) { - const [focused, setFocused] = useState(false); - const [draft, setDraft] = useState(""); - const cancelBlurCommitRef = useRef(false); - - const commit = (raw: string) => { - const parsed = Number.parseFloat(raw); - if (!Number.isFinite(parsed)) { - return; - } - const final = snapToStep(parsed, step, min, max); - if (final !== value) { - onChange(final); - } - }; - - const displayed = focused ? draft : (displayValue ?? String(value)); - - return ( - { - cancelBlurCommitRef.current = false; - setDraft(String(value)); - setFocused(true); - // Defer select() so it runs after the value swap above. - const target = e.currentTarget; - requestAnimationFrame(() => target.select()); - }} - onBlur={() => { - if (cancelBlurCommitRef.current) { - cancelBlurCommitRef.current = false; - } else { - commit(draft); - } - setFocused(false); - }} - onChange={(e) => setDraft(e.target.value)} - onKeyDown={(e) => { - if (e.key === "Enter") { - e.currentTarget.blur(); - } else if (e.key === "Escape") { - cancelBlurCommitRef.current = true; - setDraft(String(value)); - e.currentTarget.blur(); - } - }} - className={cn("panel-number-input", className)} - /> - ); -} - -function ParamSlider({ +export function ParamSlider({ label, value, min, @@ -285,6 +155,7 @@ function ParamSlider({ displayValue={displayValue} ariaLabel={label} size={valueSize ?? 4} + className="panel-number-input" disabled={disabled} /> @@ -385,8 +256,7 @@ function CollapsibleSection({ return (
{labelHref ? ( @@ -458,6 +328,7 @@ interface ChatSettingsPanelProps { onOpenChange?: (open: boolean) => void; params: InferenceParams; onParamsChange: (params: InferenceParams) => void; + modelConfig?: ReactNode; isExternalModel?: boolean; /** * Sampling-param capabilities for the active external provider, or `null` for @@ -472,21 +343,6 @@ interface ChatSettingsPanelProps { * Max Tokens floor in the slider. */ externalProviderType?: string | null; - onReloadModel?: () => void; - /** The in-flight load (id + GGUF variant + native path token), or null when - * idle. Used to show a loading state for the staged pick only — not for an - * unrelated load or a cancel's background unload. */ - loadingModel?: { - id: string; - ggufVariant?: string | null; - nativePathToken?: string | null; - } | null; - /** Loads the staged `pendingSelection` (deferred "Load on selection" flow). */ - onLoadPendingModel?: () => void; - /** Download progress (0–1) for a staged GGUF being fetched, or null when idle. */ - stagedDownloadFraction?: number | null; - /** Cancels the in-flight staged download (paired with abandoning the stage). */ - onCancelStagedDownload?: () => void; } export function ChatSettingsPanel({ @@ -494,16 +350,12 @@ export function ChatSettingsPanel({ onOpenChange, params, onParamsChange, + modelConfig = null, isExternalModel = false, providerCapabilities = null, activeExternalProvider = null, onExternalProviderChange, externalProviderType = null, - onReloadModel, - loadingModel = null, - onLoadPendingModel, - stagedDownloadFraction, - onCancelStagedDownload, }: ChatSettingsPanelProps) { // Local models show every knob; providerCapabilities is only consulted when // isExternalModel. Unknown providers fall back to the OpenAI-compat shape via @@ -518,64 +370,23 @@ export function ChatSettingsPanel({ const showPresencePenalty = !isExternalModel || Boolean(providerCapabilities?.presencePenalty); const isMobile = useIsMobile(); - const pendingSelection = useChatRuntimeStore((s) => s.pendingSelection); - // "Loading" only when the in-flight load IS this staged pick (full id + GGUF - // variant + native token match), not an unrelated load or a cancel's - // background unload. The variant matters: a different quant of the same repo - // staged mid-load must not read as this one loading. - const stagedLoading = - loadingModel != null && - pendingSelectionMatches(pendingSelection, { - id: loadingModel.id, - ggufVariant: loadingModel.ggufVariant, - nativePathToken: loadingModel.nativePathToken, - }); - // Load settings are snapshotted at click time; lock them while loading. - const modelControlsDisabled = stagedLoading; - const abandonStagedModel = useChatRuntimeStore((s) => s.abandonStagedModel); - const resetModelSettingsToLoaded = useChatRuntimeStore( - (s) => s.resetModelSettingsToLoaded, + const isLoadedGguf = useChatRuntimeStore((s) => s.activeGgufVariant) != null; + const currentCheckpoint = params.checkpoint; + const ggufContextLength = useChatRuntimeStore((s) => s.ggufContextLength); + // Direct-file / custom-folder GGUFs load without a variant label but still + // report a GGUF context, so detect them via the context and the checkpoint + // suffix too (mirrors the chat page's activeModelIsGguf). Otherwise Max Tokens + // would fall back to params.maxSeqLength instead of the loaded GGUF context. + const isGguf = + isLoadedGguf || + ggufContextLength != null || + (currentCheckpoint?.toLowerCase().endsWith(".gguf") ?? false); + const ggufMaxContextLength = useChatRuntimeStore( + (s) => s.ggufMaxContextLength, ); - // A staged GGUF pick (deferred load) shows the GGUF load knobs so they can be - // set before the single load. - const pendingIsGguf = isPendingGguf(pendingSelection); - // Short, human-readable name for the staged pick (HF ids carry an org prefix; - // native picks are already a display label). Drives the "staged, not loaded" - // callout so it's obvious the selection hasn't loaded yet. - const stagedLabel = (() => { - const id = pendingSelection?.id ?? ""; - const slash = id.lastIndexOf("/"); - const base = slash >= 0 ? id.slice(slash + 1) : id; - return base || id; - })(); - const activeNativePathToken = useChatRuntimeStore( - (s) => s.activeNativePathToken, - ); - const loadedGgufContextLength = useChatRuntimeStore((s) => s.ggufContextLength); - // A GGUF loaded from a native path / direct .gguf has no HF variant, so key - // off the same signal the status hydration uses -- variant OR native token OR - // a GGUF context -- else the GPU Memory controls hide for a loaded local GGUF. - const isLoadedGguf = - useChatRuntimeStore((s) => s.activeGgufVariant) != null || - activeNativePathToken != null || - loadedGgufContextLength != null; - // While a pick is staged the sheet configures *that* model, so its GGUF-ness - // (not the currently loaded model's) decides whether the GGUF-only controls - // show. Otherwise a staged non-GGUF Hub repo would inherit the loaded GGUF's - // context/KV/speculative controls. - const isGguf = pendingSelection != null ? pendingIsGguf : isLoadedGguf; - // The Model section (and Load button) shows for any staged pick, even when the - // currently active model is external. - const hasModelContent = - pendingSelection != null || - (!isExternalModel && (isGguf || Boolean(params.checkpoint))); + const customContextLength = useChatRuntimeStore((s) => s.customContextLength); const speculativeType = useChatRuntimeStore((s) => s.speculativeType); - const setSpeculativeType = useChatRuntimeStore((s) => s.setSpeculativeType); - const loadedSpeculativeType = useChatRuntimeStore( - (s) => s.loadedSpeculativeType, - ); const specFallbackReason = useChatRuntimeStore((s) => s.specFallbackReason); - // Only binary fallback states are solved by a newer prebuilt. const mtpUpdatable = specFallbackReason === "binary_no_mtp" || specFallbackReason === "binary_outdated"; @@ -597,65 +408,27 @@ export function ChatSettingsPanel({ `llama.cpp updated to ${result.tag ?? "the latest build"}.${reloadHint}`, ); } else { - toast.error(`llama.cpp update failed: ${result.error ?? "unknown error"}`); + toast.error( + `llama.cpp update failed: ${result.error ?? "unknown error"}`, + ); } }, [applyLlamaUpdate]); - const specDraftNMax = useChatRuntimeStore((s) => s.specDraftNMax); - const setSpecDraftNMax = useChatRuntimeStore((s) => s.setSpecDraftNMax); - const loadedSpecDraftNMax = useChatRuntimeStore( - (s) => s.loadedSpecDraftNMax, - ); - const currentCheckpoint = params.checkpoint; - const ggufContextLength = useChatRuntimeStore((s) => s.ggufContextLength); - const ggufMaxContextLength = useChatRuntimeStore( - (s) => s.ggufMaxContextLength, - ); - const ggufNativeContextLength = useChatRuntimeStore( - (s) => s.ggufNativeContextLength, - ); - const kvCacheDtype = useChatRuntimeStore((s) => s.kvCacheDtype); - const setKvCacheDtype = useChatRuntimeStore((s) => s.setKvCacheDtype); - const applyRememberedLoadSettings = useChatRuntimeStore( - (s) => s.applyRememberedLoadSettings, - ); - const loadedKvCacheDtype = useChatRuntimeStore((s) => s.loadedKvCacheDtype); - const tensorParallel = useChatRuntimeStore((s) => s.tensorParallel); - const setTensorParallel = useChatRuntimeStore((s) => s.setTensorParallel); - const loadedTensorParallel = useChatRuntimeStore( - (s) => s.loadedTensorParallel, - ); - const gpuMemoryMode = useChatRuntimeStore((s) => s.gpuMemoryMode); - const setGpuMemoryMode = useChatRuntimeStore((s) => s.setGpuMemoryMode); - const loadedGpuMemoryMode = useChatRuntimeStore((s) => s.loadedGpuMemoryMode); - const loadedIsDiffusion = useChatRuntimeStore((s) => s.loadedIsDiffusion); - const gpuLayers = useChatRuntimeStore((s) => s.gpuLayers); - const setGpuLayers = useChatRuntimeStore((s) => s.setGpuLayers); - const loadedGpuLayers = useChatRuntimeStore((s) => s.loadedGpuLayers); - const nCpuMoe = useChatRuntimeStore((s) => s.nCpuMoe); - const setNCpuMoe = useChatRuntimeStore((s) => s.setNCpuMoe); - const loadedNCpuMoe = useChatRuntimeStore((s) => s.loadedNCpuMoe); - const splitRatio = useChatRuntimeStore((s) => s.splitRatio); - const setSplitRatio = useChatRuntimeStore((s) => s.setSplitRatio); - const loadedSplitRatio = useChatRuntimeStore((s) => s.loadedSplitRatio); - const ggufLayerCount = useChatRuntimeStore((s) => s.ggufLayerCount); - const moeLayerCount = useChatRuntimeStore((s) => s.moeLayerCount); - const selectedGpuIds = useChatRuntimeStore((s) => s.selectedGpuIds); - const setSelectedGpuIds = useChatRuntimeStore((s) => s.setSelectedGpuIds); - const loadedGpuIds = useChatRuntimeStore((s) => s.loadedGpuIds); - const gpuDevices = useGpuDevices(); - const chatTemplateOverride = useChatRuntimeStore( - (s) => s.chatTemplateOverride, - ); - const loadedChatTemplateOverride = useChatRuntimeStore( - (s) => s.loadedChatTemplateOverride, - ); - const customContextLength = useChatRuntimeStore((s) => s.customContextLength); - const loadedCustomContextLength = useChatRuntimeStore( - (s) => s.loadedCustomContextLength, - ); - const setCustomContextLength = useChatRuntimeStore( - (s) => s.setCustomContextLength, - ); + const loadedEffectiveContext = customContextLength ?? ggufContextLength; + const showSpecFallback = + !isExternalModel && + isGguf && + specFallbackReason != null && + (speculativeType === "auto" || + speculativeType === "mtp" || + speculativeType === "mtp+ngram"); + const showContextVramWarning = + !isExternalModel && + isGguf && + ggufMaxContextLength != null && + loadedEffectiveContext != null && + loadedEffectiveContext > ggufMaxContextLength; + const showLoadedDiagnostics = showSpecFallback || showContextVramWarning; + const hasModelContent = showLoadedDiagnostics; const setActivePresetSource = useChatRuntimeStore( (s) => s.setActivePresetSource, ); @@ -666,170 +439,7 @@ export function ChatSettingsPanel({ const setActivePreset = useChatRuntimeStore((s) => s.setActivePreset); const settingsHydrated = useChatRuntimeStore((s) => s.settingsHydrated); - // A staged (not-yet-loaded) GGUF carries its own header context length on - // pendingSelection, so the slider can use the staged model's real ceiling - // without reading the loaded model's `ggufContextLength`. - const stagedContextLength = pendingSelection?.contextLength ?? null; - // "Remember settings next time" tick for a staged model. Seeds the store from - // the saved per-model settings on stage, so the sheet opens with what was used - // last time; the tick reflects whether a saved entry exists. - const [remember, setRemember] = useState(false); - // Keyed per quant: a different variant of the same repo has its own settings. - const pendingKey = pendingSelection - ? rememberedLoadSettingsKey(pendingSelection) - : null; - useEffect(() => { - if (!pendingKey) return; - // GGUF-only, like the stageOrLoad / Hub restore paths: every remembered - // field is a llama.cpp knob, so a non-GGUF pick has nothing to restore -- - // and applying its blob would clobber the standing gpuMemoryMode with a - // stale snapshot (the save on Load below is gated the same way). - const saved = pendingIsGguf ? loadRememberedLoadSettings(pendingKey) : null; - setRemember(saved != null); - if (saved) applyRememberedLoadSettings(saved); - }, [pendingKey, pendingIsGguf, applyRememberedLoadSettings]); - // While staging, the sheet reflects the STAGED model, so its header context - // takes precedence over the loaded model's (which may differ or be larger). - const baseContext = pendingIsGguf ? stagedContextLength : ggufContextLength; - const baseNativeContext = pendingIsGguf - ? stagedContextLength - : ggufNativeContextLength; - // Context controls render once we actually have a ceiling: for a staged GGUF, - // once its header metadata arrives (post-download); otherwise post-load. - const showContextControl = pendingIsGguf - ? stagedContextLength != null - : isLoadedGguf; - const stagedDownloading = - stagedDownloadFraction != null && stagedDownloadFraction < 1; - const ctxDisplayValue = customContextLength ?? baseContext ?? ""; - const ctxMaxValue = baseNativeContext ?? baseContext ?? null; - const kvDirty = kvCacheDtype !== loadedKvCacheDtype; - const ctxDirty = customContextLength !== loadedCustomContextLength; - const specDirty = speculativeType !== loadedSpeculativeType; - const specDraftDirty = specDraftNMax !== loadedSpecDraftNMax; - const tpDirty = tensorParallel !== (loadedTensorParallel ?? false); - // A loaded diffusion GGUF runs mode-agnostic (pins all layers on one GPU, - // ignores --fit/--gpu-layers), so the GPU Memory mode + manual controls don't - // apply -- hide them and don't let the preserved standing mode read as dirty. - // The GPU picker still applies (diffusion pins the chosen device). A staged pick - // keeps the controls (a pending pick's diffusion-ness isn't known until load). - const gpuModeApplies = - isGguf && (pendingSelection != null || !loadedIsDiffusion); - const gpuDirty = - gpuModeApplies && gpuMemoryMode !== (loadedGpuMemoryMode ?? "auto"); - const isManual = gpuModeApplies && gpuMemoryMode === "manual"; - // Manual with the GPU Layers slider at "Auto" (leftmost): --fit owns the whole - // layout, so the offload knobs (MoE, split, TP) don't apply. - const autoLayers = isManual && gpuLayers < 0; - // GPUs actually in use: the picked subset, or all visible when none picked. - const gpusInUse = selectedGpuIds ?? gpuDevices.map((d) => d.index); - // The picker must keep one GPU selected. - const singleGpuInUse = gpusInUse.length <= 1; - // TP needs at least two GPUs because tensor split is a no-op on one and may - // abort. Auto layers hides TP because --fit aborts under --split-mode tensor. - const tpDisabled = singleGpuInUse; - // Manual gpu-layers ceiling = model layer count + 1 (else a safe fallback): - // llama.cpp counts the output layer as one more offloadable layer past the - // repeating blocks ("offloaded 33/33" needs -ngl 33 on a 32-block model), so - // the slider max must reach it or full offload is unreachable. While staging, - // use the staged model's layer count (read from its header). - const stagedLayerCount = pendingSelection?.layerCount ?? null; - const modelLayerCount = pendingIsGguf ? stagedLayerCount : ggufLayerCount; - const gpuLayersMax = modelLayerCount != null ? modelLayerCount + 1 : 256; - // MoE-offload slider: shown only for MoE models, capped at their MoE-layer - // count. While staging, use the staged model's count (read from its header); - // otherwise the loaded model's. - const stagedMoeLayerCount = pendingSelection?.moeLayerCount ?? null; - const moeLayersMax = pendingIsGguf - ? (stagedMoeLayerCount ?? 0) - : (moeLayerCount ?? 0); - const showMoeSlider = isManual && !autoLayers && moeLayersMax > 0; - // gpuLayers always counts; MoE only with an explicit layer count (see above). - const manualDirty = - isManual && - (gpuLayers !== loadedGpuLayers || - (!autoLayers && nCpuMoe !== (loadedNCpuMoe ?? 0))); - // GPU picker: only meaningful on multi-GPU, and only when the reported - // indices are physical (relative ordinals from a parent CUDA_VISIBLE_DEVICES - // mask can't be mapped back to pin a device). null = use all (auto). - const showGpuPicker = - isGguf && - gpuDevices.length > 1 && - gpuDevices.every((d) => d.physicalIndex); - const isGpuChecked = (index: number) => - selectedGpuIds === null || selectedGpuIds.includes(index); - const toggleGpu = (index: number) => { - const all = gpuDevices.map((d) => d.index); - const current = selectedGpuIds ?? all; - const next = current.includes(index) - ? current.filter((i) => i !== index) - : [...current, index].sort((a, b) => a - b); - if (next.length === 0) return; // keep at least one GPU selected - setSelectedGpuIds(next.length === all.length ? null : next); - // The per-GPU split is positional, so any change to the set of GPUs in use - // invalidates it: drop it (the sliders fall back to the VRAM-weighted - // default). TP needs 2+ GPUs, so disable it when only one remains. - setSplitRatio(null); - if (next.length <= 1) { - setTensorParallel(false); - } - }; - const gpuIdsKey = (ids: number[] | null) => (ids === null ? "auto" : ids.join(",")); - const gpuIdsDirty = gpuIdsKey(selectedGpuIds) !== gpuIdsKey(loadedGpuIds); - // Per-GPU layer split (--tensor-split): manual + 2+ GPUs in use. One slider - // per GPU, each a layer count; together they sum to the GPU Layers total. - const showSplitRatio = - isManual && !autoLayers && showGpuPicker && gpusInUse.length > 1; - // The total the per-GPU counts sum to (the GPU Layers slider value); 0 under - // Auto, where the split is hidden. The devices behind the GPUs in use, for - // labels + the VRAM-weighted default. - const splitTotal = Math.max(0, Math.min(gpuLayers, gpuLayersMax)); - const gpusInUseDevices = gpusInUse.map( - (i) => gpuDevices.find((d) => d.index === i) ?? null, - ); - // Displayed per-GPU counts. splitRatio is a stable reference balance (only a - // slider edit changes it), rescaled to the current total; deriving rather than - // mutating it on GPU Layers changes keeps the balance intact when the total - // passes through low values or Auto. No saved split: free-VRAM-weighted default - // (llama.cpp's unset default splits by free VRAM, so the first edit starts from - // the default's placement, not a total-VRAM ratio that can land layers on a - // busy GPU). A genuine 0 (a full GPU) is a real weight, not missing data: the - // probe's no-data case degrades to the total server-side, and an all-zero list - // falls back to an even split in distributeByWeight. Not yet sent. - const splitCounts = - splitRatio && splitRatio.length === gpusInUse.length - ? distributeByWeight(splitTotal, splitRatio) - : distributeByWeight( - splitTotal, - gpusInUseDevices.map((d) => d?.memoryFreeGb ?? d?.memoryTotalGb ?? 1), - ); - const setSplitCount = (k: number, v: number) => - setSplitRatio(rebalanceSplit(splitTotal, splitCounts, k, v)); - const splitRatioDirty = - isManual && - !autoLayers && - JSON.stringify(splitRatio ?? null) !== JSON.stringify(loadedSplitRatio ?? null); - // Auto-fit context (Manual + Auto layers): <= 0 means "Auto" (--fit sizes it); - // a positive value pins it. Surface the length --fit chose once it's loaded. - const fitCtxAuto = autoLayers && (customContextLength ?? 0) <= 0; - const loadedAutoLayers = - loadedGpuMemoryMode === "manual" && (loadedGpuLayers ?? GPU_LAYERS_AUTO) < 0; - const fitResolvedCtx = - fitCtxAuto && loadedAutoLayers ? ggufContextLength : null; - // A saved chat-template override is a reload-time setting too, so surface - // Apply for a template-only edit (otherwise it could never be applied). - const templateDirty = chatTemplateOverride !== loadedChatTemplateOverride; - const modelSettingsDirty = - kvDirty || - ctxDirty || - specDirty || - specDraftDirty || - tpDirty || - gpuDirty || - manualDirty || - gpuIdsDirty || - splitRatioDirty || - templateDirty; + const baseContext = ggufContextLength; const [presetNameInput, setPresetNameInput] = useState(activePreset); const [systemPromptEditorOpen, setSystemPromptEditorOpen] = useState(false); const [systemPromptDraft, setSystemPromptDraft] = useState(""); @@ -855,8 +465,7 @@ export function ChatSettingsPanel({ BUILTIN_PRESETS.find((preset) => preset.name === activePreset) ?? null, [activePreset], ); - const hasUnsavedPresetChanges = useMemo( - () => { + const hasUnsavedPresetChanges = useMemo(() => { if (activePresetDefinition == null) { return false; } @@ -864,9 +473,7 @@ export function ChatSettingsPanel({ return activePresetSource === "modified"; } return !isSamePresetConfig(activePresetDefinition.params, params); - }, - [activePresetDefinition, activePresetSource, params], - ); + }, [activePresetDefinition, activePresetSource, params]); const presetSaveState = useMemo( () => getPresetSaveState({ @@ -895,6 +502,14 @@ export function ChatSettingsPanel({ const externalSelection = currentCheckpoint ? parseExternalModelId(currentCheckpoint) : null; + const maxTokensMax = isExternalModel + ? getExternalMaxOutputTokens( + externalProviderType, + externalSelection?.modelId, + ) + : isGguf && baseContext + ? baseContext + : Math.max(64, params.maxSeqLength); const showOpenAICodeExecSection = activeExternalProvider != null && providerSupportsBuiltinCodeExecution( @@ -977,8 +592,7 @@ export function ChatSettingsPanel({ return; } const fallbackPreset = - BUILTIN_PRESETS.find((preset) => preset.name === "Default") ?? - null; + BUILTIN_PRESETS.find((preset) => preset.name === "Default") ?? null; const next = customPresets.filter((preset) => preset.name !== name); setCustomPresets(next); if (activePreset === name) { @@ -1090,7 +704,7 @@ export function ChatSettingsPanel({ Run settings - + - )} -
- )} - {(speculativeType === "mtp" || - speculativeType === "mtp+ngram") && ( -
-
- - Draft Tokens - - - Max MTP draft tokens per step - (--spec-draft-n-max). Lower = less wasted - draft decode; higher = bigger speedup when - acceptance stays high. Default: 2 on GPU, - 3 on CPU/Mac. - -
- { - const raw = e.target.value; - if (raw === "") { - setSpecDraftNMax(null); - return; - } - const parsed = Number.parseInt(raw, 10); - if (Number.isFinite(parsed)) { - const clamped = Math.max(1, Math.min(16, parsed)); - setSpecDraftNMax(clamped); - } - }} - data-test-id="spec-draft-n-max-input" - aria-label="Speculative decoding draft tokens" - className="h-7 w-[88px] rounded-full border-border bg-background hover:bg-accent/50 dark:border-transparent dark:bg-white/[0.05] dark:hover:bg-white/[0.1] pl-3 py-0 text-[13px] font-medium text-nav-fg outline-none focus-visible:ring-0" - /> -
- )} - - )} - {gpuModeApplies && ( -
-
- - GPU Memory - - -
-
- Default: Unsloth - fits the model and context to your GPUs. -
-
- Manual: set GPU - Layers yourself. Leave it on Auto to let llama.cpp size - the context and offload overflow (including MoE experts) - to RAM. -
-
- -
-
- -
-
- )} - {isManual && ( - <> - - Layers to keep on the GPU (--gpu-layers); the rest run - on CPU. Auto lets llama.cpp size the split (and the - context) to fit VRAM. At the maximum, the whole model - is on the GPU. - - } - /> - {showMoeSlider && ( - - Keep the experts of this many MoE layers on the CPU - (--n-cpu-moe) to save VRAM. 0 = all experts on the - GPU; at the maximum, all are on the CPU. - - } - /> - )} - {showSplitRatio && ( -
-
- - Layers per GPU - - - Splits GPU Layers across GPUs (--tensor-split). - Without Tensor Parallelism each value is the layer - count on that GPU; with it, every GPU holds a slice - of each layer, so the values are only a ratio. - -
- {gpusInUseDevices.map((d, k) => ( - setSplitCount(k, v)} - valueSize={6} - disabled={modelControlsDisabled} - /> - ))} -
- )} - - )} - {showGpuPicker && ( -
-
- - GPUs - - - Which GPUs this model may use. Unchecked GPUs are hidden - from llama.cpp (CUDA_VISIBLE_DEVICES, or - HIP_VISIBLE_DEVICES on ROCm). Leave all checked to use - every GPU. At least one GPU must stay selected. - -
-
- {gpuDevices.map((d) => ( -
- - GPU {d.index}: {d.name} - {d.memoryTotalGb - ? ` · ${Math.round(d.memoryTotalGb)} GB` - : ""} - - toggleGpu(d.index)} - data-test-id={`gpu-pick-${d.index}`} - disabled={ - modelControlsDisabled || - (isGpuChecked(d.index) && singleGpuInUse) - } - /> -
- ))} -
-
- )} - {gpuModeApplies && !autoLayers && ( -
-
- - Tensor Parallelism - - - No effect on a single GPU. On multi-GPU setups, improves - tokens/sec during generation when using dense models. MoE - models don't benefit and can be much slower. - -
- -
- )} - - )} - {/* No persistent "enable custom code" toggle: it is consented per model - via the load-time review dialog. */} - {/* Apply/Reset belongs to the model-reload settings above (context - length, KV cache, speculative decoding). Render it here, before - the Chat Template row, so it never reads as attached to Chat - Template (which is edited via its own dialog). When a model is - staged (deferred load), Load/Cancel takes its place: there's - nothing loaded to "apply" against yet. */} - {pendingSelection ? ( -
- {stagedDownloading && ( -

- Downloading…{" "} - {Math.round((stagedDownloadFraction ?? 0) * 100)}% + : "" + }`}

- )} - {/* GGUF picks only: a non-GGUF pick shows none of the load - knobs the blob captures, so there is nothing to remember. */} - {pendingIsGguf && ( - - )} - {stagedLoading ? ( - // Mid-load: nothing to load or abandon until it settles, so disable. - - ) : ( -
+ {mtpUpdatable && llamaUpdateStatus?.update_available && ( - -
- )} -
- ) : modelSettingsDirty ? ( -
- - -
- ) : null} - {/* The template override is a load-time knob too (applied on the next - reload) and the in-flight load already snapshotted it, so lock its - editors like the sibling controls -- a mid-load save would be - silently clobbered by the load response despite its toast. */} - - - + )} + + )} + {showContextVramWarning && ( +

+ Context length exceeds the estimated VRAM capacity ( + {ggufMaxContextLength?.toLocaleString()} tokens). The + model may use system RAM. +

+ )} + + )}
- +
savePresetWithName(presetNameInput)} disabled={!(settingsHydrated && presetSaveState.canSubmit)} - variant={presetSaveState.isSaveReady ? "default" : "outline"} + variant={ + presetSaveState.isSaveReady ? "default" : "outline" + } size="sm" className={cn( "h-9 w-full rounded-full text-[13px] font-medium tracking-nav", @@ -1850,7 +912,8 @@ export function ChatSettingsPanel({ Prompt caching - Reuse compatible prompt prefixes for lower latency and cost. + Reuse compatible prompt prefixes for lower latency and + cost.
Anthropic exposes a 5 minute and a 1 hour ephemeral - cache pool. The 1 hour pool costs 2x base input on - write vs 1.25x for 5 minute, but reads stay 0.1x for - both, so a single read landing more than 5 minutes - after the write pays off the premium. + cache pool. The 1 hour pool costs 2x base input on write + vs 1.25x for 5 minute, but reads stay 0.1x for both, so + a single read landing more than 5 minutes after the + write pays off the premium.