unsloth/studio/frontend
Etherll 11e5b58827
studio: gate image input on effective vision capability (#5492)
* Studio: gate image input on a usable mmproj for GGUF vision models

* Improve image gating and model capability sync

Tighten image-handling and model capability syncing across the chat flow. Key changes:

- chat-adapter: Replace per-message current-user image check with a simpler gate that blocks if ANY image is present in the outbound payload when the selected model cannot handle vision. Show the toast reason and flip the per-thread running flag on→off to avoid hanging wait promises before throwing.

- shared-composer: Simplify and correct image-attachment gating for single vs compare modes. Use an attach-time gate that defers to send/ensureModelLoaded in compare mode, introduce attachUnavailableReason, and only block immediately for single-mode. Remove an unused models selector.

- shared-composer: Sync the runtime models[] entry with the response from ensureModelLoaded so UI/send gates read fresh capabilities (isVision, isGguf, isAudio, audioType, hasAudioInput). This addresses catalog lag (e.g., GGUF mmproj arriving after the catalog snapshot).

- UX tweak: the file-picker button no longer outright blocks on image availability; addFiles still filters images per-file and toasts appropriately.

These changes prevent mid-stream server rejections, avoid deadlocks, and ensure model capability checks are accurate when attaching images or audio.

* studio: only pass --mmproj to llama-server when effective_is_vision

When a text-only GGUF (static is_vision=False) was paired with a
family-matching mmproj path, the launcher appended both --mmproj and
--spec-default, leaving llama-server in an inconsistent state while
Studio reported is_vision=False. Gate the --mmproj flag on
effective_is_vision so the launch command tracks the runtime
capability the rest of Studio sees.

* studio: reject image content in streaming /v1/responses for non-vision GGUF

_responses_stream forwards the OpenAI request body directly to
llama-server's /v1/chat/completions, bypassing the image-vs-vision
guard that openai_chat_completions enforces for the wrapped path.
Add the same check at the top of the streaming entry point so an
SDK client that posts an image to a non-vision GGUF receives a
typed 400 instead of an opaque downstream error.

* studio: gate external chat providers in the image input helper

External selections (cohere, deepseek, mistral, openrouter, ...) live
in externalProviders, not in runtime.models[], so activeModel is
undefined for them and the helper short-circuited to allow. Result:
images attached to a non-vision external chat model were dropped
silently downstream instead of rejected up front.

Add providerTypeSupportsVision to external-providers.ts (false for
known text-only providers, true for known vision-capable ones, null
for unknown / custom self-hosted) and thread externalSupportsVision
+ externalModelLabel through the helper. shared-composer.tsx,
runtime-provider.tsx (VisionImageAdapter.add), and chat-adapter.ts
pre-stream gate all resolve the provider type and pass it.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-05-18 01:35:46 -07:00
..
public Polish/cloud to providers (#5450) 2026-05-15 19:29:21 +04:00
src studio: gate image input on effective vision capability (#5492) 2026-05-18 01:35:46 -07:00
.gitignore perf(studio): upgrade to Vite 8 + auto-install bun for faster frontend builds (#4522) 2026-03-25 04:27:41 -07:00
.gitkeep add studio root folder 2026-02-02 09:14:35 +00:00
.npmrc security: NOT affected by Mini Shai-Hulud (May-12 wave) -- forward-looking hardening only (#5397) 2026-05-13 04:58:12 -07:00
biome.json feat: add seed dataset support with configuration, preview, and builder utilities 2026-02-14 18:44:38 +01:00
components.json add studio root folder 2026-02-02 09:14:35 +00:00
data-designer.openapi (1).yaml save and import, and fixes 2026-02-04 14:32:49 +01:00
eslint.config.js Final cleanup 2026-03-12 18:28:04 +00:00
index.html Final cleanup 2026-03-12 18:28:04 +00:00
package-lock.json studio/frontend: drop unused dependencies, move type pkg to devDeps (#5477) 2026-05-16 05:49:23 -07:00
package.json studio/frontend: drop unused dependencies, move type pkg to devDeps (#5477) 2026-05-16 05:49:23 -07:00
tsconfig.app.json Relax frontend unused local check (#4388) 2026-03-17 16:04:11 -07:00
tsconfig.json cleanup 2026-02-04 13:28:39 +01:00
tsconfig.node.json cleanup 2026-02-04 13:28:39 +01:00
vite.config.ts Fix Install commands for Windows + 1 line installs (#4447) 2026-03-19 02:09:09 -07:00