unsloth/studio
Etherll 11e5b58827
studio: gate image input on effective vision capability (#5492)
* Studio: gate image input on a usable mmproj for GGUF vision models

* Improve image gating and model capability sync

Tighten image-handling and model capability syncing across the chat flow. Key changes:

- chat-adapter: Replace per-message current-user image check with a simpler gate that blocks if ANY image is present in the outbound payload when the selected model cannot handle vision. Show the toast reason and flip the per-thread running flag on→off to avoid hanging wait promises before throwing.

- shared-composer: Simplify and correct image-attachment gating for single vs compare modes. Use an attach-time gate that defers to send/ensureModelLoaded in compare mode, introduce attachUnavailableReason, and only block immediately for single-mode. Remove an unused models selector.

- shared-composer: Sync the runtime models[] entry with the response from ensureModelLoaded so UI/send gates read fresh capabilities (isVision, isGguf, isAudio, audioType, hasAudioInput). This addresses catalog lag (e.g., GGUF mmproj arriving after the catalog snapshot).

- UX tweak: the file-picker button no longer outright blocks on image availability; addFiles still filters images per-file and toasts appropriately.

These changes prevent mid-stream server rejections, avoid deadlocks, and ensure model capability checks are accurate when attaching images or audio.

* studio: only pass --mmproj to llama-server when effective_is_vision

When a text-only GGUF (static is_vision=False) was paired with a
family-matching mmproj path, the launcher appended both --mmproj and
--spec-default, leaving llama-server in an inconsistent state while
Studio reported is_vision=False. Gate the --mmproj flag on
effective_is_vision so the launch command tracks the runtime
capability the rest of Studio sees.

* studio: reject image content in streaming /v1/responses for non-vision GGUF

_responses_stream forwards the OpenAI request body directly to
llama-server's /v1/chat/completions, bypassing the image-vs-vision
guard that openai_chat_completions enforces for the wrapped path.
Add the same check at the top of the streaming entry point so an
SDK client that posts an image to a non-vision GGUF receives a
typed 400 instead of an opaque downstream error.

* studio: gate external chat providers in the image input helper

External selections (cohere, deepseek, mistral, openrouter, ...) live
in externalProviders, not in runtime.models[], so activeModel is
undefined for them and the helper short-circuited to allow. Result:
images attached to a non-vision external chat model were dropped
silently downstream instead of rejected up front.

Add providerTypeSupportsVision to external-providers.ts (false for
known text-only providers, true for known vision-capable ones, null
for unknown / custom self-hosted) and thread externalSupportsVision
+ externalModelLabel through the helper. shared-composer.tsx,
runtime-provider.tsx (VisionImageAdapter.add), and chat-adapter.ts
pre-stream gate all resolve the provider type and pass it.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-05-18 01:35:46 -07:00
..
backend studio: gate image input on effective vision capability (#5492) 2026-05-18 01:35:46 -07:00
frontend studio: gate image input on effective vision capability (#5492) 2026-05-18 01:35:46 -07:00
src-tauri studio: expose launcher capability bits on unauth /api/health (#5486) 2026-05-17 21:29:24 -07:00
__init__.py Final cleanup 2026-03-12 18:28:04 +00:00
install_llama_prebuilt.py tests/studio: lock in Windows GPU detection fix (#5106) with a synthetic CI test (#5376) 2026-05-18 00:06:01 -07:00
install_python_stack.py studio: skip flash-attn install on Blackwell GPUs (sm_100+) (#5420) 2026-05-14 18:13:50 +04:00
LICENSE.AGPL-3.0 Add AGPL-3.0 license to studio folder 2026-03-09 19:36:25 +00:00
setup.bat Final cleanup 2026-03-12 18:28:04 +00:00
setup.ps1 CI: scope GITHUB_TOKEN permissions, add MLX CI, unblock ~60 skipped tests (#5312) 2026-05-11 03:19:13 -07:00
setup.sh Route CPU-only Linux x86_64 to ggml-org/llama.cpp prebuilts (#5302) 2026-05-05 23:22:22 -07:00
Unsloth_Studio_Colab.ipynb studio: add --local to setup.sh + overlay unsloth-zoo from git main (#5252) 2026-05-02 08:51:56 +04:00