unsloth/studio/backend
Etherll 11e5b58827
studio: gate image input on effective vision capability (#5492)
* Studio: gate image input on a usable mmproj for GGUF vision models

* Improve image gating and model capability sync

Tighten image-handling and model capability syncing across the chat flow. Key changes:

- chat-adapter: Replace per-message current-user image check with a simpler gate that blocks if ANY image is present in the outbound payload when the selected model cannot handle vision. Show the toast reason and flip the per-thread running flag on→off to avoid hanging wait promises before throwing.

- shared-composer: Simplify and correct image-attachment gating for single vs compare modes. Use an attach-time gate that defers to send/ensureModelLoaded in compare mode, introduce attachUnavailableReason, and only block immediately for single-mode. Remove an unused models selector.

- shared-composer: Sync the runtime models[] entry with the response from ensureModelLoaded so UI/send gates read fresh capabilities (isVision, isGguf, isAudio, audioType, hasAudioInput). This addresses catalog lag (e.g., GGUF mmproj arriving after the catalog snapshot).

- UX tweak: the file-picker button no longer outright blocks on image availability; addFiles still filters images per-file and toasts appropriately.

These changes prevent mid-stream server rejections, avoid deadlocks, and ensure model capability checks are accurate when attaching images or audio.

* studio: only pass --mmproj to llama-server when effective_is_vision

When a text-only GGUF (static is_vision=False) was paired with a
family-matching mmproj path, the launcher appended both --mmproj and
--spec-default, leaving llama-server in an inconsistent state while
Studio reported is_vision=False. Gate the --mmproj flag on
effective_is_vision so the launch command tracks the runtime
capability the rest of Studio sees.

* studio: reject image content in streaming /v1/responses for non-vision GGUF

_responses_stream forwards the OpenAI request body directly to
llama-server's /v1/chat/completions, bypassing the image-vs-vision
guard that openai_chat_completions enforces for the wrapped path.
Add the same check at the top of the streaming entry point so an
SDK client that posts an image to a non-vision GGUF receives a
typed 400 instead of an opaque downstream error.

* studio: gate external chat providers in the image input helper

External selections (cohere, deepseek, mistral, openrouter, ...) live
in externalProviders, not in runtime.models[], so activeModel is
undefined for them and the helper short-circuited to allow. Result:
images attached to a non-vision external chat model were dropped
silently downstream instead of rejected up front.

Add providerTypeSupportsVision to external-providers.ts (false for
known text-only providers, true for known vision-capable ones, null
for unknown / custom self-hosted) and thread externalSupportsVision
+ externalModelLabel through the helper. shared-composer.tsx,
runtime-provider.tsx (VisionImageAdapter.add), and chat-adapter.ts
pre-stream gate all resolve the provider type and pass it.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-05-18 01:35:46 -07:00
..
assets Add Qwen3.6 inference defaults for Studio (#5065) 2026-04-16 11:42:42 -07:00
auth studio: security and hardening pass (auth rate-limit, sandbox, path containment, schema validation, headers) (#5375) 2026-05-13 06:12:18 -07:00
core studio: gate image input on effective vision capability (#5492) 2026-05-18 01:35:46 -07:00
loggers Studio: stop truncating long log lines as suspected base64 (#5335) 2026-05-08 13:07:18 +04:00
models Studio: warn when llama.cpp prebuilt is at least 3 days behind (#5529) 2026-05-18 00:21:50 -07:00
plugins fix(gh_client): fail fast on 401/403 auth errors instead of retrying forever (#5325) (#5329) 2026-05-08 21:57:41 +04:00
requirements studio: API external provider support for chat (OpenAI, Mistral, Gemini, Cohere, Anthropic, OpenRouter, DeepSeek, custom providers) (#4706) 2026-05-14 16:13:59 +04:00
routes studio: gate image input on effective vision capability (#5492) 2026-05-18 01:35:46 -07:00
state unsloth run: add --enable-tools/--disable-tools server-side tool policy (#5277) 2026-05-05 12:45:15 +04:00
storage studio: API external provider support for chat (OpenAI, Mistral, Gemini, Cohere, Anthropic, OpenRouter, DeepSeek, custom providers) (#4706) 2026-05-14 16:13:59 +04:00
tests fix(studio/worker): inject --gcc-install-dir for HIP source builds on Ubuntu 24.04 (#5517) 2026-05-18 01:05:30 -07:00
utils studio: extend offline DNS auto-detect to inference parent + training (#5512) 2026-05-18 00:31:33 -07:00
__init__.py Final cleanup 2026-03-12 18:28:04 +00:00
_platform_compat.py Fix Studio crash on Anaconda/conda-forge Python (#4484) 2026-03-22 05:36:55 -07:00
colab.py Fix/studio colab button message: Add fallback message for Colab Studio button when proxy URL fails (#4866) 2026-04-05 21:57:45 -07:00
main.py Studio: warn when llama.cpp prebuilt is at least 3 days behind (#5529) 2026-05-18 00:21:50 -07:00
run.py Studio: stop hint, Uvicorn log rename, reachability check + Mac UI CI retry hardening (#5503) 2026-05-17 07:44:06 -07:00
startup_banner.py Studio: stop hint, Uvicorn log rename, reachability check + Mac UI CI retry hardening (#5503) 2026-05-17 07:44:06 -07:00