unsloth/studio/backend
Daniel Han 5b41872e8b
Studio: wire OpenAI image_generation tool (#5688)
* Studio: wire OpenAI image_generation tool

OpenAI's Responses API exposes server-side image generation as a
tool entry (`{type: "image_generation"}`); the result comes back as
an `image_generation_call` output item with the base64 image on
`result`, the actual prompt used on `revised_prompt`, plus `size`,
`quality`, `output_format`, `background`. The model decides when to
call the tool based on the user's request; rendering uses one of
the gpt-image-* backbones server-side.

Available on every gpt-5.x family member plus gpt-4.1, gpt-4o, o3,
o4-mini per the docs.

Changes:

- Append `{type:"image_generation"}` to the Responses request tools
  array when `enabled_tools` carries `image_generation` AND the base
  URL points at cloud OpenAI. Non-cloud bases (ollama, llama.cpp,
  "custom" presets that collapse to provider="openai") silently drop
  the tool to avoid 400s.
- Mirror the same logic in `_build_body` (the post-expiry retry
  builder) so retries carry the same tool set as the original
  attempt.
- Handle `image_generation_call` items in
  `response.output_item.done`: emit `tool_start` with
  `arguments:{kind:"image", prompt:<revised_prompt>}` and `tool_end`
  with `image_b64`, `image_mime`, `size`, `quality`, `background`
  so the chat adapter can render an inline preview. Image bytes go
  on the tool_end chunk; no extra fields on the chat-completions
  envelope so the OpenAI SDK shape stays clean.
- Add `import time` (used for synthesised tool_call_id fallback).
- Add `test_openai_image_generation.py` with 5 cases: tool entry on
  cloud OpenAI, combined with web_search + code_execution
  (verifies all three coexist), non-cloud drop, omitted pill leaves
  body untouched, output item translation produces the expected
  tool_start + tool_end chunks.

Live verified end-to-end: `gpt-5.4-mini` with `image_generation`
tool returned an `image_generation_call` carrying ~1MB of base64
PNG plus the gpt-image backbone's revised prompt.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Use time.time_ns() for synthesised image_generation tool_call_id

Gemini medium on PR #5688: `int(time.time() * 1000)` has 1ms
resolution; two image generations resolving in the same millisecond
would collide on the synthesised id. Bump to nanoseconds.

(In practice the upstream `image_generation_call` item always carries
its own `id`; the synthesised fallback only fires when OpenAI omits
it -- rare, but cheap to harden.)

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-05-22 06:03:38 -07:00
..
assets Add Qwen3.6 inference defaults for Studio (#5065) 2026-04-16 11:42:42 -07:00
auth studio: security and hardening pass (auth rate-limit, sandbox, path containment, schema validation, headers) (#5375) 2026-05-13 06:12:18 -07:00
core Studio: wire OpenAI image_generation tool (#5688) 2026-05-22 06:03:38 -07:00
loggers Studio: stop truncating long log lines as suspected base64 (#5335) 2026-05-08 13:07:18 +04:00
models Studio: support Anthropic 1h cache TTL via prompt_cache_ttl (#5685) 2026-05-22 06:03:32 -07:00
plugins fix(gh_client): fail fast on 401/403 auth errors instead of retrying forever (#5325) (#5329) 2026-05-08 21:57:41 +04:00
requirements studio: API external provider support for chat (OpenAI, Mistral, Gemini, Cohere, Anthropic, OpenRouter, DeepSeek, custom providers) (#4706) 2026-05-14 16:13:59 +04:00
routes Studio: support Anthropic 1h cache TTL via prompt_cache_ttl (#5685) 2026-05-22 06:03:32 -07:00
state unsloth run: add --enable-tools/--disable-tools server-side tool policy (#5277) 2026-05-05 12:45:15 +04:00
storage studio: API external provider support for chat (OpenAI, Mistral, Gemini, Cohere, Anthropic, OpenRouter, DeepSeek, custom providers) (#4706) 2026-05-14 16:13:59 +04:00
tests Studio: wire OpenAI image_generation tool (#5688) 2026-05-22 06:03:38 -07:00
utils Studio: tools, thinking blocks, code execution and web search for safetensors (#5520) 2026-05-19 06:30:17 -07:00
__init__.py Final cleanup 2026-03-12 18:28:04 +00:00
_platform_compat.py Fix Studio crash on Anaconda/conda-forge Python (#4484) 2026-03-22 05:36:55 -07:00
colab.py Fix/studio colab button message: Add fallback message for Colab Studio button when proxy URL fails (#4866) 2026-04-05 21:57:45 -07:00
main.py Studio: warn when llama.cpp prebuilt is at least 3 days behind (#5529) 2026-05-18 00:21:50 -07:00
run.py Studio: stop hint, Uvicorn log rename, reachability check + Mac UI CI retry hardening (#5503) 2026-05-17 07:44:06 -07:00
startup_banner.py Studio: stop hint, Uvicorn log rename, reachability check + Mac UI CI retry hardening (#5503) 2026-05-17 07:44:06 -07:00