unsloth/studio/backend/core
Daniel Han 5b41872e8b
Studio: wire OpenAI image_generation tool (#5688)
* Studio: wire OpenAI image_generation tool

OpenAI's Responses API exposes server-side image generation as a
tool entry (`{type: "image_generation"}`); the result comes back as
an `image_generation_call` output item with the base64 image on
`result`, the actual prompt used on `revised_prompt`, plus `size`,
`quality`, `output_format`, `background`. The model decides when to
call the tool based on the user's request; rendering uses one of
the gpt-image-* backbones server-side.

Available on every gpt-5.x family member plus gpt-4.1, gpt-4o, o3,
o4-mini per the docs.

Changes:

- Append `{type:"image_generation"}` to the Responses request tools
  array when `enabled_tools` carries `image_generation` AND the base
  URL points at cloud OpenAI. Non-cloud bases (ollama, llama.cpp,
  "custom" presets that collapse to provider="openai") silently drop
  the tool to avoid 400s.
- Mirror the same logic in `_build_body` (the post-expiry retry
  builder) so retries carry the same tool set as the original
  attempt.
- Handle `image_generation_call` items in
  `response.output_item.done`: emit `tool_start` with
  `arguments:{kind:"image", prompt:<revised_prompt>}` and `tool_end`
  with `image_b64`, `image_mime`, `size`, `quality`, `background`
  so the chat adapter can render an inline preview. Image bytes go
  on the tool_end chunk; no extra fields on the chat-completions
  envelope so the OpenAI SDK shape stays clean.
- Add `import time` (used for synthesised tool_call_id fallback).
- Add `test_openai_image_generation.py` with 5 cases: tool entry on
  cloud OpenAI, combined with web_search + code_execution
  (verifies all three coexist), non-cloud drop, omitted pill leaves
  body untouched, output item translation produces the expected
  tool_start + tool_end chunks.

Live verified end-to-end: `gpt-5.4-mini` with `image_generation`
tool returned an `image_generation_call` carrying ~1MB of base64
PNG plus the gpt-image backbone's revised prompt.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Use time.time_ns() for synthesised image_generation tool_call_id

Gemini medium on PR #5688: `int(time.time() * 1000)` has 1ms
resolution; two image generations resolving in the same millisecond
would collide on the synthesised id. Bump to nanoseconds.

(In practice the upstream `image_generation_call` item always carries
its own `id`; the synthesised fallback only fires when OpenAI omits
it -- rare, but cheap to harden.)

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-05-22 06:03:38 -07:00
..
data_recipe chore(deps): bump the npm-oxc-validator group across 1 directory with 2 updates (#5667) 2026-05-22 04:46:20 -07:00
export feat(studio): MLX training tab on Apple Silicon (LoRA / full FT, VLM, export) (#5265) 2026-05-05 23:54:58 -07:00
inference Studio: wire OpenAI image_generation tool (#5688) 2026-05-22 06:03:38 -07:00
training studio: install flash-linear-attention and tilelang for Qwen3.5 family (#5434) 2026-05-18 03:49:06 -07:00
__init__.py [Studio] Show non exported models in chat UI (#4892) 2026-04-14 15:03:58 +04:00
tool_healing.py studio: extract tool-call XML parser into a reusable helper module (#5583) 2026-05-19 05:06:17 -07:00