bebc93d8fc* fix(studio): handle multimodal list content in inference text paths
Studio receives chat message content in two shapes: the legacy string
form, and the OpenAI multimodal list form
([{"type": "text", "text": ...}, {"type": "image_url", ...}]).
Several string-only paths called .strip()/re.sub()/f-string interpolation
on content directly, raising "'list' object has no attribute 'replace'"
for vision models (issue #4383), or rendering the list repr into the
prompt for the manual chat-template formatters.
Add core/inference/message_content.py with content_to_text(), a pure
helper (no heavy imports) that returns strings unchanged and joins the
text parts of a list while dropping image/audio parts. Apply it at every
string-only content site: _generate_vision_response, the audio user-text
extraction, format_chat_prompt, and the llama3/mistral/chatml/alpaca/
generic template formatters. The plain-string path is a no-op, so
existing behavior is unchanged.
Adds tests/test_message_content.py covering str/None/list/tuple,
multimodal drop, multi-part join and empty-part skipping.
* Tighten code comments (no logic change)
* studio: join multimodal text parts with newline for llama.cpp parity
llama.cpp joins multiple text content parts with a newline (common/chat.cpp),
so match that in content_to_text instead of a single space.
---------
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
38 lines
1.4 KiB
Python
38 lines
1.4 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Normalize chat-message `content` (string or OpenAI multimodal list) to text.
|
|
|
|
String-only formatting paths called string ops directly on `content` and broke
|
|
on the list form (#4383). `content_to_text` collapses either shape to a string,
|
|
dropping non-text parts. No heavy imports, so it is unit-testable alone.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import Any
|
|
|
|
|
|
def content_to_text(content: Any) -> str:
|
|
"""Plain text of a `content`: str unchanged, list/tuple text parts newline-joined
|
|
(non-text dropped), None to "", else str(content)."""
|
|
if content is None:
|
|
return ""
|
|
if isinstance(content, str):
|
|
return content
|
|
if isinstance(content, (list, tuple)):
|
|
parts = []
|
|
for item in content:
|
|
if isinstance(item, str):
|
|
if item:
|
|
parts.append(item)
|
|
elif isinstance(item, dict):
|
|
# Skip non-text parts (image_url, input_audio, ...).
|
|
part_type = item.get("type")
|
|
if part_type is not None and part_type != "text":
|
|
continue
|
|
text = item.get("text")
|
|
if isinstance(text, str) and text:
|
|
parts.append(text)
|
|
return "\n".join(parts)
|
|
return str(content)
|