ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Studio: accept audio files through Add photos & files and fix the audio gate for Gemma 4 models (#6064)

Browse source 
* Studio: sync detected model capabilities into models[] after load

The chat composer gates audio upload on activeModel.hasAudioInput, but
/api/models/list omits audio fields for default and active-GGUF entries
and the single chat load path never wrote the load response's
capability flags back into the store. Audio-capable models such as the
Gemma 4 GGUFs therefore never unlocked audio input in the main chat,
while the compare composer (which does sync) worked.

Add syncModelCapabilities and call it after a successful load and after
the status fetch in refresh, so the flags also survive F5 and are not
clobbered by stale catalog data.

* Studio: merge audio upload into the Add photos & files picker

Remove the separate Upload audio row from the composer plus menu and
register an AudioAttachmentAdapter in the shared attachment pipeline,
so the standard picker and drag-drop accept wav, mp3, m4a, ogg, flac
and webm directly. Gating matches images: the picker always lists
audio and models without audio input get a toast at add() time. The
50MB limit is kept and the file shows as a normal attachment chip.

On send the adapter emits an audio content part on the attachment and
findLatestUserAudioBase64 now also scans attachment content, so the
request still carries audio_base64 exactly as before.

* Studio: extract AudioAttachmentAdapter into its own module

Move the adapter out of runtime-provider.tsx so it is importable in
isolation, export the audio send-path and capability-sync helpers for
tests, and guard attachment id generation for non-secure contexts
(crypto.randomUUID is undefined over plain HTTP on a LAN, matching the
existing guard in startCompare).

* Studio: do not claim .webm by extension in the audio adapter

A video/webm file would match the .webm extension entry and route to
the audio adapter. Real audio webm (MediaRecorder output) always
reports the audio/webm MIME, so matching webm by MIME only keeps video
files out while keeping recorded audio working.

* Studio: only send audio from the newest user message

audio_base64 switches the backend onto the audio generation path
(generate_whisper_response ignores chat messages entirely and
generate_audio_input_response bypasses the normal streaming path), so
replaying audio from an older turn hijacked text-only follow-ups:
Whisper would retranscribe the stale clip instead of erroring cleanly,
and audio VLMs lost tools and streaming. Stop the scan at the newest
user message, matching the consumed-on-send semantics of the legacy
pendingAudio path. Regenerating the audio turn itself still resends
its audio since it is the newest user message in that run.

Also guard extractAudioPartBase64 against null parts in deserialized
history content.

* Studio: forward audio input to llama-server for GGUF models (#6096)

* Studio: forward audio input to llama-server for GGUF models

* Studio: harden GGUF audio input handling (multi-format decode, size cap)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: carry GGUF audio in the message list so it works with tools

* Studio: bound decoded audio length and make the soundfile decoder optional

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Handle audio attachment edge cases

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: gate audio file picker by loaded model capability (#6142)

* Gate audio attachments by loaded model

* Use conditional spread for audio attachment adapter

* Preserve audio fallback while filtering picker

---------

Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: oobabooga <oobabooga4@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
This commit is contained in:
Michael Han 2026-06-10 08:45:30 -07:00 • committed by GitHub
commit 8bca7bcfc9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 450 additions and 82 deletions
Download patch file Download diff file Expand all files Collapse all files

49
studio/frontend/src/features/chat/api/chat-adapter.ts
View file

@ -986,24 +986,51 @@ function findLatestUserImageBase64(messages: RunMessages): string | undefined {
return undefined;
}
function findLatestUserAudioBase64(messages: RunMessages): string | undefined {
// Message content parts (compare view CompareMessagePart type: "audio").
function extractAudioPartBase64(
part: { type: string } | null | undefined,
): string | undefined {
if (!part || part.type !== "audio" || !("audio" in part)) return undefined;
const audioPart = (
part as unknown as {
type: "audio";
audio: string | { data: string; format: string };
}
).audio;
const raw = typeof audioPart === "string" ? audioPart : audioPart?.data;
if (!raw) return undefined;
return raw.startsWith("data:") ? raw.split(",")[1] : raw;
}
// Exported for tests.
export function findLatestUserAudioBase64(
messages: RunMessages,
): string | undefined {
for (let i = messages.length - 1; i >= 0; i -= 1) {
const message = messages[i];
if (!message || message.role !== "user") continue;
// Message content parts (from compare view's CompareMessagePart with type: "audio")
for (const part of message.content ?? []) {
if (part.type === "audio" && "audio" in part) {
const audioPart = (
part as unknown as {
type: "audio";
audio: string | { data: string; format: string };
}
).audio;
const raw = typeof audioPart === "string" ? audioPart : audioPart?.data;
if (raw) return raw.startsWith("data:") ? raw.split(",")[1] : raw;
const base64 = extractAudioPartBase64(part);
if (base64) return base64;
}
// Attachment content parts (from AudioAttachmentAdapter)
if ("attachments" in message) {
for (const attachment of message.attachments ?? []) {
for (const part of attachment.content ?? []) {
const base64 = extractAudioPartBase64(part);
if (base64) return base64;
}
}
}
// Only the newest user message counts. audio_base64 switches the
// backend onto the audio generation path, so replaying audio from an
// older turn would hijack text follow-ups (Whisper would retranscribe
// the stale clip). Matches the consumed-on-send semantics of the
// legacy pendingAudio path.
break;
}
// Runtime store (main composer's audio upload).