unsloth/studio/frontend/src/features/chat/utils/chat-settings-storage.ts
Daniel Han 7d1b68079e
Studio: Anthropic fast_mode toggle and streaming refusal handling (#5715)
* Studio: add Anthropic fast_mode toggle + surface streaming refusals

Fast mode (beta `fast-mode-2026-02-01`) lets Claude Opus 4.6 and 4.7
generate output tokens up to 2.5x faster at 6x standard Opus
pricing. The toggle lives in Configuration → Provider when the
selected Anthropic model is Opus 4.6 or 4.7 and is otherwise
hidden. Backend gates the same prefixes a second time so a stale
frontend cannot make Anthropic 400 the request, and the
`fast-mode-2026-02-01` beta header is merged onto whatever other
betas the request already needed (code-execution, compaction).

Streaming refusals (`message_delta.delta.stop_reason="refusal"` on
Claude 4 models) now surface a short user-facing notice in the
assistant message before the translated OpenAI chunk emits the
existing `finish_reason="content_filter"`. Previously the chat
bubble truncated silently because the SSE stopped mid-stream with
no visible explanation. Per the upstream docs the conversation
must be reset before continuing, so the notice tells the user
exactly that.

Reference:
- https://platform.claude.com/docs/en/build-with-claude/fast-mode
- https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/handle-streaming-refusals

Tests:
- studio/backend/tests/test_anthropic_fast_mode_and_refusal.py (8 cases
  pinning fast_mode pass-through on 4.6/4.7, silent drop on Sonnet /
  Haiku / older Opus / None / False, and the refusal notice + finish
  reason on a synthetic refusal stream).

* Studio: drop refused Anthropic turns from the next request

Anthropic's streaming-refusal guidance says the refused assistant
turn must be removed or updated before the next call -- otherwise
the safety classifier keeps refusing. The PR only added a
user-visible notice; the partial assistant output (plus the notice
itself) still rode the next request via toOpenAIMessage.

Tag the refusal turn with an HTML-comment sentinel emitted alongside
the notice. The chat-adapter checks for that sentinel in
toOpenAIMessage and returns null, so the refused turn is excluded
from outboundMessages. The notice still renders in the transcript
(HTML comments don't display), so users keep the explanation.

* Studio: filter None finish_reason entries in test helper

test_refusal_maps_to_content_filter expects only ['content_filter']
in the finish_reasons list, but the post-PR refusal path emits a
user-visible content notice chunk first. Every _content_chunk
carries 'finish_reason: None' by construction; the helper was
appending those, so the assertion saw [None, 'content_filter']
instead of ['content_filter'].

None is not a finish reason -- it's just mid-stream delta noise.
Skip None values in _finish_reasons so the helper reflects what
the test names actually claim to check. Same fix applies cleanly
to the other helper usages (pause_turn test expects [] and the
sibling stop test expects ['stop'], both unaffected).

* Studio: cover Anthropic fast-mode edge cases

Adds 19 cases on top of the 9 in test_anthropic_fast_mode_and_refusal.
The base file pins the happy path; this file fills in the cliffs:

* Dated-snapshot prefix matching: claude-opus-4-7-2026-02-01 and
  claude-opus-4-6-2026-02-01 still gate fast_mode through, while
  claude-opus-4-5-2025-08-01 and claude-sonnet-4-6-2026-02-01 do not.
* Strict opt-in: a future claude-opus-4-8 or claude-opus-5 does NOT
  auto-enable fast_mode -- the prefix tuple must be bumped explicitly
  when a new family is whitelisted upstream.
* Beta-header merge: fast_mode coexists with code-execution-2025-08-25
  and compact-2026-01-12 in one comma-separated anthropic-beta header
  with no duplicates and no truncation. Pins the value to the exact
  fast-mode-2026-02-01 docs token so a typo would fail CI.
* Non-destruction: fast_mode=None produces byte-identical outbound
  body and headers to the version that omits the argument entirely.
  Same for fast_mode=False. Guarantees the upgrade path is
  non-breaking on existing Anthropic streams.
* Refusal stream ordering: the user-visible notice precedes the
  finish_reason chunk so a streaming UI paints text before flipping
  to content_filter. Refusal sentinel emitted exactly once. Notice
  rides a normal content delta chunk with finish_reason still null.
  Partial assistant deltas survive before the notice.
* Provider-side refusal coverage: a refusal on Sonnet (not just Opus)
  still emits the notice + sentinel + content_filter mapping, since
  refusal handling is not gated on fast-mode capability.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Persist fastMode, drop refused user message on retry

Two follow-ups on #5715:

1) sanitizeInferenceParams stripped fastMode. fastMode is in
   PERSISTED_INFERENCE_PARAM_KEYS but the storage sanitizer only kept
   numeric fields plus systemPrompt and trustRemoteCode, so the new
   toggle was silently dropped on reload and on the
   /api/chat/settings round-trip. Save it the same way trustRemoteCode
   is saved.

2) Refusal recovery now also drops the triggering user turn.
   Returning null from toOpenAIMessage on the assistant side left the
   user prompt that caused the refusal in the outbound history, so
   the very next request would re-trigger the same classifier.
   Anthropic's refusal-handling guidance is explicit on this: remove
   the refused turn AND the user message that triggered it before
   the next call. Implemented via a pre-pass that pops the trailing
   user message when an assistant carries the refusal sentinel.

Typecheck clean.

* Studio: out-of-band refusal signal + fast-mode prefix/usage/pricing fixes

The text sentinel for the Anthropic refusal drop signal was spoofable:
any assistant message containing the literal
<!--studio:anthropic-refusal--> would prune the prior user + assistant
pair on the next request. Move the signal onto a separate _toolEvent
chunk that the chat adapter latches into
assistant.metadata.custom.anthropicRefusal; assistant text can no
longer control the pruner.

Tighten the fast-mode model gate (backend + frontend) to require a "-"
family boundary so claude-opus-4-70 / claude-opus-4-7b style IDs do
not get speed: "fast" on a naive startswith match.

Use survivingMessages for the image / audio attachment scan so a
refused user turn does not gate or mis-attribute the next non-refused
turn.

Propagate Anthropic usage.speed onto the OpenAI-style usage chunk and
apply the documented 6x fast-mode multiplier in the cost calculator
(stacks with prompt-cache multipliers per the docs); expose the new
multiplier on the pricing snapshot for the UI tooltip.

Tests cover the tool-event chunk shape, the prefix-collision rejects,
usage.speed propagation, the 6x pricing math, and that the visible
refusal text carries no embedded sentinel.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Shorten fast-mode and refusal comments for PR #5715

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-05-25 23:37:12 -07:00

408 lines
13 KiB
TypeScript

// SPDX-License-Identifier: AGPL-3.0-only
// Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
import {
getChatSettings,
saveChatSettingsPatch,
type PersistedChatPreset,
type PersistedChatSettings,
type PersistedInferenceParams,
} from "../api/chat-settings-api";
import {
BUILTIN_PRESETS,
defaultInferenceParams,
getPresetOwnedConfigKey,
getUniquePresetName,
normalizeCustomPresets,
type ChatPresetSource,
type Preset,
} from "../presets/preset-policy";
import type { ReasoningEffort } from "../stores/chat-runtime-store";
const AUTO_TITLE_KEY = "unsloth_chat_auto_title";
const AUTO_HEAL_TOOL_CALLS_KEY = "unsloth_auto_heal_tool_calls";
const MAX_TOOL_CALLS_KEY = "unsloth_max_tool_calls_per_message";
const TOOL_CALL_TIMEOUT_KEY = "unsloth_tool_call_timeout";
const INFERENCE_PARAMS_KEY = "unsloth_chat_inference_params";
const CHAT_ACTIVE_PRESET_KEY = "unsloth_chat_active_preset";
const CHAT_ACTIVE_PRESET_SOURCE_KEY = "unsloth_chat_active_preset_source";
const REASONING_EFFORT_KEY = "unsloth_reasoning_effort";
const PRESERVE_THINKING_KEY = "unsloth_preserve_thinking";
const CHAT_PRESETS_KEY = "unsloth_chat_custom_presets";
const LEGACY_CHAT_SYSTEM_PROMPTS_KEY = "unsloth_chat_system_prompts";
const LEGACY_CHAT_SETTINGS_IMPORT_KEY =
"unsloth_chat_settings_imported_to_studio_db";
const NUMERIC_INFERENCE_FIELDS = [
"temperature",
"topP",
"topK",
"minP",
"repetitionPenalty",
"presencePenalty",
"maxSeqLength",
"maxTokens",
] as const satisfies readonly (keyof PersistedInferenceParams)[];
const CHAT_PRESET_SOURCES = new Set<string>([
"builtin-default",
"custom",
"modified",
]);
const REASONING_EFFORTS = new Set<string>([
"none",
"minimal",
"low",
"medium",
"high",
"max",
"xhigh",
]);
interface LegacySystemPromptTemplate {
name: string;
content: string;
}
function canUseStorage(): boolean {
return typeof window !== "undefined";
}
function isRecord(value: unknown): value is Record<string, unknown> {
return value != null && typeof value === "object" && !Array.isArray(value);
}
function hasKeys(value: object): boolean {
return Object.keys(value).length > 0;
}
function getStorageItem(key: string): string | null {
if (!canUseStorage()) return null;
try {
return localStorage.getItem(key);
} catch {
return null;
}
}
function isLegacySettingsImportDone(): boolean {
return getStorageItem(LEGACY_CHAT_SETTINGS_IMPORT_KEY) === "true";
}
function markLegacySettingsImportDone(): void {
if (!canUseStorage()) return;
try {
localStorage.setItem(LEGACY_CHAT_SETTINGS_IMPORT_KEY, "true");
} catch {
// ignore
}
}
function parseJson(value: string | null): unknown {
if (!value) return undefined;
try {
return JSON.parse(value) as unknown;
} catch {
return undefined;
}
}
function loadBool(key: string): boolean | undefined {
const raw = getStorageItem(key);
if (raw === "true") return true;
if (raw === "false") return false;
return undefined;
}
function loadInt(key: string, min: number): number | undefined {
const raw = getStorageItem(key);
if (raw == null || raw.trim() === "") return undefined;
const value = Number(raw);
return Number.isInteger(value) && value >= min ? value : undefined;
}
function sanitizeInferenceParams(
value: unknown,
): PersistedInferenceParams | undefined {
if (!isRecord(value)) return undefined;
const params: PersistedInferenceParams = {};
for (const field of NUMERIC_INFERENCE_FIELDS) {
const fieldValue = value[field];
if (typeof fieldValue === "number" && Number.isFinite(fieldValue)) {
params[field] = fieldValue;
}
}
if (typeof value.systemPrompt === "string") {
params.systemPrompt = value.systemPrompt;
}
if (typeof value.trustRemoteCode === "boolean") {
params.trustRemoteCode = value.trustRemoteCode;
}
// Mirror trustRemoteCode handling so the toggle survives reload
// and the /api/chat/settings round-trip.
if (typeof value.fastMode === "boolean") {
params.fastMode = value.fastMode;
}
return hasKeys(params) ? params : undefined;
}
function toFullPreset(preset: PersistedChatPreset): Preset {
return {
name: preset.name,
params: {
...defaultInferenceParams,
...preset.params,
checkpoint: defaultInferenceParams.checkpoint,
},
};
}
function sanitizeCustomPresets(
value: unknown,
): PersistedChatPreset[] | undefined {
if (!Array.isArray(value)) return undefined;
if (value.length === 0) return [];
const presets = value
.map((item): PersistedChatPreset | null => {
if (!isRecord(item) || typeof item.name !== "string") return null;
const name = item.name.trim();
if (!name) return null;
const params = sanitizeInferenceParams(item.params);
return { name, params: params ?? {} };
})
.filter((preset): preset is PersistedChatPreset => preset !== null);
if (presets.length === 0) return [];
return normalizeCustomPresets(presets.map(toFullPreset)).map(
(preset, index) => ({
name: preset.name,
params: presets[index]?.params ?? {},
}),
);
}
function sanitizePresetSource(value: unknown): ChatPresetSource | undefined {
return typeof value === "string" && CHAT_PRESET_SOURCES.has(value)
? (value as ChatPresetSource)
: undefined;
}
function sanitizeReasoningEffort(value: unknown): ReasoningEffort | undefined {
return typeof value === "string" && REASONING_EFFORTS.has(value)
? (value as ReasoningEffort)
: undefined;
}
function sanitizeBool(value: unknown): boolean | undefined {
return typeof value === "boolean" ? value : undefined;
}
function sanitizeInt(value: unknown, min: number): number | undefined {
return typeof value === "number" && Number.isInteger(value) && value >= min
? value
: undefined;
}
function sanitizeChatSettings(value: unknown): PersistedChatSettings {
if (!isRecord(value)) return {};
const settings: PersistedChatSettings = {};
const inferenceParams = sanitizeInferenceParams(value.inferenceParams);
const customPresets = sanitizeCustomPresets(value.customPresets);
const activePresetSource = sanitizePresetSource(value.activePresetSource);
const reasoningEffort = sanitizeReasoningEffort(value.reasoningEffort);
const autoTitle = sanitizeBool(value.autoTitle);
const preserveThinking = sanitizeBool(value.preserveThinking);
const autoHealToolCalls = sanitizeBool(value.autoHealToolCalls);
const maxToolCallsPerMessage = sanitizeInt(value.maxToolCallsPerMessage, 1);
const toolCallTimeout = sanitizeInt(value.toolCallTimeout, 1);
if (inferenceParams) settings.inferenceParams = inferenceParams;
if (customPresets !== undefined) settings.customPresets = customPresets;
if (typeof value.activePreset === "string" && value.activePreset.trim()) {
settings.activePreset = value.activePreset.trim();
}
if (activePresetSource) settings.activePresetSource = activePresetSource;
if (autoTitle !== undefined) settings.autoTitle = autoTitle;
if (reasoningEffort) settings.reasoningEffort = reasoningEffort;
if (preserveThinking !== undefined)
settings.preserveThinking = preserveThinking;
if (autoHealToolCalls !== undefined) {
settings.autoHealToolCalls = autoHealToolCalls;
}
if (maxToolCallsPerMessage !== undefined) {
settings.maxToolCallsPerMessage = maxToolCallsPerMessage;
}
if (toolCallTimeout !== undefined) settings.toolCallTimeout = toolCallTimeout;
return settings;
}
function loadLegacySystemPromptPresets(
existingPresets: PersistedChatPreset[],
): PersistedChatPreset[] {
const parsed = parseJson(getStorageItem(LEGACY_CHAT_SYSTEM_PROMPTS_KEY));
if (!Array.isArray(parsed)) return [];
const usedNames = new Set([
...BUILTIN_PRESETS.map((preset) => preset.name),
...existingPresets.map((preset) => preset.name),
]);
const seenConfigKeys = new Set(
[...BUILTIN_PRESETS, ...existingPresets.map(toFullPreset)].map((preset) =>
getPresetOwnedConfigKey(preset.params),
),
);
return parsed
.filter((item): item is LegacySystemPromptTemplate => {
if (!isRecord(item)) return false;
return typeof item.name === "string" && typeof item.content === "string";
})
.map((template) => ({
template,
params: {
...defaultInferenceParams,
systemPrompt: template.content,
},
}))
.filter(({ params }) => {
const configKey = getPresetOwnedConfigKey(params);
if (seenConfigKeys.has(configKey)) return false;
seenConfigKeys.add(configKey);
return true;
})
.map(({ template, params }) => ({
name: getUniquePresetName(`${template.name} Prompt`, usedNames),
params: sanitizeInferenceParams(params) ?? {},
}));
}
export function isEmptyChatSettings(settings: PersistedChatSettings): boolean {
return (
(!settings.inferenceParams || !hasKeys(settings.inferenceParams)) &&
settings.customPresets === undefined &&
settings.activePreset === undefined &&
settings.activePresetSource === undefined &&
settings.autoTitle === undefined &&
settings.reasoningEffort === undefined &&
settings.preserveThinking === undefined &&
settings.autoHealToolCalls === undefined &&
settings.maxToolCallsPerMessage === undefined &&
settings.toolCallTimeout === undefined
);
}
export function loadLegacyChatSettings(): PersistedChatSettings {
const settings: PersistedChatSettings = {};
const rawCustomPresets = getStorageItem(CHAT_PRESETS_KEY);
const rawLegacyPromptPresets = getStorageItem(LEGACY_CHAT_SYSTEM_PROMPTS_KEY);
const hasLegacyPresetStorage =
rawCustomPresets !== null || rawLegacyPromptPresets !== null;
const inferenceParams = sanitizeInferenceParams(
parseJson(getStorageItem(INFERENCE_PARAMS_KEY)),
);
const customPresets = sanitizeCustomPresets(parseJson(rawCustomPresets));
const legacyPromptPresets = loadLegacySystemPromptPresets(
customPresets ?? [],
);
const activePreset = getStorageItem(CHAT_ACTIVE_PRESET_KEY);
const activePresetSource = sanitizePresetSource(
getStorageItem(CHAT_ACTIVE_PRESET_SOURCE_KEY),
);
const reasoningEffort = sanitizeReasoningEffort(
getStorageItem(REASONING_EFFORT_KEY),
);
const autoTitle = loadBool(AUTO_TITLE_KEY);
const preserveThinking = loadBool(PRESERVE_THINKING_KEY);
const autoHealToolCalls = loadBool(AUTO_HEAL_TOOL_CALLS_KEY);
const maxToolCallsPerMessage = loadInt(MAX_TOOL_CALLS_KEY, 1);
const toolCallTimeout = loadInt(TOOL_CALL_TIMEOUT_KEY, 1);
const allCustomPresets = sanitizeCustomPresets([
...(customPresets ?? []),
...legacyPromptPresets,
]);
if (inferenceParams) settings.inferenceParams = inferenceParams;
if (hasLegacyPresetStorage && allCustomPresets !== undefined) {
settings.customPresets = allCustomPresets;
}
if (activePreset?.trim()) settings.activePreset = activePreset.trim();
if (activePresetSource) settings.activePresetSource = activePresetSource;
if (autoTitle !== undefined) settings.autoTitle = autoTitle;
if (reasoningEffort) settings.reasoningEffort = reasoningEffort;
if (preserveThinking !== undefined)
settings.preserveThinking = preserveThinking;
if (autoHealToolCalls !== undefined) {
settings.autoHealToolCalls = autoHealToolCalls;
}
if (maxToolCallsPerMessage !== undefined) {
settings.maxToolCallsPerMessage = maxToolCallsPerMessage;
}
if (toolCallTimeout !== undefined) settings.toolCallTimeout = toolCallTimeout;
return settings;
}
export async function loadChatSettingsWithLegacyImport(): Promise<PersistedChatSettings> {
let dbSettings: PersistedChatSettings;
try {
dbSettings = sanitizeChatSettings(await getChatSettings());
} catch (error) {
const legacySettings = loadLegacyChatSettings();
if (isEmptyChatSettings(legacySettings)) {
throw error;
}
return legacySettings;
}
const legacySettings = loadLegacyChatSettings();
if (isLegacySettingsImportDone()) {
if (
!isEmptyChatSettings(dbSettings) ||
isEmptyChatSettings(legacySettings)
) {
return dbSettings;
}
try {
return sanitizeChatSettings(await saveChatSettingsPatch(legacySettings));
} catch {
return legacySettings;
}
}
if (isEmptyChatSettings(legacySettings)) {
markLegacySettingsImportDone();
return dbSettings;
}
const mergedSettings = {
...legacySettings,
...dbSettings,
inferenceParams: {
...legacySettings.inferenceParams,
...dbSettings.inferenceParams,
},
};
try {
const savedSettings = sanitizeChatSettings(
await saveChatSettingsPatch(mergedSettings),
);
markLegacySettingsImportDone();
return savedSettings;
} catch {
return mergedSettings;
}
}
export async function savePersistedChatSettingsPatch(
patch: PersistedChatSettings,
options: { keepalive?: boolean } = {},
): Promise<PersistedChatSettings> {
return sanitizeChatSettings(
await saveChatSettingsPatch(sanitizeChatSettings(patch), options),
);
}