* Studio: add Anthropic fast_mode toggle + surface streaming refusals Fast mode (beta `fast-mode-2026-02-01`) lets Claude Opus 4.6 and 4.7 generate output tokens up to 2.5x faster at 6x standard Opus pricing. The toggle lives in Configuration → Provider when the selected Anthropic model is Opus 4.6 or 4.7 and is otherwise hidden. Backend gates the same prefixes a second time so a stale frontend cannot make Anthropic 400 the request, and the `fast-mode-2026-02-01` beta header is merged onto whatever other betas the request already needed (code-execution, compaction). Streaming refusals (`message_delta.delta.stop_reason="refusal"` on Claude 4 models) now surface a short user-facing notice in the assistant message before the translated OpenAI chunk emits the existing `finish_reason="content_filter"`. Previously the chat bubble truncated silently because the SSE stopped mid-stream with no visible explanation. Per the upstream docs the conversation must be reset before continuing, so the notice tells the user exactly that. Reference: - https://platform.claude.com/docs/en/build-with-claude/fast-mode - https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/handle-streaming-refusals Tests: - studio/backend/tests/test_anthropic_fast_mode_and_refusal.py (8 cases pinning fast_mode pass-through on 4.6/4.7, silent drop on Sonnet / Haiku / older Opus / None / False, and the refusal notice + finish reason on a synthetic refusal stream). * Studio: drop refused Anthropic turns from the next request Anthropic's streaming-refusal guidance says the refused assistant turn must be removed or updated before the next call -- otherwise the safety classifier keeps refusing. The PR only added a user-visible notice; the partial assistant output (plus the notice itself) still rode the next request via toOpenAIMessage. Tag the refusal turn with an HTML-comment sentinel emitted alongside the notice. The chat-adapter checks for that sentinel in toOpenAIMessage and returns null, so the refused turn is excluded from outboundMessages. The notice still renders in the transcript (HTML comments don't display), so users keep the explanation. * Studio: filter None finish_reason entries in test helper test_refusal_maps_to_content_filter expects only ['content_filter'] in the finish_reasons list, but the post-PR refusal path emits a user-visible content notice chunk first. Every _content_chunk carries 'finish_reason: None' by construction; the helper was appending those, so the assertion saw [None, 'content_filter'] instead of ['content_filter']. None is not a finish reason -- it's just mid-stream delta noise. Skip None values in _finish_reasons so the helper reflects what the test names actually claim to check. Same fix applies cleanly to the other helper usages (pause_turn test expects [] and the sibling stop test expects ['stop'], both unaffected). * Studio: cover Anthropic fast-mode edge cases Adds 19 cases on top of the 9 in test_anthropic_fast_mode_and_refusal. The base file pins the happy path; this file fills in the cliffs: * Dated-snapshot prefix matching: claude-opus-4-7-2026-02-01 and claude-opus-4-6-2026-02-01 still gate fast_mode through, while claude-opus-4-5-2025-08-01 and claude-sonnet-4-6-2026-02-01 do not. * Strict opt-in: a future claude-opus-4-8 or claude-opus-5 does NOT auto-enable fast_mode -- the prefix tuple must be bumped explicitly when a new family is whitelisted upstream. * Beta-header merge: fast_mode coexists with code-execution-2025-08-25 and compact-2026-01-12 in one comma-separated anthropic-beta header with no duplicates and no truncation. Pins the value to the exact fast-mode-2026-02-01 docs token so a typo would fail CI. * Non-destruction: fast_mode=None produces byte-identical outbound body and headers to the version that omits the argument entirely. Same for fast_mode=False. Guarantees the upgrade path is non-breaking on existing Anthropic streams. * Refusal stream ordering: the user-visible notice precedes the finish_reason chunk so a streaming UI paints text before flipping to content_filter. Refusal sentinel emitted exactly once. Notice rides a normal content delta chunk with finish_reason still null. Partial assistant deltas survive before the notice. * Provider-side refusal coverage: a refusal on Sonnet (not just Opus) still emits the notice + sentinel + content_filter mapping, since refusal handling is not gated on fast-mode capability. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Persist fastMode, drop refused user message on retry Two follow-ups on #5715: 1) sanitizeInferenceParams stripped fastMode. fastMode is in PERSISTED_INFERENCE_PARAM_KEYS but the storage sanitizer only kept numeric fields plus systemPrompt and trustRemoteCode, so the new toggle was silently dropped on reload and on the /api/chat/settings round-trip. Save it the same way trustRemoteCode is saved. 2) Refusal recovery now also drops the triggering user turn. Returning null from toOpenAIMessage on the assistant side left the user prompt that caused the refusal in the outbound history, so the very next request would re-trigger the same classifier. Anthropic's refusal-handling guidance is explicit on this: remove the refused turn AND the user message that triggered it before the next call. Implemented via a pre-pass that pops the trailing user message when an assistant carries the refusal sentinel. Typecheck clean. * Studio: out-of-band refusal signal + fast-mode prefix/usage/pricing fixes The text sentinel for the Anthropic refusal drop signal was spoofable: any assistant message containing the literal <!--studio:anthropic-refusal--> would prune the prior user + assistant pair on the next request. Move the signal onto a separate _toolEvent chunk that the chat adapter latches into assistant.metadata.custom.anthropicRefusal; assistant text can no longer control the pruner. Tighten the fast-mode model gate (backend + frontend) to require a "-" family boundary so claude-opus-4-70 / claude-opus-4-7b style IDs do not get speed: "fast" on a naive startswith match. Use survivingMessages for the image / audio attachment scan so a refused user turn does not gate or mis-attribute the next non-refused turn. Propagate Anthropic usage.speed onto the OpenAI-style usage chunk and apply the documented 6x fast-mode multiplier in the cost calculator (stacks with prompt-cache multipliers per the docs); expose the new multiplier on the pricing snapshot for the UI tooltip. Tests cover the tool-event chunk shape, the prefix-collision rejects, usage.speed propagation, the 6x pricing math, and that the visible refusal text carries no embedded sentinel. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Shorten fast-mode and refusal comments for PR #5715 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
408 lines
13 KiB
TypeScript
408 lines
13 KiB
TypeScript
// SPDX-License-Identifier: AGPL-3.0-only
|
|
// Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
import {
|
|
getChatSettings,
|
|
saveChatSettingsPatch,
|
|
type PersistedChatPreset,
|
|
type PersistedChatSettings,
|
|
type PersistedInferenceParams,
|
|
} from "../api/chat-settings-api";
|
|
import {
|
|
BUILTIN_PRESETS,
|
|
defaultInferenceParams,
|
|
getPresetOwnedConfigKey,
|
|
getUniquePresetName,
|
|
normalizeCustomPresets,
|
|
type ChatPresetSource,
|
|
type Preset,
|
|
} from "../presets/preset-policy";
|
|
import type { ReasoningEffort } from "../stores/chat-runtime-store";
|
|
|
|
const AUTO_TITLE_KEY = "unsloth_chat_auto_title";
|
|
const AUTO_HEAL_TOOL_CALLS_KEY = "unsloth_auto_heal_tool_calls";
|
|
const MAX_TOOL_CALLS_KEY = "unsloth_max_tool_calls_per_message";
|
|
const TOOL_CALL_TIMEOUT_KEY = "unsloth_tool_call_timeout";
|
|
const INFERENCE_PARAMS_KEY = "unsloth_chat_inference_params";
|
|
const CHAT_ACTIVE_PRESET_KEY = "unsloth_chat_active_preset";
|
|
const CHAT_ACTIVE_PRESET_SOURCE_KEY = "unsloth_chat_active_preset_source";
|
|
const REASONING_EFFORT_KEY = "unsloth_reasoning_effort";
|
|
const PRESERVE_THINKING_KEY = "unsloth_preserve_thinking";
|
|
const CHAT_PRESETS_KEY = "unsloth_chat_custom_presets";
|
|
const LEGACY_CHAT_SYSTEM_PROMPTS_KEY = "unsloth_chat_system_prompts";
|
|
const LEGACY_CHAT_SETTINGS_IMPORT_KEY =
|
|
"unsloth_chat_settings_imported_to_studio_db";
|
|
|
|
const NUMERIC_INFERENCE_FIELDS = [
|
|
"temperature",
|
|
"topP",
|
|
"topK",
|
|
"minP",
|
|
"repetitionPenalty",
|
|
"presencePenalty",
|
|
"maxSeqLength",
|
|
"maxTokens",
|
|
] as const satisfies readonly (keyof PersistedInferenceParams)[];
|
|
|
|
const CHAT_PRESET_SOURCES = new Set<string>([
|
|
"builtin-default",
|
|
"custom",
|
|
"modified",
|
|
]);
|
|
|
|
const REASONING_EFFORTS = new Set<string>([
|
|
"none",
|
|
"minimal",
|
|
"low",
|
|
"medium",
|
|
"high",
|
|
"max",
|
|
"xhigh",
|
|
]);
|
|
|
|
interface LegacySystemPromptTemplate {
|
|
name: string;
|
|
content: string;
|
|
}
|
|
|
|
function canUseStorage(): boolean {
|
|
return typeof window !== "undefined";
|
|
}
|
|
|
|
function isRecord(value: unknown): value is Record<string, unknown> {
|
|
return value != null && typeof value === "object" && !Array.isArray(value);
|
|
}
|
|
|
|
function hasKeys(value: object): boolean {
|
|
return Object.keys(value).length > 0;
|
|
}
|
|
|
|
function getStorageItem(key: string): string | null {
|
|
if (!canUseStorage()) return null;
|
|
try {
|
|
return localStorage.getItem(key);
|
|
} catch {
|
|
return null;
|
|
}
|
|
}
|
|
|
|
function isLegacySettingsImportDone(): boolean {
|
|
return getStorageItem(LEGACY_CHAT_SETTINGS_IMPORT_KEY) === "true";
|
|
}
|
|
|
|
function markLegacySettingsImportDone(): void {
|
|
if (!canUseStorage()) return;
|
|
try {
|
|
localStorage.setItem(LEGACY_CHAT_SETTINGS_IMPORT_KEY, "true");
|
|
} catch {
|
|
// ignore
|
|
}
|
|
}
|
|
|
|
function parseJson(value: string | null): unknown {
|
|
if (!value) return undefined;
|
|
try {
|
|
return JSON.parse(value) as unknown;
|
|
} catch {
|
|
return undefined;
|
|
}
|
|
}
|
|
|
|
function loadBool(key: string): boolean | undefined {
|
|
const raw = getStorageItem(key);
|
|
if (raw === "true") return true;
|
|
if (raw === "false") return false;
|
|
return undefined;
|
|
}
|
|
|
|
function loadInt(key: string, min: number): number | undefined {
|
|
const raw = getStorageItem(key);
|
|
if (raw == null || raw.trim() === "") return undefined;
|
|
const value = Number(raw);
|
|
return Number.isInteger(value) && value >= min ? value : undefined;
|
|
}
|
|
|
|
function sanitizeInferenceParams(
|
|
value: unknown,
|
|
): PersistedInferenceParams | undefined {
|
|
if (!isRecord(value)) return undefined;
|
|
|
|
const params: PersistedInferenceParams = {};
|
|
for (const field of NUMERIC_INFERENCE_FIELDS) {
|
|
const fieldValue = value[field];
|
|
if (typeof fieldValue === "number" && Number.isFinite(fieldValue)) {
|
|
params[field] = fieldValue;
|
|
}
|
|
}
|
|
if (typeof value.systemPrompt === "string") {
|
|
params.systemPrompt = value.systemPrompt;
|
|
}
|
|
if (typeof value.trustRemoteCode === "boolean") {
|
|
params.trustRemoteCode = value.trustRemoteCode;
|
|
}
|
|
// Mirror trustRemoteCode handling so the toggle survives reload
|
|
// and the /api/chat/settings round-trip.
|
|
if (typeof value.fastMode === "boolean") {
|
|
params.fastMode = value.fastMode;
|
|
}
|
|
return hasKeys(params) ? params : undefined;
|
|
}
|
|
|
|
function toFullPreset(preset: PersistedChatPreset): Preset {
|
|
return {
|
|
name: preset.name,
|
|
params: {
|
|
...defaultInferenceParams,
|
|
...preset.params,
|
|
checkpoint: defaultInferenceParams.checkpoint,
|
|
},
|
|
};
|
|
}
|
|
|
|
function sanitizeCustomPresets(
|
|
value: unknown,
|
|
): PersistedChatPreset[] | undefined {
|
|
if (!Array.isArray(value)) return undefined;
|
|
if (value.length === 0) return [];
|
|
|
|
const presets = value
|
|
.map((item): PersistedChatPreset | null => {
|
|
if (!isRecord(item) || typeof item.name !== "string") return null;
|
|
const name = item.name.trim();
|
|
if (!name) return null;
|
|
const params = sanitizeInferenceParams(item.params);
|
|
return { name, params: params ?? {} };
|
|
})
|
|
.filter((preset): preset is PersistedChatPreset => preset !== null);
|
|
|
|
if (presets.length === 0) return [];
|
|
return normalizeCustomPresets(presets.map(toFullPreset)).map(
|
|
(preset, index) => ({
|
|
name: preset.name,
|
|
params: presets[index]?.params ?? {},
|
|
}),
|
|
);
|
|
}
|
|
|
|
function sanitizePresetSource(value: unknown): ChatPresetSource | undefined {
|
|
return typeof value === "string" && CHAT_PRESET_SOURCES.has(value)
|
|
? (value as ChatPresetSource)
|
|
: undefined;
|
|
}
|
|
|
|
function sanitizeReasoningEffort(value: unknown): ReasoningEffort | undefined {
|
|
return typeof value === "string" && REASONING_EFFORTS.has(value)
|
|
? (value as ReasoningEffort)
|
|
: undefined;
|
|
}
|
|
|
|
function sanitizeBool(value: unknown): boolean | undefined {
|
|
return typeof value === "boolean" ? value : undefined;
|
|
}
|
|
|
|
function sanitizeInt(value: unknown, min: number): number | undefined {
|
|
return typeof value === "number" && Number.isInteger(value) && value >= min
|
|
? value
|
|
: undefined;
|
|
}
|
|
|
|
function sanitizeChatSettings(value: unknown): PersistedChatSettings {
|
|
if (!isRecord(value)) return {};
|
|
|
|
const settings: PersistedChatSettings = {};
|
|
const inferenceParams = sanitizeInferenceParams(value.inferenceParams);
|
|
const customPresets = sanitizeCustomPresets(value.customPresets);
|
|
const activePresetSource = sanitizePresetSource(value.activePresetSource);
|
|
const reasoningEffort = sanitizeReasoningEffort(value.reasoningEffort);
|
|
const autoTitle = sanitizeBool(value.autoTitle);
|
|
const preserveThinking = sanitizeBool(value.preserveThinking);
|
|
const autoHealToolCalls = sanitizeBool(value.autoHealToolCalls);
|
|
const maxToolCallsPerMessage = sanitizeInt(value.maxToolCallsPerMessage, 1);
|
|
const toolCallTimeout = sanitizeInt(value.toolCallTimeout, 1);
|
|
|
|
if (inferenceParams) settings.inferenceParams = inferenceParams;
|
|
if (customPresets !== undefined) settings.customPresets = customPresets;
|
|
if (typeof value.activePreset === "string" && value.activePreset.trim()) {
|
|
settings.activePreset = value.activePreset.trim();
|
|
}
|
|
if (activePresetSource) settings.activePresetSource = activePresetSource;
|
|
if (autoTitle !== undefined) settings.autoTitle = autoTitle;
|
|
if (reasoningEffort) settings.reasoningEffort = reasoningEffort;
|
|
if (preserveThinking !== undefined)
|
|
settings.preserveThinking = preserveThinking;
|
|
if (autoHealToolCalls !== undefined) {
|
|
settings.autoHealToolCalls = autoHealToolCalls;
|
|
}
|
|
if (maxToolCallsPerMessage !== undefined) {
|
|
settings.maxToolCallsPerMessage = maxToolCallsPerMessage;
|
|
}
|
|
if (toolCallTimeout !== undefined) settings.toolCallTimeout = toolCallTimeout;
|
|
|
|
return settings;
|
|
}
|
|
|
|
function loadLegacySystemPromptPresets(
|
|
existingPresets: PersistedChatPreset[],
|
|
): PersistedChatPreset[] {
|
|
const parsed = parseJson(getStorageItem(LEGACY_CHAT_SYSTEM_PROMPTS_KEY));
|
|
if (!Array.isArray(parsed)) return [];
|
|
|
|
const usedNames = new Set([
|
|
...BUILTIN_PRESETS.map((preset) => preset.name),
|
|
...existingPresets.map((preset) => preset.name),
|
|
]);
|
|
const seenConfigKeys = new Set(
|
|
[...BUILTIN_PRESETS, ...existingPresets.map(toFullPreset)].map((preset) =>
|
|
getPresetOwnedConfigKey(preset.params),
|
|
),
|
|
);
|
|
|
|
return parsed
|
|
.filter((item): item is LegacySystemPromptTemplate => {
|
|
if (!isRecord(item)) return false;
|
|
return typeof item.name === "string" && typeof item.content === "string";
|
|
})
|
|
.map((template) => ({
|
|
template,
|
|
params: {
|
|
...defaultInferenceParams,
|
|
systemPrompt: template.content,
|
|
},
|
|
}))
|
|
.filter(({ params }) => {
|
|
const configKey = getPresetOwnedConfigKey(params);
|
|
if (seenConfigKeys.has(configKey)) return false;
|
|
seenConfigKeys.add(configKey);
|
|
return true;
|
|
})
|
|
.map(({ template, params }) => ({
|
|
name: getUniquePresetName(`${template.name} Prompt`, usedNames),
|
|
params: sanitizeInferenceParams(params) ?? {},
|
|
}));
|
|
}
|
|
|
|
export function isEmptyChatSettings(settings: PersistedChatSettings): boolean {
|
|
return (
|
|
(!settings.inferenceParams || !hasKeys(settings.inferenceParams)) &&
|
|
settings.customPresets === undefined &&
|
|
settings.activePreset === undefined &&
|
|
settings.activePresetSource === undefined &&
|
|
settings.autoTitle === undefined &&
|
|
settings.reasoningEffort === undefined &&
|
|
settings.preserveThinking === undefined &&
|
|
settings.autoHealToolCalls === undefined &&
|
|
settings.maxToolCallsPerMessage === undefined &&
|
|
settings.toolCallTimeout === undefined
|
|
);
|
|
}
|
|
|
|
export function loadLegacyChatSettings(): PersistedChatSettings {
|
|
const settings: PersistedChatSettings = {};
|
|
const rawCustomPresets = getStorageItem(CHAT_PRESETS_KEY);
|
|
const rawLegacyPromptPresets = getStorageItem(LEGACY_CHAT_SYSTEM_PROMPTS_KEY);
|
|
const hasLegacyPresetStorage =
|
|
rawCustomPresets !== null || rawLegacyPromptPresets !== null;
|
|
const inferenceParams = sanitizeInferenceParams(
|
|
parseJson(getStorageItem(INFERENCE_PARAMS_KEY)),
|
|
);
|
|
const customPresets = sanitizeCustomPresets(parseJson(rawCustomPresets));
|
|
const legacyPromptPresets = loadLegacySystemPromptPresets(
|
|
customPresets ?? [],
|
|
);
|
|
const activePreset = getStorageItem(CHAT_ACTIVE_PRESET_KEY);
|
|
const activePresetSource = sanitizePresetSource(
|
|
getStorageItem(CHAT_ACTIVE_PRESET_SOURCE_KEY),
|
|
);
|
|
const reasoningEffort = sanitizeReasoningEffort(
|
|
getStorageItem(REASONING_EFFORT_KEY),
|
|
);
|
|
const autoTitle = loadBool(AUTO_TITLE_KEY);
|
|
const preserveThinking = loadBool(PRESERVE_THINKING_KEY);
|
|
const autoHealToolCalls = loadBool(AUTO_HEAL_TOOL_CALLS_KEY);
|
|
const maxToolCallsPerMessage = loadInt(MAX_TOOL_CALLS_KEY, 1);
|
|
const toolCallTimeout = loadInt(TOOL_CALL_TIMEOUT_KEY, 1);
|
|
const allCustomPresets = sanitizeCustomPresets([
|
|
...(customPresets ?? []),
|
|
...legacyPromptPresets,
|
|
]);
|
|
|
|
if (inferenceParams) settings.inferenceParams = inferenceParams;
|
|
if (hasLegacyPresetStorage && allCustomPresets !== undefined) {
|
|
settings.customPresets = allCustomPresets;
|
|
}
|
|
if (activePreset?.trim()) settings.activePreset = activePreset.trim();
|
|
if (activePresetSource) settings.activePresetSource = activePresetSource;
|
|
if (autoTitle !== undefined) settings.autoTitle = autoTitle;
|
|
if (reasoningEffort) settings.reasoningEffort = reasoningEffort;
|
|
if (preserveThinking !== undefined)
|
|
settings.preserveThinking = preserveThinking;
|
|
if (autoHealToolCalls !== undefined) {
|
|
settings.autoHealToolCalls = autoHealToolCalls;
|
|
}
|
|
if (maxToolCallsPerMessage !== undefined) {
|
|
settings.maxToolCallsPerMessage = maxToolCallsPerMessage;
|
|
}
|
|
if (toolCallTimeout !== undefined) settings.toolCallTimeout = toolCallTimeout;
|
|
|
|
return settings;
|
|
}
|
|
|
|
export async function loadChatSettingsWithLegacyImport(): Promise<PersistedChatSettings> {
|
|
let dbSettings: PersistedChatSettings;
|
|
try {
|
|
dbSettings = sanitizeChatSettings(await getChatSettings());
|
|
} catch (error) {
|
|
const legacySettings = loadLegacyChatSettings();
|
|
if (isEmptyChatSettings(legacySettings)) {
|
|
throw error;
|
|
}
|
|
return legacySettings;
|
|
}
|
|
|
|
const legacySettings = loadLegacyChatSettings();
|
|
if (isLegacySettingsImportDone()) {
|
|
if (
|
|
!isEmptyChatSettings(dbSettings) ||
|
|
isEmptyChatSettings(legacySettings)
|
|
) {
|
|
return dbSettings;
|
|
}
|
|
try {
|
|
return sanitizeChatSettings(await saveChatSettingsPatch(legacySettings));
|
|
} catch {
|
|
return legacySettings;
|
|
}
|
|
}
|
|
|
|
if (isEmptyChatSettings(legacySettings)) {
|
|
markLegacySettingsImportDone();
|
|
return dbSettings;
|
|
}
|
|
|
|
const mergedSettings = {
|
|
...legacySettings,
|
|
...dbSettings,
|
|
inferenceParams: {
|
|
...legacySettings.inferenceParams,
|
|
...dbSettings.inferenceParams,
|
|
},
|
|
};
|
|
try {
|
|
const savedSettings = sanitizeChatSettings(
|
|
await saveChatSettingsPatch(mergedSettings),
|
|
);
|
|
markLegacySettingsImportDone();
|
|
return savedSettings;
|
|
} catch {
|
|
return mergedSettings;
|
|
}
|
|
}
|
|
|
|
export async function savePersistedChatSettingsPatch(
|
|
patch: PersistedChatSettings,
|
|
options: { keepalive?: boolean } = {},
|
|
): Promise<PersistedChatSettings> {
|
|
return sanitizeChatSettings(
|
|
await saveChatSettingsPatch(sanitizeChatSettings(patch), options),
|
|
);
|
|
}
|