ceos/opencode
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(core): add embedded v2 session runtime and tool foundation (#30632)

Browse source
This commit is contained in:
Kit Langton 2026-06-03 23:02:17 -04:00 • committed by GitHub
commit 76ee87ead8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
215 changed files with 31398 additions and 3332 deletions
Download patch file Download diff file Expand all files Collapse all files

55
packages/llm/AGENTS.md
View file

@ -10,7 +10,7 @@
## Conventions
Per-type constructors live on the type, not as top-level re-exports. Use `Message.user(...)`, `Message.assistant(...)`, `Message.tool(...)`, `Model.make(...)`, `ToolDefinition.make(...)`, `ToolCallPart.make(...)`, `ToolResultPart.make(...)`, `ToolChoice.make(...)`, `ToolChoice.named(...)`, `SystemPart.make(...)`, and `GenerationOptions.make(...)` directly. The top-level `LLM` namespace is reserved for request-shaped call APIs: `LLM.request`, `LLM.generate`, `LLM.stream`, `LLM.updateRequest`, and `LLM.generateObject`. Two ways to construct the same thing is one too many.
Per-type constructors live on the type, not as top-level re-exports. Use `Message.system(...)`, `Message.user(...)`, `Message.assistant(...)`, `Message.tool(...)`, `Model.make(...)`, `ToolDefinition.make(...)`, `ToolCallPart.make(...)`, `ToolResultPart.make(...)`, `ToolChoice.make(...)`, `ToolChoice.named(...)`, `SystemPart.make(...)`, and `GenerationOptions.make(...)` directly. The top-level `LLM` namespace is reserved for request-shaped call APIs: `LLM.request`, `LLM.generate`, `LLM.stream`, `LLM.updateRequest`, and `LLM.generateObject`. Two ways to construct the same thing is one too many.
## Tests
@ -25,7 +25,7 @@ Primary in-repo integration point:
- `packages/opencode/src/session/llm.ts` is the session-owned orchestration layer that decides whether a request uses AI SDK or this package's native route runtime.
- `packages/opencode/src/session/llm/native-request.ts` is the lowering adapter from opencode's session/AI SDK-shaped data into this package's `LLMRequest` model.
- `packages/opencode/src/session/llm/native-runtime.ts` is the execution adapter that calls `LLMClient.stream(...)` and bridges opencode tools into this package's tool runtime.
- `packages/opencode/src/session/llm/native-runtime.ts` is the execution adapter that calls raw `LLMClient.stream(request)` and bridges one provider turn of opencode tool calls through this package's typed dispatcher.
- `packages/opencode/src/session/llm/ai-sdk.ts` keeps the default AI SDK path compatible by converting AI SDK stream parts into this package's shared `LLMEvent`s.
Keep this package independent of session concerns. Session auth, permissions, plugins, telemetry headers, and runtime selection belong in `packages/opencode/src/session/llm.ts` and its local adapters.
@ -153,7 +153,7 @@ packages/llm/src/
openai-compatible-profile.ts family defaults (deepseek, togetherai, ...)
azure.ts / amazon-bedrock.ts / cloudflare.ts / github-copilot.ts / google.ts / xai.ts / openai.ts / anthropic.ts / openrouter.ts
tool.ts typed tool() helper
tool-runtime.ts implementation helpers for LLMClient tool execution
tool-runtime.ts narrow one-call typed tool dispatcher
```
The dependency arrow points down: `providers/*.ts` files import protocol routes and auth-option utilities; protocol modules import `endpoint`, `auth`, `framing`, and transport pieces. Protocols do not import provider facades. Lower-level modules know nothing about provider catalog metadata.
@ -171,6 +171,20 @@ The dependency arrow points down: `providers/*.ts` files import protocol routes
If you find yourself copying a 3-to-5-line snippet between two protocols, lift it into `ProviderShared` next to these helpers rather than duplicating.
### Chronological System Updates
`LLMRequest.system` is the initial privileged prompt that applies ahead of the conversation. `Message.system(...)` is a separate, provider-neutral chronological operator update inside `LLMRequest.messages`; it applies only from its position in history onward and accepts text content only.
Native chronological system messages are route/model-specific. Anthropic Messages lowers them natively for Claude Opus 4.8 (`claude-opus-4-8`). Other routes and models intentionally lower the update in place into ordinary user-compatible text using this stable escaped representation:
```text
<system-update>
...
</system-update>
```
The wrapped-user fallback preserves ordering while visibly lowering authority. Never silently pass a raw chronological `role: "system"` through a route that might reject it. Do not insert raw retrieved documents, tool output, or web content into privileged chronological system updates; keep untrusted content in ordinary user/tool channels.
### Tools
Tool loops are represented in common messages and events:
@ -187,9 +201,9 @@ const followUp = LLM.request({
Routes lower these into provider-native assistant tool-call messages and tool-result messages. Streaming providers should emit `tool-input-delta` events while arguments arrive, then a final `tool-call` event with parsed input.
### Tool runtime
### Tool dispatch
`LLM.stream({ request, tools })` executes model-requested tools with full type safety. Plain `LLM.stream(request)` only streams the model; if `request.tools` contains schemas, tool calls are returned for the caller to handle. Use `toolExecution: "none"` to pass executable tool definitions as schemas without invoking handlers. Add `stopWhen` to opt into follow-up model rounds after tool results.
`LLM.stream(request)` and `LLM.generate(request)` each run exactly one provider turn. Add tool schemas to `request.tools` with `Tool.toDefinitions(tools)`. When a caller wants the package's typed one-call execution behavior, pass each canonical local `tool-call` event to `ToolRuntime.dispatch(tools, call)`.
```ts
const get_weather = tool({
@ -205,22 +219,25 @@ const get_weather = tool({
}),
})
const events = yield* LLM.stream({
request,
tools: { get_weather, get_time, ... },
stopWhen: LLM.stepCountIs(10),
}).pipe(Stream.runCollect)
const tools = { get_weather, get_time, ... }
const events = yield* LLM.stream(
LLM.updateRequest(request, { tools: Tool.toDefinitions(tools) }),
).pipe(Stream.runCollect)
const call = Array.from(events).find(LLMEvent.is.toolCall)
if (call && !call.providerExecuted) {
const dispatched = yield* ToolRuntime.dispatch(tools, call)
// Persist call + dispatched.result, then construct the next request explicitly.
}
```
The runtime:
The dispatcher:
- Adds tool definitions (derived from each tool's `parameters` Schema via `Schema.toJsonSchemaDocument`) onto `request.tools`.
- Streams the model.
- On `tool-call`: looks up the named tool, decodes input against `parameters` Schema, dispatches to the typed `execute`, encodes the result against `success` Schema, emits `tool-result`.
- Emits local `tool-result` events in the same step by default.
- Loops only when `stopWhen` is provided and the step finishes with `tool-calls`, appending the assistant + tool messages.
- On `tool-call`: looks up the named tool, decodes input against `parameters` Schema, dispatches to the typed `execute`, encodes the result against `success` Schema, and returns canonical `tool-result` events.
- Does not stream providers, construct Session events, schedule fibers, append history, count steps, or continue model rounds.
- Leaves persistence and continuation to the enclosing product flow.
Handler dependencies (services, permissions, plugin hooks, abort handling) are closed over by the consumer at tool-construction time. The runtime's only environment requirement is `RequestExecutor.Service`. Build the tools record inside an `Effect.gen` once and reuse it across many runs.
Handler dependencies (services, permissions, plugin hooks, abort handling) are closed over by the consumer at tool-construction time. Build the tools record inside an `Effect.gen` once and reuse it across many dispatches.
Errors must be expressed as `ToolFailure`. The runtime catches it and emits a `tool-error` event, then a `tool-result` of `type: "error"`, so the model can self-correct on the next step. Anything that is not a `ToolFailure` is treated as a defect and fails the stream. Three recoverable error paths produce `tool-error` events:
@ -231,8 +248,8 @@ Errors must be expressed as `ToolFailure`. The runtime catches it and emits a `t
Provider-defined / hosted tools (Anthropic `web_search` / `code_execution` / `web_fetch`, OpenAI Responses `web_search_call` / `file_search_call` / `code_interpreter_call` / `mcp_call` / `local_shell_call` / `image_generation_call` / `computer_use_call`) pass through the runtime untouched:
- Routes surface the model's call as a `tool-call` event with `providerExecuted: true`, and the provider's result as a matching `tool-result` event with `providerExecuted: true`.
- The runtime detects `providerExecuted` on `tool-call` and **skips client dispatch** — no handler is invoked and no `tool-error` is raised for "unknown tool". The provider already executed it.
- Both events are appended to the assistant message in `assistantContent` so the next round's history carries the call + result for context. Anthropic encodes them back as `server_tool_use` + `web_search_tool_result` (or `code_execution_tool_result` / `web_fetch_tool_result`) blocks; OpenAI Responses callers typically use `previous_response_id` instead of resending hosted-tool items.
- Callers detect `providerExecuted` on `tool-call` and **skip local dispatch** — no handler is invoked and no `tool-error` is raised for "unknown tool". The provider already executed it.
- Callers that continue should retain both events in explicit history when the protocol requires it. Anthropic encodes them back as `server_tool_use` + `web_search_tool_result` (or `code_execution_tool_result` / `web_fetch_tool_result`) blocks; OpenAI Responses callers typically use `previous_response_id` instead of resending hosted-tool items.
Add provider-defined tools to `request.tools` (no runtime entry needed). The matching route must know how to lower the tool definition into the provider-native shape; right now Anthropic accepts `web_search` / `code_execution` / `web_fetch` and OpenAI Responses accepts the hosted tool names listed above.

39
packages/llm/example/tutorial.ts
View file

@ -1,5 +1,5 @@
import { Config, Effect, Formatter, Layer, Schema, Stream } from "effect"
import { LLM, LLMClient, ProviderID, Tool } from "@opencode-ai/llm"
import { LLM, LLMClient, Message, ProviderID, Tool, ToolRuntime } from "@opencode-ai/llm"
import { Route, Auth, Endpoint, Framing, Protocol, RequestExecutor, WebSocketExecutor } from "@opencode-ai/llm/route"
import { OpenAI } from "@opencode-ai/llm/providers"
@ -84,9 +84,9 @@ const streamText = LLM.stream(request).pipe(
Stream.runDrain,
)
// 5. Tools are typed with Effect Schema. Passing tools to `LLMClient.stream`
// adds their definitions to the request and dispatches matching tool calls.
// Add `stopWhen` to opt into follow-up model rounds after tool results.
// 5. Tools are typed with Effect Schema. Provider turns remain explicit:
// advertise definitions on the request, stream one turn, dispatch local calls,
// then persist/build follow-up history in the enclosing product flow.
const tools = {
get_weather: Tool.make({
description: "Get current weather for a city.",
@ -96,24 +96,29 @@ const tools = {
}),
}
const streamWithTools = LLM.stream({
request: LLM.request({
const streamWithTools = Effect.gen(function* () {
const request = LLM.request({
model,
prompt: "Use get_weather for San Francisco, then answer in one sentence.",
generation: { maxTokens: 80, temperature: 0 },
}),
tools,
stopWhen: LLM.stepCountIs(3),
}).pipe(
Stream.tap((event) =>
Effect.sync(() => {
tools: Tool.toDefinitions(tools),
})
const events = Array.from(yield* LLM.stream(request).pipe(Stream.runCollect))
for (const event of events) {
if (event.type === "tool-call") console.log("tool call", event.name, event.input)
if (event.type === "tool-result") console.log("tool result", event.name, event.result)
if (event.type === "text-delta") process.stdout.write(event.text)
}),
),
Stream.runDrain,
)
if (event.type !== "tool-call" || event.providerExecuted) continue
const dispatched = yield* ToolRuntime.dispatch(tools, event)
console.log("tool result", event.name, dispatched.result)
// A durable agent would persist these messages before starting another
// raw model turn. This tutorial keeps the boundary visible instead.
const followUp = LLM.updateRequest(request, {
messages: [...request.messages, Message.assistant([event]), Message.tool({ ...event, result: dispatched.result })],
})
console.log("follow-up history messages:", followUp.messages.length)
}
})
// 6. `generateObject` is the structured-output helper. It forces a synthetic
// tool call internally, so the same call site works across providers instead of

13
packages/llm/src/index.ts
View file

@ -8,7 +8,9 @@ export type {
Service as LLMClientService,
} from "./route/client"
export * from "./schema"
export { Tool, ToolFailure, toDefinitions, tool } from "./tool"
export { Tool, ToolFailure, toDefinitions } from "./tool"
export { ToolRuntime } from "./tool-runtime"
export type { DispatchResult as ToolDispatchResult, ToolSettlement } from "./tool-runtime"
export type {
AnyExecutableTool,
AnyTool,
@ -17,16 +19,11 @@ export type {
Tool as ToolShape,
ToolExecute,
ToolExecuteContext,
ToolModelOutputInput,
Tools,
ToolSchema,
ToolToModelOutput,
} from "./tool"
export type {
RunOptions as ToolRunOptions,
RuntimeState as ToolRuntimeState,
StopCondition as ToolStopCondition,
ToolExecution,
} from "./tool-runtime"
export * as LLM from "./llm"
export type {
Definition as ProviderDefinition,

11
packages/llm/src/llm.ts
View file

@ -16,7 +16,7 @@ import {
type ContentPart,
ToolResultPart,
} from "./schema"
import { make as makeTool, type ToolSchema } from "./tool"
import { make as makeTool, toDefinitions, type ToolSchema } from "./tool"
export type ModelInput = SchemaModelInput
@ -46,8 +46,6 @@ export const generate = LLMClient.generate
export const stream = LLMClient.stream
export const stepCountIs = LLMClient.stepCountIs
export const requestInput = (input: LLMRequest): RequestInput => ({
...LLMRequest.input(input),
})
@ -115,13 +113,10 @@ const runGenerateObject = Effect.fn("LLM.generateObject")(function* (
) {
const baseRequest = request(options)
const generateRequest = LLMRequest.update(baseRequest, {
tools: toDefinitions({ [GENERATE_OBJECT_TOOL_NAME]: tool }),
toolChoice: ToolChoice.named(GENERATE_OBJECT_TOOL_NAME),
})
const response = yield* LLMClient.generate({
request: generateRequest,
tools: { [GENERATE_OBJECT_TOOL_NAME]: tool },
toolExecution: "none",
})
const response = yield* LLMClient.generate(generateRequest)
const call = response.toolCalls.find(
(event) => LLMEvent.is.toolCall(event) && event.name === GENERATE_OBJECT_TOOL_NAME,
)

69
packages/llm/src/protocols/anthropic-messages.ts
View file

@ -128,6 +128,7 @@ type AnthropicToolResultBlock = Schema.Schema.Type<typeof AnthropicToolResultBlo
const AnthropicMessage = Schema.Union([
Schema.Struct({ role: Schema.Literal("user"), content: Schema.Array(AnthropicUserBlock) }),
Schema.Struct({ role: Schema.Literal("assistant"), content: Schema.Array(AnthropicAssistantBlock) }),
Schema.Struct({ role: Schema.Literal("system"), content: Schema.Array(AnthropicTextBlock) }),
]).pipe(Schema.toTaggedUnion("role"))
type AnthropicMessage = Schema.Schema.Type<typeof AnthropicMessage>
@ -340,13 +341,79 @@ const lowerToolResultContent = Effect.fn("AnthropicMessages.lowerToolResultConte
return yield* Effect.forEach(content, lowerToolResultContentItem)
})
// Mid-conversation system messages are a native Claude API feature only for
// Opus 4.8. Other Anthropic models intentionally use the same visible wrapped-
// user fallback as non-Anthropic routes rather than sending a role they reject.
const supportsNativeSystemUpdates = (request: LLMRequest) => String(request.model.id) === "claude-opus-4-8"
const endsInServerToolUse = (message: LLMRequest["messages"][number]) => {
const last = message.content.at(-1)
return message.role === "assistant" && last?.type === "tool-call" && last.providerExecuted === true
}
const endsInLocalToolUse = (message: LLMRequest["messages"][number]) => {
const last = message.content.at(-1)
return message.role === "assistant" && last?.type === "tool-call" && last.providerExecuted !== true
}
const validateNativeSystemUpdate = Effect.fn("AnthropicMessages.validateNativeSystemUpdate")(function* (
messages: LLMRequest["messages"],
index: number,
) {
const previous = messages[index - 1]
const next = messages[index + 1]
if (!previous)
return yield* invalid("Anthropic Messages chronological system updates cannot be the first message; use LLMRequest.system")
if (previous.role === "system")
return yield* invalid("Anthropic Messages chronological system updates cannot be consecutive")
if (endsInLocalToolUse(previous))
return yield* invalid("Anthropic Messages chronological system updates cannot appear between a local tool call and its tool result")
if (previous.role !== "user" && previous.role !== "tool" && !endsInServerToolUse(previous))
return yield* invalid(
"Anthropic Messages chronological system updates must follow a user message, tool result, or assistant server tool use",
)
if (next?.role === "system")
return yield* invalid("Anthropic Messages chronological system updates cannot be consecutive")
if (next && next.role !== "assistant")
return yield* invalid("Anthropic Messages chronological system updates must end the messages array or immediately precede an assistant message")
})
const lowerNativeSystemUpdate = Effect.fn("AnthropicMessages.lowerNativeSystemUpdate")(function* (
message: LLMRequest["messages"][number],
breakpoints: Cache.Breakpoints,
) {
const content = yield* ProviderShared.systemUpdateText("Anthropic Messages", message)
return {
role: "system" as const,
content: content.map((part) => ({
type: "text" as const,
text: part.text,
cache_control: cacheControl(breakpoints, part.cache),
})),
}
})
const lowerMessages = Effect.fn("AnthropicMessages.lowerMessages")(function* (
request: LLMRequest,
breakpoints: Cache.Breakpoints,
) {
const messages: AnthropicMessage[] = []
for (const message of request.messages) {
for (const [index, message] of request.messages.entries()) {
if (message.role === "system") {
if (supportsNativeSystemUpdates(request)) {
yield* validateNativeSystemUpdate(request.messages, index)
messages.push(yield* lowerNativeSystemUpdate(message, breakpoints))
continue
}
const part = yield* ProviderShared.wrappedSystemUpdate("Anthropic Messages", message)
const block = { type: "text" as const, text: part.text, cache_control: cacheControl(breakpoints, part.cache) }
const previous = messages.at(-1)
if (previous?.role === "user") messages[messages.length - 1] = { role: "user", content: [...previous.content, block] }
else messages.push({ role: "user", content: [block] })
continue
}
if (message.role === "user") {
const content: AnthropicUserBlock[] = []
for (const part of message.content) {

49
packages/llm/src/protocols/bedrock-converse.ts
View file

@ -8,6 +8,8 @@ import {
type CacheHint,
type FinishReason,
type LLMRequest,
type ProviderMetadata,
type ReasoningPart,
type ToolCallPart,
type ToolDefinition,
type ToolResultPart,
@ -237,6 +239,13 @@ const lowerToolChoice = (toolChoice: NonNullable<LLMRequest["toolChoice"]>) =>
tool: (name) => ({ tool: { name } }) as const,
})
const bedrockMetadata = (metadata: Record<string, unknown>): ProviderMetadata => ({ bedrock: metadata })
const reasoningSignature = (part: ReasoningPart) => {
const bedrock = part.providerMetadata?.bedrock
return part.encrypted ?? (ProviderShared.isRecord(bedrock) && typeof bedrock.signature === "string" ? bedrock.signature : undefined)
}
const lowerToolCall = (part: ToolCallPart): BedrockToolUseBlock => ({
toolUse: {
toolUseId: part.id,
@ -281,6 +290,15 @@ const lowerMessages = Effect.fn("BedrockConverse.lowerMessages")(function* (
const messages: BedrockMessage[] = []
for (const message of request.messages) {
if (message.role === "system") {
const part = yield* ProviderShared.wrappedSystemUpdate("Bedrock Converse", message)
const content = textWithCache(breakpoints, part.text, part.cache)
const previous = messages.at(-1)
if (previous?.role === "user") messages[messages.length - 1] = { role: "user", content: [...previous.content, ...content] }
else messages.push({ role: "user", content })
continue
}
if (message.role === "user") {
const content: BedrockUserBlock[] = []
for (const part of message.content) {
@ -315,7 +333,7 @@ const lowerMessages = Effect.fn("BedrockConverse.lowerMessages")(function* (
if (part.type === "reasoning") {
content.push({
reasoningContent: {
reasoningText: { text: part.text, signature: part.encrypted },
reasoningText: { text: part.text, signature: reasoningSignature(part) },
},
})
continue
@ -425,6 +443,7 @@ interface ParserState {
readonly pendingFinish: { readonly reason: FinishReason; readonly usage?: Usage } | undefined
readonly hasToolCalls: boolean
readonly lifecycle: Lifecycle.State
readonly reasoningSignatures: Readonly<Record<number, string>>
}
const step = (state: ParserState, event: BedrockEvent) =>
@ -468,17 +487,19 @@ const step = (state: ParserState, event: BedrockEvent) =>
] as const
}
if (event.contentBlockDelta?.delta?.reasoningContent?.text) {
if (event.contentBlockDelta?.delta?.reasoningContent) {
const index = event.contentBlockDelta.contentBlockIndex
const reasoning = event.contentBlockDelta.delta.reasoningContent
const events: LLMEvent[] = []
return [
{
...state,
lifecycle: Lifecycle.reasoningDelta(
state.lifecycle,
events,
`reasoning-${event.contentBlockDelta.contentBlockIndex}`,
event.contentBlockDelta.delta.reasoningContent.text,
),
lifecycle: reasoning.text
? Lifecycle.reasoningDelta(state.lifecycle, events, `reasoning-${index}`, reasoning.text)
: state.lifecycle,
reasoningSignatures: reasoning.signature
? { ...state.reasoningSignatures, [index]: reasoning.signature }
: state.reasoningSignatures,
},
events,
] as const
@ -501,15 +522,17 @@ const step = (state: ParserState, event: BedrockEvent) =>
}
if (event.contentBlockStop) {
const result = yield* ToolStream.finish(ADAPTER, state.tools, event.contentBlockStop.contentBlockIndex)
const index = event.contentBlockStop.contentBlockIndex
const result = yield* ToolStream.finish(ADAPTER, state.tools, index)
const events: LLMEvent[] = []
const resultEvents = result.events ?? []
const lifecycle = resultEvents.length
? Lifecycle.stepStart(state.lifecycle, events)
: Lifecycle.reasoningEnd(
Lifecycle.textEnd(state.lifecycle, events, `text-${event.contentBlockStop.contentBlockIndex}`),
Lifecycle.textEnd(state.lifecycle, events, `text-${index}`),
events,
`reasoning-${event.contentBlockStop.contentBlockIndex}`,
`reasoning-${index}`,
state.reasoningSignatures[index] ? bedrockMetadata({ signature: state.reasoningSignatures[index] }) : undefined,
)
events.push(...resultEvents)
return [
@ -518,6 +541,9 @@ const step = (state: ParserState, event: BedrockEvent) =>
hasToolCalls: resultEvents.some(LLMEvent.is.toolCall) ? true : state.hasToolCalls,
lifecycle,
tools: result.tools,
reasoningSignatures: Object.fromEntries(
Object.entries(state.reasoningSignatures).filter(([key]) => key !== String(index)),
),
},
events,
] as const
@ -591,6 +617,7 @@ export const protocol = Protocol.make({
pendingFinish: undefined,
hasToolCalls: false,
lifecycle: Lifecycle.initial(),
reasoningSignatures: {},
}),
step,
onHalt,

53
packages/llm/src/protocols/gemini.ts
View file

@ -10,6 +10,7 @@ import {
type FinishReason,
type LLMRequest,
type MediaPart,
type ProviderMetadata,
type TextPart,
type ToolCallPart,
type ToolDefinition,
@ -136,6 +137,7 @@ interface ParserState {
readonly nextToolCallId: number
readonly usage?: Usage
readonly lifecycle: Lifecycle.State
readonly reasoningSignature?: string
}
const mediaData = ProviderShared.mediaBytes
@ -181,14 +183,32 @@ const lowerToolConfig = (toolChoice: NonNullable<LLMRequest["toolChoice"]>) =>
const lowerUserPart = (part: TextPart | MediaPart) =>
part.type === "text" ? { text: part.text } : { inlineData: { mimeType: part.mediaType, data: mediaData(part) } }
const googleMetadata = (metadata: Record<string, unknown>): ProviderMetadata => ({ google: metadata })
const thoughtSignature = (providerMetadata: ProviderMetadata | undefined) => {
const google = providerMetadata?.google
return ProviderShared.isRecord(google) && typeof google.thoughtSignature === "string"
? google.thoughtSignature
: undefined
}
const lowerToolCall = (part: ToolCallPart) => ({
functionCall: { name: part.name, args: part.input },
thoughtSignature: thoughtSignature(part.providerMetadata),
})
const lowerMessages = Effect.fn("Gemini.lowerMessages")(function* (request: LLMRequest) {
const contents: GeminiContent[] = []
for (const message of request.messages) {
if (message.role === "system") {
const part = yield* ProviderShared.wrappedSystemUpdate("Gemini", message)
const previous = contents.at(-1)
if (previous?.role === "user") contents[contents.length - 1] = { role: "user", parts: [...previous.parts, { text: part.text }] }
else contents.push({ role: "user", parts: [{ text: part.text }] })
continue
}
if (message.role === "user") {
const parts: Array<Schema.Schema.Type<typeof GeminiContentPart>> = []
for (const part of message.content) {
@ -210,7 +230,7 @@ const lowerMessages = Effect.fn("Gemini.lowerMessages")(function* (request: LLMR
continue
}
if (part.type === "reasoning") {
parts.push({ text: part.text, thought: true })
parts.push({ text: part.text, thought: true, thoughtSignature: thoughtSignature(part.providerMetadata) })
continue
}
if (part.type === "tool-call") {
@ -326,7 +346,15 @@ const finish = (state: ParserState): ReadonlyArray<LLMEvent> =>
state.finishReason || state.usage
? (() => {
const events: LLMEvent[] = []
Lifecycle.finish(state.lifecycle, events, {
const lifecycle = state.reasoningSignature
? Lifecycle.reasoningEnd(
state.lifecycle,
events,
"reasoning-0",
googleMetadata({ thoughtSignature: state.reasoningSignature }),
)
: state.lifecycle
Lifecycle.finish(lifecycle, events, {
reason: mapFinishReason(state.finishReason, state.hasToolCalls),
usage: state.usage,
})
@ -350,11 +378,20 @@ const step = (state: ParserState, event: GeminiEvent) => {
let hasToolCalls = nextState.hasToolCalls
let lifecycle = nextState.lifecycle
let nextToolCallId = nextState.nextToolCallId
let reasoningSignature = nextState.reasoningSignature
for (const part of candidate.content.parts) {
if ("thoughtSignature" in part && part.thoughtSignature && "thought" in part && part.thought)
reasoningSignature = part.thoughtSignature
if ("text" in part && part.text.length > 0) {
lifecycle = part.thought
? Lifecycle.reasoningDelta(lifecycle, events, "reasoning-0", part.text)
? Lifecycle.reasoningDelta(
lifecycle,
events,
"reasoning-0",
part.text,
part.thoughtSignature ? googleMetadata({ thoughtSignature: part.thoughtSignature }) : undefined,
)
: Lifecycle.textDelta(lifecycle, events, "text-0", part.text)
continue
}
@ -363,7 +400,14 @@ const step = (state: ParserState, event: GeminiEvent) => {
const input = part.functionCall.args
const id = `tool_${nextToolCallId++}`
lifecycle = Lifecycle.stepStart(lifecycle, events)
events.push(LLMEvent.toolCall({ id, name: part.functionCall.name, input }))
events.push(
LLMEvent.toolCall({
id,
name: part.functionCall.name,
input,
providerMetadata: part.thoughtSignature ? googleMetadata({ thoughtSignature: part.thoughtSignature }) : undefined,
}),
)
hasToolCalls = true
}
}
@ -374,6 +418,7 @@ const step = (state: ParserState, event: GeminiEvent) => {
hasToolCalls,
lifecycle,
nextToolCallId,
reasoningSignature,
finishReason: candidate.finishReason ?? nextState.finishReason,
},
events,

29
packages/llm/src/protocols/openai-chat.ts
View file

@ -1,4 +1,4 @@
import { Array as Arr, Effect, Schema } from "effect"
import { Effect, Schema } from "effect"
import { Route } from "../route/client"
import { Auth } from "../route/auth"
import { Endpoint } from "../route/endpoint"
@ -9,6 +9,7 @@ import {
Usage,
type FinishReason,
type LLMRequest,
type ReasoningPart,
type TextPart,
type ToolCallPart,
type ToolDefinition,
@ -202,14 +203,19 @@ const lowerAssistantMessage = Effect.fn("OpenAIChat.lowerAssistantMessage")(func
message: OpenAIChatRequestMessage,
) {
const content: TextPart[] = []
const reasoning: ReasoningPart[] = []
const toolCalls: OpenAIChatAssistantToolCall[] = []
for (const part of message.content) {
if (!ProviderShared.supportsContent(part, ["text", "tool-call"]))
return yield* ProviderShared.unsupportedContent("OpenAI Chat", "assistant", ["text", "tool-call"])
if (!ProviderShared.supportsContent(part, ["text", "reasoning", "tool-call"]))
return yield* ProviderShared.unsupportedContent("OpenAI Chat", "assistant", ["text", "reasoning", "tool-call"])
if (part.type === "text") {
content.push(part)
continue
}
if (part.type === "reasoning") {
reasoning.push(part)
continue
}
if (part.type === "tool-call") {
toolCalls.push(lowerToolCall(part))
continue
@ -219,7 +225,8 @@ const lowerAssistantMessage = Effect.fn("OpenAIChat.lowerAssistantMessage")(func
role: "assistant" as const,
content: content.length === 0 ? null : ProviderShared.joinText(content),
tool_calls: toolCalls.length === 0 ? undefined : toolCalls,
reasoning_content: openAICompatibleReasoningContent(message.native?.openaiCompatible),
reasoning_content:
reasoning.length > 0 ? reasoning.map((part) => part.text).join("") : openAICompatibleReasoningContent(message.native?.openaiCompatible),
}
})
@ -242,7 +249,19 @@ const lowerMessage = Effect.fn("OpenAIChat.lowerMessage")(function* (message: Op
const lowerMessages = Effect.fn("OpenAIChat.lowerMessages")(function* (request: LLMRequest) {
const system: OpenAIChatMessage[] =
request.system.length === 0 ? [] : [{ role: "system", content: ProviderShared.joinText(request.system) }]
return [...system, ...Arr.flatten(yield* Effect.forEach(request.messages, lowerMessage))]
const messages = [...system]
for (const message of request.messages) {
if (message.role === "system") {
const part = yield* ProviderShared.wrappedSystemUpdate("OpenAI Chat", message)
const previous = messages.at(-1)
if (previous?.role === "user")
messages[messages.length - 1] = { role: "user", content: `${previous.content}\n${part.text}` }
else messages.push({ role: "user", content: part.text })
continue
}
messages.push(...(yield* lowerMessage(message)))
}
return messages
})
const lowerOptions = Effect.fn("OpenAIChat.lowerOptions")(function* (request: LLMRequest) {

26
packages/llm/src/protocols/openai-responses.ts
View file

@ -291,6 +291,13 @@ const lowerReasoning = (part: ReasoningPart): OpenAIResponsesReasoningInput | un
}
}
const hostedToolItemID = (part: ToolResultPart) => {
const openai = part.providerMetadata?.openai
return ProviderShared.isRecord(openai) && typeof openai.itemId === "string" && openai.itemId.length > 0
? openai.itemId
: undefined
}
const lowerUserContent = Effect.fn("OpenAIResponses.lowerUserContent")(function* (
part: LLMRequest["messages"][number]["content"][number],
) {
@ -332,6 +339,15 @@ const lowerMessages = Effect.fn("OpenAIResponses.lowerMessages")(function* (requ
const store = OpenAIOptions.store(request)
for (const message of request.messages) {
if (message.role === "system") {
const part = yield* ProviderShared.wrappedSystemUpdate("OpenAI Responses", message)
const previous = input.at(-1)
if (previous && "role" in previous && previous.role === "user")
input[input.length - 1] = { role: "user", content: [...previous.content, { type: "input_text", text: part.text }] }
else input.push({ role: "user", content: [{ type: "input_text", text: part.text }] })
continue
}
if (message.role === "user") {
input.push({ role: "user", content: yield* Effect.forEach(message.content, lowerUserContent) })
continue
@ -341,6 +357,7 @@ const lowerMessages = Effect.fn("OpenAIResponses.lowerMessages")(function* (requ
const content: TextPart[] = []
const reasoningItems: Record<string, OpenAIResponsesReasoningInput> = {}
const reasoningReferences = new Set<string>()
const hostedToolReferences = new Set<string>()
const flushText = () => {
if (content.length === 0) return
input.push({ role: "assistant", content: content.map((part) => ({ type: "output_text", text: part.text })) })
@ -373,13 +390,22 @@ const lowerMessages = Effect.fn("OpenAIResponses.lowerMessages")(function* (requ
}
if (part.type === "tool-call") {
flushText()
if (part.providerExecuted === true) continue
input.push(lowerToolCall(part))
continue
}
if (part.type === "tool-result" && part.providerExecuted === true) {
flushText()
const itemID = hostedToolItemID(part)
if (store !== false && itemID && !hostedToolReferences.has(itemID)) input.push({ type: "item_reference", id: itemID })
if (itemID) hostedToolReferences.add(itemID)
continue
}
return yield* ProviderShared.unsupportedContent("OpenAI Responses", "assistant", [
"text",
"reasoning",
"tool-call",
"tool-result",
])
}
flushText()

38
packages/llm/src/protocols/shared.ts
View file

@ -9,6 +9,7 @@ import {
type ContentPart,
type LLMRequest,
type MediaPart,
type TextPart,
type ToolResultPart,
} from "../schema"
export { isRecord } from "../utils/record"
@ -104,6 +105,43 @@ export const parseJson = (route: string, input: string, message: string) =>
*/
export const joinText = (parts: ReadonlyArray<{ readonly text: string }>) => parts.map((part) => part.text).join("\n")
const escapeSystemUpdateText = (text: string) => text.replaceAll("&", "&amp;").replaceAll("<", "&lt;").replaceAll(">", "&gt;")
/**
* Stable fallback representation for chronological `Message.system(...)`
* updates on routes that do not support that privileged role natively. The
* wrapper remains visibly lower-authority user text, preserves the original
* temporal position, and XML-escapes content so it cannot close the wrapper.
*/
export const wrapSystemUpdate = (parts: ReadonlyArray<{ readonly text: string }>) =>
`<system-update>\n${escapeSystemUpdateText(joinText(parts))}\n</system-update>`
/**
* Chronological system updates deliberately accept text only. Do not insert
* raw retrieved, tool, or web content into privileged updates: keep untrusted
* data in ordinary user/tool messages instead.
*/
export const systemUpdateText = Effect.fn("ProviderShared.systemUpdateText")(function* (
route: string,
message: LLMRequest["messages"][number],
) {
const content: TextPart[] = []
for (const part of message.content) {
if (!supportsContent(part, ["text"])) return yield* unsupportedContent(route, "system", ["text"])
content.push(part)
}
return content
})
/** Lower an unsupported privileged update into visible, in-order user text. */
export const wrappedSystemUpdate = Effect.fn("ProviderShared.wrappedSystemUpdate")(function* (
route: string,
message: LLMRequest["messages"][number],
) {
const content = yield* systemUpdateText(route, message)
return { type: "text" as const, text: wrapSystemUpdate(content), cache: content.at(-1)?.cache }
})
/**
* Parse the streamed JSON input of a tool call. Treats an empty string as
* `"{}"` — providers occasionally finish a tool call without ever emitting

10
packages/llm/src/protocols/utils/lifecycle.ts
View file

@ -36,8 +36,14 @@ export const reasoningStart = (
return { ...stepped, reasoning: new Set([...stepped.reasoning, id]) }
}
export const reasoningDelta = (state: State, events: LLMEvent[], id: string, text: string): State => {
const started = reasoningStart(state, events, id)
export const reasoningDelta = (
state: State,
events: LLMEvent[],
id: string,
text: string,
providerMetadata?: ProviderMetadata,
): State => {
const started = reasoningStart(state, events, id, providerMetadata)
events.push(LLMEvent.reasoningDelta({ id, text }))
return started
}

44
packages/llm/src/route/client.ts
View file

@ -10,8 +10,6 @@ import { WebSocketExecutor } from "./transport"
import type { Protocol } from "./protocol"
import { applyCachePolicy } from "../cache-policy"
import * as ProviderShared from "../protocols/shared"
import * as ToolRuntime from "../tool-runtime"
import type { Tools } from "../tool"
import type { LLMError, LLMEvent, PreparedRequestOf, ProtocolID, ProviderOptions } from "../schema"
import {
GenerationOptions,
@ -158,12 +156,10 @@ export interface Interface {
export interface StreamMethod {
(request: LLMRequest): Stream.Stream<LLMEvent, LLMError>
<T extends Tools>(options: ToolRuntime.RunOptions<T>): Stream.Stream<LLMEvent, LLMError>
}
export interface GenerateMethod {
(request: LLMRequest): Effect.Effect<LLMResponse, LLMError>
<T extends Tools>(options: ToolRuntime.RunOptions<T>): Effect.Effect<LLMResponse, LLMError>
}
export class Service extends Context.Service<Service, Interface>()("@opencode/LLMClient") {}
@ -376,19 +372,10 @@ const streamRequestWith = (runtime: TransportRuntime) => (request: LLMRequest) =
}),
)
const isToolRunOptions = (input: LLMRequest | ToolRuntime.RunOptions<Tools>): input is ToolRuntime.RunOptions<Tools> =>
"request" in input && "tools" in input
const streamWith = (streamRequest: (request: LLMRequest) => Stream.Stream<LLMEvent, LLMError>): StreamMethod =>
((input: LLMRequest | ToolRuntime.RunOptions<Tools>) => {
if (isToolRunOptions(input)) return ToolRuntime.stream({ ...input, stream: streamRequest })
return streamRequest(input)
}) as StreamMethod
const generateWith = (stream: Interface["stream"]) =>
Effect.fn("LLM.generate")(function* (input: LLMRequest | ToolRuntime.RunOptions<Tools>) {
Effect.fn("LLM.generate")(function* (request: LLMRequest) {
return new LLMResponse(
yield* stream(input as never).pipe(
yield* stream(request).pipe(
Stream.runFold(
() => ({ events: [] as LLMEvent[], usage: undefined as LLMResponse["usage"] }),
(acc, event) => {
@ -404,22 +391,18 @@ const generateWith = (stream: Interface["stream"]) =>
export const prepare = <Body = unknown>(request: LLMRequest) =>
prepareWith(request) as Effect.Effect<PreparedRequestOf<Body>, LLMError>
export function stream(request: LLMRequest): Stream.Stream<LLMEvent, LLMError>
export function stream<T extends Tools>(options: ToolRuntime.RunOptions<T>): Stream.Stream<LLMEvent, LLMError>
export function stream(input: LLMRequest | ToolRuntime.RunOptions<Tools>) {
export function stream(request: LLMRequest): Stream.Stream<LLMEvent, LLMError> {
return Stream.unwrap(
Effect.gen(function* () {
return (yield* Service).stream(input as never)
return (yield* Service).stream(request)
}),
)
) as Stream.Stream<LLMEvent, LLMError>
}
export function generate(request: LLMRequest): Effect.Effect<LLMResponse, LLMError>
export function generate<T extends Tools>(options: ToolRuntime.RunOptions<T>): Effect.Effect<LLMResponse, LLMError>
export function generate(input: LLMRequest | ToolRuntime.RunOptions<Tools>) {
export function generate(request: LLMRequest): Effect.Effect<LLMResponse, LLMError> {
return Effect.gen(function* () {
return yield* (yield* Service).generate(input as never)
})
return yield* (yield* Service).generate(request)
}) as Effect.Effect<LLMResponse, LLMError>
}
export const streamRequest = (request: LLMRequest) =>
@ -432,12 +415,10 @@ export const streamRequest = (request: LLMRequest) =>
export const layer: Layer.Layer<Service, never, RequestExecutor.Service> = Layer.effect(
Service,
Effect.gen(function* () {
const stream = streamWith(
streamRequestWith({
http: yield* RequestExecutor.Service,
webSocket: Option.getOrUndefined(yield* Effect.serviceOption(WebSocketExecutor.Service)),
}),
)
const stream = streamRequestWith({
http: yield* RequestExecutor.Service,
webSocket: Option.getOrUndefined(yield* Effect.serviceOption(WebSocketExecutor.Service)),
})
return Service.of({ prepare: prepareWith as Interface["prepare"], stream, generate: generateWith(stream) })
}),
)
@ -450,5 +431,4 @@ export const LLMClient = {
prepare,
stream,
generate,
stepCountIs: ToolRuntime.stepCountIs,
} as const

10
packages/llm/src/schema/events.ts
View file

@ -1,7 +1,7 @@
import { Schema } from "effect"
import { ContentBlockID, FinishReason, ProtocolID, ProviderMetadata, RouteID, ToolCallID } from "./ids"
import { ModelSchema } from "./options"
import { ToolResultValue } from "./messages"
import { ToolOutput, ToolResultValue } from "./messages"
/**
* Token usage reported by an LLM provider.
@ -163,6 +163,7 @@ export const ToolResult = Schema.Struct({
id: ToolCallID,
name: Schema.String,
result: ToolResultValue,
output: Schema.optional(ToolOutput),
providerExecuted: Schema.optional(Schema.Boolean),
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.ToolResult" })
@ -252,7 +253,12 @@ export const LLMEvent = Object.assign(llmEventTagged, {
ToolInputDelta.make({ ...input, id: toolCallID(input.id) }),
toolInputEnd: (input: WithID<ToolInputEnd, ToolCallID>) => ToolInputEnd.make({ ...input, id: toolCallID(input.id) }),
toolCall: (input: WithID<ToolCall, ToolCallID>) => ToolCall.make({ ...input, id: toolCallID(input.id) }),
toolResult: (input: WithID<ToolResult, ToolCallID>) => ToolResult.make({ ...input, id: toolCallID(input.id) }),
toolResult: (input: WithID<ToolResult, ToolCallID>) =>
ToolResult.make({
...input,
id: toolCallID(input.id),
output: input.output === undefined ? undefined : ToolOutput.make(input.output.structured, input.output.content),
}),
toolError: (input: WithID<ToolError, ToolCallID>) => ToolError.make({ ...input, id: toolCallID(input.id) }),
stepFinish: (input: WithUsage<StepFinish>) =>
StepFinish.make({

2
packages/llm/src/schema/ids.ts
View file

@ -30,7 +30,7 @@ export type ReasoningEffort = Schema.Schema.Type<typeof ReasoningEffort>
export const TextVerbosity = Schema.Literals(["low", "medium", "high"])
export type TextVerbosity = Schema.Schema.Type<typeof TextVerbosity>
export const MessageRole = Schema.Literals(["user", "assistant", "tool"])
export const MessageRole = Schema.Literals(["system", "user", "assistant", "tool"])
export type MessageRole = Schema.Schema.Type<typeof MessageRole>
export const FinishReason = Schema.Literals(["stop", "length", "tool-calls", "content-filter", "error", "unknown"])

134
packages/llm/src/schema/messages.ts
View file

@ -51,6 +51,56 @@ export type ToolResultMediaPart = Schema.Schema.Type<typeof ToolResultMediaPart>
export const ToolResultContentPart = Schema.Union([TextPart, ToolResultMediaPart])
export type ToolResultContentPart = Schema.Schema.Type<typeof ToolResultContentPart>
export class ToolTextContent extends Schema.Class<ToolTextContent>("Tool.TextContent")({
type: Schema.Literal("text"),
text: Schema.String,
}) {}
export const ToolFileSource = Schema.Union([
Schema.Struct({ type: Schema.Literal("data"), data: Schema.String }),
Schema.Struct({ type: Schema.Literal("url"), url: Schema.String }),
Schema.Struct({ type: Schema.Literal("file"), uri: Schema.String }),
]).pipe(Schema.toTaggedUnion("type"))
export type ToolFileSource = Schema.Schema.Type<typeof ToolFileSource>
export class ToolFileContent extends Schema.Class<ToolFileContent>("Tool.FileContent")({
type: Schema.Literal("file"),
source: ToolFileSource,
mime: Schema.String,
name: Schema.optional(Schema.String),
}) {}
/** Ordered, provider-independent content shown to models and UIs after a tool succeeds. */
export const ToolContent = Schema.Union([ToolTextContent, ToolFileContent]).pipe(Schema.toTaggedUnion("type"))
export type ToolContent = Schema.Schema.Type<typeof ToolContent>
export const toolText = (value: ConstructorParameters<typeof ToolTextContent>[0]) => new ToolTextContent(value)
export const toolFile = (value: ConstructorParameters<typeof ToolFileContent>[0]) => new ToolFileContent(value)
const inlineData = (uri: string) => {
if (!uri.startsWith("data:")) return undefined
const match = /^data:[^;,]+;base64,(.*)$/s.exec(uri)
if (!match) throw new Error("Tool file data URI must contain raw base64 bytes")
return match[1]!
}
const legacyInlineData = (value: string) => {
const data = inlineData(value)
if (data !== undefined) return data
if (/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/.test(value)) return value
throw new Error("Legacy tool-result media must contain raw base64 bytes or a base64 data URI")
}
/** Convert a legacy attachment URI without guessing unknown string semantics. */
export const toolFileSourceFromUri = (uri: string): ToolFileSource => {
const data = inlineData(uri)
if (data !== undefined) return { type: "data", data }
const url = URL.parse(uri)
if (url?.protocol === "file:") return { type: "file", uri }
if (url?.protocol === "http:" || url?.protocol === "https:") return { type: "url", url: uri }
throw new Error(`Unsupported tool file URI: ${uri}`)
}
const isToolResultValue = (value: unknown): value is ToolResultValue =>
isRecord(value) &&
(value.type === "text" || value.type === "json" || value.type === "error" || value.type === "content") &&
@ -86,6 +136,80 @@ export const ToolResultValue = Object.assign(
)
export type ToolResultValue = Schema.Schema.Type<typeof ToolResultValue>
export interface ToolOutput {
readonly structured: unknown
readonly content: ReadonlyArray<ToolContent>
}
export const ToolOutput = Object.assign(
Schema.Struct({
structured: Schema.Unknown,
content: Schema.Array(ToolContent),
}).annotate({ identifier: "LLM.ToolOutput" }),
{
make: (structured: unknown, content: ReadonlyArray<ToolContent> = []): ToolOutput => ({
structured,
content: content.map((item) =>
item.type === "text"
? toolText({ type: "text", text: item.text })
: toolFile({ type: "file", source: item.source, mime: item.mime, name: item.name }),
),
}),
fromResultValue: (result: ToolResultValue): ToolOutput | undefined => {
switch (result.type) {
case "json":
return { structured: result.value, content: [] }
case "text":
return { structured: {}, content: [toolText({ type: "text", text: toolResultText(result.value) })] }
case "content":
return {
structured: {},
content: result.value.map((item) =>
item.type === "text"
? toolText({ type: "text", text: item.text })
: toolFile({
type: "file",
source: { type: "data", data: legacyInlineData(item.data) },
mime: item.mediaType,
name: item.filename,
}),
),
}
case "error":
return undefined
}
},
toResultValue: (output: ToolOutput): ToolResultValue => {
if (output.content.length === 0) return { type: "json", value: output.structured }
if (output.content.length === 1 && output.content[0]?.type === "text")
return { type: "text", value: output.content[0].text }
const unsupported = output.content.find((item) => item.type === "file" && item.source.type !== "data")
if (unsupported?.type === "file")
return {
type: "error",
value: `Tool file source "${unsupported.source.type}" must be materialized to inline data before provider conversion`,
}
return {
type: "content",
value: output.content.map((item) => {
if (item.type === "text") return { type: "text", text: item.text }
if (item.source.type !== "data") throw new Error("Unmaterialized tool file source reached provider conversion")
return { type: "media", mediaType: item.mime, data: item.source.data, filename: item.name }
}),
}
},
},
)
const toolResultText = (value: unknown) => {
if (typeof value === "string") return value
try {
return JSON.stringify(value) ?? String(value)
} catch {
return String(value)
}
}
export const ToolCallPart = Object.assign(
Schema.Struct({
type: Schema.Literal("tool-call"),
@ -157,6 +281,7 @@ export class Message extends Schema.Class<Message>("LLM.Message")({
export namespace Message {
export type ContentInput = string | ContentPart | ReadonlyArray<ContentPart>
export type SystemContentInput = string | TextPart | ReadonlyArray<TextPart>
export type Input = Omit<ConstructorParameters<typeof Message>[0], "content"> & {
readonly content: ContentInput
}
@ -175,6 +300,14 @@ export namespace Message {
export const assistant = (content: ContentInput) => make({ role: "assistant", content })
/**
* Add an operator-authored instruction at this chronological point in the
* conversation. This is distinct from the initial `LLMRequest.system`
* prompt. Keep raw retrieved, tool, and web content out of privileged system
* updates; pass that untrusted content through ordinary user/tool channels.
*/
export const system = (content: SystemContentInput) => make({ role: "system", content })
export const tool = (result: ToolResultPart | Parameters<typeof ToolResultPart.make>[0]) =>
make({ role: "tool", content: ["type" in result ? result : ToolResultPart.make(result)] })
}
@ -183,6 +316,7 @@ export class ToolDefinition extends Schema.Class<ToolDefinition>("LLM.ToolDefini
name: Schema.String,
description: Schema.String,
inputSchema: JsonSchema,
outputSchema: Schema.optional(JsonSchema),
cache: Schema.optional(CacheHint),
metadata: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
native: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),

370
packages/llm/src/tool-runtime.ts
View file

@ -1,340 +1,74 @@
import { Effect, Stream } from "effect"
import type { Concurrency } from "effect/Types"
import {
type ContentPart,
type FinishReason,
type LLMError,
LLMEvent,
LLMRequest,
Message,
type ProviderMetadata,
ToolCallPart,
ToolFailure,
ToolResultPart,
ToolResultValue,
type ToolResultValue as ToolResultValueType,
Usage,
} from "./schema"
import { type AnyTool, type ExecutableTools, type Tools, toDefinitions } from "./tool"
import { Effect } from "effect"
import { LLMEvent, type ToolCallPart, ToolFailure, ToolOutput, ToolResultValue, type ToolOutput as ToolOutputType, type ToolResultValue as ToolResultValueType } from "./schema"
import { type AnyTool, type Tools } from "./tool"
export interface RuntimeState {
readonly step: number
readonly request: LLMRequest
export interface ToolSettlement {
readonly result: ToolResultValueType
readonly output?: ToolOutputType
}
export type StopCondition = (state: RuntimeState) => boolean
export type ToolExecution = "auto" | "none"
interface RunOptionsBase {
readonly request: LLMRequest
readonly concurrency?: Concurrency
readonly stopWhen?: StopCondition
export interface DispatchResult extends ToolSettlement {
readonly events: ReadonlyArray<LLMEvent>
}
export type RunOptions<T extends Tools> = RunOptionsAuto<T & ExecutableTools> | RunOptionsNone<T>
export interface RunOptionsAuto<T extends ExecutableTools> extends RunOptionsBase {
readonly request: LLMRequest
readonly tools: T
readonly toolExecution?: "auto"
}
export interface RunOptionsNone<T extends Tools> extends RunOptionsBase {
readonly request: LLMRequest
readonly tools: T
/** Advertise tool schemas but leave model-emitted tool calls for the caller. */
readonly toolExecution: "none"
}
export type StreamOptions<T extends Tools> = RunOptions<T> & {
readonly stream: (request: LLMRequest) => Stream.Stream<LLMEvent, LLMError>
}
export const stepCountIs =
(count: number): StopCondition =>
(state) =>
state.step + 1 >= count
/**
* Run a model with typed tools. This helper owns tool orchestration, while the
* caller supplies the actual model stream function. It can advertise schemas
* only (`toolExecution: "none"`), execute one step, or continue model rounds
* when `stopWhen` is provided.
*/
export const stream = <T extends Tools>(options: StreamOptions<T>): Stream.Stream<LLMEvent, LLMError> => {
const concurrency = options.concurrency ?? 10
const tools = options.tools as Tools
const runtimeTools = toDefinitions(tools)
const runtimeToolNames = new Set(runtimeTools.map((tool) => tool.name))
const initialRequest =
runtimeTools.length === 0
? options.request
: LLMRequest.update(options.request, {
tools: [...options.request.tools.filter((tool) => !runtimeToolNames.has(tool.name)), ...runtimeTools],
})
const loop = (
request: LLMRequest,
step: number,
usage: Usage | undefined,
providerMetadata: ProviderMetadata | undefined,
): Stream.Stream<LLMEvent, LLMError> =>
Stream.unwrap(
Effect.gen(function* () {
const state: StepState = {
assistantContent: [],
toolCalls: [],
finishReason: undefined,
usage: undefined,
providerMetadata: undefined,
}
const modelStream = options
.stream(request)
.pipe(Stream.map((event) => indexStep(event, step)))
.pipe(Stream.tap((event) => Effect.sync(() => accumulate(state, event))))
.pipe(Stream.filter((event) => event.type !== "finish"))
const continuation = Stream.unwrap(
Effect.gen(function* () {
const totalUsage = addUsage(usage, state.usage)
const totalProviderMetadata = mergeProviderMetadata(providerMetadata, state.providerMetadata)
const finishStream = Stream.fromIterable([
LLMEvent.finish({
reason: state.finishReason ?? "unknown",
usage: totalUsage,
providerMetadata: totalProviderMetadata,
}),
])
if (state.finishReason !== "tool-calls" || state.toolCalls.length === 0) return finishStream
if (options.toolExecution === "none") return finishStream
const dispatched = yield* Effect.forEach(
state.toolCalls,
(call) =>
dispatch(tools, call).pipe(Effect.map((result) => [call, result.result, result.error] as const)),
{ concurrency },
)
const resultStream = Stream.fromIterable(
dispatched.flatMap(([call, result, error]) => emitEvents(call, result, error)),
)
if (!options.stopWhen) return resultStream.pipe(Stream.concat(finishStream))
if (options.stopWhen({ step, request })) return resultStream.pipe(Stream.concat(finishStream))
return resultStream.pipe(
Stream.concat(
loop(
followUpRequest(
request,
state,
dispatched.map(([call, result]) => [call, result] as const),
),
step + 1,
totalUsage,
totalProviderMetadata,
),
),
)
}),
)
return modelStream.pipe(Stream.concat(continuation))
}),
)
return loop(initialRequest, 0, undefined, undefined)
}
const indexStep = (event: LLMEvent, index: number): LLMEvent => {
if (event.type === "step-start") return LLMEvent.stepStart({ index })
if (event.type === "step-finish") return LLMEvent.stepFinish({ ...event, index })
return event
}
interface StepState {
assistantContent: ContentPart[]
toolCalls: ToolCallPart[]
finishReason: FinishReason | undefined
usage: Usage | undefined
providerMetadata: ProviderMetadata | undefined
}
const accumulate = (state: StepState, event: LLMEvent) => {
if (event.type === "text-delta") {
appendStreamingText(state, "text", event.text, undefined)
return
}
if (event.type === "reasoning-delta") {
appendStreamingText(state, "reasoning", event.text, undefined)
return
}
if (event.type === "reasoning-end") {
appendStreamingText(state, "reasoning", "", event.providerMetadata)
return
}
if (event.type === "text-end") {
appendStreamingText(state, "text", "", event.providerMetadata)
return
}
if (event.type === "tool-call") {
const part = ToolCallPart.make({
id: event.id,
name: event.name,
input: event.input,
providerExecuted: event.providerExecuted,
providerMetadata: event.providerMetadata,
})
state.assistantContent.push(part)
if (!event.providerExecuted) state.toolCalls.push(part)
return
}
if (event.type === "tool-result" && event.providerExecuted) {
state.assistantContent.push(
ToolResultPart.make({
id: event.id,
name: event.name,
result: event.result,
providerExecuted: true,
providerMetadata: event.providerMetadata,
}),
)
return
}
if (event.type === "step-finish") {
state.finishReason = event.reason === "stop" && state.toolCalls.length > 0 ? "tool-calls" : event.reason
state.usage = addUsage(state.usage, event.usage)
state.providerMetadata = mergeProviderMetadata(state.providerMetadata, event.providerMetadata)
return
}
if (event.type === "finish") {
state.finishReason ??= event.reason
state.usage ??= event.usage
state.providerMetadata = mergeProviderMetadata(state.providerMetadata, event.providerMetadata)
}
}
const addUsage = (left: Usage | undefined, right: Usage | undefined) => {
if (!left) return right
if (!right) return left
type UsageKey =
| "inputTokens"
| "outputTokens"
| "nonCachedInputTokens"
| "cacheReadInputTokens"
| "cacheWriteInputTokens"
| "reasoningTokens"
| "totalTokens"
const sum = (key: UsageKey) =>
left[key] === undefined && right[key] === undefined ? undefined : (left[key] ?? 0) + (right[key] ?? 0)
return new Usage({
inputTokens: sum("inputTokens"),
outputTokens: sum("outputTokens"),
nonCachedInputTokens: sum("nonCachedInputTokens"),
cacheReadInputTokens: sum("cacheReadInputTokens"),
cacheWriteInputTokens: sum("cacheWriteInputTokens"),
reasoningTokens: sum("reasoningTokens"),
totalTokens: sum("totalTokens"),
providerMetadata: mergeProviderMetadata(left.providerMetadata, right.providerMetadata),
})
}
const sameProviderMetadata = (left: ProviderMetadata | undefined, right: ProviderMetadata | undefined) =>
left === right || JSON.stringify(left) === JSON.stringify(right)
const mergeProviderMetadata = (left: ProviderMetadata | undefined, right: ProviderMetadata | undefined) => {
if (!left) return right
if (!right) return left
return Object.fromEntries(
Array.from(new Set([...Object.keys(left), ...Object.keys(right)])).map((provider) => [
provider,
{ ...left[provider], ...right[provider] },
]),
)
}
const appendStreamingText = (
state: StepState,
type: "text" | "reasoning",
text: string,
providerMetadata: ProviderMetadata | undefined,
) => {
const last = state.assistantContent.at(-1)
if (last?.type === type && text.length === 0) {
state.assistantContent[state.assistantContent.length - 1] = {
...last,
providerMetadata: mergeProviderMetadata(last.providerMetadata, providerMetadata),
}
return
}
if (last?.type === type && sameProviderMetadata(last.providerMetadata, providerMetadata)) {
state.assistantContent[state.assistantContent.length - 1] = { ...last, text: `${last.text}${text}` }
return
}
state.assistantContent.push({ type, text, providerMetadata })
}
const dispatch = (
tools: Tools,
call: ToolCallPart,
): Effect.Effect<{ result: ToolResultValueType; error?: unknown }> => {
/** Execute one canonical tool call without owning provider IO or continuation. */
export const dispatch = (tools: Tools, call: ToolCallPart): Effect.Effect<DispatchResult> => {
const tool = tools[call.name]
if (!tool) return Effect.succeed({ result: { type: "error" as const, value: `Unknown tool: ${call.name}` } })
if (!tool) return Effect.succeed(result(call, { type: "error", value: `Unknown tool: ${call.name}` }))
if (!tool.execute)
return Effect.succeed({ result: { type: "error" as const, value: `Tool has no execute handler: ${call.name}` } })
return Effect.succeed(result(call, { type: "error", value: `Tool has no execute handler: ${call.name}` }))
return decodeAndExecute(tool, call).pipe(
Effect.map((value) => result(call, value)),
Effect.catchTag("LLM.ToolFailure", (failure) =>
Effect.succeed({
result: { type: "error" as const, value: failure.message } satisfies ToolResultValueType,
error: failure.error,
}),
Effect.succeed(result(call, { type: "error", value: failure.message }, failure.error)),
),
Effect.map((result) => ("result" in result ? result : { result })),
)
}
const decodeAndExecute = (tool: AnyTool, call: ToolCallPart): Effect.Effect<ToolResultValueType, ToolFailure> =>
const decodeAndExecute = (tool: AnyTool, call: ToolCallPart): Effect.Effect<ToolSettlement, ToolFailure> =>
tool._decode(call.input).pipe(
Effect.mapError((error) => new ToolFailure({ message: `Invalid tool input: ${error.message}` })),
Effect.flatMap((decoded) => tool.execute!(decoded, { id: call.id, name: call.name })),
Effect.flatMap((value) =>
tool._encode(value).pipe(
Effect.mapError(
(error) =>
new ToolFailure({
message: `Tool returned an invalid value for its success schema: ${error.message}`,
}),
Effect.flatMap((decoded) =>
tool.execute!(decoded, { id: call.id, name: call.name }).pipe(
Effect.flatMap((value) =>
tool._encode(value).pipe(
Effect.mapError(
(error) =>
new ToolFailure({
message: `Tool returned an invalid value for its success schema: ${error.message}`,
}),
),
),
),
Effect.map((encoded) => {
if (tool._legacyResult && ToolResultValue.is(encoded))
return { result: encoded, output: ToolOutput.fromResultValue(encoded) }
const output = tool._project(decoded, call.id, encoded)
const result = ToolOutput.toResultValue(output)
return result.type === "error" ? { result } : { result, output }
}),
),
),
Effect.map(
(encoded): ToolResultValueType => (ToolResultValue.is(encoded) ? encoded : { type: "json", value: encoded }),
),
)
const emitEvents = (call: ToolCallPart, result: ToolResultValueType, error: unknown): ReadonlyArray<LLMEvent> =>
result.type === "error"
? [
LLMEvent.toolError({ id: call.id, name: call.name, message: String(result.value), error }),
LLMEvent.toolResult({ id: call.id, name: call.name, result }),
]
: [LLMEvent.toolResult({ id: call.id, name: call.name, result })]
const result = (
call: ToolCallPart,
value: ToolResultValueType | ToolSettlement,
error?: unknown,
): DispatchResult => {
const settlement = ToolResultValue.is(value) ? { result: value } : value
return {
result: settlement.result,
output: settlement.output,
events:
settlement.result.type === "error"
? [
LLMEvent.toolError({ id: call.id, name: call.name, message: String(settlement.result.value), error }),
LLMEvent.toolResult({ id: call.id, name: call.name, result: settlement.result }),
]
: [LLMEvent.toolResult({ id: call.id, name: call.name, result: settlement.result, output: settlement.output })],
}
}
const followUpRequest = (
request: LLMRequest,
state: StepState,
dispatched: ReadonlyArray<readonly [ToolCallPart, ToolResultValueType]>,
) =>
LLMRequest.update(request, {
messages: [
...request.messages,
Message.assistant(state.assistantContent),
...dispatched.map(([call, result]) => Message.tool({ id: call.id, name: call.name, result })),
],
})
export const ToolRuntime = { stream, stepCountIs } as const
export const ToolRuntime = { dispatch } as const

64
packages/llm/src/tool.ts
View file

@ -1,6 +1,6 @@
import { Effect, JsonSchema, Schema } from "effect"
import type { ToolCallPart, ToolDefinition as ToolDefinitionClass } from "./schema"
import { ToolDefinition, ToolFailure } from "./schema"
import type { ToolCallPart, ToolContent, ToolDefinition as ToolDefinitionClass, ToolOutput as ToolOutputType } from "./schema"
import { ToolDefinition, ToolFailure, ToolOutput, toolText } from "./schema"
/**
* Schema constraint for tool parameters / success values: no decoding or
@ -18,6 +18,16 @@ export type ToolExecute<Parameters extends ToolSchema<any>, Success extends Tool
context?: ToolExecuteContext,
) => Effect.Effect<Schema.Schema.Type<Success>, ToolFailure>
export interface ToolModelOutputInput<Parameters, Output> {
readonly callID: ToolCallPart["id"]
readonly parameters: Parameters
readonly output: Output
}
export type ToolToModelOutput<Parameters extends ToolSchema<any>, Success extends ToolSchema<any>> = (
input: ToolModelOutputInput<Schema.Schema.Type<Parameters>, Success["Encoded"]>,
) => ReadonlyArray<ToolContent>
/**
* A type-safe LLM tool. Each tool bundles its own description, parameter
* Schema and success Schema. The execute handler is optional: omit it when you
@ -28,22 +38,27 @@ export type ToolExecute<Parameters extends ToolSchema<any>, Success extends Tool
* the stream.
*
* Internally each tool also carries memoized codecs and a precomputed
* `ToolDefinition` so the runtime doesn't rebuild them per invocation.
* `ToolDefinition` so callers do not rebuild them per invocation.
*/
export interface Tool<Parameters extends ToolSchema<any>, Success extends ToolSchema<any>> {
readonly description: string
readonly parameters: Parameters
readonly success: Success
readonly execute?: ToolExecute<Parameters, Success>
readonly toModelOutput?: ToolToModelOutput<Parameters, Success>
/** @internal */
readonly _decode: (input: unknown) => Effect.Effect<Schema.Schema.Type<Parameters>, Schema.SchemaError>
/** @internal */
readonly _encode: (value: Schema.Schema.Type<Success>) => Effect.Effect<unknown, Schema.SchemaError>
/** @internal */
readonly _project: (parameters: Schema.Schema.Type<Parameters>, callID: ToolCallPart["id"], output: unknown) => ToolOutputType
/** @internal */
readonly _legacyResult: boolean
/** @internal */
readonly _definition: ToolDefinitionClass
}
export type AnyTool = Tool<ToolSchema<any>, ToolSchema<any>>
export type AnyTool = Tool<any, any>
export type ExecutableTool<Parameters extends ToolSchema<any>, Success extends ToolSchema<any>> = Tool<
Parameters,
@ -52,7 +67,7 @@ export type ExecutableTool<Parameters extends ToolSchema<any>, Success extends T
readonly execute: ToolExecute<Parameters, Success>
}
export type AnyExecutableTool = ExecutableTool<ToolSchema<any>, ToolSchema<any>>
export type AnyExecutableTool = ExecutableTool<any, any>
export type ExecutableTools = Record<string, AnyExecutableTool>
@ -61,12 +76,15 @@ type TypedToolConfig = {
readonly parameters: ToolSchema<any>
readonly success: ToolSchema<any>
readonly execute?: ToolExecute<ToolSchema<any>, ToolSchema<any>>
readonly toModelOutput?: ToolToModelOutput<ToolSchema<any>, ToolSchema<any>>
}
type DynamicToolConfig = {
readonly description: string
readonly jsonSchema: JsonSchema.JsonSchema
readonly outputSchema?: JsonSchema.JsonSchema
readonly execute?: (params: unknown, context?: ToolExecuteContext) => Effect.Effect<unknown, ToolFailure>
readonly toModelOutput?: (input: ToolModelOutputInput<unknown, unknown>) => ReadonlyArray<ToolContent>
}
/**
@ -97,30 +115,36 @@ type DynamicToolConfig = {
* })
* ```
*
* In both modes the produced tool flows through `toDefinitions(...)` and the
* runtime identically.
* In both modes the produced tool flows through `toDefinitions(...)`
* identically.
*/
export function make<Parameters extends ToolSchema<any>, Success extends ToolSchema<any>>(config: {
readonly description: string
readonly parameters: Parameters
readonly success: Success
readonly execute: ToolExecute<Parameters, Success>
readonly toModelOutput?: ToolToModelOutput<Parameters, Success>
}): ExecutableTool<Parameters, Success>
export function make<Parameters extends ToolSchema<any>, Success extends ToolSchema<any>>(config: {
readonly description: string
readonly parameters: Parameters
readonly success: Success
readonly execute?: undefined
readonly toModelOutput?: ToolToModelOutput<Parameters, Success>
}): Tool<Parameters, Success>
export function make(config: {
readonly description: string
readonly jsonSchema: JsonSchema.JsonSchema
readonly outputSchema?: JsonSchema.JsonSchema
readonly execute: (params: unknown, context?: ToolExecuteContext) => Effect.Effect<unknown, ToolFailure>
readonly toModelOutput?: (input: ToolModelOutputInput<unknown, unknown>) => ReadonlyArray<ToolContent>
}): AnyExecutableTool
export function make(config: {
readonly description: string
readonly jsonSchema: JsonSchema.JsonSchema
readonly outputSchema?: JsonSchema.JsonSchema
readonly execute?: undefined
readonly toModelOutput?: (input: ToolModelOutputInput<unknown, unknown>) => ReadonlyArray<ToolContent>
}): AnyTool
export function make(config: TypedToolConfig | DynamicToolConfig): AnyTool {
if ("jsonSchema" in config) {
@ -129,12 +153,16 @@ export function make(config: TypedToolConfig | DynamicToolConfig): AnyTool {
parameters: Schema.Unknown as ToolSchema<unknown>,
success: Schema.Unknown as ToolSchema<unknown>,
execute: config.execute,
toModelOutput: config.toModelOutput,
_decode: Effect.succeed,
_encode: Effect.succeed,
_project: (parameters, callID, output) => project(config.toModelOutput, parameters, callID, output),
_legacyResult: config.toModelOutput === undefined,
_definition: new ToolDefinition({
name: "",
description: config.description,
inputSchema: config.jsonSchema,
outputSchema: config.outputSchema,
}),
}
}
@ -143,18 +171,20 @@ export function make(config: TypedToolConfig | DynamicToolConfig): AnyTool {
parameters: config.parameters,
success: config.success,
execute: config.execute,
toModelOutput: config.toModelOutput,
_decode: Schema.decodeUnknownEffect(config.parameters),
_encode: Schema.encodeEffect(config.success),
_project: (parameters, callID, output) => project(config.toModelOutput, parameters, callID, output),
_legacyResult: false,
_definition: new ToolDefinition({
name: "",
description: config.description,
inputSchema: toJsonSchema(config.parameters),
outputSchema: toJsonSchema(config.success),
}),
}
}
export const tool = make
/**
* A record of named tools. The record key becomes the tool name on the wire.
*/
@ -162,8 +192,7 @@ export type Tools = Record<string, AnyTool>
/**
* Convert a tools record into the `ToolDefinition[]` shape that
* `LLMRequest.tools` expects. The runtime calls this internally; consumers
* that build `LLMRequest` themselves can use it too.
* `LLMRequest.tools` expects.
*
* Tool names come from the record keys, so the per-tool cached
* `_definition` is rebuilt with the correct name here. The JSON Schema body
@ -176,6 +205,7 @@ export const toDefinitions = (tools: Tools): ReadonlyArray<ToolDefinitionClass>
name,
description: item._definition.description,
inputSchema: item._definition.inputSchema,
outputSchema: item._definition.outputSchema,
}),
)
@ -185,6 +215,18 @@ const toJsonSchema = (schema: Schema.Top): JsonSchema.JsonSchema => {
return { ...document.schema, $defs: document.definitions }
}
const project = (
toModelOutput: ((input: ToolModelOutputInput<any, any>) => ReadonlyArray<ToolContent>) | undefined,
parameters: unknown,
callID: ToolCallPart["id"],
output: unknown,
): ToolOutputType =>
ToolOutput.make(
output,
toModelOutput?.({ callID, parameters, output }) ??
(typeof output === "string" ? [toolText({ type: "text", text: output })] : []),
)
export { ToolFailure }
export * as Tool from "./tool"

144
packages/llm/test/lib/tool-runtime.ts
View file

@ -1,8 +1,142 @@
import { Effect, Stream } from "effect"
import { LLMClient } from "../../src/route"
import type { Tools } from "../../src/tool"
import type { RunOptions } from "../../src/tool-runtime"
import {
LLMEvent,
LLMRequest,
Message,
type ContentPart,
type ProviderMetadata,
type ToolCallPart,
ToolResultPart,
type ToolResultValue,
type Usage,
} from "../../src/schema"
import { type Tools, toDefinitions } from "../../src/tool"
import { ToolRuntime } from "../../src/tool-runtime"
type CompatRunOptions<T extends Tools> = RunOptions<T> & { readonly maxSteps?: number }
interface RunOptions<T extends Tools> {
readonly request: LLMRequest
readonly tools: T
readonly maxSteps?: number
}
export const runTools = <T extends Tools>(options: CompatRunOptions<T>) =>
LLMClient.stream({ ...options, stopWhen: options.stopWhen ?? LLMClient.stepCountIs(options.maxSteps ?? 10) })
/** Test-owned continuation loop. Production callers must own durable history. */
export const runTools = <T extends Tools>(options: RunOptions<T>) =>
Stream.unwrap(
Effect.gen(function* () {
const names = new Set(Object.keys(options.tools))
let request = LLMRequest.update(options.request, {
tools: [...options.request.tools.filter((tool) => !names.has(tool.name)), ...toDefinitions(options.tools)],
})
let usage: Usage | undefined
const events: LLMEvent[] = []
for (let step = 0; step < (options.maxSteps ?? 10); step++) {
const streamed = Array.from(yield* LLMClient.stream(request).pipe(Stream.runCollect))
const state = stepState(streamed)
usage = addUsage(usage, state.usage)
events.push(...streamed.filter((event) => event.type !== "finish").map((event) => indexStep(event, step)))
if (state.toolCalls.length === 0) {
events.push(LLMEvent.finish({ reason: state.reason, usage, providerMetadata: state.providerMetadata }))
return Stream.fromIterable(events)
}
const dispatched = yield* Effect.forEach(
state.toolCalls,
(call) => ToolRuntime.dispatch(options.tools, call).pipe(Effect.map((result) => [call, result] as const)),
{ concurrency: 10 },
)
events.push(...dispatched.flatMap(([, result]) => result.events))
if (step + 1 >= (options.maxSteps ?? 10)) {
events.push(LLMEvent.finish({ reason: state.reason, usage, providerMetadata: state.providerMetadata }))
return Stream.fromIterable(events)
}
request = LLMRequest.update(request, {
messages: [
...request.messages,
Message.assistant(state.assistantContent),
...dispatched.map(([call, dispatched]) =>
Message.tool({ id: call.id, name: call.name, result: dispatched.result }),
),
],
})
}
return Stream.fromIterable(events)
}),
)
const indexStep = (event: LLMEvent, index: number): LLMEvent => {
if (event.type === "step-start") return LLMEvent.stepStart({ index })
if (event.type === "step-finish") return LLMEvent.stepFinish({ ...event, index })
return event
}
const stepState = (events: ReadonlyArray<LLMEvent>) => {
const assistantContent: ContentPart[] = []
const toolCalls: ToolCallPart[] = []
let reason: Extract<LLMEvent, { type: "finish" }>["reason"] = "unknown"
let usage: Usage | undefined
let providerMetadata: ProviderMetadata | undefined
for (const event of events) {
if (event.type === "text-delta" || event.type === "reasoning-delta") {
appendText(assistantContent, event.type === "text-delta" ? "text" : "reasoning", event.text)
} else if (event.type === "text-end" || event.type === "reasoning-end") {
appendText(assistantContent, event.type === "text-end" ? "text" : "reasoning", "", event.providerMetadata)
} else if (event.type === "tool-call") {
assistantContent.push(event)
if (!event.providerExecuted) toolCalls.push(event)
} else if (event.type === "tool-result" && event.providerExecuted && event.result !== undefined) {
assistantContent.push(
ToolResultPart.make({
id: event.id,
name: event.name,
result: event.result,
providerExecuted: true,
providerMetadata: event.providerMetadata,
}),
)
} else if (event.type === "finish") {
reason = event.reason
usage = event.usage
providerMetadata = event.providerMetadata
}
}
return { assistantContent, toolCalls, reason, usage, providerMetadata }
}
const appendText = (
content: ContentPart[],
type: "text" | "reasoning",
text: string,
providerMetadata?: ProviderMetadata,
) => {
const last = content.at(-1)
if (last?.type === type) {
content[content.length - 1] = { ...last, text: `${last.text}${text}`, providerMetadata: providerMetadata ?? last.providerMetadata }
return
}
content.push({ type, text, providerMetadata })
}
const addUsage = (left: Usage | undefined, right: Usage | undefined): Usage | undefined => {
if (!left) return right
if (!right) return left
const sum = (key: keyof Usage) =>
typeof left[key] !== "number" && typeof right[key] !== "number"
? undefined
: ((left[key] as number | undefined) ?? 0) + ((right[key] as number | undefined) ?? 0)
return {
inputTokens: sum("inputTokens"),
outputTokens: sum("outputTokens"),
nonCachedInputTokens: sum("nonCachedInputTokens"),
cacheReadInputTokens: sum("cacheReadInputTokens"),
cacheWriteInputTokens: sum("cacheWriteInputTokens"),
reasoningTokens: sum("reasoningTokens"),
totalTokens: sum("totalTokens"),
} as Usage
}

19
packages/llm/test/llm.test.ts
View file

@ -1,5 +1,5 @@
import { describe, expect, test } from "bun:test"
import { LLM, LLMResponse } from "../src"
import { CacheHint, LLM, LLMResponse } from "../src"
import * as OpenAIChat from "../src/protocols/openai-chat"
import * as OpenAIResponses from "../src/protocols/openai-responses"
import { LLMRequest, Message, Model, ToolCallPart, ToolChoice, ToolDefinition, ToolResultPart } from "../src/schema"
@ -135,6 +135,23 @@ describe("llm constructors", () => {
])
})
test("builds chronological text-only system updates separately from the initial system prompt", () => {
const update = Message.system([{ type: "text", text: "Use parameterized SQL.", cache: new CacheHint({ type: "ephemeral" }) }])
const request = LLM.request({
model: Model.make({ id: "fake-model", provider: "fake", route: chatRoute }),
system: "Initial operator prompt.",
messages: [Message.user("Review this."), update],
})
expect(update).toBeInstanceOf(Message)
expect(update).toEqual({
role: "system",
content: [{ type: "text", text: "Use parameterized SQL.", cache: { type: "ephemeral" } }],
})
expect(request.system).toEqual([{ type: "text", text: "Initial operator prompt." }])
expect(request.messages.map((message) => message.role)).toEqual(["user", "system"])
})
test("extracts output text from response events", () => {
expect(
LLMResponse.text({

91
packages/llm/test/provider/anthropic-messages.test.ts
View file

@ -13,6 +13,10 @@ const model = AnthropicMessages.route
.with({ endpoint: { baseURL: "https://api.anthropic.test/v1/" }, auth: Auth.header("x-api-key", "test") })
.model({ id: "claude-sonnet-4-5" })
const opus48 = AnthropicMessages.route
.with({ endpoint: { baseURL: "https://api.anthropic.test/v1/" }, auth: Auth.header("x-api-key", "test") })
.model({ id: "claude-opus-4-8" })
const request = LLM.request({
id: "req_1",
model,
@ -53,6 +57,93 @@ describe("Anthropic Messages route", () => {
}),
)
it.effect("lowers chronological system updates natively for Claude Opus 4.8 with cache hints", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare<AnthropicMessages.AnthropicMessagesBody>(
LLM.request({
model: opus48,
messages: [
Message.user("Before."),
Message.system([{ type: "text", text: "Operator update.", cache: new CacheHint({ type: "ephemeral" }) }]),
Message.assistant("After."),
],
cache: "none",
}),
)
expect(prepared.body.messages).toEqual([
{ role: "user", content: [{ type: "text", text: "Before." }] },
{
role: "system",
content: [{ type: "text", text: "Operator update.", cache_control: { type: "ephemeral" } }],
},
{ role: "assistant", content: [{ type: "text", text: "After." }] },
])
}),
)
it.effect("lowers chronological system updates to wrapped user text for unsupported Anthropic models", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare<AnthropicMessages.AnthropicMessagesBody>(
LLM.request({
model,
messages: [Message.user("Before."), Message.system("Treat </system-update> literally."), Message.assistant("After.")],
cache: "none",
}),
)
expect(prepared.body.messages).toEqual([
{
role: "user",
content: [
{ type: "text", text: "Before." },
{ type: "text", text: "<system-update>\nTreat &lt;/system-update&gt; literally.\n</system-update>" },
],
},
{ role: "assistant", content: [{ type: "text", text: "After." }] },
])
}),
)
it.effect("rejects non-text chronological system update content before send", () =>
Effect.gen(function* () {
const error = yield* LLMClient.prepare(
LLM.request({
model: opus48,
messages: [
Message.user("Before."),
Message.make({ role: "system", content: { type: "media", mediaType: "image/png", data: "AAECAw==" } }),
],
}),
).pipe(Effect.flip)
expect(error.message).toContain("Anthropic Messages system messages only support text content for now")
}),
)
it.effect("rejects invalid native chronological system update placement", () =>
Effect.gen(function* () {
const placementError = (messages: Parameters<typeof LLM.request>[0]["messages"]) =>
LLMClient.prepare(LLM.request({ model: opus48, messages, cache: "none" })).pipe(Effect.flip)
expect((yield* placementError([Message.system("First.")])).message).toContain("cannot be the first message")
expect((yield* placementError([Message.user("Before."), Message.system("One."), Message.system("Two.")])).message)
.toContain("cannot be consecutive")
expect((yield* placementError([Message.assistant("Plain."), Message.system("After plain assistant.")])).message)
.toContain("must follow a user message, tool result, or assistant server tool use")
expect(
(
yield* placementError([
Message.user("Use the tool."),
Message.assistant([ToolCallPart.make({ id: "call_1", name: "lookup", input: {} })]),
Message.system("Too early."),
Message.tool({ id: "call_1", name: "lookup", result: "Done." }),
])
).message,
).toContain("cannot appear between a local tool call and its tool result")
}),
)
it.effect("prepares tool call and tool result messages", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare<AnthropicMessages.AnthropicMessagesBody>(

53
packages/llm/test/provider/bedrock-converse.test.ts
View file

@ -5,6 +5,7 @@ import { Effect } from "effect"
import { CacheHint, LLM, Message, ToolCallPart, ToolChoice } from "../../src"
import { LLMClient } from "../../src/route"
import { AmazonBedrock } from "../../src/providers"
import * as BedrockConverse from "../../src/protocols/bedrock-converse"
import { it } from "../lib/effect"
import { fixedResponse } from "../lib/http"
import {
@ -82,6 +83,23 @@ describe("Bedrock Converse route", () => {
}),
)
it.effect("lowers chronological system updates to wrapped user text in order", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare<BedrockConverse.BedrockConverseBody>(
LLM.request({
model,
messages: [Message.user("Before."), Message.system("Update."), Message.assistant("After.")],
cache: "none",
}),
)
expect(prepared.body.messages).toEqual([
{ role: "user", content: [{ text: "Before." }, { text: "<system-update>\nUpdate.\n</system-update>" }] },
{ role: "assistant", content: [{ text: "After." }] },
])
}),
)
it.effect("prepares tool config with toolSpec and toolChoice", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare(
@ -279,6 +297,41 @@ describe("Bedrock Converse route", () => {
}),
)
it.effect("preserves streamed reasoning signatures for continuation lowering", () =>
Effect.gen(function* () {
const body = eventStreamBody(
["messageStart", { role: "assistant" }],
["contentBlockDelta", { contentBlockIndex: 0, delta: { reasoningContent: { text: "Let me think." } } }],
["contentBlockDelta", { contentBlockIndex: 0, delta: { reasoningContent: { signature: "sig_1" } } }],
["contentBlockStop", { contentBlockIndex: 0 }],
["messageStop", { stopReason: "end_turn" }],
)
const response = yield* LLMClient.generate(baseRequest).pipe(Effect.provide(fixedBytes(body)))
const reasoning = response.events.find((event) => event.type === "reasoning-end")
expect(reasoning).toEqual({
type: "reasoning-end",
id: "reasoning-0",
providerMetadata: { bedrock: { signature: "sig_1" } },
})
const prepared = yield* LLMClient.prepare<BedrockConverse.BedrockConverseBody>(
LLM.request({
model,
messages: [
Message.assistant([
{ type: "reasoning", text: "Let me think.", providerMetadata: reasoning?.providerMetadata },
]),
],
cache: "none",
}),
)
expect(prepared.body.messages).toEqual([
{ role: "assistant", content: [{ reasoningContent: { reasoningText: { text: "Let me think.", signature: "sig_1" } } }] },
])
}),
)
it.effect("emits provider-error for throttlingException", () =>
Effect.gen(function* () {
const body = eventStreamBody(

82
packages/llm/test/provider/gemini.test.ts
View file

@ -35,6 +35,22 @@ describe("Gemini route", () => {
}),
)
it.effect("lowers chronological system updates to wrapped user text in order", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare<Gemini.GeminiBody>(
LLM.request({
model,
messages: [Message.user("Before."), Message.system("Update."), Message.assistant("After.")],
}),
)
expect(prepared.body.contents).toEqual([
{ role: "user", parts: [{ text: "Before." }, { text: "<system-update>\nUpdate.\n</system-update>" }] },
{ role: "model", parts: [{ text: "After." }] },
])
}),
)
it.effect("prepares multimodal user input and tool history", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare(
@ -241,6 +257,72 @@ describe("Gemini route", () => {
}),
)
it.effect("preserves thoughtSignature for reasoning and tool-call continuation", () =>
Effect.gen(function* () {
const body = sseEvents({
candidates: [
{
content: {
role: "model",
parts: [
{ text: "thinking", thought: true },
{ text: "", thought: true, thoughtSignature: "thought_sig" },
{ functionCall: { name: "lookup", args: { query: "weather" } }, thoughtSignature: "tool_sig" },
],
},
finishReason: "STOP",
},
],
})
const response = yield* LLMClient.generate(
LLM.updateRequest(request, {
tools: [{ name: "lookup", description: "Lookup data", inputSchema: { type: "object" } }],
}),
).pipe(Effect.provide(fixedResponse(body)))
const reasoning = response.events.find((event) => event.type === "reasoning-start")
const reasoningEnd = response.events.find((event) => event.type === "reasoning-end")
const toolCall = response.events.find((event) => event.type === "tool-call")
expect(reasoning).toEqual({
type: "reasoning-start",
id: "reasoning-0",
providerMetadata: undefined,
})
expect(reasoningEnd).toEqual({
type: "reasoning-end",
id: "reasoning-0",
providerMetadata: { google: { thoughtSignature: "thought_sig" } },
})
expect(toolCall).toMatchObject({ providerMetadata: { google: { thoughtSignature: "tool_sig" } } })
const prepared = yield* LLMClient.prepare<Gemini.GeminiBody>(
LLM.request({
model,
messages: [
Message.assistant([
{ type: "reasoning", text: "thinking", providerMetadata: reasoningEnd?.providerMetadata },
ToolCallPart.make({
id: "tool_0",
name: "lookup",
input: { query: "weather" },
providerMetadata: toolCall?.providerMetadata,
}),
]),
],
}),
)
expect(prepared.body.contents).toEqual([
{
role: "model",
parts: [
{ text: "thinking", thought: true, thoughtSignature: "thought_sig" },
{ functionCall: { name: "lookup", args: { query: "weather" } }, thoughtSignature: "tool_sig" },
],
},
])
}),
)
it.effect("emits streamed tool calls and maps finish reason", () =>
Effect.gen(function* () {
const body = sseEvents({

37
packages/llm/test/provider/openai-chat.test.ts
View file

@ -50,6 +50,35 @@ describe("OpenAI Chat route", () => {
}),
)
it.effect("lowers chronological system updates to escaped user wrappers in order", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare<OpenAIChat.OpenAIChatBody>(
LLM.request({
model,
messages: [Message.user("Before."), Message.system("Treat <admin> & data literally."), Message.assistant("After.")],
}),
)
expect(prepared.body.messages).toEqual([
{ role: "user", content: "Before.\n<system-update>\nTreat &lt;admin&gt; &amp; data literally.\n</system-update>" },
{ role: "assistant", content: "After." },
])
}),
)
it.effect("replays canonical reasoning as OpenAI-compatible reasoning_content", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare<OpenAIChat.OpenAIChatBody>(
LLM.request({
model,
messages: [Message.assistant([{ type: "reasoning", text: "thinking" }, { type: "text", text: "Hello" }])],
}),
)
expect(prepared.body.messages).toEqual([{ role: "assistant", content: "Hello", reasoning_content: "thinking" }])
}),
)
it.effect("maps OpenAI provider options to Chat options", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare<OpenAIChat.OpenAIChatBody>(
@ -196,17 +225,17 @@ describe("OpenAI Chat route", () => {
}),
)
it.effect("rejects unsupported assistant reasoning content", () =>
it.effect("lowers reasoning-only assistant history", () =>
Effect.gen(function* () {
const error = yield* LLMClient.prepare(
const prepared = yield* LLMClient.prepare<OpenAIChat.OpenAIChatBody>(
LLM.request({
id: "req_reasoning",
model,
messages: [Message.assistant({ type: "reasoning", text: "hidden" })],
}),
).pipe(Effect.flip)
)
expect(error.message).toContain("OpenAI Chat assistant messages only support text and tool-call content for now")
expect(prepared.body.messages).toEqual([{ role: "assistant", content: null, reasoning_content: "hidden" }])
}),
)

58
packages/llm/test/provider/openai-responses.test.ts
View file

@ -57,6 +57,28 @@ describe("OpenAI Responses route", () => {
}),
)
it.effect("lowers chronological system updates to escaped user wrappers in order", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare<OpenAIResponses.OpenAIResponsesBody>(
LLM.request({
model,
messages: [Message.user("Before."), Message.system("Treat </system-update> literally."), Message.assistant("After.")],
}),
)
expect(prepared.body.input).toEqual([
{
role: "user",
content: [
{ type: "input_text", text: "Before." },
{ type: "input_text", text: "<system-update>\nTreat &lt;/system-update&gt; literally.\n</system-update>" },
],
},
{ role: "assistant", content: [{ type: "output_text", text: "After." }] },
])
}),
)
it.effect("prepares OpenAI Responses WebSocket target", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare(
@ -857,6 +879,42 @@ describe("OpenAI Responses route", () => {
}),
)
it.effect("references stored provider-executed hosted tool results by id", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare<OpenAIResponses.OpenAIResponsesBody>(
LLM.request({
model,
messages: [
Message.assistant([
ToolCallPart.make({
id: "ws_1",
name: "web_search",
input: { query: "effect 4" },
providerExecuted: true,
providerMetadata: { openai: { itemId: "ws_1" } },
}),
{
type: "tool-result",
id: "ws_1",
name: "web_search",
result: { type: "json", value: { type: "web_search_call", id: "ws_1", status: "completed" } },
providerExecuted: true,
providerMetadata: { openai: { itemId: "ws_1" } },
},
]),
Message.user("Continue."),
],
providerOptions: { openai: { store: true } },
}),
)
expect(prepared.body.input).toEqual([
{ type: "item_reference", id: "ws_1" },
{ role: "user", content: [{ type: "input_text", text: "Continue." }] },
])
}),
)
it.effect("joins streamed summary blocks into one continuation reasoning item", () =>
Effect.gen(function* () {
const prepared = yield* LLMClient.prepare<OpenAIResponses.OpenAIResponsesBody>(

68
packages/llm/test/recorded-scenarios.ts
View file

@ -5,15 +5,17 @@ import {
LLMEvent,
LLMResponse,
Message,
ToolRuntime,
ToolChoice,
ToolDefinition,
toDefinitions,
type ContentPart,
type FinishReason,
type LLMRequest,
type Model,
} from "../src"
import { LLMClient } from "../src/route"
import { tool } from "../src/tool"
import { Tool } from "../src/tool"
export const weatherToolName = "get_weather"
@ -40,7 +42,7 @@ export const weatherTool = ToolDefinition.make({
},
})
export const weatherRuntimeTool = tool({
export const weatherRuntimeTool = Tool.make({
description: weatherTool.description,
parameters: Schema.Struct({ city: Schema.String }),
success: Schema.Struct({ temperature: Schema.Number, condition: Schema.String }),
@ -87,14 +89,60 @@ const restroomImage = () =>
)
export const runWeatherToolLoop = (request: LLMRequest) =>
LLMClient.stream({
request,
tools: { [weatherToolName]: weatherRuntimeTool },
stopWhen: LLMClient.stepCountIs(10),
}).pipe(
Stream.runCollect,
Effect.map((events) => Array.from(events)),
)
Effect.gen(function* () {
const tools = { [weatherToolName]: weatherRuntimeTool }
let next = LLM.updateRequest(request, { tools: toDefinitions(tools) })
const events: LLMEvent[] = []
for (let step = 0; step < 10; step++) {
const response = yield* LLMClient.generate(next)
events.push(...response.events.filter((event) => event.type !== "finish"))
const calls = response.events.filter(LLMEvent.is.toolCall).filter((call) => !call.providerExecuted)
if (calls.length === 0) {
const finish = response.events.find(LLMEvent.is.finish)
if (finish) events.push(finish)
return events
}
const dispatched = yield* Effect.forEach(calls, (call) =>
ToolRuntime.dispatch(tools, call).pipe(Effect.map((result) => [call, result] as const)),
)
events.push(...dispatched.flatMap(([, result]) => result.events))
next = LLM.updateRequest(next, {
messages: [
...next.messages,
Message.assistant(assistantContent(response.events)),
...dispatched.map(([call, result]) => Message.tool({ id: call.id, name: call.name, result: result.result })),
],
})
}
throw new Error("Weather tool loop exceeded 10 steps")
})
const assistantContent = (events: ReadonlyArray<LLMEvent>) => {
const content: ContentPart[] = []
for (const event of events) {
if (event.type === "text-delta" || event.type === "reasoning-delta") {
const type = event.type === "text-delta" ? "text" : "reasoning"
const last = content.at(-1)
if (last?.type === type) {
content[content.length - 1] = { ...last, text: `${last.text}${event.text}` }
} else {
content.push({ type, text: event.text })
}
continue
}
if (event.type === "text-end" || event.type === "reasoning-end") {
const type = event.type === "text-end" ? "text" : "reasoning"
const last = content.at(-1)
if (last?.type === type) content[content.length - 1] = { ...last, providerMetadata: event.providerMetadata }
continue
}
if (event.type === "tool-call") content.push(event)
}
return content
}
export const expectFinish = (
events: ReadonlyArray<LLMEvent>,

11
packages/llm/test/schema.test.ts
View file

@ -43,6 +43,17 @@ describe("llm schema", () => {
expect(decoded.model.route.id).toBe("openai-responses")
})
test("decodes chronological system messages", () => {
const decoded = decodeLLMRequest({
model,
system: [],
messages: [{ role: "system", content: [{ type: "text", text: "Operator update." }] }],
tools: [],
})
expect(decoded.messages[0]).toMatchObject({ role: "system", content: [{ type: "text", text: "Operator update." }] })
})
test("rejects invalid event type", () => {
expect(() => decodeLLMEvent({ type: "bogus" })).toThrow()
})

270
packages/llm/test/tool-runtime.test.ts
View file

@ -1,11 +1,11 @@
import { describe, expect } from "bun:test"
import { Effect, Schema, Stream } from "effect"
import { GenerationOptions, LLM, LLMEvent, LLMRequest, LLMResponse, ToolChoice } from "../src"
import { GenerationOptions, LLM, LLMEvent, LLMRequest, LLMResponse, ToolChoice, ToolContent, ToolOutput, toolFileSourceFromUri, toDefinitions } from "../src"
import { Auth, LLMClient } from "../src/route"
import * as AnthropicMessages from "../src/protocols/anthropic-messages"
import * as OpenAIChat from "../src/protocols/openai-chat"
import * as OpenAIResponses from "../src/protocols/openai-responses"
import { tool, ToolFailure, type ToolExecuteContext } from "../src/tool"
import { Tool, ToolFailure, type ToolExecuteContext } from "../src/tool"
import { ToolRuntime } from "../src/tool-runtime"
import { it } from "./lib/effect"
import * as TestToolRuntime from "./lib/tool-runtime"
@ -26,7 +26,7 @@ const baseRequest = LLM.request({
})
const weatherFailureCause = new Error("weather lookup denied")
const get_weather = tool({
const get_weather = Tool.make({
description: "Get current weather for a city.",
parameters: Schema.Struct({ city: Schema.String }),
success: Schema.Struct({ temperature: Schema.Number, condition: Schema.String }),
@ -38,7 +38,7 @@ const get_weather = tool({
}),
})
const schema_only_weather = tool({
const schema_only_weather = Tool.make({
description: "Get current weather for a city.",
parameters: Schema.Struct({ city: Schema.String }),
success: Schema.Struct({ temperature: Schema.Number, condition: Schema.String }),
@ -140,9 +140,161 @@ describe("LLMClient tools", () => {
}),
)
it.effect("projects encoded typed tool success into canonical model content", () =>
Effect.gen(function* () {
const calls: unknown[] = []
const projected = Tool.make({
description: "Project an encoded success.",
parameters: Schema.Struct({ prefix: Schema.String }),
success: Schema.Struct({ count: Schema.NumberFromString }),
execute: () => Effect.succeed({ count: 2 }),
toModelOutput: (input) => {
calls.push(input)
return [{ type: "text", text: `${input.parameters.prefix}:${input.output.count}` }]
},
})
const dispatched = yield* ToolRuntime.dispatch(
{ projected },
LLMEvent.toolCall({ id: "call_projected", name: "projected", input: { prefix: "count" } }),
)
expect(calls).toEqual([{ callID: "call_projected", parameters: { prefix: "count" }, output: { count: "2" } }])
expect(dispatched.result).toEqual({ type: "text", value: "count:2" })
expect(dispatched.output).toEqual({ structured: { count: "2" }, content: [{ type: "text", text: "count:2" }] })
expect(dispatched.events).toEqual([
LLMEvent.toolResult({
id: "call_projected",
name: "projected",
result: { type: "text", value: "count:2" },
output: { structured: { count: "2" }, content: [{ type: "text", text: "count:2" }] },
}),
])
}),
)
it.effect("uses the narrow default projection for encoded typed success", () =>
Effect.gen(function* () {
const text = Tool.make({
description: "Return text.",
parameters: Schema.Struct({}),
success: Schema.String,
execute: () => Effect.succeed("hello"),
})
const json = Tool.make({
description: "Return JSON.",
parameters: Schema.Struct({}),
success: Schema.Struct({ ok: Schema.Boolean }),
execute: () => Effect.succeed({ ok: true }),
})
expect((yield* ToolRuntime.dispatch({ text }, LLMEvent.toolCall({ id: "call_text", name: "text", input: {} }))).output)
.toEqual({ structured: "hello", content: [{ type: "text", text: "hello" }] })
expect((yield* ToolRuntime.dispatch({ json }, LLMEvent.toolCall({ id: "call_json", name: "json", input: {} }))).output)
.toEqual({ structured: { ok: true }, content: [] })
}),
)
it.effect("models canonical tool files with explicit data, url, and file sources", () =>
Effect.sync(() => {
const decode = Schema.decodeUnknownSync(ToolContent)
expect(decode({ type: "file", source: { type: "data", data: "AAAA" }, mime: "image/png" })).toEqual({
type: "file",
source: { type: "data", data: "AAAA" },
mime: "image/png",
})
expect(decode({ type: "file", source: { type: "url", url: "https://example.test/image.png" }, mime: "image/png" })).toEqual({
type: "file",
source: { type: "url", url: "https://example.test/image.png" },
mime: "image/png",
})
expect(decode({ type: "file", source: { type: "file", uri: "file:///tmp/image.png" }, mime: "image/png" })).toEqual({
type: "file",
source: { type: "file", uri: "file:///tmp/image.png" },
mime: "image/png",
})
}),
)
it.effect("converts canonical data files deliberately and rejects unmaterialized sources", () =>
Effect.sync(() => {
expect(
ToolOutput.toResultValue(
ToolOutput.make({}, [{ type: "file", source: { type: "data", data: "AAAA" }, mime: "image/png" }]),
),
).toEqual({ type: "content", value: [{ type: "media", mediaType: "image/png", data: "AAAA" }] })
expect(
ToolOutput.toResultValue(
ToolOutput.make({}, [{ type: "file", source: { type: "url", url: "https://example.test/image.png" }, mime: "image/png" }]),
),
).toEqual({ type: "error", value: 'Tool file source "url" must be materialized to inline data before provider conversion' })
expect(
ToolOutput.toResultValue(
ToolOutput.make({}, [{ type: "file", source: { type: "file", uri: "file:///tmp/image.png" }, mime: "image/png" }]),
),
).toEqual({ type: "error", value: 'Tool file source "file" must be materialized to inline data before provider conversion' })
expect(toolFileSourceFromUri("data:image/png;base64,AAAA")).toEqual({ type: "data", data: "AAAA" })
expect(toolFileSourceFromUri("https://example.test/image.png")).toEqual({ type: "url", url: "https://example.test/image.png" })
expect(toolFileSourceFromUri("file:///tmp/image.png")).toEqual({ type: "file", uri: "file:///tmp/image.png" })
expect(() => toolFileSourceFromUri("opaque-value")).toThrow("Unsupported tool file URI")
expect(() =>
ToolOutput.fromResultValue({
type: "content",
value: [{ type: "media", mediaType: "image/png", data: "https://example.test/image.png" }],
}),
).toThrow("Legacy tool-result media must contain raw base64 bytes or a base64 data URI")
}),
)
it.effect("settles projected url files as materialization errors", () =>
Effect.gen(function* () {
const remote = Tool.make({
description: "Return a remote file.",
parameters: Schema.Struct({}),
success: Schema.Struct({ ok: Schema.Boolean }),
execute: () => Effect.succeed({ ok: true }),
toModelOutput: () => [
{ type: "file", source: { type: "url", url: "https://example.test/image.png" }, mime: "image/png" },
],
})
const dispatched = yield* ToolRuntime.dispatch(
{ remote },
LLMEvent.toolCall({ id: "call_remote", name: "remote", input: {} }),
)
expect(dispatched.output).toBeUndefined()
expect(dispatched.result).toEqual({
type: "error",
value: 'Tool file source "url" must be materialized to inline data before provider conversion',
})
expect(dispatched.events.map((event) => event.type)).toEqual(["tool-error", "tool-result"])
}),
)
it.effect("derives typed output schemas and preserves dynamic output schemas", () =>
Effect.sync(() => {
const [typed] = toDefinitions({ get_weather })
const schema = { type: "object", properties: { result: { type: "string" } } } as const
const [dynamic] = toDefinitions({
dynamic: Tool.make({ description: "Dynamic tool.", jsonSchema: { type: "object" }, outputSchema: schema }),
})
expect(typed?.outputSchema).toMatchObject({
type: "object",
properties: { condition: { type: "string" } },
required: ["temperature", "condition"],
additionalProperties: false,
})
expect(Reflect.get(Reflect.get(typed?.outputSchema ?? {}, "properties") as object, "temperature")).toBeDefined()
expect(dynamic?.outputSchema).toEqual(schema)
}),
)
it.effect("preserves content tool results from dynamic tools", () =>
Effect.gen(function* () {
const screenshot = tool({
const screenshot = Tool.make({
description: "Capture a screenshot.",
jsonSchema: { type: "object", properties: {} },
execute: () =>
@ -156,7 +308,7 @@ describe("LLMClient tools", () => {
})
const events = Array.from(
yield* LLMClient.stream({ request: baseRequest, tools: { screenshot } }).pipe(
yield* TestToolRuntime.runTools({ request: baseRequest, tools: { screenshot }, maxSteps: 1 }).pipe(
Stream.runCollect,
Effect.provide(
scriptedResponses([sseEvents(toolCallChunk("call_1", "screenshot", "{}"), finishChunk("tool_calls"))]),
@ -179,6 +331,32 @@ describe("LLMClient tools", () => {
}),
)
it.effect("does not mistake dynamic tool output fields for dispatcher state", () =>
Effect.gen(function* () {
const callerOwned = { type: "json" as const, value: { ok: true }, events: ["caller-owned"] }
const eventful = Tool.make({
description: "Return an events field.",
jsonSchema: { type: "object", properties: {} },
execute: () => Effect.succeed(callerOwned),
})
const dispatched = yield* ToolRuntime.dispatch(
{ eventful },
LLMEvent.toolCall({ id: "call_1", name: "eventful", input: {} }),
)
expect(dispatched.result).toEqual(callerOwned)
expect(dispatched.events).toEqual([
LLMEvent.toolResult({
id: "call_1",
name: "eventful",
result: callerOwned,
output: { structured: { ok: true }, content: [] },
}),
])
}),
)
it.effect("executes tool calls for one step without looping by default", () =>
Effect.gen(function* () {
const layer = scriptedResponses([
@ -187,7 +365,7 @@ describe("LLMClient tools", () => {
])
const events = Array.from(
yield* LLMClient.stream({ request: baseRequest, tools: { get_weather } }).pipe(
yield* TestToolRuntime.runTools({ request: baseRequest, tools: { get_weather }, maxSteps: 1 }).pipe(
Stream.runCollect,
Effect.provide(layer),
),
@ -201,7 +379,7 @@ describe("LLMClient tools", () => {
it.effect("passes tool call context to execute", () =>
Effect.gen(function* () {
let context: ToolExecuteContext | undefined
const contextual = tool({
const contextual = Tool.make({
description: "Capture tool context.",
parameters: Schema.Struct({ value: Schema.String }),
success: Schema.Struct({ ok: Schema.Boolean }),
@ -234,11 +412,9 @@ describe("LLMClient tools", () => {
])
const events = Array.from(
yield* LLMClient.stream({
request: baseRequest,
tools: { get_weather: schema_only_weather },
toolExecution: "none",
}).pipe(Stream.runCollect, Effect.provide(layer)),
yield* LLMClient.stream(
LLMRequest.update(baseRequest, { tools: toDefinitions({ get_weather: schema_only_weather }) }),
).pipe(Stream.runCollect, Effect.provide(layer)),
)
expect(events.find(LLMEvent.is.toolCall)).toMatchObject({ type: "tool-call", id: "call_1" })
@ -500,74 +676,6 @@ describe("LLMClient tools", () => {
}),
)
it.effect("emits one final finish with aggregate usage", () =>
Effect.gen(function* () {
let calls = 0
const events = Array.from(
yield* ToolRuntime.stream({
request: baseRequest,
tools: { get_weather },
stopWhen: ToolRuntime.stepCountIs(2),
stream: () =>
Stream.fromIterable<LLMEvent>(
calls++ === 0
? [
LLMEvent.stepStart({ index: 0 }),
LLMEvent.toolCall({ id: "call_1", name: "get_weather", input: { city: "Paris" } }),
LLMEvent.stepFinish({
index: 0,
reason: "tool-calls",
usage: { inputTokens: 1, outputTokens: 2, totalTokens: 3 },
}),
LLMEvent.finish({
reason: "tool-calls",
usage: { inputTokens: 1, outputTokens: 2, totalTokens: 3 },
}),
]
: [
LLMEvent.stepStart({ index: 0 }),
LLMEvent.textDelta({ id: "text_1", text: "Done." }),
LLMEvent.stepFinish({
index: 0,
reason: "stop",
usage: { inputTokens: 4, outputTokens: 5, totalTokens: 9 },
}),
LLMEvent.finish({ reason: "stop", usage: { inputTokens: 4, outputTokens: 5, totalTokens: 9 } }),
],
),
}).pipe(Stream.runCollect),
)
expect(events.filter(LLMEvent.is.stepFinish).map((event) => event.index)).toEqual([0, 1])
expect(events.filter(LLMEvent.is.finish)).toHaveLength(1)
expect(events.find(LLMEvent.is.finish)?.usage).toMatchObject({
inputTokens: 5,
outputTokens: 7,
totalTokens: 12,
})
}),
)
it.effect("stops follow-up when stopWhen returns true after the first step", () =>
Effect.gen(function* () {
const layer = scriptedResponses([
sseEvents(toolCallChunk("call_1", "get_weather", '{"city":"Paris"}'), finishChunk("tool_calls")),
sseEvents(deltaChunk({ role: "assistant", content: "Should not run." }), finishChunk("stop")),
])
const events = Array.from(
yield* TestToolRuntime.runTools({
request: baseRequest,
tools: { get_weather },
stopWhen: (state) => state.step >= 0,
}).pipe(Stream.runCollect, Effect.provide(layer)),
)
expect(events.filter(LLMEvent.is.finish)).toHaveLength(1)
expect(events.find(LLMEvent.is.toolResult)).toMatchObject({ type: "tool-result", id: "call_1" })
}),
)
it.effect("does not dispatch provider-executed tool calls", () =>
Effect.gen(function* () {
let streams = 0

24
packages/llm/test/tool.types.ts
View file

@ -1,30 +1,38 @@
import { Effect, Schema } from "effect"
import { LLM } from "../src"
import { LLM, LLMRequest, ToolRuntime, toDefinitions } from "../src"
import * as OpenAIChat from "../src/protocols/openai-chat"
import { Auth } from "../src/route"
import { tool } from "../src/tool"
import { Tool } from "../src/tool"
const request = LLM.request({
model: OpenAIChat.route.with({ auth: Auth.bearer("fixture") }).model({ id: "gpt-4o-mini" }),
prompt: "Use the tool.",
})
const executable = tool({
const executable = Tool.make({
description: "Get weather.",
parameters: Schema.Struct({ city: Schema.String }),
success: Schema.Struct({ forecast: Schema.String }),
execute: (input) => Effect.succeed({ forecast: input.city }),
})
const schemaOnly = tool({
const schemaOnly = Tool.make({
description: "Get weather.",
parameters: Schema.Struct({ city: Schema.String }),
success: Schema.Struct({ forecast: Schema.String }),
})
LLM.stream({ request, tools: { executable } })
LLM.generate({ request, tools: { executable }, stopWhen: LLM.stepCountIs(2) })
LLM.stream({ request, tools: { schemaOnly }, toolExecution: "none" })
Tool.make({
description: "Encode success before projection.",
parameters: Schema.Struct({ city: Schema.String }),
success: Schema.Struct({ forecast: Schema.NumberFromString }),
execute: () => Effect.succeed({ forecast: 1 }),
toModelOutput: ({ callID, parameters, output }) => [{ type: "text", text: `${callID}:${parameters.city}:${output.forecast}` }],
})
// @ts-expect-error Handler-less tools can only be passed with toolExecution: "none".
LLM.stream(request)
LLM.generate(LLMRequest.update(request, { tools: toDefinitions({ schemaOnly }) }))
ToolRuntime.dispatch({ executable }, { type: "tool-call", id: "call_1", name: "executable", input: { city: "Paris" } })
// @ts-expect-error High-level tool orchestration overloads are intentionally not supported.
LLM.stream({ request, tools: { schemaOnly } })