refactor(file): add ripgrep search service (#22295)

This commit is contained in:
Kit Langton 2026-04-13 10:04:32 -04:00 committed by GitHub
commit 6fdb8ab90d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
12 changed files with 665 additions and 348 deletions

View file

@ -0,0 +1,137 @@
# HttpApi migration
Practical notes for an eventual migration of `packages/opencode` server routes from the current Hono handlers to Effect `HttpApi`, either as a full replacement or as a parallel surface.
## Goal
Use Effect `HttpApi` where it gives us a better typed contract for:
- route definition
- request decoding and validation
- typed success and error responses
- OpenAPI generation
- handler composition inside Effect
This should be treated as a later-stage HTTP boundary migration, not a prerequisite for ongoing service, route-handler, or schema work.
## Core model
`HttpApi` is definition-first.
- `HttpApi` is the root API
- `HttpApiGroup` groups related endpoints
- `HttpApiEndpoint` defines a single route and its request / response schemas
- handlers are implemented separately from the contract
This is a better fit once route inputs and outputs are already moving toward Effect Schema-first models.
## Why it is relevant here
The current route-effectification work is already pushing handlers toward:
- one `AppRuntime.runPromise(Effect.gen(...))` body
- yielding services from context
- using typed Effect errors instead of Promise wrappers
That work is a good prerequisite for `HttpApi`. Once the handler body is already a composed Effect, the remaining migration is mostly about replacing the Hono route declaration and validator layer.
## What HttpApi gives us
### Contracts
Request params, query, payload, success payloads, and typed error payloads are declared in one place using Effect Schema.
### Validation and decoding
Incoming data is decoded through Effect Schema instead of hand-maintained Zod validators per route.
### OpenAPI
`HttpApi` can derive OpenAPI from the API definition, which overlaps with the current `describeRoute(...)` and `resolver(...)` pattern.
### Typed errors
`Schema.TaggedErrorClass` maps naturally to endpoint error contracts.
## Likely fit for opencode
Best fit first:
- JSON request / response endpoints
- route groups that already mostly delegate into services
- endpoints whose request and response models can be defined with Effect Schema
Harder / later fit:
- SSE endpoints
- websocket endpoints
- streaming handlers
- routes with heavy Hono-specific middleware assumptions
## Current blockers and gaps
### Schema split
Many route boundaries still use Zod-first validators. That does not block all experimentation, but full `HttpApi` adoption is easier after the domain and boundary types are more consistently Schema-first with `.zod` compatibility only where needed.
### Mixed handler styles
Many current `server/instance/*.ts` handlers still call async facades directly. Migrating those to composed `Effect.gen(...)` handlers is the low-risk step to do first.
### Non-JSON routes
The server currently includes SSE, websocket, and streaming-style endpoints. Those should not be the first `HttpApi` targets.
### Existing Hono integration
The current server composition, middleware, and docs flow are Hono-centered today. That suggests a parallel or incremental adoption plan is safer than a flag day rewrite.
## Recommended strategy
### 1. Finish the prerequisites first
- continue route-handler effectification in `server/instance/*.ts`
- continue schema migration toward Effect Schema-first DTOs and errors
- keep removing service facades
### 2. Start with one parallel group
Introduce one small `HttpApi` group for plain JSON endpoints only. Good initial candidates are the least stateful endpoints in:
- `server/instance/question.ts`
- `server/instance/provider.ts`
- `server/instance/permission.ts`
Avoid `session.ts`, SSE, websocket, and TUI-facing routes first.
### 3. Reuse existing services
Do not re-architect business logic during the HTTP migration. `HttpApi` handlers should call the same Effect services already used by the Hono handlers.
### 4. Run in parallel before replacing
Prefer mounting an experimental `HttpApi` surface alongside the existing Hono routes first. That lowers migration risk and lets us compare:
- handler ergonomics
- OpenAPI output
- auth and middleware integration
- test ergonomics
### 5. Migrate JSON route groups gradually
If the parallel slice works well, migrate additional JSON route groups one at a time. Leave streaming-style endpoints on Hono until there is a clear reason to move them.
## Proposed first steps
- [ ] add one small spike that defines an `HttpApi` group for a simple JSON route set
- [ ] use Effect Schema request / response types for that slice
- [ ] keep the underlying service calls identical to the current handlers
- [ ] compare generated OpenAPI against the current Hono/OpenAPI setup
- [ ] document how auth, instance lookup, and error mapping would compose in the new stack
- [ ] decide after the spike whether `HttpApi` should stay parallel, replace only some groups, or become the long-term default
## Rule of thumb
Do not start with the hardest route file.
If `HttpApi` is adopted here, it should arrive after the handler body is already Effect-native and after the relevant request / response models have moved to Effect Schema.

View file

@ -230,55 +230,9 @@ Still open at the service-shape level:
- [ ] `SyncEvent``sync/index.ts` (deferred pending sync with James)
- [ ] `Workspace``control-plane/workspace.ts` (deferred pending sync with James)
## Tool interface → Effect
## Tool migration
`Tool.Def.execute` and `Tool.Info.init` already return `Effect` on this branch, and the current tools in `src/tool/*.ts` have been migrated to the Effect-native `Tool.define(...)` shape.
The remaining work here is follow-on cleanup rather than the top-level tool interface migration:
1. Remove internal `Effect.promise(...)` bridges where practical
2. Keep replacing raw platform helpers with Effect services inside tool bodies
3. Update remaining callers and tests to prefer `yield* info.init()` / `Tool.init(...)` over older Promise-oriented patterns
### Tool migration details
With `Tool.Info.init()` now effectful, use this transitional pattern for migrated tools that still need Promise-based boundaries internally:
- `Tool.defineEffect(...)` should `yield*` the services the tool depends on and close over them in the returned tool definition.
- Keep the bridge at the Promise boundary only inside the tool body when required by external APIs. Do not return Promise-based init callbacks from `Tool.define()`.
- If a tool starts requiring new services, wire them into `ToolRegistry.defaultLayer` so production callers resolve the same dependencies as tests.
Tool tests should use the existing Effect helpers in `packages/opencode/test/lib/effect.ts`:
- Use `testEffect(...)` / `it.live(...)` instead of creating fake local wrappers around effectful tools.
- Yield the real tool export, then initialize it: `const info = yield* ReadTool`, `const tool = yield* info.init()`.
- Run tests inside a real instance with `provideTmpdirInstance(...)` or `provideInstance(tmpdirScoped(...))` so instance-scoped services resolve exactly as they do in production.
This keeps migrated tool tests aligned with the production service graph today, and makes the eventual `Tool.Info``Effect` cleanup mostly mechanical later.
Individual tools, ordered by value:
- [x] `apply_patch.ts` — HIGH: multi-step orchestration, error accumulation, Bus events
- [x] `bash.ts` — HIGH: shell orchestration, quoting, timeout handling, output capture
- [x] `read.ts` — HIGH: effectful interface migrated; still has raw fs/readline internals tracked below
- [x] `edit.ts` — HIGH: multi-step diff/format/publish pipeline, FileWatcher lock
- [x] `grep.ts` — MEDIUM: spawns ripgrep → ChildProcessSpawner, timeout handling
- [x] `write.ts` — MEDIUM: permission checks, diagnostics polling, Bus events
- [x] `codesearch.ts` — MEDIUM: HTTP + SSE + manual timeout → HttpClient + Effect.timeout
- [x] `webfetch.ts` — MEDIUM: fetch with UA retry, size limits → HttpClient
- [x] `websearch.ts` — MEDIUM: MCP over HTTP → HttpClient
- [x] `task.ts` — MEDIUM: task state management
- [x] `ls.ts` — MEDIUM: bounded directory listing over ripgrep-backed traversal
- [x] `multiedit.ts` — MEDIUM: sequential edit orchestration over `edit.ts`
- [x] `glob.ts` — LOW: simple async generator
- [x] `lsp.ts` — LOW: dispatch switch over LSP operations
- [x] `question.ts` — LOW: prompt wrapper
- [x] `skill.ts` — LOW: skill tool adapter
- [x] `todo.ts` — LOW: todo persistence wrapper
- [x] `invalid.ts` — LOW: invalid-tool fallback
- [x] `plan.ts` — LOW: plan file operations
`batch.ts` was removed from `src/tool/` and is no longer tracked here.
Tool-specific migration guidance and checklist live in `tools.md`.
## Effect service adoption in already-migrated code
@ -298,11 +252,7 @@ Some already-effectified areas still use raw `Filesystem.*` or `Process.spawn` i
`util/filesystem.ts` is still used widely across `src/`, and raw `fs` / `fs/promises` imports still exist in multiple tooling and infrastructure files. As services and tools are effectified, they should switch from `Filesystem.*` to yielding `AppFileSystem.Service` where possible — this should happen naturally during each migration, not as a separate sweep.
Current raw fs users that will convert during tool migration:
- `tool/read.ts` — fs.createReadStream, readline
- `file/ripgrep.ts` — fs/promises
- `patch/index.ts` — fs, fs/promises
Tool-specific filesystem cleanup notes live in `tools.md`.
## Primitives & utilities
@ -344,47 +294,14 @@ For each service, the migration is roughly:
- `ShareNext` — migrated 2026-04-11. Swapped remaining async callers to `AppRuntime.runPromise(ShareNext.Service.use(...))`, removed the `makeRuntime(...)` facade, and kept instance bootstrap on the shared app runtime.
- `SessionTodo` — migrated 2026-04-10. Already matched the target service shape in `session/todo.ts`: single namespace, traced Effect methods, and no `makeRuntime(...)` facade remained; checklist updated to reflect the completed migration.
- `Storage` — migrated 2026-04-10. One production caller (`Session.diff`) and all storage.test.ts tests converted to effectful style. Facades and `makeRuntime` removed.
- `SessionRunState` — migrated 2026-04-11. Single caller in `server/routes/session.ts` converted; facade removed.
- `Account` — migrated 2026-04-11. Callers in `server/routes/experimental.ts` and `cli/cmd/account.ts` converted; facade removed.
- `SessionRunState` — migrated 2026-04-11. Single caller in `server/instance/session.ts` converted; facade removed.
- `Account` — migrated 2026-04-11. Callers in `server/instance/experimental.ts` and `cli/cmd/account.ts` converted; facade removed.
- `Instruction` — migrated 2026-04-11. Test-only callers converted; facade removed.
- `FileTime` — migrated 2026-04-11. Test-only callers converted; facade removed.
- `FileWatcher` — migrated 2026-04-11. Callers in `project/bootstrap.ts` and test converted; facade removed.
- `Question` — migrated 2026-04-11. Callers in `server/routes/question.ts` and test converted; facade removed.
- `Question` — migrated 2026-04-11. Callers in `server/instance/question.ts` and test converted; facade removed.
- `Truncate` — migrated 2026-04-11. Caller in `tool/tool.ts` and test converted; facade removed.
## Route handler effectification
Route handlers should wrap their entire body in a single `AppRuntime.runPromise(Effect.gen(...))` call, yielding services from context rather than calling facades one-by-one. This eliminates multiple `runPromise` round-trips and lets handlers compose naturally.
```ts
// Before — one facade call per service
;async (c) => {
await SessionRunState.assertNotBusy(id)
await Session.removeMessage({ sessionID: id, messageID })
return c.json(true)
}
// After — one Effect.gen, yield services from context
;async (c) => {
await AppRuntime.runPromise(
Effect.gen(function* () {
const state = yield* SessionRunState.Service
const session = yield* Session.Service
yield* state.assertNotBusy(id)
yield* session.removeMessage({ sessionID: id, messageID })
}),
)
return c.json(true)
}
```
When migrating, always use `{ concurrency: "unbounded" }` with `Effect.all` — route handlers should run independent service calls in parallel, not sequentially.
Route files to convert (each handler that calls facades should be wrapped):
- [ ] `server/routes/session.ts` — heaviest; uses Session, SessionPrompt, SessionRevert, SessionCompaction, SessionShare, SessionSummary, SessionRunState, Agent, Permission, Bus
- [ ] `server/routes/global.ts` — uses Config, Project, Provider, Vcs, Snapshot, Agent
- [ ] `server/routes/provider.ts` — uses Provider, Auth, Config
- [ ] `server/routes/question.ts` — uses Question
- [ ] `server/routes/pty.ts` — uses Pty
- [ ] `server/routes/experimental.ts` — uses Account, ToolRegistry, Agent, MCP, Config
Route-handler migration guidance and checklist live in `routes.md`.

View file

@ -0,0 +1,66 @@
# Route handler effectification
Practical reference for converting server route handlers in `packages/opencode` to a single `AppRuntime.runPromise(Effect.gen(...))` body.
## Goal
Route handlers should wrap their entire body in a single `AppRuntime.runPromise(Effect.gen(...))` call, yielding services from context rather than calling facades one-by-one.
This eliminates multiple `runPromise` round-trips and lets handlers compose naturally.
```ts
// Before - one facade call per service
;async (c) => {
await SessionRunState.assertNotBusy(id)
await Session.removeMessage({ sessionID: id, messageID })
return c.json(true)
}
// After - one Effect.gen, yield services from context
;async (c) => {
await AppRuntime.runPromise(
Effect.gen(function* () {
const state = yield* SessionRunState.Service
const session = yield* Session.Service
yield* state.assertNotBusy(id)
yield* session.removeMessage({ sessionID: id, messageID })
}),
)
return c.json(true)
}
```
## Rules
- Wrap the whole handler body in one `AppRuntime.runPromise(Effect.gen(...))` call when the handler is service-heavy.
- Yield services from context instead of calling async facades repeatedly.
- When independent service calls can run in parallel, use `Effect.all(..., { concurrency: "unbounded" })`.
- Prefer one composed Effect body over multiple separate `runPromise(...)` calls in the same handler.
## Current route files
Current instance route files live under `src/server/instance`, not `server/routes`.
The main migration targets are:
- [ ] `server/instance/session.ts` — heaviest; still has many direct facade calls for Session, SessionPrompt, SessionRevert, SessionCompaction, SessionShare, SessionSummary, Agent, Bus
- [ ] `server/instance/global.ts` — still has direct facade calls for Config and instance lifecycle actions
- [ ] `server/instance/provider.ts` — still has direct facade calls for Config and Provider
- [ ] `server/instance/question.ts` — partially converted; still worth tracking here until it consistently uses the composed style
- [ ] `server/instance/pty.ts` — still calls Pty facades directly
- [ ] `server/instance/experimental.ts` — mixed state; some handlers are already composed, others still use facades
Additional route files that still participate in the migration:
- [ ] `server/instance/index.ts` — Vcs, Agent, Skill, LSP, Format
- [ ] `server/instance/file.ts` — Ripgrep, File, LSP
- [ ] `server/instance/mcp.ts` — MCP facade-heavy
- [ ] `server/instance/permission.ts` — Permission
- [ ] `server/instance/workspace.ts` — Workspace
- [ ] `server/instance/tui.ts` — Bus and Session
- [ ] `server/instance/middleware.ts` — Session and Workspace lookups
## Notes
- Some handlers already use `AppRuntime.runPromise(Effect.gen(...))` in isolated places. Keep pushing those files toward one consistent style.
- Route conversion is closely tied to facade removal. As services lose `makeRuntime`-backed async exports, route handlers should switch to yielding the service directly.

View file

@ -0,0 +1,99 @@
# Schema migration
Practical reference for migrating data types in `packages/opencode` from Zod-first definitions to Effect Schema with Zod compatibility shims.
## Goal
Use Effect Schema as the source of truth for domain models, IDs, inputs, outputs, and typed errors.
Keep Zod available at existing HTTP, tool, and compatibility boundaries by exposing a `.zod` field derived from the Effect schema.
## Preferred shapes
### Data objects
Use `Schema.Class` for structured data.
```ts
export class Info extends Schema.Class<Info>("Foo.Info")({
id: FooID,
name: Schema.String,
enabled: Schema.Boolean,
}) {
static readonly zod = zod(Info)
}
```
If the class cannot reference itself cleanly during initialization, use the existing two-step pattern:
```ts
const _Info = Schema.Struct({
id: FooID,
name: Schema.String,
})
export const Info = Object.assign(_Info, {
zod: zod(_Info),
})
```
### Errors
Use `Schema.TaggedErrorClass` for domain errors.
```ts
export class NotFoundError extends Schema.TaggedErrorClass<NotFoundError>()("FooNotFoundError", {
id: FooID,
}) {}
```
### IDs and branded leaf types
Keep branded/schema-backed IDs as Effect schemas and expose `static readonly zod` for compatibility when callers still expect Zod.
## Compatibility rule
During migration, route validators, tool parameters, and any existing Zod-based boundary should consume the derived `.zod` schema instead of maintaining a second hand-written Zod schema.
The default should be:
- Effect Schema owns the type
- `.zod` exists only as a compatibility surface
- new domain models should not start Zod-first unless there is a concrete boundary-specific need
## When Zod can stay
It is fine to keep a Zod-native schema temporarily when:
- the type is only used at an HTTP or tool boundary
- the validator depends on Zod-only transforms or behavior not yet covered by `zod()`
- the migration would force unrelated churn across a large call graph
When this happens, prefer leaving a short note or TODO rather than silently creating a parallel schema source of truth.
## Ordering
Migrate in this order:
1. Shared leaf models and `schema.ts` files
2. Exported `Info`, `Input`, `Output`, and DTO types
3. Tagged domain errors
4. Service-local internal models
5. Route and tool boundary validators that can switch to `.zod`
This keeps shared types canonical first and makes boundary updates mostly mechanical.
## Checklist
- [ ] Shared `schema.ts` leaf models are Effect Schema-first
- [ ] Exported `Info` / `Input` / `Output` types use `Schema.Class` where appropriate
- [ ] Domain errors use `Schema.TaggedErrorClass`
- [ ] Migrated types expose `.zod` for back compatibility
- [ ] Route and tool validators consume derived `.zod` instead of duplicate Zod definitions
- [ ] New domain models default to Effect Schema first
## Notes
- Use `@/util/effect-zod` for all Schema -> Zod conversion.
- Prefer one canonical schema definition. Avoid maintaining parallel Zod and Effect definitions for the same domain type.
- Keep the migration incremental. Converting the domain model first is more valuable than converting every boundary in the same change.

View file

@ -0,0 +1,96 @@
# Tool migration
Practical reference for the current tool-migration state in `packages/opencode`.
## Status
`Tool.Def.execute` and `Tool.Info.init` already return `Effect` on this branch, and the built-in tool surface is now largely on the target shape.
The current exported tools in `src/tool` all use `Tool.define(...)` with Effect-based initialization, and nearly all of them already build their tool body with `Effect.gen(...)` and `Effect.fn(...)`.
So the remaining work is no longer "convert tools to Effect at all". The remaining work is mostly:
1. remove Promise and raw platform bridges inside individual tool bodies
2. swap tool internals to Effect-native services like `AppFileSystem`, `HttpClient`, and `ChildProcessSpawner`
3. keep tests and callers aligned with `yield* info.init()` and real service graphs
## Current shape
`Tool.define(...)` is already the Effect-native helper here.
- `init` is an `Effect`
- `info.init()` returns an `Effect`
- `execute(...)` returns an `Effect`
That means a tool does not need a separate `Tool.defineEffect(...)` helper to count as migrated. A tool is effectively migrated when its init and execute path stay Effect-native, even if some internals still bridge to Promise-based or raw APIs.
## Tests
Tool tests should use the existing Effect helpers in `packages/opencode/test/lib/effect.ts`:
- Use `testEffect(...)` / `it.live(...)` instead of creating fake local wrappers around effectful tools.
- Yield the real tool export, then initialize it: `const info = yield* ReadTool`, `const tool = yield* info.init()`.
- Run tests inside a real instance with `provideTmpdirInstance(...)` or `provideInstance(tmpdirScoped(...))` so instance-scoped services resolve exactly as they do in production.
This keeps tool tests aligned with the production service graph and makes follow-up cleanup mostly mechanical.
## Exported tools
These exported tool definitions already exist in `src/tool` and are on the current Effect-native `Tool.define(...)` path:
- [x] `apply_patch.ts`
- [x] `bash.ts`
- [x] `codesearch.ts`
- [x] `edit.ts`
- [x] `glob.ts`
- [x] `grep.ts`
- [x] `invalid.ts`
- [x] `ls.ts`
- [x] `lsp.ts`
- [x] `multiedit.ts`
- [x] `plan.ts`
- [x] `question.ts`
- [x] `read.ts`
- [x] `skill.ts`
- [x] `task.ts`
- [x] `todo.ts`
- [x] `webfetch.ts`
- [x] `websearch.ts`
- [x] `write.ts`
Notes:
- `batch.ts` is no longer a current tool file and should not be tracked here.
- `truncate.ts` is an Effect service used by tools, not a tool definition itself.
- `mcp-exa.ts`, `external-directory.ts`, and `schema.ts` are support modules, not standalone tool definitions.
## Follow-up cleanup
Most exported tools are already on the intended Effect-native shape. The remaining cleanup is narrower than the old checklist implied.
Current spot cleanups worth tracking:
- [ ] `read.ts` — still bridges to Node stream / `readline` helpers and Promise-based binary detection
- [ ] `bash.ts` — already uses Effect child-process primitives; only keep tracking shell-specific platform bridges and parser/loading details as they come up
- [ ] `webfetch.ts` — already uses `HttpClient`; remaining work is limited to smaller boundary helpers like HTML text extraction
- [ ] `file/ripgrep.ts` — adjacent to tool migration; still has raw fs/process usage that affects `grep.ts` and `ls.ts`
- [ ] `patch/index.ts` — adjacent to tool migration; still has raw fs usage behind patch application
Notable items that are already effectively on the target path and do not need separate migration bullets right now:
- `apply_patch.ts`
- `grep.ts`
- `write.ts`
- `codesearch.ts`
- `websearch.ts`
- `ls.ts`
- `multiedit.ts`
- `edit.ts`
## Filesystem notes
Current raw fs users that still appear relevant here:
- `tool/read.ts``fs.createReadStream`, `readline`
- `file/ripgrep.ts``fs/promises`
- `patch/index.ts``fs`, `fs/promises`