1dcb9ea8cd
Benchmarking showed llama-server through Studio's web layer runs at 284 TPS vs 357 TPS direct (~20% overhead, +0.72ms per token). The generator code itself has zero overhead -- the bottleneck is entirely in how the route handler and middleware pipeline yield SSE chunks. Three structural changes: 1. Convert LoggingMiddleware from BaseHTTPMiddleware to pure ASGI middleware. BaseHTTPMiddleware routes every response body chunk through an anyio memory channel (two coroutine context switches per chunk). The pure ASGI version wraps the send callable directly. 2. Replace per-token asyncio.to_thread / run_in_executor with a single background thread feeding an asyncio.Queue via call_soon_threadsafe (~2us vs ~50us per token). All three streaming loops (gguf_stream_chunks, gguf_tool_stream, stream_chunks) now use the shared _run_gen_to_queue helper. 3. Pre-format JSON for content-token chunks (the 99% hot path) instead of constructing 3 Pydantic objects and calling model_dump_json per token. Pydantic is still used for the first chunk (role), final chunk (finish_reason), and usage chunk where correctness matters more than speed. Additionally, request.is_disconnected() is now checked every 20 tokens instead of every token. The cancel_event mechanism still provides prompt cancellation since the watcher in llama_cpp.py closes the httpx response on cancel. |
||
|---|---|---|
| .. | ||
| assets | ||
| auth | ||
| core | ||
| loggers | ||
| models | ||
| plugins | ||
| requirements | ||
| routes | ||
| state | ||
| storage | ||
| tests | ||
| utils | ||
| __init__.py | ||
| _platform_compat.py | ||
| colab.py | ||
| main.py | ||
| run.py | ||