e945f43652
Adds a separate streaming server (Option B) that bypasses the main
app's asyncio event loop for GGUF token streaming.
When UNSLOTH_FAST_SSE=1 or UNSLOTH_STREAM_SERVER=1 is set, a
lightweight FastAPI app starts in a daemon thread on a random port.
The frontend negotiates a one-time token via /api/inference/stream-url
and streams directly from this server, falling back to the baseline
/v1/chat/completions transparently on failure.
Architecture:
- streaming_server.py: standalone FastAPI app with three paths:
- Path A (hot): async httpx.AsyncClient streaming direct to
llama-server. No cumulative-to-delta conversion needed since
llama-server sends delta tokens natively via OpenAI SSE.
- Path B: tool calling via asyncio.to_thread (tool execution
is the bottleneck, not the streaming proxy).
- Path C: non-streaming one-shot JSON response.
- stream_token_store.py: thread-safe one-time token store (10s TTL)
shared between the main app and streaming server.
- routes/inference.py: /stream-url endpoint for token issuance,
/direct-stream for Option C, /internal/consume-stream-token for
future subprocess mode.
- llama_cpp.py: --api-key support so the streaming server can
authenticate to llama-server directly.
- chat-api.ts: transparent fast-path negotiation with silent fallback.
Handles: vision (image_url + legacy image_base64), thinking models
(reasoning_content -> <think> tags), CORS preflight, friendly error
messages, stream_options usage/timings passthrough.
Only sends repeat_penalty when the client explicitly provides it,
avoiding the 24% TPS penalty from repetition scanning.
|
||
|---|---|---|
| .. | ||
| public | ||
| src | ||
| .gitignore | ||
| .gitkeep | ||
| biome.json | ||
| components.json | ||
| data-designer.openapi (1).yaml | ||
| eslint.config.js | ||
| index.html | ||
| package.json | ||
| tsconfig.app.json | ||
| tsconfig.json | ||
| tsconfig.node.json | ||
| vite.config.ts | ||