09399ab877
The disconnect check is an async round-trip that adds overhead on every loop iteration. Since the cancel watcher in llama_cpp.py already handles connection teardown (closes the streaming response on cancel), this route-layer check is a secondary safety net that does not need to run on every single token. Check every 20 tokens across all 4 streaming paths: - gguf_tool_stream (tool-enabled GGUF) - gguf_stream_chunks (standard GGUF) - audio_input_generate (audio/whisper input) - generic backend stream (non-GGUF fallback) |
||
|---|---|---|
| .. | ||
| backend | ||
| frontend | ||
| __init__.py | ||
| install_llama_prebuilt.py | ||
| install_python_stack.py | ||
| LICENSE.AGPL-3.0 | ||
| setup.bat | ||
| setup.ps1 | ||
| setup.sh | ||
| Unsloth_Studio_Colab.ipynb | ||