* fix(studio/llama_cpp): disable trust_env on the loopback health probe
_wait_for_health() polls http://127.0.0.1:<port>/health with the default
httpx trust_env=True, so an ambient HTTP(S)_PROXY in the environment is
applied to the loopback request. A proxy that returns 503 for 127.0.0.1
makes every probe fail, so the loop runs until timeout and Studio load
hangs (trust_env=False returns 200 immediately).
Pass trust_env=False so the local readiness probe never goes through a
proxy. This mirrors the existing trust_env=False handling in the sibling
llama_http / external_provider HTTP clients.
* test(offline_gguf_cache): accept trust_env kwarg in fake_get mock
_wait_for_health now calls httpx.get(..., trust_env=False); update the retry test's fake_get to accept the kwarg so it doesn't raise TypeError.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/llama_cpp): bypass proxies for loopback clients
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/routes): bypass proxies for llama streams
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: serialize non-streaming responses once and pool the proxy client
Two safe latency wins on the OpenAI/Anthropic-compatible endpoints that leave
the streaming generation paths untouched (they keep Connection: close and
max_keepalive_connections=0 so a client disconnect still stops GPU decode).
1. Non-streaming responses used JSONResponse(content=model.model_dump()), which
builds a dict and then re-runs json.dumps. Serialize once with
model.model_dump_json() via a small _model_json_response helper. The body is
byte-identical (nulls preserved), about 3x faster to encode in a microbench.
2. The non-streaming completions and embeddings proxies built a fresh
httpx.AsyncClient per request. Route them through one pooled client
(core/inference/llama_http) closed on shutdown; streaming generation keeps
its own per-request close-only client. About 5x faster per call to the
local llama-server in a microbench.
The existing API-monitor tests for the non-streaming completions, embeddings
and passthrough paths now patch nonstreaming_client instead of httpx.AsyncClient
to match the pooled client, so they stay deterministic.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make the pooled non-streaming client per event loop
Review follow-up on the shared httpx client. It was a single module-global
instance, which has two lifecycle problems the per-request client did not:
1. After aclose() in lifespan shutdown, nonstreaming_client() kept handing back
the closed client, so a second lifespan in the same process (repeated
TestClient, embedded restart) failed with "client has been closed".
2. An httpx client binds its transport to the loop it first runs on, so reuse
from another loop could raise "Event loop is closed".
Hold one client per running loop in a WeakKeyDictionary, recreate when missing
or closed, and close all on shutdown. Single-loop production is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
