ba9405b908* Studio: surface prompt-cache token counts in /v1/chat/completions usage chunk
Studio's Anthropic and OpenAI Responses proxies already capture
cache_creation_input_tokens, cache_read_input_tokens (Anthropic) and
input_tokens_details.cached_tokens (OpenAI), but they were only written
to the structlog stream. Browser and SDK clients had no way to compute
"how many tokens hit the prompt cache" without scraping the server log,
so the chat UI could not show users how much money the cache was
saving on each turn.
This change emits one extra OpenAI include_usage-style chunk
(choices: [] with a populated usage block) just before the existing
[DONE] for Anthropic and after the final finish_reason chunk for
OpenAI Responses (both response.completed and response.incomplete).
The chunk shape:
usage.prompt_tokens_details.cached_tokens
normalised cache-read count, present for both providers.
usage.cache_creation_input_tokens
Anthropic-only; tokens billed at the cache-write premium.
usage.cache_read_input_tokens
Anthropic-only; same value as cached_tokens, kept for callers
that already key off the native Anthropic name.
Smoke verified end to end against a live Studio (claude-haiku-4-5
and gpt-4o-mini) plus 7 new unit tests on the helper and the two
streaming paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Anthropic: include cache buckets in prompt_tokens / total_tokens
Anthropic's `input_tokens` field excludes the cache buckets -- the
real prompt size is `input_tokens + cache_creation_input_tokens +
cache_read_input_tokens`. Previously the new usage chunk reported
only `input_tokens` as `prompt_tokens`, which heavily undercounted
cache-hit turns (e.g. an 18.9k-token cache_read turn looked like an
8-token prompt) and broke any downstream context / cost display fed
by `prompt_tokens` or `total_tokens`.
Fix `_build_usage_chunk` to sum all three input buckets for the
Anthropic provider while keeping the OpenAI Responses path unchanged
(OpenAI already folds cached tokens into `input_tokens`). The native
`cache_creation_input_tokens` / `cache_read_input_tokens` keys and
`prompt_tokens_details.cached_tokens` mirror are still emitted, so
clients keep full visibility of the cache split.
Tests updated to assert the summed shape.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
|
||
|---|---|---|
| .. | ||
| assets | ||
| auth | ||
| core | ||
| loggers | ||
| models | ||
| plugins | ||
| requirements | ||
| routes | ||
| state | ||
| storage | ||
| tests | ||
| utils | ||
| __init__.py | ||
| _platform_compat.py | ||
| colab.py | ||
| main.py | ||
| run.py | ||
| startup_banner.py | ||