Two overlapping /chat/completions requests could both read from the shared resp_queue, consuming and dropping each other's token events. Replace the request_id filtering (which silently dropped non-matching messages) with a threading.Lock that serializes generation — correct for single-GPU inference. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| data_recipe | ||
| export | ||
| inference | ||
| training | ||
| __init__.py | ||