ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Studio: extend llama.cpp first-token timeout (#5841)

Browse source 
* fix: extend llama.cpp first-token timeout

* fix: timeout label pluralization

* studio: distinguish llama stream timeout phases

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix/adjust timeout handling for PR #5841

* Fix lint failure for PR #5841

* Fix/adjust stream timeout handling for PR #5841

* Fix/adjust first token timeout for PR #5841

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix/adjust passthrough timeouts for PR #5841

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix/adjust preheader stream cancellation for PR #5841

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix/adjust timeout PR diff for PR #5841

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix/adjust Python 3.9 stream iteration for PR #5841

* Fix first body timeout for PR #5841

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix first token timeout deadlines for PR #5841

---------

Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
This commit is contained in:
Lee Jackson 2026-06-12 17:41:38 +01:00 • committed by GitHub
commit 31439d9eed
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 524 additions and 206 deletions
Download patch file Download diff file Expand all files Collapse all files

7
studio/backend/tests/test_llama_cpp_tool_loop.py
View file

@ -52,11 +52,16 @@ def _make_backend(monkeypatch, streams: list[list[str]], payloads: list[dict]):
payload,
_cancel_event,
headers = None,
first_token_deadline = None,
):
payloads.append(copy.deepcopy(payload))
yield type("FakeResponse", (), {"status_code": 200, "chunks": streams.pop(0)})()
def fake_iter_text_cancellable(response, _cancel_event):
def fake_iter_text_cancellable(
response,
_cancel_event,
first_token_deadline = None,
):
yield from response.chunks
monkeypatch.setattr(backend, "_stream_with_retry", fake_stream_with_retry)