* Studio: harden the data-recipe and inference consumer loops against pump death
Follow-up to #6643. The same single-unsupervised-consumer pattern the training
pump had lives in two sibling loops, with the same failure mode: one bad event
kills the only thread that updates the in-memory state every UI surface reads,
while the worker subprocess keeps running.
- data_recipe JobManager._pump_loop: a malformed worker log line that makes
parse_log_message raise no longer kills the pump. Guard _handle_event, the
queue read, and the worker-exit finalize, and broaden _drain_queue so a drain
error still finalizes the job instead of leaving it wedged "active" (which also
leaked the workflow-scoped API key until its 24h expiry).
- inference InferenceOrchestrator._dispatcher_loop: guard the routing body so a
malformed response or a mailbox put error can't kill the dispatcher and hang
every in-flight generation (callers key liveness on the subprocess, not on
this thread).
Adds regression tests for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: extend consumer-loop hardening to RAG, hub, auth, and stream-reader paths
Continuation of the data-recipe and inference pump hardening: the same
"background producer updates in-memory state that a single unsupervised
consumer surfaces to the UI" pattern shows up in several more Studio paths,
each able to silently freeze a UI surface while the worker keeps running.
RAG ingestion SSE (core/rag/ingestion.py):
- job_events polled the queue with a blocking get and never noticed client
disconnect or a dead worker, so a closed tab or a producer that died
without emitting a terminal event left the stream hanging. It now polls
with a timeout, emits heartbeats, ends on terminal job status, caps idle
time, and always pops the job registry in finally.
- Added _reap_finished_jobs() and call it from start_ingestion so finished
job state does not accumulate.
Startup reconcile (storage/rag_db.py, main.py):
- reconcile_orphaned_ingestion_jobs() marks ingestion jobs (and their
documents) that were left non-terminal by a previous crash as failed, so
the UI does not show jobs stuck "running" forever after a restart. Wired
in at startup next to cleanup_orphaned_runs().
Hub download watcher (hub/services/download_lifecycle.py):
- _watch() could leave a job pinned "running" if finalize raised. Body is
now guarded: on failure it logs and sets the job to error, and always
invalidates the hf cache scan in finally.
External provider stream (core/inference/external_provider.py):
- read timeout was None (no stall ceiling); set to 300s so a wedged
upstream surfaces as an error instead of an indefinitely hung stream.
Auth store (auth/storage.py):
- Enable WAL + busy_timeout on the auth DB so token validation (read on
every request) and login writes stop serialising on the rollback journal.
Matches studio_db / rag_db / providers_db.
Login rate limiter (routes/auth.py):
- _LOGIN_IP_BUCKETS could grow unbounded under spoofed-IP traffic; cap it
and prune stale buckets, mirroring the per-account bucket handling.
Training progress SSE (routes/training.py):
- Break promptly on client disconnect instead of waiting for the next
yield to fail on a closed socket, matching the export / data-recipe SSE
routes.
llama-server stdout drain (core/inference/llama_cpp.py):
- Broaden the drain guard so an unexpected decode/read error logs at debug
and stops the drainer cleanly instead of escaping the thread.
Frontend stream readers (chat-api.ts, rag-api.ts):
- Wrap the SSE read loops in try/finally + reader.cancel() so early return
([DONE]), thrown errors, and consumer aborts release the reader lock
instead of holding it until GC.
Tests:
- test_training_progress_stream_nan: fake request now implements the async
is_disconnected() the route polls, matching the other SSE route fakes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address Codex review feedback on the consumer-loop hardening
Four follow-ups from the automated review, all on code this PR introduced:
- Data-recipe pump (manager.py): a queue read that keeps raising an error
outside the read's narrow catch set (e.g. a broken queue pipe after the
child died) hit the `continue` guard and skipped the dead-worker finalize
below, spinning forever and leaving the job wedged "active" with its
workflow key unretired. On a read failure, fall through to finalize when
the worker is no longer alive. Added a regression test.
- RAG ingestion SSE (ingestion.py): the 5-minute idle cap could end the
stream while the job was still pending/running (a large document spends
minutes in embedding/storing with no per-batch progress event). The route
then sends [DONE], and the client treats a no-terminal-frame end as
completion, marking the document indexed mid-ingestion. Drop the idle cap:
while the worker is alive and non-terminal we keep heartbeating; the stream
ends only on terminal DB status, the None sentinel, or client disconnect.
- Login rate limiter (auth.py): the per-IP path pruned but then added the
new IP unconditionally, so a spoofed-source-IP spray kept _LOGIN_IP_BUCKETS
unbounded and made every new IP pay a full-dict prune scan. Gate the add on
the cap, mirroring the account path.
- Hub download watcher (download_lifecycle.py): if finalize raised before it
reaped (proc.wait) and dropped the worker (e.g. an I/O error draining
stderr), the crash path published a terminal state while the live Popen
stayed registered and kept writing the cache, and the terminal set_job let
claim() admit a retry on the same repo. Terminate + drop the worker before
setting the terminal state.
* Studio: keep login throttling working when the per-IP bucket dict saturates
Review follow-up. The previous cap fix skipped creating a bucket for a new IP
once _LOGIN_IP_BUCKETS was full, returning ip_fails=0. Under a sustained spray
that also fills the account dict, every failure from such an IP then looked
first-seen and _login_blocked had no bucket to enforce, so the cap effectively
disabled throttling once saturated.
Bound the dict with a FIFO eviction instead: if the IP is new and the dict is
full, reclaim expired buckets (rate-limited so a burst of distinct IPs can't
make each failure an O(n) sweep) and, if still full, evict the oldest-inserted
IP. The new IP always gets a real bucket, so a saturating (e.g. spoofed
X-Forwarded-For) spray stays throttled while memory stays bounded. Added a
regression test that saturates the dict and asserts a later IP is still blocked.
* Studio: address Codex review (RAG queue lifecycle, stream error, orphan chunks)
Three follow-ups on the Phase 6 changes:
- RAG ingestion SSE (ingestion.py): job_events removed the per-job queue in its
finally on ANY exit, including an early client disconnect while the worker is
still running. That dropped the worker's later events (the queue is the only
one _emit writes to) and made a reconnect find no queue and receive only
[DONE], which the client treats as completion. Only drop the queue on a
terminal exit (None sentinel / terminal DB status); leftover terminal queues
are still swept by _reap_finished_jobs. Added queue-lifecycle tests.
- External provider stream (routes/inference.py): once the 300s read timeout can
fire, the stream's except path failed the monitor but ended without an error
frame or [DONE], so the chat client saw a bare EOF and saved the timed-out
answer as a successful partial with no error. Emit an SSE error frame (and
[DONE]) on stream failure so the client surfaces it.
- RAG startup reconcile (storage/rag_db.py): marking a half-ingested document
failed left its chunks/fts/vec rows intact, and retrieval filters by scope not
status, so a failed document could still be retrieved and cited. Purge the
document's chunks when reconciling it to failed (the doc row stays for
re-ingest).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: release the remaining SSE stream readers (training, data-recipe, export)
reviewer.py follow-up. The chat and RAG SSE readers were wrapped in
try/finally + reader.cancel(), but the other three readers built on the same
response.body.getReader() pattern were left without it: streamTrainingProgress,
streamRecipeJobEvents, and streamExportLogs leak the ReadableStreamDefaultReader
lock (held until GC) when the consumer aborts, returns early, or a parse/callback
throws. Wrap each in try/finally + reader.cancel() (export already had a
try/catch, so it only needed the finally). All five frontend SSE readers now
release the reader symmetrically.
* Tighten resilience comments and docstrings
Condense the verbose explanatory comments and internal-helper docstrings added
in this branch to shorter, clearer forms. Comment/whitespace only; verified no
code changed via AST diff. No behaviour change.
* Studio: keep chunks for completed docs during ingestion reconcile
Startup reconciliation flips orphaned (non-terminal) ingestion jobs to failed and
purges the document's chunks so a failed source can't be retrieved. But it dropped
the chunks unconditionally, so a document the worker had already committed as
'completed' before the crash (only its job row left non-terminal) lost every chunk
while still reporting 'completed'. That leaves an empty source that retrieval can't
return and dedup (status != 'failed') blocks from re-ingest.
Only purge chunks when the document UPDATE actually transitions it to failed; an
already-completed document keeps its chunks. Adds reconcile regression tests for
both the completed-doc and genuine in-flight-orphan cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: drop a finished RAG job's queue when the client disconnects
job_events kept the per-job queue until it consumed the None sentinel, so a UI
that stops on the terminal event (its reader.cancel aborts the stream before
[DONE]) left the queue registered until the next _reap_finished_jobs sweep; a
batch of uploads followed by idling retained them all.
_run writes the terminal DB status before emitting the terminal event, so on
generator exit, drop the queue when the job's DB row is already terminal (worker
done, nothing to resume) and keep it only while the worker is still running. Adds
a disconnect-after-terminal-event regression test.
* Remove stray async task output files committed by mistake
* Studio: harden login IP throttle and end progress stream on disconnect
Two Codex review items:
Login per-IP throttle: when the per-IP bucket dict saturated, FIFO eviction could
drop a still-hot (blocked) bucket, so an IP could flood the dict with distinct
(or spoofed) source IPs to push out its own bucket and retry as first-seen. Stop
evicting hot buckets; a new IP that can't fit now shares a bounded overflow
counter that still trips the per-IP threshold, so a saturating spray stays
throttled and no live counter is reset.
Progress SSE: on client disconnect the polling loop only broke and fell through
to the unconditional final 'complete' frame, so a buffered or proxying consumer
could read a still-active run as completed. Return from the generator instead.
Adds regression tests for both (spray cannot reset a hot bucket; disconnect while
active emits no complete frame).
* Studio: shard the login overflow counter and stop cancelling chat stream after [DONE]
Two Codex review items:
Login throttle overflow: the single shared overflow counter meant that once a
saturating spray pushed it past the per-IP threshold, _login_blocked returned 429
for every new unbucketed source IP, before credentials were checked -- a global
login denial. Shard the overflow into a fixed array of counters keyed by hash(ip),
so a hot shard only throttles the IPs that map to it while a single source's
repeated failures still concentrate in one shard and stay throttled. Memory stays
bounded and no live bucket is evicted. Adds a regression test that a hot overflow
shard does not block an unrelated IP.
Chat stream: the reader.cancel() in the SSE finally fired even after a natural
[DONE]/EOF. The backend finalizes its api-monitor entry right after yielding the
sentinel (the local pass-through finishes after the last yield), so a client
cancel there can be observed as a disconnect and mark a completed request as
cancelled. Track natural completion and only cancel on an early/abnormal exit.
(No frontend unit test: the Studio frontend has no test harness.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: give prep-timeout test fakes an is_disconnected method
The progress stream now ends on client disconnect (await request.is_disconnected()
before falling through to the terminal frame). After merging that into the
prep-timeout tests added later on main, their _FakeRequest/_ReconnectRequest must
provide is_disconnected or the generator raises AttributeError under CI.
* Studio: keep the login overflow throttle when bucket capacity frees up
_login_blocked only consulted the per-IP overflow shard while the bucket dict was
still at capacity. If a slot freed before the 60s window expired (e.g. another
IP's successful login calls _clear_login_bucket), a source counted in a hot shard
stopped being blocked and its next failure got a fresh per-IP bucket, resetting
the throttle the overflow path exists to preserve. Always max in the IP's shard
(shards are empty outside saturation, so it is a no-op in the common case). Adds a
regression test that a hot source stays throttled after a bucket frees.
* Studio: clear a login IP's overflow throttle on successful login
_clear_login_bucket reset the per-IP and per-account buckets on a successful
login but not the overflow shard, so after the dict saturated and an IP was
counted in overflow, a later successful login left those entries behind and the
next failed attempt could immediately return 429.
Store overflow entries as (timestamp, ip) so a source is throttled by its own
count within the shard (also removing cross-IP collateral within a shard), and
drop just that IP's entries in _clear_login_bucket. Adds a regression test that a
successful login clears the overflow throttle.
* Studio: bound the login overflow shard memory under high-cardinality spray
The per-IP overflow tracked failures in a time-pruned deque of (timestamp, ip)
tuples, so a spoofed-X-Forwarded-For spray of distinct one-off IPs grew memory and
the per-check scan with request cardinality for the whole window -- undermining
the bucket cap that exists to bound memory. Replace each shard with a fixed-
capacity dict (ip -> [count, window_start]): O(1) lookups, and when a shard is
full a one-off IP evicts the lowest-count entry (Space-Saving) so memory is hard-
bounded while a persistent attacker keeps a high count and is never evicted. Adds
a regression test that shards stay within the per-shard cap under a 5000-IP spray.
* Studio: purge chunks for already-failed docs during ingestion reconcile
The reconcile chunk-purge was gated on the documents UPDATE actually flipping a
non-terminal doc to failed. A doc the worker had already marked 'failed' before
the crash (job row left non-terminal) was not re-flipped, so its committed chunks
were kept and stayed retrievable/citable, since retrieval filters by scope not
status. Purge chunks whenever the document is not 'completed' (failed, in-flight,
or gone), preserving the completed-doc carve-out. Adds a regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: don't inherit an evicted IP's count onto a new overflow source
When a full overflow shard evicted the lowest-count entry, the new source
inherited that count (Space-Saving base + 1). If a shard was saturated with hot
entries, an unrelated new IP could land at/over the threshold and be 429'd after a
single attempt -- cross-IP collateral despite the per-source-isolation intent.
New entries now start clean at count 1; the only cost is that a heavy hitter that
is the lowest-count entry in a fully saturated shard can briefly reset, which is
preferable to blocking a bystander. Adds a regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry overflow failures into a new IP bucket on transition
_login_blocked took max(per-IP bucket, overflow shard) rather than combining them,
so a source could log (threshold-1) failures in overflow during saturation and,
once a bucket slot freed, another (threshold-1) in a fresh bucket within the same
window -- roughly doubling the per-IP limit. When a saturated-era IP first gets a
real bucket, migrate its windowed overflow count into that bucket (and drop the
overflow entry) so the combined failures throttle at the intended limit. Adds a
regression test.
* Studio: reconcile a completed doc's orphaned job to completed, not failed
When a crash left an ingestion job non-terminal after its document was already
committed as completed, reconcile marked the job failed. After restart the upload
UI has no in-memory SSE queue and falls back to getJob(), which treats a failed
job as an indexing failure and removes/toasts a document that is actually
searchable. Mark the job completed (keeping its chunks) when its document is
completed. Extends the completed-doc reconcile test to assert the job status.
* Studio: clamp the overflow failure count migrated into a login bucket
A saturated source could accrue an unbounded overflow count, then materialize
one deque entry per recorded failure when a bucket slot freed, allocating an
arbitrarily large deque under the login lock. Only at-or-above the per-IP
threshold matters for blocking, so cap the count there at the record and take
sites; the migration is now bounded without weakening the limit.
* Studio: keep the RAG job stream alive on a transient status read
The heartbeat poll read the job row unguarded; a momentarily-locked DB would
raise out of job_events, which the SSE route turns into a terminal error frame,
and the UI drops a document whose worker is still running. Treat a failed status
read as non-terminal: heartbeat and retry, and keep the queue so a reconnect can
resume.
* Studio: set busy_timeout before journal_mode on the auth DB
Switching journal_mode needs a lock, so if a refresh-token write already holds
one, journal_mode=WAL raises SQLITE_BUSY and the shared try leaves the
connection on SQLite's default zero lock wait. Set busy_timeout first so the
switch waits instead of failing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: use an isolated Node.js for the frontend build instead of replacing the system Node/npm
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address Node isolation review (no-Node probe crash, PATH refresh, OXC provisioning, venv python, runtime node resolver)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust Node isolation for PR #6533
* Studio Node: don't cache a negative node resolution; accept Node metadata in setup.sh ownership guard
- node_runtime: memoize only a version-adequate executable so a Node installed
by a separate-process 'studio update' is picked up without a backend restart.
- setup.sh: _studio_owned_adoptable also accepts UNSLOTH_NODE_PREBUILT_INFO.json,
matching the setup.ps1 Node ownership guard (custom-home parity).
* Studio setup.ps1: skip OXC npm install gracefully when npm is absent
Mirror setup.sh's `command -v npm` guard so a pip-installed Studio with no
system Node skips the OXC runtime install (validator degrades at runtime) instead
of exit 1 aborting the whole setup. Tighten test_node_probe_guard.ps1's probe
regex so it only matches the two system-version probes, not this new npm guard.
* Wire test_node_probe_guard.ps1 into Windows CI for PR #6533
* Harden isolated Node install and probes for PR #6533
- install_node_prebuilt.py: keep an existing, still-usable isolated Node
when nodejs.org's dist index is unreachable instead of aborting the
update on a transient outage (existing_install_usable + tolerant fetch).
- install_node_prebuilt.py: pin NPM_CONFIG_PREFIX/npm_config_prefix and
drop NODE_PATH in _run_node so any npm -g stays inside the isolated
prefix; Windows npm otherwise writes to %APPDATA%\npm.
- install_node_prebuilt.py: resolve tar hard-link targets against the
archive root (symlink targets stay link-parent relative).
- setup.ps1: wrap the system node/npm probes in try/catch so a present
but broken shim degrades to the bundled Node instead of aborting setup.
- setup.ps1: run the isolated Node install with the handed-off/venv Python
(ReusedSetupPython); the main resolver runs later and bare python may be
a Store stub this early.
- setup.sh: log when the OXC validator runtime is skipped for missing npm,
matching setup.ps1.
- node_runtime.py: move the version-floor comment onto _version_meets_floor.
- Tests for the offline-reuse and broken-shim paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim verbose comments across the Studio Node installer for PR #6533
Comments-only pass: collapse the multi-line section banners to single lines,
drop comments that restate obvious code, and tighten the remaining docstrings
and "why" notes without losing intent. No code changes (verified with an AST
comment-only check on the Python files and a non-comment-diff scan on setup.sh
and setup.ps1). Net 109 fewer lines; the install, decision, and probe-guard
suites stay green.
* Harden Node install from review: validated Python, version floor, legacy home, lock race
For PR #6533, addressing the latest review pass:
- setup.ps1: run the isolated Node install with the validated reused/venv Python.
An incompatible reused interpreter (old venv, conda, stale UNSLOTH_SETUP_PYTHON)
is no longer used; fall back to the resolved python instead.
- setup.ps1: a STUDIO_HOME/UNSLOTH_STUDIO_HOME override equal to the legacy default
now uses the legacy sibling node dir (~/.unsloth/node), matching the runtime
resolver and setup.sh, so OXC can find the Node it installed.
- install_node_prebuilt.py: reject an explicit --node-version below the floor
(^20.19 || >=22.12 || >=23) instead of installing a Node the build cannot use.
- install_node_prebuilt.py: atomically rename a stale install lock before unlinking
so two concurrent runs without filelock cannot both acquire it.
Tests added for the version floor (parametrized + explicit-below-floor rejection).
Full install suite: 937 passed, 1 skipped; setup.ps1 parses; decision tests green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address latest review: armv7l + later-fetch offline reuse for PR #6533
- install_node_prebuilt.py: reject 32-bit ARM (armv7l) up front. Node 24 LTS
ships no linux-armv7l build, so the old path failed late with a confusing
"no sha256"; it now fails fast with a clear unsupported-architecture error.
- install_node_prebuilt.py: extend the offline-reuse fallback to the SHASUMS and
archive fetches. If index.json resolves a newer Node but a later download fails
and a usable isolated Node is already on disk, keep it instead of aborting a
non-force update.
Tests added: armv7l/armhf are unsupported; a SHASUMS failure keeps an existing
usable Node and re-raises when none is present. Full install suite: 941 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add UNSLOTH_STUDIO_HOME node-dir tests (install side + resolver) for PR #6533
* Add regression tests pinning the reuse path read-only and isolating installer writes
Lock in the two invariants behind the isolated-Node design: reusing a good
system Node never mutates the user's Node/npm, and the installer's own npm
calls only ever write inside its install_dir.
- tests/studio/install/test_install_node_prebuilt_logic.py: assert _run_node
redirects NPM_CONFIG_PREFIX/npm_config_prefix into install_dir and drops an
inherited NODE_PATH; assert _ensure_npm_floor scopes the npm self-upgrade to
install_dir (never -g against the system) and is a no-op once npm meets the floor.
- tests/sh/test_system_node_readonly.sh (new, wired into studio-backend-ci.yml):
the setup.sh NODE_SOURCE=system arm runs no global install and sets no
NPM_CONFIG_PREFIX, with a positive control that the bundled arm does.
- tests/studio/test_node_decision.ps1: symmetric structural guard that the prefix
pin and the only global install (bun) live in the bundled branch, not the system arm.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Reap Studio child processes when the parent dies abnormally
Standalone `unsloth studio` launches orphaned cloudflared and llama-server when
the parent exited without running the cooperative shutdown path (terminal-window
close, Task Manager End Task, SIGKILL): the children reparented to init and kept
running, leaving an authenticated Cloudflare tunnel up for days.
Add utils/process_lifetime.py: a parent-owned Windows Job Object
(JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, children auto-inherit) plus Linux
PR_SET_PDEATHSIG, behind a best-effort helper that mirrors the desktop app's
windows_job.rs. initialize_parent_lifetime() runs at the top of run_server;
long-lived spawns (cloudflared, llama-server, RAG embedder, llama.cpp updater)
get the PDEATHSIG preexec, multiprocessing workers are adopted into the job, and
_graceful_shutdown plus atexit gain a terminate_all() backstop sweep. The
cooperative shutdown path is otherwise unchanged.
Verified on Linux: killing the parent now reaps cloudflared and llama-server
within ~2s instead of orphaning them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: add real Windows kill-on-job-close integration test
Spawn a parent that installs the job and a child that inherits it, terminate
the parent, and assert the child is reaped. Skipped off Windows. Also make the
liveness probe Windows-safe (os.kill(pid, 0) terminates on Windows).
* Fix Win64 handle truncation in the Job Object calls
Set explicit argtypes so the 64-bit job/process handles are not marshaled as
c_int (which truncated them on Win64, failing AssignProcessToJobObject). Assert
install success in the Windows integration test.
* Bind multiprocessing workers to parent death; harden the sweep
Review follow-ups:
- Multiprocessing workers (inference/export/training/data-recipe/Xet) cannot be
given a preexec_fn by the parent, so adopt_pid alone left them orphanable on a
Linux SIGKILL. They now bind themselves with PR_SET_PDEATHSIG at startup via
bind_current_process_to_parent_lifetime(), wired into the shared
run_without_native_path_secret entrypoint and the Xet child entry.
- Wire the previously-missed data-recipe worker through adopt_pid.
- terminate_all now honors its timeout: SIGTERM, wait, then SIGKILL the
survivors, so cooperative children can exit cleanly.
- Track adopted pids with a /proc starttime identity and add forget_pid, so the
shutdown sweep never signals a recycled pid.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Michael Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Trim and tighten code comments and docstrings across studio/ Python. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.
* Studio: harden stdio MCP gating and fix transport edge cases
- Gate the Data Recipe stdio path behind UNSLOTH_STUDIO_ALLOW_STDIO_MCP so a hosted deployment cannot spawn local processes through recipes
- Enforce the gate inside _client() so the transport sink cannot spawn when disabled
- keep_alive=False so stdio probes/calls do not leave orphan subprocesses
- Force OAuth off for stdio servers on create and update
- Drop stored headers when a server switches transport type
- Reject a command whose first token is a URL scheme
- Add MCP gate and improvement tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests: skip Data Recipe stdio tests when data_designer is absent
The data_designer plugin is only installed in the Studio test job, so guard
the two build_mcp_providers tests with importorskip so the core matrix skips
them instead of failing on ModuleNotFoundError.
---------
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat(recipes): round-trip local model variants
* feat(recipes): add local model selector
* feat(recipes): wire selector into model editors
* fix(recipes): clear stale model state on relink
* feat(recipes): load selected local models for jobs
* chore(frontend): simplify biome scripts
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(recipes): handle local selector edge cases
* fix(recipes): polish local model selector behavior
* fix(recipes): delay local model restore until terminal runs
* fix(recipes): accept resolved default gguf variants
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* ci: add advisory lockfile supply-chain audit
Adds a fast, focused workflow that scans every checked-in npm and
cargo lockfile on PRs touching one. Default behaviour is advisory:
only public indicator-of-compromise strings, versions on the public
known-malicious list, and structurally broken lockfiles fail the
build. Structural anomalies (missing integrity hashes, non-default
registry, etc.) surface as :⚠️: annotations without gating
merges, so reviewers see the audit result inline on every PR
without changing the existing install behaviour.
Also commits the two missing npm lockfiles the audit needs:
studio/package-lock.json (Tauri CLI holder for desktop release)
and studio/backend/core/data_recipe/oxc-validator/package-lock.json
(oxc-parser runtime for the data-recipe validator). studio/setup.sh,
studio/setup.ps1, build.sh, and pyproject.toml are intentionally
left alone so the existing install path keeps working unchanged.
Audit script behaviour:
default mode -> exits 1 only on blocked-known-malicious,
known-ioc-string, malformed-lockfile,
missing-lockfile, unreadable-lockfile, or
missing-toml-parser
--strict -> promotes every finding to blocking (opt-in)
Adds a try/except around lockfile reads so a permissions error
prints a finding instead of crashing CI with a raw traceback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test(security): update cargo regression test for advisory mode
`scripts/lockfile_supply_chain_audit.py` now classifies
`non-registry-cargo-source` as an advisory finding by default
(returns exit 0 with a `:⚠️:` annotation) rather than
unconditionally blocking with exit 1. Update the existing
`test_malicious_cargo_lockfile_refused` to pass --strict so it
keeps verifying the "refuse to install" behavior it is named for,
and add a second test that pins the default-mode behavior:
advisory finding emitted, exit code 0.
* audit: escape Finding for GH Actions annotations
`:⚠️:` and `::error::` workflow commands truncate the
annotation message at the first newline unless the message is
%-encoded per the workflow-commands spec. Since `Finding.__str__`
returns three lines (kind+path, package, detail), the package
and detail fields were being dropped from the GitHub Actions UI.
Add a `_gha_escape()` helper that applies the spec'd escapes
(`%` -> `%25`, then `\r` -> `%0D`, then `\n` -> `%0A`; the `%`
replacement must happen first so the subsequent escapes are not
double-encoded), wrap every Finding rendered into a workflow
command with it, and pin both the helper and the end-to-end
single-line emission with two new regression tests.
Caught by gemini-code-assist on PR #5604.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat(studio): add Tauri native GGUF intake
* feat(studio): polish native GGUF intake
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): load backend helpers during local setup
* fix(studio): acquire native load lease before unload
* Studio: harden native path lease verification and Tauri intake
- Wrap path.resolve(strict=True) and Path.stat() in NativePathLeaseError so a deleted or unmounted GGUF returns 400 instead of leaking the full filesystem path through the generic load_model/validate_model handler.
- Re-apply _reject_network_or_device_path to the resolved canonical path for defense in depth after symlink resolution.
- Replace try/except ValueError pattern in the device-path guard with Path.is_relative_to; the previous shape silently swallowed NativePathLeaseError (which subclasses ValueError) so /dev,/proc,/sys were never actually rejected.
- Broaden the lease redaction regex and dict-key check (Python and Rust diagnostics) to cover both native_path_lease and nativePathLease so the camelCase form emitted by Tauri/frontend payloads is also redacted.
- Hoist the redact_native_paths import to module top in loggers/handlers; the recursive filter no longer pays a per-record import lookup.
- Persist activeNativePathToken in the chat runtime store so the rollback branch can mint a fresh lease and reload the previous native GGUF when a new load fails after unload; clear it in clearCheckpoint and overwrite it on each successful load.
- use-native-drop: read options through a ref so the Tauri onDragDropEvent listener is registered once and stays attached across option changes; reject ambiguous multi-file drops up front instead of silently registering only the first GGUF.
- pick_native_model: use an async pick_file with a tokio oneshot channel instead of blocking_pick_file so the Tokio worker is not held for the duration of the OS dialog.
- registerNativeModelPath: drop the duplicate sourceKind argument; the Rust command parameter is source_kind.
- install_python_stack: insert the script directory (studio/) on sys.path; the previous insert pointed at studio/backend/ which does not satisfy `from backend.utils.wheel_utils import ...`.
* install_python_stack: keep _BACKEND_DIR on sys.path
Restore the studio/backend insertion. Although the immediately following `from backend.utils.wheel_utils import (...)` is satisfied by studio/ already being on sys.path[0] when invoked as `python studio/install_python_stack.py`, wheel_utils itself runs `from utils.native_path_leases import ...`, which requires studio/backend/ to be importable. Without the backend insertion, the existing tests/python/test_install_python_stack.py collection fails with ModuleNotFoundError: No module named 'utils'.
* Studio: tighten native path lease lifecycle and Tauri intake IPC
- register_native_model_path now hardcodes NativePathSourceKind::Drop on the Rust side and the frontend stops sending source_kind. The previous JS payload (source_kind only) never reached the Rust deserializer because Tauri's default ArgumentCase::Camel maps the Rust parameter source_kind to the JS key sourceKind, so drag/drop registration silently failed. Hardcoding the source kind also keeps audit metadata trustworthy on this command.
- Add native_path_secret_removed_for_child_start context manager and wrap multiprocessing.Process.start() at the inference, export, training, and data-recipe job spawn sites. The previous wrapper-only scrub left UNSLOTH_STUDIO_NATIVE_PATH_LEASE_SECRET visible to spawn-platform import-time worker code. The wrapper run_without_native_path_secret stays as defense-in-depth inside the child.
- Stop passing exc_info=True from the native-grant load/validate error logs in routes/inference.py. The structlog filter_sensitive_data processor runs before the renderer, so ConsoleRenderer formatted tracebacks bypassed redaction; the redacted str(e) preserves the message text.
- Replace the os.path.normcase string equality on the resolved canonical path with Path.samefile (with a normcase fallback) so Windows leases that differ only in extended-length \\?\ prefix or short-name spelling are accepted.
- Wrap consumeNativePathToken in its own try/catch in the chat runtime rollback. If the previous native-model token has aged out of TOKEN_TTL we now surface a clear modelsError instead of silently swallowing the rollback inside the outer catch.
- Reject non-ASCII lease strings in _split_lease and convert UnicodeEncodeError / binascii.Error / ValueError raised by _b64decode into NativePathLeaseError so verify_native_path_lease never escapes raw exceptions to the route handler.
- Tighten dropStateForPaths to mark multi-file payloads invalid so the overlay matches the post-fix drop handler that rejects the same payload.
- Replace the one-shot fetch in useNativePathLeasesSupported with a delayed-retry loop so the picker/drop becomes available once the backend is up rather than staying disabled for the rest of the session after a transient failure.
- Drop the unused setActiveNativePathToken setter; the value is set via setState directly in use-chat-model-runtime.
- Add a toast on auto-load failure in use-native-drop so a collapsed model selector does not hide the error.
- Burn the lease nonce before _validate_current_stat so a stat-failed lease is single-use even if a later state change happens to match the original size/mtime.
* Studio: cache lease secret, harden native path stat checks, polish intake UX
- Cache the decoded UNSLOTH_STUDIO_NATIVE_PATH_LEASE_SECRET on first verify and validate that it is base64-decodable and at least 32 bytes. Subsequent _decode_secret calls return from the cache and never touch os.environ, so concurrent /api/inference/load and /api/health requests no longer race with native_path_secret_removed_for_child_start scrubbing the env. native_path_leases_supported now wraps _decode_secret so the health flag matches what verify_native_path_lease actually accepts.
- Replace path.is_file()/is_dir() + path.stat() with os.lstat() in _validate_current_stat and explicitly reject S_ISLNK; size and mtime checks now refer to the link itself, closing the same-size+same-mtime symlink-swap window that the prior follow-symlink stat() left open.
- Add an issued_at_ms < expires_at_ms sanity check in _validate_payload to reject internally inconsistent (HMAC-protected) lease payloads.
- Sort _NATIVE_PATH_REDACTIONS by length (descending) before iterating in redact_native_paths so a longer registered path is replaced before a shorter prefix path; otherwise logs containing /foo/X.gguf.bak after only /foo/X.gguf was registered would leak the .bak suffix.
- classify_existing_path now re-checks the canonical path with symlink_metadata after canonicalize, so a regular file that is replaced with a symlink in the small canonicalize window is rejected at registration.
- ModelSelector renders the local file picker as its own block (not in the eject ternary), so a user with an active model can still replace it via the picker rather than only via drag/drop.
- useNativePathLeasesSupported caps the readiness probe at MAX_READINESS_POLLS (60 = ~5 minutes) and aborts the in-flight fetch on unmount via AbortController, so a permanently-disabled backend stops generating sustained traffic and hot-reload no longer leaks open connections.
- useChooseNativeModel returns a stable useCallback closure and guards the OS dialog with a useRef so rapid double-clicks cannot open multiple dialogs and orphan Rust tokens.
- Branch the multi-file drop toast: if no GGUF was present we say "Only .gguf model files can be dropped here." and otherwise "Drop a single .gguf model file." so users dropping non-GGUF attachments get an accurate explanation.
* native_path_leases: lstat the signed canonical path before resolving
The earlier change to lstat inside _validate_current_stat operates on grant.canonical_path, which is the post-resolve target. If the user atomically replaces the originally-signed file with a symlink to a different file of identical size and mtime, path.resolve(strict=True) follows the symlink, samefile returns True (both ends share the new inode), and the lstat in _validate_current_stat sees the regular target file rather than the symlink, so the swap goes undetected.
Add an os.lstat on the signed canonical path before path.resolve(strict=True), and reject S_ISLNK there. The lstat in _validate_current_stat stays as defense-in-depth for swaps that occur strictly between resolve and stat.
* Studio: scrub native lease secret before mp.Queue spawn and tighten lease lifecycle
- Move _CTX.Queue / _CTX.Event / _CTX.Process construction inside native_path_secret_removed_for_child_start at the inference, export, training and data-recipe spawn sites. The first Queue creation lazily spawns Python's multiprocessing.resource_tracker child, so when it ran outside the scrub context the tracker process inherited the lease secret. Reproduced via the proc filesystem environ entry; the wrapped order keeps the tracker clean.
- native_path_secret_removed_for_child_start now refcounts entries: the env var is popped on the first entry and restored only when the last context exits. Concurrent training/inference/export starts no longer serialize on the env lock across the entire proc.start yield, while still guaranteeing the env stays empty for the duration of every overlapping spawn.
- run_without_native_path_secret now also nulls the module-level cached lease secret. With the existing spawn-only multiprocessing context the cache is irrelevant in practice, but a future fork caller would otherwise inherit the in-memory secret even though the env var was scrubbed.
- filter_sensitive_data now applies the native lease key check on the top-level event_dict, not only on nested dicts, so a logger call that includes a lease value as a top-level keyword field actually redacts it (the bare value does not match the prefix-anchored regex).
- chat-page loadNativeModelIntent now passes intent.id to clearModelIntent so a second drag-drop during an in-flight first auto-load is not wiped from the chip area when the first resolves.
- Bump useNativePathLeasesSupported's MAX_READINESS_POLLS from 60 to 720 so first-run installs that compile llama.cpp from source or download large CUDA wheels (well past 5 minutes) don't permanently disable the native picker.
* native_path_leases: serialize first-decode against scrub context
_decode_secret used a separate _SECRET_INIT_LOCK from the env scrub's _NATIVE_PATH_ENV_LOCK, so the very first decode (before the cache is populated) could race a concurrent native_path_secret_removed_for_child_start and read os.environ during the env-empty window, raising "Native path grants require the managed desktop backend." Subsequent calls hit the cache and were already safe.
Acquire _NATIVE_PATH_ENV_LOCK around the env read inside _SECRET_INIT_LOCK and fall back to _SCRUB_SAVED_SECRET when the scrub has temporarily popped the env var. Lock ordering (init then env) is consistent with no other caller, so no deadlock.
* Studio: surface native model load errors and harden native path label cache
- Native model load and validate now bubble up the actual exception (with
paths redacted) and apply the same friendly-error rewrite the non-native
path uses, so users see "CUDA OOM", "trust_remote_code required", etc.
instead of a generic "Failed to load native model: <label>".
- run_without_native_path_secret now also nulls _SCRUB_SAVED_SECRET so a
forked grandchild that imports native_path_leases cannot recover the
secret via the scrub-aware fallback in _decode_secret.
- _NATIVE_PATH_LABELS now has its own 10000-entry cap independent of the
100-entry redaction list, so display_label_for_native_path no longer
falls back to returning the raw canonical path after 101 native paths
in one session. Redaction list keeps the 100-entry cap for log-scan
performance.
- _validate_payload now also rejects null bytes in display_label, which
is echoed back in HTTP responses and log lines.
* Studio: harden native path lease validation and chained native rollback
- child_env_without_native_path_secret now copies os.environ under
_NATIVE_PATH_ENV_LOCK so a concurrent scrub-context env pop cannot
raise RuntimeError: dictionary changed size during iteration in a
background hardware scan or other env reader.
- _validate_payload and grant construction route every signed numeric
field (version, issued_at_ms, expires_at_ms, size_bytes, modified_ms)
through new _required_int / _optional_int helpers that wrap raw int()
ValueError into NativePathLeaseError. The single upstream catcher
produces 400 instead of 500 for malformed signed payloads.
- verify_native_path_lease now runs _validate_current_stat before
_consume_nonce, so a transient stat error on the canonical path no
longer permanently burns the nonce. Concurrent verifies still
serialize through _consume_nonce, so single-use is preserved.
- Chained native model rollback now restores activeNativePathToken in
the chat runtime store after a successful rollback loadModel. Without
this, a second consecutive failed switch could not re-roll-back
because the store token had been overwritten by the failed attempt.
- validate_model now applies the same not_supported_hints friendly
rewrite to native model errors that load_model already does, so a
native .gguf that fails validation with an upstream "is not supported"
message gets the same actionable wording as the non-native branch.
* Studio: harden native path log redaction, status disclosure, and chip lifecycle
- structlog processor chain now runs format_exc_info before
filter_sensitive_data so traceback strings are produced (and then
redacted) rather than passed through as untouched (type, value, tb)
tuples that the JSON or console renderer formats after the redaction
filter has already finished.
- native_path_secret_removed_for_child_start clears _CACHED_LEASE_SECRET
in addition to popping the env var, so a fork during the scrub window
cannot inherit the cached bytes via the parent's heap. Parent verify
calls during the window keep working through the existing scrub-aware
fallback in _decode_secret.
- load_model's except ValueError handler now redacts native paths and
uses the native model log label when native_grant_backed is true.
Previously a ValueError raised after lease verification (e.g. from
ModelConfig.from_identifier or downstream GGUF parsing) returned the
raw exception string in the HTTP response body.
- llama_cpp_backend now records the native display label at GGUF load
time, and /api/inference/status prefers it over the redaction store.
After a Python backend restart the redaction store is empty; the
attribute keeps the friendly label, and an absolute model_identifier
with no other label source falls back to the basename so the canonical
path no longer appears in active_model.
- reveal_path_token uses native "reveal and select" commands on macOS
(open -R) and Windows (explorer /select,) so the file is highlighted
in the file manager. Linux keeps the existing parent-directory open.
- Native model rollback that fails because the previous token cannot be
consumed now throws a rollback-specific Error, and the outer empty
catch was replaced with one that re-throws the rollback error. The
rollback-specific message now reaches the user instead of being
overwritten by the original load error message.
- NativeModelChip tracks the Rust token's expiresAtMs on a single
setTimeout, disables the Load button at expiry, and relabels it
"Select again" with an explanatory tooltip so users do not click into
a guaranteed-failure path after the 15-minute TTL elapses.
* Studio: tighten native artifact policy, mmproj sibling check, and intake UX
- is_open_safe_artifact no longer grants Open for directories. Reveal
already handles directory navigation, so the change closes the
attack surface where a macOS .app artifact could be launched via
open_path_token + open::that_detached.
- Display labels are sanitized in classify_existing_path. Control
characters in filenames (newlines, tabs, NUL et al.) are replaced
with spaces and the label is trimmed and capped, so a file named
with embedded newlines cannot inject forged log lines or scramble
the UI status panel.
- validate_entry_path skips the size_bytes/modified_ms equality check
when the operation is Reveal or Open. Cloud-sync agents (Dropbox,
iCloud Drive, OneDrive) routinely rewrite extended-attribute
metadata which bumps mtime, and the user expects Reveal/Open to
remain available for files in synced folders.
- llama_cpp_backend gains a _native_grant_backed flag at GGUF load
success. /api/inference/status only applies the absolute-path
basename fallback when that flag is true, so a non-native absolute
local GGUF still reports its canonical model_identifier and unload
by identifier keeps working.
- Native vision GGUFs now run through _validate_native_mmproj_companion
before llama-server starts: the companion mmproj must be a regular
file, not a symlink, and must live in the same resolved directory as
the granted GGUF. This stops a hostile sibling or symlinked mmproj
from being loaded under a single-file lease.
- Chained native rollback restructured: the rollback loadModel + state
+ refresh runs inside its own try/catch that swallows so the outer
throw error surfaces the ORIGINAL load failure. The native-token
consume-failure case still throws the rollback-specific message
early, before the inner block runs, so its actionable guidance is
preserved.
- Loading-model state and the duplicate-load guard in the chat runtime
hook now compare both the model id and the native path token. Two
drops or picks with the same basename in different folders no longer
silently dedup; the second token is honored.
- chat-page loadNativeModelIntent awaits selectModel before clearing
the pending intent. If selectModel returns early via dedup or
throws, the chip and its token stay so the user can retry instead
of losing the selection.
- NativeModelChip's Reveal button is disabled when the lease has
expired (Rust would reject it anyway), and the Load button label
reads "Expired" instead of "Select again" so the disabled element
no longer promises an action it cannot perform.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: add github_repo seed reader and GitHub Support Bot recipe
Adds a first-party Data Designer seed reader that scrapes GitHub issues,
pull requests, and commits from one or more repositories via the GraphQL
API, and a learning recipe (GitHub Support Bot) that turns those rows into
synthetic support Q&A pairs for fine-tuning.
Backend (new plugin studio/backend/plugins/data-designer-github-repo-seed):
* GitHubRepoSeedSource config: repos, token (falls back to GH_TOKEN /
GITHUB_TOKEN env var), item_types (issues / pulls / commits),
per-resource limit (0 means all), max_comments_per_item.
* Rate-limit-aware GraphQL client (GitHubClient + RepoScraper) shared
across repos; flattens each item into a uniform row with columns
item_type, repo, number, title, body, state, author, created_at,
closed_at, url, labels, comments.
* Registered via the data_designer.plugins entry point.
Frontend:
* New seed_github block variant so the seed node card shows
"GitHub repositories" instead of the generic "Document file"
placeholder, with its own icon and inline summary (repo count +
item-type list).
* Rewritten seed dialog github_repo form: repos textarea pre-filled with
unslothai/unsloth + unslothai/unsloth-zoo, password input for the GH
token, items-per-repo number with an "All" toggle, and the noisier
options (item types, max comments, include comments) tucked under an
Advanced collapsible.
* Local model auto-load on Run: if a recipe uses an is_local provider
and the inference server is not already serving that model, the
executions hook calls /api/inference/load first. Removes the "open
/chat to load a model" prerequisite that users kept tripping on.
* Honor the recipe's run.rows value in the Run dialog (previously the
store reset to 5 regardless of what the template shipped).
Recipe (studio/frontend/src/features/data-recipes/learning-recipes/
github-support-bot.json):
* Defaults to the Local Model provider + unsloth/gemma-4-E2B-it-GGUF.
* Scrapes unslothai/unsloth and unslothai/unsloth-zoo, issues and pulls,
up to 100 items per resource.
* Two LLM blocks: normalized_question (llm-text) rewrites each thread
into a clean support question, support_answer (llm-structured)
produces JSON with answer / diagnosis_questions / cites / confidence.
* Run defaults to 10 rows for a quick smoke test.
Verified end-to-end on a running Studio: card renders, source-data
dialog is pre-populated, All toggle disables the limit input, the
recipe executes and produces rows against a loaded local GGUF.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: improve GitHub recipe support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: speed up GitHub scraper and harden the support-bot recipe
Addresses a perf issue found while demoing the github_repo seed reader:
Scraper is too slow at scale. The PRs GraphQL query pulls deeply nested
fields (reviewThreads, reviews, commits, timelineItems, etc.) so the
page size was pinned at 3 to stay under GitHub's node-count ceiling. 100
PRs meant 34 serial round trips. Added lighter query variants
(PRS_PAGE_QUERY_LIGHT, ISSUES_PAGE_QUERY_LIGHT) that drop the fields the
Studio flatten layer does not use (it only reads title, body, state,
author, labels, comments). With the light query PR pages can safely go
to 25 per page and issues to 50. The plugin scraper now passes
light=True to RepoScraper so Studio always uses the fast path; the heavy
query remains available for other callers.
Recipe defaults are now demo-ready with production knobs called out:
- max_parallel_requests: 1 and max_tokens: 800 so small local models
stay stable when running the support_answer structured column.
- support_answer prompt trimmed to 80-200 words so gemma-4-E2B GGUF can
actually comply with the schema. The canonical 150-300 word codex
prompt is still documented in the node3 markdown note for
production upgrades.
* Studio: rename GitHub recipe to 'GitHub Scraper' and add Easy mode
Changes the recipe framing from a single-purpose 'Support Bot' pipeline
to a general-purpose scraper that produces {user_request,
grounded_response} training pairs. Aligns with the canonical
github_data_gatherer dataset (11 enrichment tasks mirrored in pr_requests_20
/ issue_requests_20 on the input side and explain_pr / issue_fix_plan /
issue_solution on the output side).
Recipe JSON changes:
- columns[0] renamed normalized_question -> user_request, prompt now
inverts a GitHub thread into a realistic user ask instead of
normalising it.
- columns[1] renamed support_answer -> coauthor_response, emits
{response, followups, cites, task, confidence} and branches on
issue vs PR thread type.
- Notes rewritten to document the 11-task catalog and the canonical
production prompt to paste in for a full dataset backfill.
Frontend: Easy mode for github_repo recipes. The drag-and-drop canvas is
hidden behind an 'Advanced' tab; Easy mode is the default for any recipe
whose seed_source_type is github_repo. The Easy form reuses the existing
GithubRepoSeedForm (promoted to exported), adds a rows input bound to
previewRows, a model field bound to the model_config, and a single Run
button that calls runPreview() directly (no modal). Non-github recipes
see the same Editor / Runs tabs as before.
View mode persists per-recipe-id in localStorage under
recipe-studio:view-mode:<recipeId>.
* Studio: auto-detect server GH_TOKEN and widen Easy-mode detection
The GitHub seed form now fetches /api/data-recipe/seed/github/env-token
on mount and, when the server exposes a GH_TOKEN / GITHUB_TOKEN env var
and the token field is blank, shows a small 'Using server env var' badge
and swaps the placeholder text. The token value itself is never returned
to the UI.
Widens Easy-mode detection in recipe-studio-page.tsx so that recipes
saved before ui.seed_source_type was persisted also get the Easy tab:
falls back to recipe.seed_config.source.seed_type, which is always
present for github_repo seeds.
* fix: polish GitHub recipe UI
* Studio: default llama-server --threads to -1 (auto)
Previously we passed --threads only when the caller set an explicit
value, which meant llama-server fell back to its internal default.
That default has varied across llama.cpp builds (some versions use
hardware concurrency including hyperthreads, which hurts throughput on
CPU-heavy inference). Always passing --threads -1 pins the behaviour
to llama.cpp's auto-detect (physical cores).
Caller-supplied n_threads still wins when non-None.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch Easy mode to Runs pane on run start
Easy mode had no progress island or canvas overlay, so after clicking Run
the only visible state was the button label flipping to "Running..." while
the screen otherwise stayed identical. This reads as stuck even though the
job is progressing.
Wire an onExecutionStart callback from recipe-studio-page.tsx through to
useRecipeExecutions so that when a run is kicked off from easy mode, the
page flips to the executions view where the Runs sidebar, progress bar,
rate/ETA panel, and live log are rendered. Advanced/editor mode keeps its
existing behavior and stays on the canvas (it already has the floating
ExecutionProgressIsland).
* fix: clean up GitHub scraper layout
* Studio: forward llm-structured output_format as llama-server response_format
Local GGUF runs of llm-structured columns used to generate the full
max_tokens budget before the prompt-level "return JSON in a ```json
fence" instruction got parsed. Small models (e.g. gemma-4-E2B-it)
routinely broke format, so each row took ~65s and frequently failed
with "No parsable JSON structure within ```json markdown fence".
For any local-provider model_config referenced by an llm-structured
column, clone the model_config and inject response_format into the
clone's inference_parameters. Uses llama.cpp server's flat shape
(tools/server/README.md):
{"type": "json_schema", "schema": <output_format>}
Not the OpenAI-nested form; data_designer's OpenAI adapter forwards
response_format verbatim via facade._COMPLETION_REQUEST_FIELDS, and
llama-server's documented schema path expects the flat variant.
The clone is per (model_alias, column) so:
- llm-text / llm-judge columns that share the same alias keep
free-form sampling.
- Each structured column gets its own schema, so columns with
different output_formats don't collide.
Effect on gemma-4-E2B-it demos: every row parses cleanly, and the
model terminates immediately after the closing brace instead of
running to max_tokens. Net wall-clock is usually faster even though
grammar-constrained sampling is slightly slower per token.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: flip Easy to Runs pane before validation scrape, not after
Previously onExecutionStart fired inside runExecution, which runs AFTER
validateRecipe() -- and validation re-invokes the seed reader. For the
github_repo reader that is a full GraphQL scrape, so the user sat on a
"Running..." button with an otherwise unchanged Easy form for 10-15s
before anything moved.
Call onExecutionStart at the top of runWithValidation, right after we
have a payload to send. The view flips immediately; ensureLocalModelLoaded
+ validateRecipe now run against the Runs pane instead of a frozen Easy
form. runExecution still calls onExecutionStart downstream, but the
callback is idempotent (the page's easy -> executions guard skips the
second call), so no behaviour change for runs that pass validation.
If validation fails the toast + runErrors path still fires; the Easy
form's error banner still reads runErrors when the user switches back.
* Studio: unify data-recipe workflow auth on sk-unsloth-* keys
The previous commit (a61b4cc9) assumed storage.create_api_key(..., internal=True)
and storage.revoke_internal_api_key(key_id) existed, but those helpers were
only in the working tree, never committed. Recipe runs in local-model mode
were therefore crashing with 500 when _inject_local_providers tried to mint
a workflow key. This commit ships the missing pieces.
auth/storage.py:
- api_keys schema gains is_internal INTEGER DEFAULT 0 (with a guarded
ALTER TABLE migration so existing auth.db files upgrade in place).
- create_api_key takes an internal=False kwarg; internal keys are flagged
so they can be hidden from user-facing listings.
- list_api_keys takes include_internal=False so UIs never see workflow keys.
- New revoke_internal_api_key(key_id): id-only revoke for keys minted by
non-user subjects (the JobManager does not know a username).
core/data_recipe/jobs/manager.py:
- JobManager.start accepts internal_api_key_id and stores it on Job so
lifecycle handlers can revoke eagerly.
- _handle_event revokes on EVENT_JOB_COMPLETED / _ERROR / _CANCELLED.
- _pump_loop subprocess-died fallback also retires the key so a crashed
worker cannot leak a live sk-unsloth-* beyond its TTL.
- Revocation is best-effort (swallow exceptions) -- the 24h TTL is the
safety net if storage hiccups.
core/data_recipe/jobs/types.py:
- Job dataclass gains internal_api_key_id: int | None = None.
Replaces the bespoke 24h JWT path that jobs.py used to mint for local
providers. One mint/revoke/verify surface for every API key the server
issues, and revocation is now eager (seconds, not 24h) instead of TTL-only.
* Studio: plug workflow-key leak on unexpected create_job errors
Review follow-up on the sk-unsloth-* workflow-key lifecycle in
create_job. Previously the revoke handlers wrapped mgr.start(...) but
only caught RuntimeError and ValueError, and get_job_manager() sat
outside the try block entirely. Any other exception type (TypeError
from a mismatched kwarg, OSError from the queue write, etc.) would
bubble up to FastAPI and leave the minted key live until its 24h TTL.
Fix: one try block covers both get_job_manager() and mgr.start(), with
a trailing except Exception that revokes and re-raises. The
RuntimeError -> 409 and ValueError -> 400 paths are unchanged so
specific client-facing status codes still surface. Revocation is still
best-effort (_revoke_internal_api_key_safe swallows errors) because we
never want revoke failures to mask the original crash.
Severity is low -- the key can't bootstrap longer access and the 24h
TTL bounds the window -- but the reviewer's point stands: eager revoke
on every failure path is the right invariant.
* Studio: nest response_format under extra_body so pydantic accepts it
The previous commit dropped response_format at the top level of a cloned
model_config's inference_parameters, which BuilderConfig rejected with:
ValidationError: Extra inputs are not permitted [type=extra_forbidden]
data_designer.model_configs.1.inference_parameters.response_format
data_designer's BaseInferenceParams is a pydantic model with extra=forbid
and only a fixed set of fields (temperature, top_p, max_tokens,
max_parallel_requests, timeout, extra_body). The pass-through path for
anything the schema doesn't know about is `extra_body`, which the
OpenAI SDK spreads into the chat-completions request body at the top
level -- which is exactly where llama-server reads response_format from.
Inject under extra_body (merging with any existing extra_body contents)
so the clone validates. llama-server still receives
{"type": "json_schema", "schema": <output_format>} at the top level of
the request body, which is the flat shape llama.cpp's server expects.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: forward response_format to llama-server and fence-wrap the reply
Two-part fix for the llm-structured data-recipe path:
(1) The /v1/chat/completions proxy was dropping response_format. The
route's passthrough branch only triggered on tools / tool messages, so
requests carrying a JSON schema fell into the non-passthrough GGUF path
which calls generate_chat_completion (no response_format kwarg). The
schema never reached llama-server, so guided decoding was a no-op and
the model emitted free-form text that happened to parse a fraction of
the time. Widen the passthrough trigger and teach _build_passthrough_payload
to forward response_format so llama-server's GBNF grammar actually runs.
Guided decoding does not require supports_tools, so split the condition:
a request is now passthrough-routed if it carries tools/tool messages
(existing behavior) OR carries response_format (new). The vision guard,
streaming fork, and tools-choice defaulting are unchanged.
(2) data_designer's llm-structured parser looks for a ```json ... ```
markdown fence and discards anything else. Guided decoding emits only
the JSON object (the GBNF grammar has no fence tokens), so a
100%-valid schema-constrained run still ended up 0 ok / N failed with
"No parsable JSON structure within ```json markdown fence". In
_openai_passthrough_non_streaming, wrap each choice's content in the
expected fence when the caller asked for guided decoding. Already-fenced
content is left alone so other clients that prefer raw JSON are not
affected; the wrap is scoped to requests that carried response_format.
Net effect on the GitHub Support Bot recipe on a local GGUF: schema
actually binds during sampling, content arrives wrapped in the fence
data_designer expects, and generation terminates immediately after the
closing brace instead of running out to max_tokens.
* Studio: Easy mode runs a full run, capped at the user's row count
Easy mode used to call runPreview, which produces a test run: no
artifact persisted, reduced progress tracking, and framed in the Runs
pane as "Test run". The whole point of the form is to let a user kick
off a real dataset build with one click, so wire it to runFull instead
and bind the Rows input to fullRows (not previewRows).
runFull requires a non-empty fullRunName. The Easy form has no run-name
input, so seed a default on mount whenever Easy is active and
fullRunName is still empty. Uses `<recipe name> <iso-timestamp>` so
each Easy run gets a stable-ish default that still sorts chronologically
in the Runs pane. User can override it from the Advanced run dialog
before clicking Run.
Rename GithubScraperEasyView's rows props from previewRows/setPreviewRows
to rows/setRows so the view stays agnostic to which hook state the page
chooses to bind. Loading indicator now follows fullLoading.
* Studio: clamp GitHub scrape page size and memoize the materialization
Two wins for the "before Generating fires" gap on small previews:
(1) scrape_{issues,prs,commits} hardcoded per_page (50 / 25 / 100) and
only checked the trial limit AFTER the page was written, so a 1-row
Easy run still asked GitHub for a full 50-issue + 25-PR page, wrote
them all to JSONL, and then stopped because total_new already exceeded
the trial cap. Cap per_page at min(page_cap, trial_limit) so
github_limit=1 actually asks for first:1.
(2) GitHubRepoSeedReader.get_dataset_uri used to scrape fresh on every
invocation. data_designer calls the seed reader multiple times per
recipe job (validation, preview, per-column sampling), so a 2-repo
Easy preview ran the full GraphQL scrape three times back-to-back,
burning ~15s of dead air before any LLM generation began.
Added a module-level in-process cache keyed on
(repos, item_types, limit, include_comments, max_comments_per_item,
sha256(token)[:16]) that stores the JSONL path of the first
materialization. Subsequent calls with the same signature return the
cached path, guarded by a staleness check that drops the entry if the
file was tmp-cleaned. Raw token values never land in the key.
Net effect on a 1-row Easy run, 2 repos, limit=1: 2 GraphQL round
trips instead of ~12, and the first-to-Generating gap collapses from
~15s to roughly 2-3s.
* Studio: make Easy mode Rows input editable instead of snapping to 1
The Rows to generate input used type="number" with value bound directly
to the rows state and an onChange that coerced any non-positive parse
result back to 1. The moment the user pressed backspace to clear the
field, the parent re-rendered with value=1 and the caret jumped, making
it impossible to change the value without arrowing the browser's +/-
spinner.
Switch to a text input with inputMode="numeric" and pattern="[0-9]*"
(so mobile still shows a numeric keyboard, and the browser drops the
spinner buttons the user did not want). Add a local rowsText buffer so
the field can hold transient empty / partial digit strings while
editing without fighting the parent state; the canonical rows value
only advances when the buffer parses to a valid integer in [1, 10000],
and onBlur clamps back to 1 or 10000 if the user left it out of range.
No behavior change for valid numeric edits - the downstream runFull()
still sees a clean positive integer.
* Studio: expand dataset cells horizontally by column on click
Click a long cell to expand that whole column. Click again to collapse.
Replaces the prior row-level vertical expansion which made it hard to
compare cells across columns. State is scoped per execution and per
column; the row itself is no longer a click target.
* Studio: force expanded dataset column to grow wide enough to read
* Studio: disable thinking for local recipe inference and plumb the kwarg
Reasoning-capable models (gemma-3n, qwen3.5, etc.) emit a
<think>...</think> preamble ahead of the answer by default, which
roughly doubles the generated token count per row on a local GGUF
and pushes the actual answer past data_designer's json-fence regex
on llm-structured columns. Recipes want the terse answer, not the
scratchpad.
Two halves of the fix:
(1) routes/data_recipe/jobs.py: when _inject_local_providers walks
the recipe's model_configs to point them at the local endpoint, also
stash chat_template_kwargs={"enable_thinking": false} under each
config's inference_parameters.extra_body. OpenAI SDK spreads
extra_body into the top-level request body, so llama-server and the
Studio /v1/chat/completions route both see it.
(2) routes/inference.py: the chat-completions route previously
dropped chat_template_kwargs on the floor because the whitelist
body builder only forwarded known fields.
- At the top of openai_chat_completions, lift
chat_template_kwargs.enable_thinking from payload.model_extra
onto the typed payload.enable_thinking field when the caller
did not set the latter, so the non-passthrough GGUF path's
generate_chat_completion(...) call honors the override.
- Teach _build_passthrough_payload to forward a
chat_template_kwargs dict, and have _build_openai_passthrough_body
derive that dict from payload.enable_thinking so
response_format requests (structured columns) also land at
llama-server with the reasoning preamble suppressed.
Net effect on a 10-row support-bot run with gemma-4-E2B-it-GGUF:
responses arrive without <think> tags, wall-clock per call drops
roughly in half, and structured columns stop leaking reasoning
tokens through the GBNF-constrained output.
* Studio: update GitHub Support Bot learning recipe with maintainer layout
Replace the template with the hand-laid-out export from the maintainer
so note nodes ship with real x/y positions (scattered around the
graph instead of all stacked at x=480) and the edges / canvas pan look
correct on first load. Also picks up the maintainer's prompt tweaks and
output schema names (coauthor_response / user_request / followups / task /
cites / confidence).
Diff is mostly ui.nodes positions and prompt bodies; runtime shape is
unchanged (seed_config / columns still target model_1 against the Local
Model provider).
* Studio: auto-size dataset sample columns; wide text gets a wide column
Drop the per-column click-to-expand toggle and the 180-char truncation.
Every column now renders its full value. Columns with long text get a
min-w of 48rem so the text is readable without wrapping into a tall
block; narrow-content columns get a 12rem min-w. The table wrapper
already has overflow-x-auto, so wide-column totals cause a horizontal
scrollbar instead of cramming everything into the viewport.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix GitHub scrape progress
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add resetApiBase export for test setup
* Studio: rename github-support-bot output columns to User / Assistant
Previously emitted user_request and coauthor_response, which did not
match the canonical User / Assistant chat-pair shape that downstream
SFT consumers expect. Renamed the columns in the recipe JSON (columns,
UI node ids, edges, notes, prompt Jinja refs) and the matching copy in
the learning-recipes index, data-recipes-page, and easy view.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* add unsloth studio desktop app
* Fix review findings
- studio/src-tauri/tauri.conf.json: retarget updater to staging repo
(danielhanchen/unsloth-staging-2); switch to unslothai/unsloth on upstream merge.
- studio/src-tauri/linux/postremove.sh: drop the interactive read loop and the
/home/* iteration. Package maintainer scripts must stay non-interactive and
must not touch other users' data.
- studio/frontend/src/app/auth-guards.ts: honor tauriAutoAuth() boolean. Failed
auto-auth now redirects to /login; requireGuest/requirePasswordChangeFlow
only redirect to /chat when auth succeeds. The new early-return on failed
auth is intentional so the login / change-password flows remain reachable
when desktop auth is not yet established.
- studio/frontend/src/config/env.ts: keep fetched=false on health failure so
later calls retry instead of caching the client-side platform guess.
- studio/src-tauri/src/install.rs: pick the available system package manager
(apt-get, dnf, zypper, pacman); AppImage bundles run on non-Debian distros.
- studio/frontend/src/lib/open-link.ts + markdown-text/sources callers: return
boolean from openLink so callers only preventDefault on handled URLs; relative
hrefs now navigate natively.
- studio/frontend/src/features/settings/tabs/about-tab.tsx: fetch(apiUrl(...))
so the version request targets the backend port in desktop mode. The bare
/api/health predates the Tauri webview (blame: the earlier onboarding commit,
which ran with same-origin frontend/backend); in desktop mode the webview
origin is tauri://localhost so the bare path fails.
- install.ps1: gate the install_python_stack.py hotfix on a sentinel comment
instead of a content regex; append the sentinel after applying so reruns
are unambiguous.
- unsloth_cli/commands/studio.py _write_auth_secret: use the atomic mkstemp +
os.replace path on Windows too; chmod calls are wrapped in try/except OSError.
- studio/src-tauri/src/preflight.rs probe_existing_backends: fan out the health
probes concurrently; desktop-auth status still runs sequentially per candidate.
reqwest::Client is internally Arc-wrapped so the in-loop .clone() is a
refcount bump, not a deep clone; annotated inline.
- studio/src-tauri/src/preflight.rs run_cli_probe: wait() after kill() to reap
the child, matching probe_cli_capability.
- studio/src-tauri/src/process.rs + main.rs: add stop_backend_detached and use
it from the tray quit handler so the 5s graceful-wait does not block the
Tauri main loop. RunEvent::Exit keeps the synchronous safety-net call.
- studio/backend/main.py: drop the permissive localhost CORS regex in
api-only mode; the explicit allow_origins list is sufficient.
- .github/workflows/release-desktop.yml: drop max-parallel: 1 so platform
builds run in parallel, and lift releaseBody to an env var so the three
tauri-action invocations share one source of truth.
* Fix review findings (loop 2)
- studio/backend/auth/storage.py update_password: clear_desktop_secret()
alongside clear_bootstrap_password() so rotating the admin password
also revokes any previously provisioned .desktop_secret. Without this,
an old local desktop credential keeps minting fresh admin tokens via
/api/auth/desktop-login after a password rotation.
- studio/src-tauri/src/desktop_auth.rs provision_desktop_auth: wrap
cmd.output().await in tokio::time::timeout(30s). DESKTOP_AUTH_LOCK is
held across the whole desktop_auth flow, and previously a hanging
`unsloth studio provision-desktop-auth` subprocess would pin the lock
indefinitely and freeze every subsequent desktop_auth call.
* Add review tests
* Consolidate review tests
Merge review-added tests into the existing studio/backend/tests/test_desktop_auth.py
(the PR's authoritative desktop-auth test file). Drops three scaffolding files under
tests/python/ in favor of five focused tests next to the tests they extend:
- test_update_password_clears_desktop_secret (runtime)
- test_update_password_on_unknown_user_leaves_desktop_secret_intact (runtime)
- test_cli_provisioning_delegates_to_storage_create_desktop_secret (source-level)
- test_cli_connect_auth_db_reads_storage_db_path (source-level)
- test_desktop_auth_provision_has_bounded_timeout (Rust source-level)
* Revert auth-guards.ts Tauri branches to unconditional form
The review loop on PR 5144 introduced a regression: the isTauri branch of
requireAuth redirected to /login when tauriAutoAuth() returned false, and
requireGuest / requirePasswordChangeFlow silently fell through on the same
condition. The Tauri desktop app authenticates via a local auto-generated
secret; it must never surface /login or /change-password to the user. A
failed auto-auth should let the startup layer retry, not expose a password
form.
Restore the three Tauri branches to the author's original unconditional
form (requireAuth: return; requireGuest / requirePasswordChangeFlow: throw
redirect({to: '/chat'})). Keep the rest of the review fixes -- the
apiUrl() fetch wrapping, authRedirect helper, and fetchAuthStatus refactor
are all legitimate improvements and are preserved.
* Revert release-desktop.yml to author's version
The review loop's workflow-file tweaks (drop max-parallel: 1, lift releaseBody
to an env var) are cosmetic. OAuth tokens cannot push workflow-file changes,
and fine-grained PATs cannot honor maintainerCanModify on a third-party fork.
Reverting the workflow file to wasimysaid's version lets the push go through
without needing a classic PAT with both repo and workflow scopes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Daniel Han <unslothai@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: allow non-LLM recipes to run without provider block
* feat: reorder execution tabs and add generation-aware data tab empty state
* fix: add accessibility attrs to data tab spinner and use literal ellipsis
* fix(studio): use shared spinner, stub provider, and hide unused LLM metrics
Backend: inject stub model provider for sampler-only recipes so
DataDesigner init does not reject empty provider lists.
Frontend: use shared Spinner component, hide LLM columns metric
and model usage card when recipe has no LLM columns.
* Fix tab reset and terminal auto-scroll regressions for PR #4805
Reset detailTab to "data" when switching between executions so
the Data tab default is applied consistently, not only on first
mount. Also add detailTab to the terminal scroll effect deps so
auto-scroll-to-bottom fires when the user opens the Overview tab
after landing on Data.
* Guard terminal scroll reset to only fire on Overview tab
The previous scroll effect ran on every tab switch, which could
reset the user's manual scroll position if they scrolled up in
the terminal and briefly switched tabs. Now the scroll-to-bottom
and sticky-bottom reset only fires when navigating to the
Overview tab.
* Use None for stub provider api_key instead of literal string
The stub ModelProvider that satisfies the DataDesigner registry
for non-LLM recipes should not carry a fake credential string.
Using None avoids sending an Authorization header if the provider
is ever inadvertently invoked.
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Strip <think> blocks from LLM assist model output
* Add debug logging for raw LLM assist output
* Quiet llama-server logs, use structlog in llm_assist
* Fix think-tag stripping when response is inside tags
* Remove debug logging of raw model output
* Clarify GGUF download logs: show cache hit vs actual download
* Clarify heuristic-detected mapping in UI text
* Default helper model to Qwen3-4B-Instruct-2507 UD-Q4_K_XL
* Remove package-lock.json from tracking, add to .gitignore
* Auto-open mapping dialog on Start Training for custom_heuristic format
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use last think block when extracting inner content (review feedback)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* miscallenous studio
* chore: upload dataset misc
* chore: redudancy studio cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: adress the pr comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: adress comments about recipes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
- Added `log_lines` field to track and display runtime logs for executions.
- Enhanced progress tracking with terminal-like log outputs and live log scrolling.
- Introduced detailed "model usage" and "dropped columns" analysis in `ExecutionsView`.
- Optimized UI components for displaying dataset metrics, including input/output token averages.
- Added logic to calculate and manage column-level progress for job executions.
- Introduced `progress_columns_total` and `_column_done` fields for more granular progress updates.
- Improved overall progress computation by considering total columns and individual progress per column.
- Extracted shared execution utilities into `execution-helpers.ts` for reusability across features.
- Replaced deprecated `/preview` endpoint and its logic with unified job execution handling.
- Consolidated job execution flows ("Preview" and "Full Run") into shared `runJobExecution` logic.
- Enhanced execution progress tracking with support for column-level progress reporting.
- Added support for handling execution job events and improved error reporting from the backend.
- Updated backend to better manage dataset access errors and provide more informative error messages.
- Cleaned up redundant code in `use-recipe-studio-actions` and streamlined execution APIs.
- Introduced backend changes to handle dataset pagination with limit, offset, and total row support.
- Updated frontend execution view with dataset pagination controls, including "Next" and "Prev" buttons.
- Extended recipe execution logic to manage dataset pagination details like page number, page size, and total records.
- Introduced "Full Run" support in execution logic, including progress tracking, cancellation, and job status updates.
- Extended backend to manage full execution jobs, handle dataset previews, and return detailed analysis and artifacts.
- Updated frontend components to support full runs, with execution sorting, live updates, and detailed execution views.
- Enhanced `ExecutionsView` with progress indicators, status filtering, and dataset preview capabilities.
- Added IndexedDB schema migration to track additional execution metadata.
Daniel Han