* Studio setup.sh: cope with fresh CUDA toolkits like 13.3
CUDA 13.3 shipped today. Three loose ends in studio/setup.sh surfaced
during the llama.cpp build path:
1. setup.ps1 already aborts cleanly when the CUDA toolkit is below
llama.cpp's minimum (12.4) via #4517, but setup.sh still hit the
generic cmake failure described in #4437. Added a min-version check
that downgrades to a CPU build for nvcc < 12.4 with a clear message
pointing to the toolkit archive.
2. The first day a new CUDA toolkit ships, its host-compiler whitelist
lags whatever gcc/clang the distro is on, so nvcc rejects the host
compiler with a wall of "#error -- unsupported GNU version" before
any real compile runs. NVCC_PREPEND_FLAGS now carries
-allow-unsupported-compiler so the build moves on instead.
3. The Linux CUDA/ROCm configure failure path had no symmetry with the
macOS Metal fallback: a single nvcc failure left BUILD_OK=false and
no llama.cpp at all. Generalised the existing Metal -> CPU fallback
to cover any GPU_BACKEND, so a CUDA configure or build failure now
transparently retries with the CPU args and the user still ends up
with a working llama-server.
Pulled the version probe out into _nvcc_meets_llama_minimum so it can
be unit-tested. Added tests/sh/test_nvcc_meets_llama_minimum.sh and two
extra cases in tests/sh/test_get_torch_index_url.sh covering the legacy
"CUDA Version: 13.3" header (driver-reported) and the future 13.7
case. Wired the new test into tests/run_all.sh and the studio-backend
CI workflow.
* tests: relax pr4562 regression to allow generic GPU fallback label
* studio tests: assert setup.sh exports NVCC_PREPEND_FLAGS=-allow-unsupported-compiler
The -allow-unsupported-compiler flag is the core of the fresh-CUDA-toolkit fix
(it lets nvcc accept a host gcc/clang newer than its release-time whitelist, so
CUDA 13.3 day-one builds do not abort on '#error -- unsupported GNU version'),
but it had no automated coverage. Add a source-pattern test asserting the flag
is present, delivered via NVCC_PREPEND_FLAGS so it also covers cmake's CUDA
compiler-id probe, and kept out of CMAKE_ARGS for bash word-splitting safety.
* studio/setup.ps1: allow unsupported host compiler for CUDA build (Windows parity)
Mirror the Linux setup.sh headline fix from this PR on Windows. A freshly
released CUDA toolkit ships with a host-compiler whitelist that lags the
installed toolchain, so nvcc can reject the host with
"#error -- unsupported Microsoft Visual Studio version!" before any real
compile runs (the MSVC analogue of the gcc wall the Linux side hit on
CUDA 13.3). Set NVCC_PREPEND_FLAGS=-allow-unsupported-compiler in the CUDA
build branch so both cmake's configure-time CUDA compiler-id probe and the
cmake --build step proceed. The flag disables the host version check only and
is a no-op when the compiler is already supported.
Set via the process environment (not the $CmakeArgs array), after the
Refresh-Environment calls that re-sanitize CUDA env vars, and appended
idempotently to any value the user already set.
Validated with PowerShell 7.6.2: full setup.ps1 AST parse is clean and the
snippet is idempotent (empty -> set, existing -> append once, no duplicate).
Needs real Windows + CUDA CI to exercise the actual nvcc/MSVC build.
Adds test_setup_ps1_exports_allow_unsupported_compiler asserting the flag is
present, env-delivered, kept out of $CmakeArgs, and scoped to the CUDA-on branch.
* studio: tighten code comments added in this PR
Shorten the verbose multi-line comments and test docstrings introduced by
this PR (setup.sh, setup.ps1, and the shell/python tests) to be succinct
while preserving the rationale. No code or test-assertion changes.
* Studio: keep web search/code pills off on model load if user disabled them
* Studio: avoid redundant localStorage reads when resolving tool pills on load
* fix(studio/colab): merge iframe+keepalive into start(), add proxy_headers to uvicorn
- Move serve_kernel_port_as_iframe and keepalive loop into colab.start()
so both run in the same cell execution context, eliminating the race
where the proxy URL was shown before the iframe cell had a chance to run
- Add a 2s sleep after run_server() before show_link() to give Colab's
proxy infrastructure time to register the bound port
- Add proxy_headers=True and forwarded_allow_ips="*" to uvicorn Config
so X-Forwarded-Proto/Host from Colab's reverse proxy are trusted
- Simplify notebook start cell (no more separate iframe cell needed)
* fix(studio/colab): fix iframe blocking and server thread crash in Colab
Two root causes for the long-standing proxy/iframe breakage:
1. SecurityHeadersMiddleware set X-Frame-Options: DENY and
frame-ancestors 'none' unconditionally, blocking
serve_kernel_port_as_iframe regardless of server health.
Fix: detect Colab via COLAB_BACKEND_URL/COLAB_GPU env vars,
relax frame-ancestors to *.prod.colab.dev and omit X-Frame-Options.
2. asyncio.run() in the daemon thread conflicted with nest_asyncio's
global patches applied on the main thread, causing the server to
crash silently after ready_event fired.
Fix: use explicit new_event_loop() + run_until_complete() in the
daemon thread to bypass nest_asyncio's asyncio.run patch.
Also replace blind time.sleep(2) with a health endpoint poll so the
link and iframe are only shown once the server is truly reachable.
* fix(studio/colab): use reliable /content + google.colab path for Colab detection
COLAB_BACKEND_URL and COLAB_GPU env vars aren't consistently set across
all Colab runtime versions. Use /content dir + google.colab package path
as a more reliable signal, computed once at module load.
* fix(studio/colab): fix port mismatch, health-check silence, and CSP framing
Four bugs causing the iframe and URL button to always fail:
1. Port not propagated back: run_server auto-increments when 8888 is taken,
but start() kept using the original port for show_link() and
serve_kernel_port_as_iframe() — now reads app.state.server_port.
2. Silent health-check failure: the poll loop never checked whether any
attempt succeeded; on all-fail it continued and showed a dead link —
now exits early with a clear error message.
3. CSP frame-ancestors too narrow: '*.prod.colab.dev' only matches one
subdomain level; actual Colab proxy URLs are two levels deep
(e.g. foo.region.prod.colab.dev), and the parent frame may also be
colab.research.google.com or a sandboxed null-origin output iframe —
changed to '*' in Colab mode (single-user sandbox, no security loss).
4. _IS_COLAB detection hardcoded python3.10/3.11 paths: Python 3.12+
Colab runtimes wouldn't match when env vars aren't set — replaced with
a glob over python3.*/dist-packages/google/colab.
* fix(studio/colab): harden Colab startup against every known failure mode
colab.py:
- get_colab_url: retry eval_js up to 3x (10s timeout each), validate that
result is a real https:// URL containing the port before accepting it;
log a clear warning when falling back to localhost
- show_link: safe short_url truncation (try/except around str.index so an
unexpected URL shape never blocks the link card from rendering); also
emit the URL via logger so it's visible in cell text output even if
HTML display is suppressed
- start: detect "already running" at entry — on cell re-run Studio is
still healthy on port 8888; skip re-launch and go straight to
show+iframe so the user never ends up with mismatched port state
- start: wrap run_server in try/except (SystemExit + Exception) so
startup errors surface as readable messages rather than cell crashes
- start: check frontend_path/index.html exists, not just the directory
- start: remove unused `import sys`
- start / keepalive: catch KeyboardInterrupt so interrupting the cell
prints a clean "stopped" message instead of a raw traceback
- extract _is_studio_healthy() and _show_and_embed() helpers to
deduplicate the fast-path and normal-path logic
main.py:
- _build_csp: in Colab mode, extend script-src to include
*.prod.colab.dev and *.googleusercontent.com (Colab injects scripts
from these origins into the output iframe scaffolding)
- _build_csp: in Colab mode, extend connect-src with blob:, data:,
wss://*.prod.colab.dev, and wss://*.googleusercontent.com so
WebSocket streams and Colab kernel traffic are not blocked by CSP
* fix(studio/colab): fix iframe width responsiveness and height sizing
Replace serve_kernel_port_as_iframe with a raw CSS iframe for two
reasons:
1. Width responsiveness: serve_kernel_port_as_iframe sets the width as
an HTML attribute (width="100%") which Colab's output machinery can
bake into a fixed pixel value on first render, causing the Studio to
stop following the notebook panel width when it opens/closes or the
window resizes. A CSS style property (style="width:100%") participates
in normal reflow and always tracks the parent container width.
2. Height sizing: the hardcoded height=1200 was too tall on short monitors
(forced outer-page scroll) and wasted space on tall ones. A small JS
snippet reads screen.availHeight and sets height to ~82% of the screen,
clamped to [600, 1100]px, with a resize listener that re-fits on zoom
changes and panel open/close events.
Also eliminate the double eval_js call: _show_and_embed now fetches the
Colab proxy URL once and passes it to show_link via the new _url kwarg,
so google.colab.kernel.proxyPort is only called once per invocation.
Falls back to serve_kernel_port_as_iframe if IPython.display.HTML is
unavailable for any reason.
* fix(studio/colab): fix link button + add fullscreen hover button to iframe
Link button: target="_blank" is blocked by Colab's output sandbox.
Switch to onclick="window.open(url,'_blank')" which the sandbox allows.
Fullscreen: add a small button that appears on hover in the top-right
corner of the iframe. Clicking it calls requestFullscreen() on the
wrapper div and stretches the iframe to 100vh/100vw. Exits back to
normal on fullscreen change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* revert(studio/colab): remove fullscreen button
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/colab): address review feedback
- Wrap both urlopen calls in with statements to prevent socket/fd leaks
- Replace JS resize listener with CSS height:82vh — simpler, responsive,
and no risk of leaked window listeners on cell re-runs
- Use importlib.util.find_spec("google.colab") instead of a glob path
to detect Colab; more robust across Python versions and venv layouts
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio/colab): fall back to href navigation when window.open is blocked
window.open from a cross-origin sandboxed Colab output iframe can be
silently blocked by the browser (returns null, no exception). The old
code returned false unconditionally, so a blocked popup left the button
doing nothing. Now: if window.open succeeds the new tab opens and the
href is suppressed; if it returns null the browser follows the href,
navigating the output cell to Studio — always does something useful.
* fix(studio/colab): remove button, give iframe a branded header bar
The "Open Unsloth Studio" button was unreliable in Colab's sandboxed
output context regardless of how window.open was called. Since the
iframe already loads Studio inline, the button added no value and
confused users with a URL that 404s outside the output cell.
Replace the separate link card + bare iframe with a single block:
a slim black header bar (Unsloth logo + truncated URL) flush on top
of the full-height responsive iframe. Cleaner and removes the broken
button entirely.
* studio: gate uvicorn proxy_headers/forwarded_allow_ips behind _IS_COLAB
forwarded_allow_ips="*" was applied unconditionally, so every Studio
deployment trusted X-Forwarded-* headers from any client. Only Colab needs
that, because its reverse proxy fronts the kernel. For a normal
local/standalone Studio this is an unwanted relaxation, especially when bound
to 0.0.0.0.
Now proxy_headers/forwarded_allow_ips are only set when _IS_COLAB. Standalone
runs fall back to uvicorn's defaults (proxy_headers honored from loopback
only), restoring the prior security posture, while Colab keeps the wide trust
its proxy requires.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
meta-pytorch/OpenEnv was transferred to huggingface/OpenEnv. The old
URL still works via GitHub's redirect from a clean clone, but uv's git
cache can fail to follow it on some machines — surfacing as a
'could not read Username for github.com' credential prompt mid-install.
Point the requirement at the canonical huggingface/OpenEnv.git (same
HEAD), which also sidesteps any stale cache entry keyed on the old URL.
The completion toast reported only document count. Capture each job's
num_chunks from its complete event into the index-progress store and sum
across the batch, so the toast reads 'N documents and M chunks indexed'.
Already-indexed (deduped) files contribute 0 new chunks.
Uploading several documents (or a folder) produced a separate toast per
file, which piled up. Replace the per-job toast stack with a single
aggregate toast driven by a new index-progress-store that's populated at
addDoc time — so it counts queued files (held by the concurrency
semaphore) in the denominator, which the per-job rag-store map can't see.
- index-progress-store: one entry per file in the batch
(queued/indexing/ready/error + 0..1 progress).
- Both addDoc paths register each file on entry and update it through
the lifecycle (setIndexing after acquiring a slot; setProgress on job
progress events; setReady/setError on terminal).
- ingestion-toast-stack: renders ONE toast — 'Indexing document(s) · X/Y
· Z%' with a progress bar while in flight (overall = completed files +
in-flight fractions, over total), 'RAG index ready · N documents
indexed' (+ failures) when the last finishes, then auto-dismisses.
Single-file uploads still read naturally ('Indexing document' / '1
document indexed').
IngestionProgress / rag-store jobs are untouched (still used by the
KB detail panel). Not build/UI verified here (no bun).
Uploading many docs (or a folder) previously spawned an ingestion
subprocess per file all at once, thrashing the GPU/CPU. Add a
configurable concurrency limit and a folder picker.
- ragIndexConcurrency setting (default 1) in the chat runtime store,
persisted like the other RAG scalar settings; exposed as a 'Parallel
indexing' slider (1-8) at the bottom of the sidebar Retrieval section.
- New rag-index-queue.ts semaphore: each document upload acquires a slot
before it starts and releases it once its ingestion job finishes
(complete / error / already-indexed), so bulk uploads drain at the
configured rate. Wired into both composer upload paths
(use-thread-doc-uploads + shared-composer).
- Folder upload: a second 'Attach a folder' button on the RAG attach
control uses a webkitdirectory input; every compatible file is routed
through the same queue. Multi-file select already worked (the input has
'multiple' and loops addDoc).
- Content-hash dedup (shipped earlier) means re-scanning a folder skips
already-indexed files.
Not build/UI verified here (no bun); needs bun typecheck + a browser
check of bulk/folder upload draining at the set concurrency.
External providers (OpenAI/Anthropic/Gemini) can't run the local
search_knowledge_base tool loop, so give them RAG by prefetching:
studio retrieves before calling the provider, injects the chunks into
the user prompt, and surfaces it as a synthetic tool call. Local models
are untouched (they keep tool-based RAG + decomposition).
Backend:
- New POST /api/rag/prefetch: momentarily loads the pre-cached helper
(gemma-4-E2B-it-GGUF) via LlamaCppBackend(kill_orphans=False) to
decompose the question into up to 3 queries, retrieves+merges+dedups
per query, unloads the helper. Raw single-query fallback if the helper
can't load. New core/rag/query_decompose.py owns the helper lifecycle.
- Factored the retrieval body of /search into _execute_search, reused by
both endpoints.
Frontend:
- prefetchRag() client.
- chat-adapter external branch: gated on isExternalRequest + ragToolEnabled
+ scope!=off + ragScopeHasDocs (no docs -> no prefetch, prior behavior
preserved). Formats hits as <chunk id=N> (parseChunks shape), injects
into the last user message (send-only; not shown in the user bubble),
seeds a synthetic search_knowledge_base tool-call part so the existing
chunk-card UI + [N] citations + source badges all work unchanged.
- Extends PR #5674's disabled-tool guard: when RAG is off, reinforce
'no document search (RAG) capabilities'; when prefetch ran, point the
model at the injected excerpts instead.
- RAG pill enabled for external providers regardless of supports_tools.
Not build/UI verified here (no bun/GPU/keys); needs bun typecheck+test
and a browser round-trip with real provider keys.
RAG retrieval runs entirely through the local search_knowledge_base
tool. If the loaded model doesn't support tool calling (e.g. a
safetensors model whose template advertises tools in an unparsable
emission format, so supports_tools is suppressed), enabling RAG does
nothing — the model never calls the tool. The pill stayed lit and
clickable, which was misleading.
Gate the RAG pill on supportsTools (in addition to modelLoaded), the
same condition web/code use when there's no provider builtin. Applied
to both composer surfaces (shared-composer and the in-thread
RagToggle), with a 'RAG needs a model that supports tool calling'
tooltip on the disabled state.
The backend dedups re-uploads and the sidepanel shows the doc once, but
the composer's pending-doc chips are created per addDoc call, so each
re-upload of an already-indexed file appended another 'Ready' chip for
the same document. In the already_indexed branch, if a chip with the
returned documentId already exists, drop the chip we just added instead
of marking it ready — so the composer shows each document only once.
Re-uploading the same file into the same scope (KB or thread) used to
parse, chunk, caption and embed it all over again, creating a duplicate
set of chunks. Dedup by content hash instead:
- schema: add rag_documents.content_hash (sha256 of the bytes) via the
standard PRAGMA/ALTER migration, plus (scope, content_hash) indexes.
- upload: _save_upload now streams the bytes through sha256 and returns
the digest alongside path/name/size.
- _start_ingestion: before inserting, look for a COMPLETED row in the
same scope with the same hash. If found, delete the redundant upload
from disk and return the existing document_id with already_indexed=
true and an empty job_id — no ingestion job is started. Only
'completed' rows dedup, so a failed/in-flight prior attempt can still
retry. Scope-local: the same file in two KBs is indexed in each.
- frontend: UploadResponse.already_indexed flows through the rag-store
(skips job subscription) into both upload paths, which mark the chip
ready immediately and toast '<file> is already indexed'.
Pre-existing rows have NULL content_hash and won't dedup until
re-uploaded once under the new path. Not build/UI-verified here (no bun
in this env); needs typecheck + browser check.
docx-preview defaults to blob: URLs for embedded images, which DOMPurify
strips from img src (blob: isn't in its default allowed-URI list), so
figures vanished after sanitize. Switch docx-preview to useBase64URL so
images inline as data: URIs, and add ADD_DATA_URI_TAGS: ['img'] to the
DOMPurify config so those data: URIs survive sanitization. Script /
handler / javascript: stripping is unchanged.
Previously a DOCX citation only showed the extracted snippet + a
Download button (Risk #3: never render a user-supplied .docx inline).
Add a faithful inline render that keeps that guarantee:
- New PreviewDocxView renders the .docx with docx-preview into an
off-screen element, then injects DOMPurify-sanitized HTML into the
live DOM (keeping <style> for docx-preview's scoped layout CSS).
Script tags, event handlers and javascript: URLs are stripped, so
a malicious .docx can't execute in the app origin.
- preview-store now fetches the raw bytes for docx and exposes them
via previewBlob, but deliberately keeps previewBlobUrl = null — no
object URL is created, so the 'open raw original inline' path stays
disabled (Risk #3) and Download remains the only raw-file path.
- preview-panel routes docx -> PreviewDocxView when a blob is present,
falling back to the text-view snippet otherwise. isInlineBlobAllowed
still returns false for docx, so html/unknown behaviour is unchanged.
- Deps: docx-preview + dompurify added to package.json.
- Tests updated: docx now asserts bytes-fetched-without-object-URL.
Not build/UI-verified in this environment (deps not installed here);
needs bun install + browser check.
- Add human-readable stage labels for caption_images ('Captioning
images') and extract_images ('Extracting images') so the raw
underscore stage names no longer leak into the progress toast.
- On completion, the toast title is now 'RAG index ready' (was
'Indexed') and the body reads '1 document and N chunk(s) indexed'
(was 'Indexed N chunks'), with chunk pluralization.
CodeQL flagged information exposure through an exception in the /warmup
and /reranker/precache endpoints: both returned str(exc) in the JSON
body, exposing internal paths and stack details to the client. Keep
the full exception in the server-side warning log and return a generic
error message ('Failed to load embedder' / 'Failed to download
reranker') to the caller instead. The frontend only surfaces the
message in a toast, so a generic string is sufficient.
Two changes to the RAG captioning log output:
- Drop the noisy per-image and path-selection info lines
(using-chat-VLM, loading-helper, per-image done). Only the
'caption_images: invoked' and 'caption_images: complete' lines
remain; warnings for genuine failures (helper load, per-image
request, helper unload) are kept.
- Configure structlog at the top of the ingestion subprocess worker
with the same env the parent uses. The worker runs in a spawned
process where structlog was never set up, so its logs fell back to
structlog's dev ConsoleRenderer ([info] ...) instead of the JSON
renderer the rest of the app uses. Now captioner/parser logs from
the subprocess match the parent's JSON format.
The toast stack and the chat settings sheet were both z-50, so an open
side panel (rendered later in the DOM) covered the indexing toast. Bump
the stack to z-[9999] — comfortably above the sheet's z-50 — so the
ingestion toast stays visible like the Sonner reranker toast does.
The thumbnail-rail refactor hoisted <Document> to wrap both the rail and
the main page so the PDF loads once. That moved the width-measuring scroll
container INSIDE <Document>, which only renders its children after the PDF
finishes loading. The old `useEffect(..., [])` ran on component mount —
when the container was still absent — so the ResizeObserver never attached,
`width` stayed null, and the main <Page> collapsed to width 0.
Replace the mount-effect measurement with a callback ref: the
ResizeObserver now attaches the instant the container node mounts,
regardless of when that happens relative to PDF load. Disconnects cleanly
on unmount / re-attach.
Verified: tsc clean, vite build succeeds, preview-pdf-smoke (incl. the
resize/debounce case) passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: honor --ctx-size and other forwarded args from `unsloth studio run` in Studio's context-fit logic
* refactor: extract resolve_requested_ctx as single source of truth
The test helper was reimplementing the two-line
'ctx_override = parse_ctx_override(...); requested_ctx = ctx_override
if ctx_override is not None else n_ctx' pattern locally, so the test
asserted against its own reimplementation rather than production logic.
Extract the conditional into resolve_requested_ctx and have both the
production caller and the test use it.
* fix(studio): honor pass-through cache type flags in KV VRAM estimate
Studio's KV cache VRAM estimate computed from the first-class
cache_type_kv even when the user passed -ctk/--cache-type-k/-ctv/
--cache-type-v via extras. Those flags reached llama-server fine
(last-wins on the CLI) but the pre-launch estimate kept using the
default f16 bytes-per-element, so GPU placement decisions could be
off when the user lowered cache precision via pass-through.
Adds parse_cache_override + resolve_cache_type_kv in llama_server_args.py
(mirroring parse_ctx_override / resolve_requested_ctx), wires both into
load_model alongside the existing ctx resolution, and adds focused
unit tests for the parser + resolver.
Follow-up to @rolandtannous review on #5815.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
assistant-ui mints a fresh __LOCALID_* draft id on every page load, so
RAG docs uploaded under the previous draft id were orphaned after a
refresh — the doc panel queries useThreadDocuments(activeThreadId) and
the new id had nothing.
Persist activeThreadId in localStorage and, on the first settled render
after load, have ActiveThreadSync ask aui to switchToThread(persisted)
when it differs from the freshly-minted draft. Because the draft was
already persisted to the backend by initialize()/ensureThreadRecord
when its first doc was uploaded, the adapter's fetch() resolves it and
aui adopts it as mainThreadId. That keeps aui's mainThreadId and our
activeThreadId unified, so the earlier divergence (uploads under the
persisted id vs chat-completion reading aui's fresh id) can't recur —
unlike the reverted localStorage-only attempt, the chat-adapter's
unstable_threadId now equals the persisted id after the switch.
A one-shot ref ensures we only re-adopt on initial load; user-driven
new-chat / thread switches still flow through normally. If the
persisted draft was never initialized (no doc/message, not in the
backend), switchToThread rejects and we fall back to the fresh draft.
