* Studio: free chat model VRAM at training start only when the GPU is tight The training start route unconditionally tore down the transformers/MLX inference subprocess before training, and never stopped the llama.cpp GGUF server at all, so a loaded GGUF chat model kept holding VRAM for the whole run. Conversely the HF model was always unloaded even when there was plenty of room to keep it. Make the unload VRAM aware and cover every inference backend: - Add routes/training_vram.py with summarize_resident_chat(), can_keep_chat_during_training() and free_chat_models_for_training(). The keep/unload decision reuses the same estimator and live per device free VRAM reader the training GPU selection already uses (auto_select_gpu_ids, estimate_required_model_memory_gb, get_visible_gpu_utilization), so the probe agrees with the placement computed later in start_training. - When a chat model is resident and training fits alongside it with a conservative margin (required_gb * 1.15 + 4 GB), keep it loaded so the user can train and chat at the same time; on a multi GPU box training lands on a different GPU and both coexist. Otherwise unload the HF/MLX orchestrator and the llama.cpp GGUF server before training starts. - The export subprocess shutdown stays unconditional and now runs first so its freed VRAM is reflected in the decision. Default deny: non CUDA backends, unestimable models, or any probe error fall back to the previous always unload behavior. Adds tests/test_training_vram_coexistence.py and updates two existing route tests in test_gpu_selection.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: per-GPU floor for explicit GPU lists + don't unload chat on invalid gpu_ids Address review feedback on the chat coexistence probe: - Explicit gpu_ids mode now enforces a per-GPU floor in addition to the aggregate free-VRAM check, mirroring auto_select_gpu_ids' min_per_gpu_N. Without it, an uneven split such as free [45, 10] for a 40 GB job passed the aggregate threshold and kept chat loaded even though the 10 GB GPU could not hold its training shard, risking an OOM. - Invalid explicit gpu_ids (ids outside the visible set, or a UUID/MIG mask) make resolve_requested_gpu_ids raise. That request is rejected with a 400 before training starts, so leave the resident chat model untouched instead of unloading it. - Tighten the target_modules / gpu_ids type hints to List[str] / List[int]. Adds tests for the per-GPU floor (uneven split unloads, even split keeps) and for invalid gpu_ids keeping the chat model loaded. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: only free chat VRAM once training will start; handle in-flight and CPU-only chat Address the second review pass on the chat-coexistence path: - Run the chat/export VRAM teardown as a before_spawn hook inside TrainingBackend.start_training, fired only after the start guards pass. Previously the route freed chat VRAM before calling start_training, so a refused start (e.g. a lingering pump thread) would tear down the resident chat model even though no training job began. - Treat an in-flight HF chat load (loading_models set, no active model yet) as not safely sizeable: free it rather than risk both OOMing as the load keeps allocating after training starts. - Do not count or tear down a GGUF llama-server confirmed to run entirely on CPU (_gpu_offload_active is False): it holds no VRAM, so killing it cannot help training fit. Adds tests for the before_spawn hook (runs on start, skipped when a subprocess is alive or a pump thread will not die, survives a hook error), the in-flight load flag, and the CPU-only GGUF exclusion in both the resident summary and the unload path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: treat any in-flight chat load (HF swap / mid-start GGUF) as unsafe to keep Tighten the in-flight detection in summarize_resident_chat so the keep check never sizes a load that is still allocating: - Flag loading on ANY non-empty loading_models, not only when active_model_name is empty. load_model adds the new model to loading_models before clearing the old active_model_name, so a replacement load during a swap was previously sized as a normal resident and could OOM as the new model finishes loading. - Flag a GGUF server that is active but not yet healthy (is_loaded False) as in-flight: it is still mmaping/offloading layers, so its final VRAM footprint is unknown. Consolidates the signal into a single resident["loading"] flag; the route frees the chat model whenever it is set. Adds tests for the replacement HF load and the mid-start GGUF cases. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments in chat/training VRAM coexistence (comments only) * Studio: run before_spawn VRAM hook only after GPU-selection validation Reviewers found the before_spawn hook fired before prepare_gpu_selection validated gpu_ids (and before config build), so a refused start (invalid gpu_ids -> 400, or a bad grad-clip value) could still tear down chat/export VRAM. Move the hook to immediately before proc.start(), once all synchronous validation and process construction have passed. This also fixes the route's in-flight-chat loading branch, since that teardown runs inside the same hook. Add test_hook_skipped_when_gpu_selection_rejects. * Studio: recompute GPU auto-selection after the before_spawn VRAM hook Codex P2: with before_spawn moved after prepare_gpu_selection, placement was frozen against the pre-teardown VRAM state while the hook freed export/chat afterward. Auto-selection could pin training onto a GPU the hook then cleared (or onto a kept chat model). Split validation from placement: explicit gpu_ids are still validated before the hook (raise -> 400, no teardown; explicit placement is VRAM-independent), but VRAM-dependent auto-selection now runs after the hook so it sees the freed memory. Add test_auto_placement_runs_after_hook and test_explicit_placement_validated_before_hook. * Studio: allow chatting during training (lift sidebar gate + VRAM-aware load guard) (#6335) * Studio: allow chatting during training (lift sidebar gate + VRAM-aware load guard) The sidebar disabled New Chat, project, and home navigation while a training run was active, so users could not chat during training even though the backend serves inference fine alongside a run. This removes that gate and adds a backend guard so the one genuinely risky operation, loading a new local chat model mid-training, is refused with a clear 409 when it would not fit beside the run. Frontend (app-sidebar.tsx): drop the chatDisabled = isTrainingRunning gate and its consumers. Navigation triggers no model load on its own, so chat stays usable during training. Backend (routes/training_vram.py, routes/inference.py): add can_load_chat_during_training plus a load/validate guard that sizes the same effective load the loader performs (LoRA 4-bit to 16-bit resolved first, HF auto placement via auto_select_gpu_ids, explicit multi-GPU per-GPU floor, GGUF sized from on-disk shards and companions or the selected remote variant). It is a no-op when training is inactive, never blocks external providers or already-resident models, and default-denies only on a CUDA sizing failure so a load can never OOM the run. Validate refuses early with the real settings so the frontend does not unload the resident chat model for a load that would be rejected. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address review feedback for chat-during-training load guard - Run the load/validate VRAM guard via asyncio.to_thread so the sync nvidia-smi + HF metadata work never blocks the event loop. - Size the GGUF KV cache at the requested context (_estimate_gguf_kv_gb) and add it to the local GGUF estimate so large-context picks are not under-counted. - Keep the requested quantization when adapter_config.json is malformed (not a JSON object) instead of raising in _effective_load_in_4bit. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: size the training load guard at the launcher's effective GGUF context The GGUF KV-cache estimate used max_seq_length only, but the llama.cpp launcher honors a user --ctx-size/-c in llama_extra_args. A load such as max_seq_length=4096 with --ctx-size 131072 was sized against a 4k cache while the server allocates 131k, so the guard could approve a long-context GGUF load that then OOMs training. Size the guard's KV at the larger of max_seq_length and the parsed --ctx-size (reusing the launcher's own parse_ctx_override), keeping the conservative f16 cache so the estimate is never smaller than what the server allocates. The chat model picker also validated with the raw max_seq_length while /load sizes with resolveLoadMaxSeqLength, so validate could pass, unload the current model, then have /load reject the native-context load. Validate now uses the same effective context; the load path is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: size the GGUF training guard at the server parallel-slot count The KV-cache estimate assumed a single slot, but llama-server allocates the cache across --parallel slots (app.state.llama_parallel_slots). On a Studio launched with --parallel N>1 the guard under-sized the cache N-fold and could approve a GGUF chat load that then OOMs training. Thread the same slot count the loader uses into the guard's KV estimate; default 1 leaves single-slot setups unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim comments for chat-during-training guard * Studio: keep chat generation alive across navigation; Train spinner + Return to Chat Hoist the base chat runtime above the routed outlet so navigating to Train (or any tab) no longer aborts an in-flight generation; only an explicit Stop cancels. Add a Train sidebar spinner and swap New Chat to Return to Chat while a run is active, with a lightweight completion watch so the spinner clears from any tab. Also respawn a chat llama-server killed mid-session and guard unreadable HF cache dirs that 500'd the hub model list. * Studio: show Return to Chat on the Train tab whenever a chat is live Previously the top sidebar item only swapped to Return to Chat while training was running; on the Train tab with an idle/just-finished run it stayed New Chat, which started a fresh thread and cancelled an in-flight generation. Show Return to Chat (and navigate back, preserving the run) whenever a generation is running or its thread is still active, or training is in progress. * Studio: keep a running chat alive when starting a New Chat Starting a New Chat (or switching threads) while a generation was in flight remounted the single-chat runtime provider, which detached the in-flight run and cut the previous chat off (it showed up frozen / empty when reopened). Key the single-chat view by project instead of by thread or new-chat nonce so the provider stays mounted and assistant-ui switches to a fresh thread in place. The previous generation keeps streaming in the background and autosaves on completion, and returning to that thread reattaches the live run instead of reloading a half-saved one. Also: - "Return to Chat" now lands on the thread that is still generating rather than the empty new chat that became active after New Chat. - Skip the explicit /inference/cancel POST when an abort comes from a runtime detach (navigation / background switch) rather than an explicit Stop, so a backgrounded generation is never cancelled behind the scenes. * Studio: make model export non-blocking and inline The Export tab opened a full-screen modal that trapped focus, could not be closed or cancelled while running, and showed no progress. It also stopped training and unloaded the chat model before loading, so export could not run alongside them. Export now mirrors the training runtime pattern: - Inline panel embedded where the Export Model button was, with no modal or backdrop, so the rest of the UI stays usable during an export. - Global export runtime store plus an app-root lifecycle hook, so a run keeps going and streaming across navigation and is reflected on the Export nav item from any tab. - The worker log stream now stays connected across the load to export phase boundary instead of stranding on "Waiting for worker output". - Progress bar driven by phase and quant index (quant N of M for GGUF), with elapsed time and a working Cancel. - load-checkpoint no longer stops training or unloads inference; export loads in its own subprocess in parallel and surfaces out-of-memory as a clear error. - Add POST /api/export/cancel and is_export_active on /api/export/status. * Studio: show Return to Chat on the Export tab too Extend the New Chat to Return to Chat swap to the Export route so leaving a running chat for Export offers a way back to the live generation, matching the Train tab. * Studio: smooth out Export animations and polish the panel - Drop the height-based reveal animations (source switch, run panel, quant picker, hub fields) that caused flashing and reflow; use instant swaps and quick opacity fades instead. - Method and quant cards now transition colors only, with no transition-all or hover lift, so selecting a method or quant is crisp instead of jumpy. - Auto-scroll the export panel into view when it opens and add a scroll-to-bottom button when its output is below the fold, like Chat. - Show Return to Chat on the Export tab while an export is running, matching how training drives it on the Train tab. - Surface the current phase or stage in the live output before the first worker line arrives so the panel never looks stuck while progress is advancing. * Studio: show Return to Chat on every non-chat tab Generalize the Return to Chat swap from just Train/Export to any non-chat route (Recipes, Projects, Hub, ...) so a running or active chat is always one click away, instead of showing New Chat there. * Studio: stream export logs over the Cloudflare tunnel; drop janky export animations Exporting over a --secure Cloudflare quick tunnel showed "connecting..." with no logs while the progress bar advanced. Cloudflare buffers text/event-stream and only flushes when the stream closes, so the SSE log stream never reached the browser during the run (direct localhost is unaffected, which is why this only showed up over the tunnel). Add a tunnel-safe JSON poll fallback (GET /api/export/logs?since=) that the runtime lifecycle hook polls while a run is active. Short JSON responses are not buffered by the proxy, so logs show up in near real time over the tunnel. It shares the orchestrator's monotonic seq cursor with the SSE stream and the store de-dupes by seq, so the two transports run together (SSE on localhost, poll over the tunnel) without double-printing. A successful poll marks the panel "streaming" instead of leaving it stuck on "connecting...". Also remove the framer-motion AnimatePresence reveals from the export config and run panel (quant picker, hub fields, the inline run panel, and the live log section). The expand/slide animations flashed and felt clunky; the sections now render in place. * Studio: recover export over the Cloudflare tunnel when the blocking POST times out (524) A model export over a --secure Cloudflare quick tunnel showed "Request failed (524)" even though the export succeeded on the backend (the GGUF was written). Cloudflare returns 524 when a single request takes longer than ~100s to respond, and a GGUF conversion routinely runs for minutes, so the blocking per-method export POST is cut off while the backend keeps going. Confirm completion via short status polls instead of relying on the long POST response (the same approach that fixed log streaming): - The orchestrator records each finished op's outcome (status / output_path / error) with a monotonic seq, exposed on GET /api/export/status. - parseJson now preserves the HTTP status; a 524/520/522/523/502/503 or a status-less network drop is classified as a recoverable transport error. - runExport wraps each phase (load, every export method, each GGUF quant): on a recoverable failure it keeps the run alive (logs keep streaming, the panel shows "reconnecting...") and polls status until the still-running op finishes, then settles from the recorded result, recovering the output path for the success banner. A real 4xx still fails immediately; localhost still uses the fast POST response. applyBackendStatus also settles a reloaded run from the last-op record. Verified over the tunnel: a 3m14s gemma-4-E4B-it GGUF export now ends on the success banner with the output path instead of 524. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep the export method + logs visible after navigating away mid-export While an export was running, navigating to another tab and back to Export remounted the page and reset the local form state (exportMethod, quant levels), so the method card showed unselected and the run panel's log area was hidden until the card was re-clicked. The run itself lives in the global store and was unaffected. Seed exportMethod / quantLevels from the active run's summary via lazy useState initializers on (re)mount, and gate the panel's log area on the live run (isExporting / logLines / the run's method) rather than only the local form selection. The card stays selected and the logs/progress stay visible across navigation; nothing changes when no run is active. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Studio: address export/training review findings - Export: guard Start against an empty GGUF quant selection so an inline-panel run with no quant can't settle as success with no file produced. - Export: thread the source HF token into the background load so gated/private HF source exports (and gated bases) authenticate, matching the consent path. - Export: only settle a recovered (non-owned) run as a finished export when the last backend op was an export, not a standalone load_checkpoint. - Training: free the export subprocess whenever an export is active, not only once a checkpoint is loaded, so an in-flight export load can't race training for VRAM (current_checkpoint is unset during the load phase). --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
207 lines
7 KiB
Python
207 lines
7 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Pydantic schemas for Export API."""
|
|
|
|
from pathlib import Path, PureWindowsPath
|
|
|
|
from pydantic import BaseModel, Field, field_validator
|
|
from typing import List, Optional, Literal, Dict, Any
|
|
|
|
|
|
def _validate_save_directory(value: str) -> str:
|
|
"""Validate save_directory — allows absolute paths (user may want a different drive)."""
|
|
if value is None:
|
|
raise ValueError("save_directory is required")
|
|
raw = str(value).strip()
|
|
if not raw:
|
|
raise ValueError("save_directory must not be empty")
|
|
if "\x00" in raw:
|
|
raise ValueError("save_directory may not contain null bytes")
|
|
if any(ch in raw for ch in ("\r", "\n")):
|
|
raise ValueError("save_directory may not contain control characters")
|
|
path = Path(raw).expanduser()
|
|
path_parts = (*path.parts, *PureWindowsPath(raw).parts, *raw.replace("\\", "/").split("/"))
|
|
if any(len(part) > 255 for part in path_parts if part not in ("", ".", "/", "\\")):
|
|
raise ValueError("save_directory path components must be <= 255 characters")
|
|
if (
|
|
".." in path.parts
|
|
or ".." in PureWindowsPath(raw).parts
|
|
or ".." in raw.replace("\\", "/").split("/")
|
|
):
|
|
raise ValueError("save_directory may not contain '..' segments")
|
|
return raw
|
|
|
|
|
|
class LoadCheckpointRequest(BaseModel):
|
|
"""Request for loading a checkpoint into the export backend."""
|
|
|
|
checkpoint_path: str = Field(..., description = "Path to the checkpoint directory")
|
|
max_seq_length: int = Field(
|
|
2048,
|
|
ge = 128,
|
|
le = 32768,
|
|
description = "Maximum sequence length for loading the model",
|
|
)
|
|
load_in_4bit: bool = Field(
|
|
True,
|
|
description = "Whether to load the model in 4-bit quantization",
|
|
)
|
|
trust_remote_code: bool = Field(
|
|
False,
|
|
description = "Allow loading models with custom code. Only enable for checkpoints/base models you trust.",
|
|
)
|
|
approved_remote_code_fingerprint: Optional[str] = Field(
|
|
None,
|
|
description = "sha256 fingerprint from the remote-code scan, pinning user approval of this exact custom-code version.",
|
|
)
|
|
hf_token: Optional[str] = Field(
|
|
None,
|
|
description = "Hugging Face token used to scan/load gated checkpoints and their base models.",
|
|
)
|
|
|
|
|
|
class ExportStatusResponse(BaseModel):
|
|
"""Current export backend status."""
|
|
|
|
current_checkpoint: Optional[str] = Field(
|
|
None,
|
|
description = "Path to the currently loaded checkpoint, if any",
|
|
)
|
|
is_vision: bool = Field(
|
|
False,
|
|
description = "True if the loaded checkpoint is a vision model",
|
|
)
|
|
is_peft: bool = Field(
|
|
False,
|
|
description = "True if the loaded checkpoint is a PEFT (LoRA) model",
|
|
)
|
|
is_export_active: bool = Field(
|
|
False,
|
|
description = "True while a load / export / cleanup operation is running",
|
|
)
|
|
# Recovery fields: when a blocking export POST is cut off by a Cloudflare tunnel
|
|
# timeout (524 at ~100s), the client polls this endpoint to learn the real
|
|
# outcome of the operation that kept running on the backend.
|
|
active_op_kind: Optional[str] = Field(
|
|
None,
|
|
description = "Kind of the currently running op (load_checkpoint / export_* / cleanup)",
|
|
)
|
|
last_op_seq: int = Field(
|
|
0,
|
|
description = "Monotonic counter of finished ops; client baseline to detect 'my op finished'",
|
|
)
|
|
last_op_kind: Optional[str] = Field(
|
|
None,
|
|
description = "Kind of the most recently finished op",
|
|
)
|
|
last_op_status: Optional[str] = Field(
|
|
None,
|
|
description = "Outcome of the most recently finished op: success / error / cancelled",
|
|
)
|
|
last_op_output_path: Optional[str] = Field(
|
|
None,
|
|
description = "Output path of the most recently finished op, if it produced one",
|
|
)
|
|
last_op_error: Optional[str] = Field(
|
|
None,
|
|
description = "Error message of the most recently finished op, if it failed",
|
|
)
|
|
|
|
|
|
class ExportOperationResponse(BaseModel):
|
|
"""Generic response for export operations."""
|
|
|
|
success: bool = Field(..., description = "True if the operation succeeded")
|
|
message: str = Field(..., description = "Human-readable status or error message")
|
|
details: Optional[Dict[str, Any]] = Field(
|
|
default = None,
|
|
description = "Optional extra details about the operation",
|
|
)
|
|
|
|
|
|
class ExportCommonOptions(BaseModel):
|
|
"""Common options for export operations that save locally and/or push to Hub."""
|
|
|
|
save_directory: str = Field(
|
|
...,
|
|
description = "Local directory where the exported artifacts will be written",
|
|
)
|
|
|
|
@field_validator("save_directory", mode = "before")
|
|
@classmethod
|
|
def _check_save_directory(cls, v):
|
|
return _validate_save_directory(v)
|
|
|
|
push_to_hub: bool = Field(
|
|
False,
|
|
description = "If True, also push the exported model to the Hugging Face Hub",
|
|
)
|
|
repo_id: Optional[str] = Field(
|
|
None,
|
|
description = "Hugging Face Hub repository ID (username/model-name)",
|
|
)
|
|
hf_token: Optional[str] = Field(
|
|
None,
|
|
description = "Hugging Face access token used for Hub operations",
|
|
)
|
|
private: bool = Field(
|
|
False,
|
|
description = "If True, create a private repository on the Hub (where applicable)",
|
|
)
|
|
base_model_id: Optional[str] = Field(
|
|
None,
|
|
description = "HuggingFace model ID of the base model (for model card metadata)",
|
|
)
|
|
|
|
|
|
class ExportMergedModelRequest(ExportCommonOptions):
|
|
"""Request for exporting a merged PEFT model."""
|
|
|
|
format_type: Literal["16-bit (FP16)", "4-bit (FP4)"] = Field(
|
|
"16-bit (FP16)",
|
|
description = "Export precision / format for the merged model",
|
|
)
|
|
|
|
|
|
class ExportBaseModelRequest(ExportCommonOptions):
|
|
"""Request for exporting a non-PEFT (base) model."""
|
|
|
|
# Uses fields from ExportCommonOptions only
|
|
|
|
|
|
class ExportGGUFRequest(BaseModel):
|
|
"""Request for exporting the current model to GGUF format."""
|
|
|
|
save_directory: str = Field(
|
|
...,
|
|
description = "Directory where GGUF files will be saved",
|
|
)
|
|
|
|
@field_validator("save_directory", mode = "before")
|
|
@classmethod
|
|
def _check_save_directory(cls, v):
|
|
return _validate_save_directory(v)
|
|
|
|
quantization_method: str = Field(
|
|
"Q4_K_M",
|
|
description = 'GGUF quantization method (e.g. "Q4_K_M")',
|
|
)
|
|
push_to_hub: bool = Field(
|
|
False,
|
|
description = "If True, also push GGUF artifacts to the Hugging Face Hub",
|
|
)
|
|
repo_id: Optional[str] = Field(
|
|
None,
|
|
description = "Hugging Face Hub repository ID for GGUF upload",
|
|
)
|
|
hf_token: Optional[str] = Field(
|
|
None,
|
|
description = "Hugging Face token for GGUF upload",
|
|
)
|
|
|
|
|
|
class ExportLoRAAdapterRequest(ExportCommonOptions):
|
|
"""Request for exporting only the LoRA adapter (not merged)."""
|
|
|
|
# Uses fields from ExportCommonOptions only
|