Compare commits

...
Sign in to create a new pull request.

7 commits

Author SHA1 Message Date
pre-commit-ci[bot]
446f829da4 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-03-18 07:25:33 +00:00
Daniel Han
fc2d577d0d Fix studio setup: venv isolation, centralized .venv_t5, uv targeting
- All platforms (including Colab) now create ~/.unsloth/studio/.venv
  with --without-pip fallback for broken ensurepip environments
- Add --python sys.executable to uv pip install in install_python_stack.py
  so uv targets the correct venv instead of system Python
- Centralize .venv_t5 bootstrap in transformers_version.py with proper
  validation (checks required packages exist, not just non-empty dir)
- Replace ~150 lines of duplicated install code across 3 worker files
  with calls to the shared _ensure_venv_t5_exists() helper
- Use uv-if-present with pip fallback; do not install uv at runtime
- Add site.addsitedir() shim in colab.py so notebook cells can import
  studio packages from the venv without system-Python double-install
- Update .venv_t5 packages: huggingface_hub 1.3.0->1.7.1, add hf_xet
- Bump transformers pin 4.57.1->4.57.6 in requirements + constraints
- Add Fast-Install helper to setup.ps1 with uv+pip fallback
- Keep Colab-specific completion banner in setup.sh
2026-03-18 07:24:03 +00:00
Daniel Han
f43091bf34
Merge branch 'main' into fix/studio-venv-colab 2026-03-18 00:22:55 -07:00
Daniel Han
2b1e53d1bd Fix frontend rebuild detection and npm dependency issues
Addresses reviewer feedback on the frontend caching logic:

1. setup.sh: Fix broken find command that caused exit under pipefail.
   The piped `find | xargs find -newer` had paths after the expression
   which GNU find rejects. Replaced with a simpler `find -maxdepth 1
   -type f -newer dist/` that checks ALL top-level files (catches
   index.html, bun.lock, etc. that the extension allowlist missed).

2. setup.sh: Guard oxc-validator npm install behind `command -v npm`
   check. When the frontend build is skipped (dist/ is cached), Node
   bootstrap is also skipped, so npm may not be available.

3. setup.ps1: Replace Get-ChildItem -Include with explicit path
   probing for src/ and public/. PowerShell's -Include without a
   trailing wildcard silently returns nothing, so src/public changes
   were never detected. Also check ALL top-level files instead of
   just .json/.ts/.js/.mjs extensions.
2026-03-18 07:01:45 +00:00
Daniel Han
153c203a89 Fix frontend rebuild detection and decouple oxc-validator install
Address review feedback:

- Check entire frontend/ directory for changes, not just src/.
  The build also depends on package.json, vite.config.ts,
  tailwind.config.ts, public/, and other config files. A change
  to any of these now triggers a rebuild.
- Move oxc-validator npm install outside the frontend build gate
  in setup.sh so it always runs on setup, matching setup.ps1
  which already had it outside the gate.
2026-03-18 05:59:42 +00:00
Daniel Han
5e36f09664 Rebuild frontend when source files are newer than dist/
Instead of only checking if dist/ exists, compare source file timestamps
against the dist/ directory. If any file in frontend/src/ is newer than
dist/, trigger a rebuild. This handles the case where a developer pulls
new frontend changes and re-runs setup -- stale assets get rebuilt
automatically.
2026-03-18 05:44:16 +00:00
Daniel Han
1f075a932b Cache frontend build across setup runs
Skip the frontend npm install + build if frontend/dist already exists.
Previously setup.ps1 nuked node_modules and package-lock.json on every
run, and both scripts always rebuilt even when dist/ was already present.

On a git clone editable install, the first setup run still builds the
frontend as before. Subsequent runs skip it, saving several minutes.
To force a rebuild, delete frontend/dist and re-run setup.
2026-03-18 05:14:37 +00:00
10 changed files with 304 additions and 233 deletions

View file

@ -7,8 +7,26 @@ Uses Colab's built-in proxy - no external tunneling needed!
"""
from pathlib import Path
import site
import sys
def _bootstrap_studio_venv() -> None:
"""Expose the Studio venv's site-packages to the current interpreter.
On Colab, notebook cells run outside the venv subshell. Instead of
installing the full stack into system Python, we add the venv's
site-packages so that packages like structlog, fastapi, etc. are
importable from notebook cells.
"""
venv_lib = Path.home() / ".unsloth" / "studio" / ".venv" / "lib"
for sp in venv_lib.glob("python*/site-packages"):
if str(sp) not in sys.path:
site.addsitedir(str(sp))
_bootstrap_studio_venv()
# Add backend to path early so local modules like loggers can be imported
backend_path = str(Path(__file__).parent)
if backend_path not in sys.path:

View file

@ -40,59 +40,25 @@ def _activate_transformers_version(model_name: str) -> None:
if backend_path not in sys.path:
sys.path.insert(0, backend_path)
from utils.transformers_version import needs_transformers_5, _resolve_base_model
from utils.transformers_version import (
needs_transformers_5,
_resolve_base_model,
_ensure_venv_t5_exists,
_VENV_T5_DIR,
)
resolved = _resolve_base_model(model_name)
if needs_transformers_5(resolved):
venv_t5 = os.path.join(
os.path.expanduser("~"), ".unsloth", "studio", ".venv_t5"
)
if os.path.isdir(venv_t5):
sys.path.insert(0, venv_t5)
logger.info("Activated transformers 5.x from %s", venv_t5)
else:
# Fallback: pip install at runtime (slower, ~10-15s)
logger.warning(".venv_t5 not found at %s — installing at runtime", venv_t5)
import subprocess as sp
os.makedirs(venv_t5, exist_ok = True)
r1 = sp.run(
[
sys.executable,
"-m",
"pip",
"install",
"--target",
venv_t5,
"--no-deps",
"transformers==5.3.0",
],
stdout = sp.PIPE,
stderr = sp.STDOUT,
if not _ensure_venv_t5_exists():
raise RuntimeError(
f"Cannot activate transformers 5.x: .venv_t5 missing at {_VENV_T5_DIR}"
)
r2 = sp.run(
[
sys.executable,
"-m",
"pip",
"install",
"--target",
venv_t5,
"--no-deps",
"huggingface_hub==1.3.0",
],
stdout = sp.PIPE,
stderr = sp.STDOUT,
)
if r1.returncode != 0 or r2.returncode != 0:
raise RuntimeError(
f"Failed to install transformers 5.x into {venv_t5}. "
f"pip returncode: transformers={r1.returncode}, huggingface_hub={r2.returncode}"
)
sys.path.insert(0, venv_t5)
if _VENV_T5_DIR not in sys.path:
sys.path.insert(0, _VENV_T5_DIR)
logger.info("Activated transformers 5.x from %s", _VENV_T5_DIR)
# Propagate to child subprocesses (e.g. GGUF converter)
_pp = os.environ.get("PYTHONPATH", "")
os.environ["PYTHONPATH"] = venv_t5 + (os.pathsep + _pp if _pp else "")
os.environ["PYTHONPATH"] = _VENV_T5_DIR + (os.pathsep + _pp if _pp else "")
else:
logger.info("Using default transformers (4.57.x) for %s", model_name)

View file

@ -42,59 +42,25 @@ def _activate_transformers_version(model_name: str) -> None:
if backend_path not in sys.path:
sys.path.insert(0, backend_path)
from utils.transformers_version import needs_transformers_5, _resolve_base_model
from utils.transformers_version import (
needs_transformers_5,
_resolve_base_model,
_ensure_venv_t5_exists,
_VENV_T5_DIR,
)
resolved = _resolve_base_model(model_name)
if needs_transformers_5(resolved):
venv_t5 = os.path.join(
os.path.expanduser("~"), ".unsloth", "studio", ".venv_t5"
)
if os.path.isdir(venv_t5):
sys.path.insert(0, venv_t5)
logger.info("Activated transformers 5.x from %s", venv_t5)
else:
# Fallback: pip install at runtime (slower, ~10-15s)
logger.warning(".venv_t5 not found at %s — installing at runtime", venv_t5)
import subprocess as sp
os.makedirs(venv_t5, exist_ok = True)
r1 = sp.run(
[
sys.executable,
"-m",
"pip",
"install",
"--target",
venv_t5,
"--no-deps",
"transformers==5.3.0",
],
stdout = sp.PIPE,
stderr = sp.STDOUT,
if not _ensure_venv_t5_exists():
raise RuntimeError(
f"Cannot activate transformers 5.x: .venv_t5 missing at {_VENV_T5_DIR}"
)
r2 = sp.run(
[
sys.executable,
"-m",
"pip",
"install",
"--target",
venv_t5,
"--no-deps",
"huggingface_hub==1.3.0",
],
stdout = sp.PIPE,
stderr = sp.STDOUT,
)
if r1.returncode != 0 or r2.returncode != 0:
raise RuntimeError(
f"Failed to install transformers 5.x into {venv_t5}. "
f"pip returncode: transformers={r1.returncode}, huggingface_hub={r2.returncode}"
)
sys.path.insert(0, venv_t5)
if _VENV_T5_DIR not in sys.path:
sys.path.insert(0, _VENV_T5_DIR)
logger.info("Activated transformers 5.x from %s", _VENV_T5_DIR)
# Propagate to child subprocesses (e.g. GGUF converter)
_pp = os.environ.get("PYTHONPATH", "")
os.environ["PYTHONPATH"] = venv_t5 + (os.pathsep + _pp if _pp else "")
os.environ["PYTHONPATH"] = _VENV_T5_DIR + (os.pathsep + _pp if _pp else "")
else:
logger.info("Using default transformers (4.57.x) for %s", model_name)

View file

@ -36,59 +36,25 @@ def _activate_transformers_version(model_name: str) -> None:
if backend_path not in sys.path:
sys.path.insert(0, backend_path)
from utils.transformers_version import needs_transformers_5, _resolve_base_model
from utils.transformers_version import (
needs_transformers_5,
_resolve_base_model,
_ensure_venv_t5_exists,
_VENV_T5_DIR,
)
resolved = _resolve_base_model(model_name)
if needs_transformers_5(resolved):
venv_t5 = os.path.join(
os.path.expanduser("~"), ".unsloth", "studio", ".venv_t5"
)
if os.path.isdir(venv_t5):
sys.path.insert(0, venv_t5)
logger.info("Activated transformers 5.x from %s", venv_t5)
else:
# Fallback: pip install at runtime (slower, ~10-15s)
logger.warning(".venv_t5 not found at %s — installing at runtime", venv_t5)
import subprocess as sp
os.makedirs(venv_t5, exist_ok = True)
r1 = sp.run(
[
sys.executable,
"-m",
"pip",
"install",
"--target",
venv_t5,
"--no-deps",
"transformers==5.3.0",
],
stdout = sp.PIPE,
stderr = sp.STDOUT,
if not _ensure_venv_t5_exists():
raise RuntimeError(
f"Cannot activate transformers 5.x: .venv_t5 missing at {_VENV_T5_DIR}"
)
r2 = sp.run(
[
sys.executable,
"-m",
"pip",
"install",
"--target",
venv_t5,
"--no-deps",
"huggingface_hub==1.3.0",
],
stdout = sp.PIPE,
stderr = sp.STDOUT,
)
if r1.returncode != 0 or r2.returncode != 0:
raise RuntimeError(
f"Failed to install transformers 5.x into {venv_t5}. "
f"pip returncode: transformers={r1.returncode}, huggingface_hub={r2.returncode}"
)
sys.path.insert(0, venv_t5)
if _VENV_T5_DIR not in sys.path:
sys.path.insert(0, _VENV_T5_DIR)
logger.info("Activated transformers 5.x from %s", _VENV_T5_DIR)
# Propagate to child subprocesses (e.g. GGUF converter)
_pp = os.environ.get("PYTHONPATH", "")
os.environ["PYTHONPATH"] = venv_t5 + (os.pathsep + _pp if _pp else "")
os.environ["PYTHONPATH"] = _VENV_T5_DIR + (os.pathsep + _pp if _pp else "")
else:
logger.info("Using default transformers (4.57.x) for %s", model_name)

View file

@ -11,4 +11,4 @@ git+https://github.com/meta-pytorch/OpenEnv.git
# executorch>=1.0.1 # 41.5 MB - no imports in unsloth/zoo/studio
torch-c-dlpack-ext
sentence_transformers==5.2.0
transformers==4.57.1
transformers==4.57.6

View file

@ -1,6 +1,6 @@
# Single-env pins for unsloth + studio + data-designer
# Keep compatible with unsloth transformers bounds.
transformers==4.57.1
transformers==4.57.6
trl==0.23.1
huggingface-hub==0.36.2

View file

@ -26,6 +26,7 @@ import json
import structlog
from loggers import get_logger
import os
import shutil
import subprocess
import sys
from pathlib import Path
@ -58,7 +59,7 @@ _tokenizer_class_cache: dict[str, bool] = {}
# Versions
TRANSFORMERS_5_VERSION = "5.3.0"
TRANSFORMERS_DEFAULT_VERSION = "4.57.1"
TRANSFORMERS_DEFAULT_VERSION = "4.57.6"
# Pre-installed directory for transformers 5.x — created by setup.sh / setup.ps1
_VENV_T5_DIR = str(Path.home() / ".unsloth" / "studio" / ".venv_t5")
@ -216,15 +217,55 @@ def _purge_modules() -> int:
return len(to_remove)
def _ensure_venv_t5_exists() -> bool:
"""Ensure .venv_t5/ exists. Install at runtime if missing."""
if os.path.isdir(_VENV_T5_DIR) and os.listdir(_VENV_T5_DIR):
return True
_VENV_T5_PACKAGES = (
f"transformers=={TRANSFORMERS_5_VERSION}",
"huggingface_hub==1.7.1",
"hf_xet==1.4.2",
)
logger.warning(".venv_t5 not found at %s — installing at runtime", _VENV_T5_DIR)
os.makedirs(_VENV_T5_DIR, exist_ok = True)
for pkg in (f"transformers=={TRANSFORMERS_5_VERSION}", "huggingface_hub==1.3.0"):
cmd = [
def _venv_t5_is_valid() -> bool:
"""Return True if .venv_t5/ has all required packages installed."""
if not os.path.isdir(_VENV_T5_DIR) or not os.listdir(_VENV_T5_DIR):
return False
# Check that the key package directories actually exist
for pkg_spec in _VENV_T5_PACKAGES:
pkg_name = pkg_spec.split("==")[0].replace("-", "_")
if not any(
(Path(_VENV_T5_DIR) / d).is_dir()
for d in (pkg_name, pkg_name.replace("_", "-"))
):
return False
return True
def _install_to_venv_t5(pkg: str) -> bool:
"""Install a single package into .venv_t5/, preferring uv then pip."""
# Try uv first (faster) if already on PATH -- do NOT install uv at runtime
if shutil.which("uv"):
result = subprocess.run(
[
"uv",
"pip",
"install",
"--python",
sys.executable,
"--target",
_VENV_T5_DIR,
"--no-deps",
pkg,
],
stdout = subprocess.PIPE,
stderr = subprocess.STDOUT,
text = True,
)
if result.returncode == 0:
return True
logger.warning("uv install of %s failed, falling back to pip", pkg)
# Fallback to pip
result = subprocess.run(
[
sys.executable,
"-m",
"pip",
@ -233,12 +274,28 @@ def _ensure_venv_t5_exists() -> bool:
_VENV_T5_DIR,
"--no-deps",
pkg,
]
result = subprocess.run(
cmd, stdout = subprocess.PIPE, stderr = subprocess.STDOUT, text = True
)
if result.returncode != 0:
logger.error("pip install failed:\n%s", result.stdout)
],
stdout = subprocess.PIPE,
stderr = subprocess.STDOUT,
text = True,
)
if result.returncode != 0:
logger.error("install failed:\n%s", result.stdout)
return False
return True
def _ensure_venv_t5_exists() -> bool:
"""Ensure .venv_t5/ exists with all required packages. Install if missing."""
if _venv_t5_is_valid():
return True
logger.warning(
".venv_t5 not found or incomplete at %s -- installing at runtime", _VENV_T5_DIR
)
os.makedirs(_VENV_T5_DIR, exist_ok = True)
for pkg in _VENV_T5_PACKAGES:
if not _install_to_venv_t5(pkg):
return False
logger.info("Installed transformers 5.x to %s", _VENV_T5_DIR)
return True

View file

@ -140,15 +140,15 @@ def _bootstrap_uv() -> bool:
global UV_NEEDS_SYSTEM
if not shutil.which("uv"):
return False
# Probe: try a dry-run install without --system.
# If uv can't find a venv it exits with code 2.
# Probe: try a dry-run install targeting the current Python explicitly.
# Without --python, uv can ignore the activated venv on some platforms.
probe = subprocess.run(
["uv", "pip", "install", "--dry-run", "pip"],
["uv", "pip", "install", "--dry-run", "--python", sys.executable, "pip"],
stdout = subprocess.PIPE,
stderr = subprocess.STDOUT,
)
if probe.returncode != 0:
# Retry with --system to confirm it works
# Retry with --system (some envs need it when uv can't find a venv)
probe_sys = subprocess.run(
["uv", "pip", "install", "--dry-run", "--system", "pip"],
stdout = subprocess.PIPE,
@ -204,6 +204,10 @@ def _build_uv_cmd(args: tuple[str, ...]) -> list[str]:
cmd = ["uv", "pip", "install"]
if UV_NEEDS_SYSTEM:
cmd.append("--system")
# Always pass --python so uv targets the correct environment.
# Without this, uv can ignore an activated venv and install into
# the system Python (observed on Colab and similar environments).
cmd.extend(["--python", sys.executable])
cmd.extend(_translate_pip_args_for_uv(args))
cmd.append("--torch-backend=auto")
return cmd

View file

@ -748,9 +748,39 @@ Write-Host ""
# ==========================================================================
# PHASE 2: Frontend build (skip if pip-installed -- already bundled)
# ==========================================================================
$DistDir = Join-Path $FrontendDir "dist"
# Skip build if dist/ exists and no tracked input is newer than dist/.
# Checks src/, public/, package.json, config files -- not just src/.
$NeedFrontendBuild = $true
if ($IsPipInstall) {
$NeedFrontendBuild = $false
Write-Host "[OK] Running from pip install - frontend already bundled, skipping build" -ForegroundColor Green
} else {
} elseif (Test-Path $DistDir) {
$DistTime = (Get-Item $DistDir).LastWriteTime
$NewerFile = $null
# Check src/ and public/ recursively (probe paths directly, not via -Include)
foreach ($subDir in @("src", "public")) {
$subPath = Join-Path $FrontendDir $subDir
if (Test-Path $subPath) {
$NewerFile = Get-ChildItem -Path $subPath -Recurse -File -ErrorAction SilentlyContinue |
Where-Object { $_.LastWriteTime -gt $DistTime } | Select-Object -First 1
if ($NewerFile) { break }
}
}
# Also check ALL top-level files (package.json, index.html, bun.lock, configs, etc.)
if (-not $NewerFile) {
$NewerFile = Get-ChildItem -Path $FrontendDir -File -ErrorAction SilentlyContinue |
Where-Object { $_.LastWriteTime -gt $DistTime } |
Select-Object -First 1
}
if (-not $NewerFile) {
$NeedFrontendBuild = $false
Write-Host "[OK] Frontend already built and up to date -- skipping build" -ForegroundColor Green
} else {
Write-Host "[INFO] Frontend source changed since last build -- rebuilding..." -ForegroundColor Yellow
}
}
if ($NeedFrontendBuild -and -not $IsPipInstall) {
Write-Host ""
Write-Host "Building frontend..." -ForegroundColor Cyan
# npm writes warnings to stderr; lower ErrorActionPreference so PS doesn't
@ -758,9 +788,6 @@ if ($IsPipInstall) {
$prevEAP_npm = $ErrorActionPreference
$ErrorActionPreference = "Continue"
Push-Location $FrontendDir
# Remove stale node_modules and package-lock.json to avoid version conflicts
if (Test-Path "node_modules") { Remove-Item -Recurse -Force "node_modules" }
if (Test-Path "package-lock.json") { Remove-Item -Force "package-lock.json" }
npm install 2>&1 | Out-Null
if ($LASTEXITCODE -ne 0) {
Pop-Location
@ -845,7 +872,35 @@ $ErrorActionPreference = "Continue"
$ActivateScript = Join-Path $VenvDir "Scripts\Activate.ps1"
. $ActivateScript
pip install --upgrade pip 2>&1 | Out-Null
# Try to use uv (much faster than pip), fall back to pip if unavailable
$UseUv = $false
if (Get-Command uv -ErrorAction SilentlyContinue) {
$UseUv = $true
} else {
Write-Host " Installing uv package manager..." -ForegroundColor Cyan
try {
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" 2>&1 | Out-Null
Refresh-Environment
# Re-activate venv since Refresh-Environment rebuilds PATH from
# registry and drops the venv's Scripts directory
. $ActivateScript
if (Get-Command uv -ErrorAction SilentlyContinue) { $UseUv = $true }
} catch { }
}
# Helper: install a package, preferring uv with pip fallback
function Fast-Install {
param([Parameter(ValueFromRemainingArguments=$true)]$Args_)
if ($UseUv) {
$VenvPy = (Get-Command python).Source
$result = & uv pip install --python $VenvPy @Args_ 2>&1
if ($LASTEXITCODE -eq 0) { return }
}
& python -m pip install @Args_ 2>&1
}
Fast-Install --upgrade pip | Out-Null
# if (-not $IsPipInstall) {
# # Running from repo: copy requirements and do editable install
@ -882,14 +937,14 @@ Write-Host "[OK] TORCHINDUCTOR_CACHE_DIR set to $TorchCacheDir (avoids MAX_PATH
$CuTag = Get-PytorchCudaTag
Write-Host " Installing PyTorch with CUDA support ($CuTag)..." -ForegroundColor Cyan
pip install torch torchvision torchaudio --index-url "https://download.pytorch.org/whl/$CuTag" 2>&1 | Out-Null
Fast-Install torch torchvision torchaudio --index-url "https://download.pytorch.org/whl/$CuTag" | Out-Null
# Install Triton for Windows (enables torch.compile without it training can hang)
# Install Triton for Windows (enables torch.compile -- without it training can hang)
Write-Host " Installing Triton for Windows..." -ForegroundColor Cyan
pip install "triton-windows<3.7" 2>&1 | Out-Null
Fast-Install "triton-windows<3.7" | Out-Null
Write-Host "[OK] Triton for Windows installed (enables torch.compile)" -ForegroundColor Green
# Ordered heavy dependency installation shared cross-platform script
# Ordered heavy dependency installation -- shared cross-platform script
Write-Host " Running ordered dependency installation..." -ForegroundColor Cyan
python "$PSScriptRoot\install_python_stack.py"
# Restore ErrorActionPreference after pip/python work
@ -898,7 +953,7 @@ $ErrorActionPreference = $prevEAP
# ── Pre-install transformers 5.x into .venv_t5/ ──
# Models like GLM-4.7-Flash need transformers>=5.3.0. Instead of pip-installing
# at runtime (slow, ~10-15s), we pre-install into a separate directory.
# The training subprocess just prepends .venv_t5/ to sys.path instant switch.
# The training subprocess just prepends .venv_t5/ to sys.path -- instant switch.
Write-Host ""
Write-Host " Pre-installing transformers 5.x for newer model support..." -ForegroundColor Cyan
$VenvT5Dir = Join-Path $env:USERPROFILE ".unsloth\studio\.venv_t5"
@ -906,17 +961,13 @@ if (Test-Path $VenvT5Dir) { Remove-Item -Recurse -Force $VenvT5Dir }
New-Item -ItemType Directory -Path $VenvT5Dir -Force | Out-Null
$prevEAP_t5 = $ErrorActionPreference
$ErrorActionPreference = "Continue"
pip install --target $VenvT5Dir --no-deps "transformers==5.3.0" 2>&1 | Out-Null
if ($LASTEXITCODE -ne 0) {
Write-Host "[FAIL] Could not install transformers 5.3.0 into .venv_t5/" -ForegroundColor Red
$ErrorActionPreference = $prevEAP_t5
exit 1
}
pip install --target $VenvT5Dir --no-deps "huggingface_hub==1.3.0" 2>&1 | Out-Null
if ($LASTEXITCODE -ne 0) {
Write-Host "[FAIL] Could not install huggingface_hub 1.3.0 into .venv_t5/" -ForegroundColor Red
$ErrorActionPreference = $prevEAP_t5
exit 1
foreach ($pkg in @("transformers==5.3.0", "huggingface_hub==1.7.1", "hf_xet==1.4.2")) {
Fast-Install --target $VenvT5Dir --no-deps $pkg | Out-Null
if ($LASTEXITCODE -ne 0) {
Write-Host "[FAIL] Could not install $pkg into .venv_t5/" -ForegroundColor Red
$ErrorActionPreference = $prevEAP_t5
exit 1
}
}
$ErrorActionPreference = $prevEAP_t5
Write-Host "[OK] Transformers 5.x pre-installed to .venv_t5/" -ForegroundColor Green

View file

@ -41,13 +41,25 @@ if [[ "$keynames" == *$'\nCOLAB_'* ]]; then
fi
# ── Detect whether frontend needs building ──
# Only skip when BOTH conditions are true:
# 1. We're inside site-packages (PyPI / pip install, not editable)
# 2. dist/ already exists (pre-built in the wheel)
# Otherwise always (re)build — handles upgrades, editable installs, and
# pip-from-source where dist/ was never built.
if [[ "$SCRIPT_DIR" == */site-packages/* ]] && [ -d "$SCRIPT_DIR/frontend/dist" ]; then
echo "✅ Frontend pre-built (PyPI) — skipping Node/npm check."
# Skip if dist/ exists AND no tracked input is newer than dist/.
# Checks src/, public/, package.json, config files -- not just src/.
# This handles: PyPI installs (dist/ bundled), repeat runs (no changes),
# and upgrades/pulls (source newer than dist/ triggers rebuild).
_NEED_FRONTEND_BUILD=true
if [ -d "$SCRIPT_DIR/frontend/dist" ]; then
# Check top-level config files (package.json, vite.config.ts, index.html, bun.lock, etc.)
_changed=$(find "$SCRIPT_DIR/frontend" -maxdepth 1 -type f -newer "$SCRIPT_DIR/frontend/dist" 2>/dev/null | head -1)
# Check src/ and public/ recursively
if [ -z "$_changed" ]; then
_changed=$(find "$SCRIPT_DIR/frontend/src" "$SCRIPT_DIR/frontend/public" \
-type f -newer "$SCRIPT_DIR/frontend/dist" 2>/dev/null | head -1)
fi
if [ -z "$_changed" ]; then
_NEED_FRONTEND_BUILD=false
fi
fi
if [ "$_NEED_FRONTEND_BUILD" = false ]; then
echo "✅ Frontend already built and up to date -- skipping Node/npm check."
else
NEED_NODE=true
if command -v node &>/dev/null && command -v npm &>/dev/null; then
@ -146,12 +158,17 @@ run_quiet "npm run build" npm run build
_restore_gitignores
trap - EXIT
cd "$SCRIPT_DIR/backend/core/data_recipe/oxc-validator"
run_quiet "npm install (oxc validator runtime)" npm install
cd "$SCRIPT_DIR"
echo "✅ Frontend built to frontend/dist"
fi # end frontend dist check
fi # end frontend build check
# ── oxc-validator runtime (needs npm -- skip if not available) ──
if [ -d "$SCRIPT_DIR/backend/core/data_recipe/oxc-validator" ] && command -v npm &>/dev/null; then
cd "$SCRIPT_DIR/backend/core/data_recipe/oxc-validator"
run_quiet "npm install (oxc validator runtime)" npm install
cd "$SCRIPT_DIR"
fi
# ── 6. Python venv + deps ──
@ -223,50 +240,73 @@ install_python_stack() {
python "$SCRIPT_DIR/install_python_stack.py"
}
if [ "$IS_COLAB" = true ]; then
# Colab: install packages directly without venv
install_python_stack
else
# Local: create venv under ~/.unsloth/studio/ (shared location, not in repo)
STUDIO_HOME="$HOME/.unsloth/studio"
VENV_DIR="$STUDIO_HOME/.venv"
VENV_T5_DIR="$STUDIO_HOME/.venv_t5"
mkdir -p "$STUDIO_HOME"
# Create venv under ~/.unsloth/studio/ (shared location, not in repo).
# All platforms (including Colab) use the same isolated venv so that
# studio dependencies are never installed into the system Python.
STUDIO_HOME="$HOME/.unsloth/studio"
VENV_DIR="$STUDIO_HOME/.venv"
VENV_T5_DIR="$STUDIO_HOME/.venv_t5"
mkdir -p "$STUDIO_HOME"
# Clean up legacy in-repo venvs if they exist
[ -d "$REPO_ROOT/.venv" ] && rm -rf "$REPO_ROOT/.venv"
[ -d "$REPO_ROOT/.venv_overlay" ] && rm -rf "$REPO_ROOT/.venv_overlay"
[ -d "$REPO_ROOT/.venv_t5" ] && rm -rf "$REPO_ROOT/.venv_t5"
# Clean up legacy in-repo venvs if they exist
[ -d "$REPO_ROOT/.venv" ] && rm -rf "$REPO_ROOT/.venv"
[ -d "$REPO_ROOT/.venv_overlay" ] && rm -rf "$REPO_ROOT/.venv_overlay"
[ -d "$REPO_ROOT/.venv_t5" ] && rm -rf "$REPO_ROOT/.venv_t5"
rm -rf "$VENV_DIR"
rm -rf "$VENV_T5_DIR"
"$BEST_PY" -m venv "$VENV_DIR"
rm -rf "$VENV_DIR"
rm -rf "$VENV_T5_DIR"
# Try creating venv with pip; fall back to --without-pip + bootstrap
# (some environments like Colab have broken ensurepip)
if ! "$BEST_PY" -m venv "$VENV_DIR" 2>/dev/null; then
"$BEST_PY" -m venv --without-pip "$VENV_DIR"
source "$VENV_DIR/bin/activate"
cd "$SCRIPT_DIR"
install_python_stack
curl -sS https://bootstrap.pypa.io/get-pip.py | python > /dev/null
else
source "$VENV_DIR/bin/activate"
fi
# ── 6b. Pre-install transformers 5.x into .venv_t5/ ──
# Models like GLM-4.7-Flash need transformers>=5.3.0. Instead of pip-installing
# at runtime (slow, ~10-15s), we pre-install into a separate directory.
# The training subprocess just prepends .venv_t5/ to sys.path — instant switch.
echo ""
echo " Pre-installing transformers 5.x for newer model support..."
mkdir -p "$VENV_T5_DIR"
run_quiet "pip install transformers 5.x" pip install --target "$VENV_T5_DIR" --no-deps "transformers==5.3.0"
run_quiet "pip install huggingface_hub for t5" pip install --target "$VENV_T5_DIR" --no-deps "huggingface_hub==1.3.0"
echo "✅ Transformers 5.x pre-installed to $VENV_T5_DIR/"
# ── Ensure uv is available (much faster than pip) ──
USE_UV=false
if command -v uv &>/dev/null; then
USE_UV=true
elif curl -LsSf https://astral.sh/uv/install.sh | sh > /dev/null 2>&1; then
export PATH="$HOME/.local/bin:$PATH"
command -v uv &>/dev/null && USE_UV=true
fi
# ── 7. WSL: pre-install GGUF build dependencies ──
# On WSL, sudo requires a password and can't be entered during GGUF export
# (runs in a non-interactive subprocess). Install build deps here instead.
if grep -qi microsoft /proc/version 2>/dev/null; then
echo ""
echo "⚠️ WSL detected — installing build dependencies for GGUF export..."
echo " You may be prompted for your password."
sudo apt-get update -y
sudo apt-get install -y build-essential cmake curl git libcurl4-openssl-dev
echo "✅ GGUF build dependencies installed"
# Helper: install a package, preferring uv with pip fallback
fast_install() {
if [ "$USE_UV" = true ]; then
uv pip install --python "$(command -v python)" "$@" && return 0
fi
python -m pip install "$@"
}
cd "$SCRIPT_DIR"
install_python_stack
# ── 6b. Pre-install transformers 5.x into .venv_t5/ ──
# Models like GLM-4.7-Flash need transformers>=5.3.0. Instead of pip-installing
# at runtime (slow, ~10-15s), we pre-install into a separate directory.
# The training subprocess just prepends .venv_t5/ to sys.path -- instant switch.
echo ""
echo " Pre-installing transformers 5.x for newer model support..."
mkdir -p "$VENV_T5_DIR"
run_quiet "install transformers 5.x" fast_install --target "$VENV_T5_DIR" --no-deps "transformers==5.3.0"
run_quiet "install huggingface_hub for t5" fast_install --target "$VENV_T5_DIR" --no-deps "huggingface_hub==1.7.1"
run_quiet "install hf_xet for t5" fast_install --target "$VENV_T5_DIR" --no-deps "hf_xet==1.4.2"
echo "✅ Transformers 5.x pre-installed to $VENV_T5_DIR/"
# ── 7. WSL: pre-install GGUF build dependencies ──
# On WSL, sudo requires a password and can't be entered during GGUF export
# (runs in a non-interactive subprocess). Install build deps here instead.
if grep -qi microsoft /proc/version 2>/dev/null; then
echo ""
echo "⚠️ WSL detected -- installing build dependencies for GGUF export..."
echo " You may be prompted for your password."
sudo apt-get update -y
sudo apt-get install -y build-essential cmake curl git libcurl4-openssl-dev
echo "✅ GGUF build dependencies installed"
fi
# ── 8. Build llama.cpp binaries for GGUF inference + export ──
@ -405,6 +445,9 @@ if [ "$IS_COLAB" = true ]; then
echo "╠══════════════════════════════════════╣"
echo "║ Unsloth Studio is ready to start ║"
echo "║ in your Colab notebook! ║"
echo "║ ║"
echo "║ from colab import start ║"
echo "║ start() ║"
echo "╚══════════════════════════════════════╝"
else
echo "╔══════════════════════════════════════╗"