ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
unsloth/.gitignore
Daniel Han 6d27160dcc
 Studio: graceful recovery ladder when llama-server hard-crashes at startup (#6291) 
* Studio: fall back to text-only when a vision projector hard-crashes llama-server

The text-only mmproj fallback (#6075) only fired when llama-server printed a
recognizable projector-format error ("Unknown projector type", exit -6). An
installed llama.cpp that predates a model's projector can instead SIGSEGV
(exit -11) with no parseable output, e.g. unsloth/Qwen3.5-4B-MTP-GGUF +
mmproj-F16 on an older gfx1151 prebuilt: llama-server crashes on load, the
--fit off retry crashes the same way, and load_model gives up with a hard 500
instead of dropping vision.

Generalize the decision: a vision (--mmproj) launch killed by a signal (POSIX
returncode < 0, e.g. -11 SIGSEGV / -6 SIGABRT; Windows 0xC0000000+ access
violation) is treated like a projector incompatibility, so the load retries
once text-only. The retry is skipped if a cancel/unload is pending, mirroring
the MTP guard. Clean non-zero exits (bad GGUF, port bind) and hung processes
keep their own handling; non-vision launches are unaffected.

Reproduced and verified on gfx1151 (Radeon 8060S, ROCm 7.2.1): a current
prebuilt (llama.cpp b9596) loads the exact model + args fine, confirming the
crash is a stale prebuilt. With a wrapper that SIGSEGVs on --mmproj, Studio now
recovers: the load returns 200 (is_vision=false) and serves at ~31 tok/s
text-only instead of failing. New _is_signal_crash helper plus tests pin the
decision.

Also normalize a few em-dashes to ASCII punctuation in existing comments.

* Studio: refine mmproj hard-crash fallback (signal scope + last argv)

- Limit _is_signal_crash to genuine program faults (SIGSEGV, SIGABRT,
  SIGILL, SIGFPE, SIGBUS) and Windows 0xC0000000+ statuses. SIGKILL,
  SIGTERM and SIGINT no longer count, so an OOM-killer, unload or
  supervisor kill is not masked as a projector incompatibility.
- Strip --mmproj from the last attempted argv so the text-only retry
  keeps --fit off / --spec-default instead of resurrecting the original
  spec flags (matters for MTP vision models on an older llama.cpp).
- Drop stray temp files committed by mistake and gitignore the "~" dir
  so they cannot be re-added.

* Studio: tighten comments in mmproj hard-crash fallback

* Studio: retry --flash-attn off before dropping vision on a startup crash

When llama-server hard-crashes at startup, the recovery chain now tries the
least-destructive mitigation first. Flash-attention kernels SIGSEGV at load on
some ROCm/GPU builds (often inside the vision tower's attention); disabling
flash attention keeps BOTH vision and MTP, so a hard program fault with
--flash-attn on now retries once with --flash-attn off before the MTP-drop or
the text-only (mmproj-strip) fallbacks. _is_signal_crash already gates this to
genuine faults (SIGSEGV/SIGABRT/SIGILL/SIGFPE/SIGBUS), so an OOM-kill or unload
(SIGKILL/SIGTERM/SIGINT) does not trigger a retry.

Field context: a gfx1151 user crashes loading a vision GGUF even on the latest
prebuilt, so an update cannot help, and the same model and args load fine on
another gfx1151 box, pointing at a runtime/flash-attn fault. New
_with_flash_attn_off helper plus tests. Verified on hardware with a wrapper
that SIGSEGVs on --flash-attn on: Studio recovers with is_vision=true (vision
and MTP intact) instead of failing or losing vision.

* Studio: name the OOM kill on a too-large model load

When the OS kills llama-server with no diagnostic output (SIGKILL/SIGTERM,
almost always the OOM killer, e.g. a BF16 model too large for the WSL VM's
RAM cap), the recovery ladder correctly does not retry an external kill, so
this is the message the user sees. It fell through to the generic "is the
GGUF valid / out of memory" text. Make it actionable: name the signal and
point at a smaller or more quantized GGUF, a lower context length, or raising
the WSL memory limit. Output-based diagnoses still win and a hard fault keeps
the generic fallback.

* Studio: refuse a model too large for system RAM on a unified-memory APU

On gfx1150/gfx1151 APUs the weights load into shared system RAM (GGML
unified memory). _get_gpu_free_memory reports the full ROCm/APU budget as
free (often ~100 GB), but under WSL the VM's RAM cap is the real ceiling.
Studio trusted the budget, spawned a load larger than RAM, and the OS killed
it mid-flight, taking the Studio process with it (a silent "Terminated" with
no error, the model resident in RAM not VRAM).

Add a pre-flight guard on the APU path: if the weights exceed available
system RAM (psutil, then /proc/meminfo), refuse before spawning with a clear
message (smaller/more-quantized GGUF, lower context, or raise the WSL memory
limit). Weights only so KV/context auto-reduction is not double-counted;
unknown RAM never refuses; non-APU and discrete-GPU paths are untouched.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: refine recovery ladder (keep diagnosed errors, flip all flash-attn)

Two review points on the hard-crash recovery ladder:

1. The signal-only text-only fallback stripped --mmproj on any hard fault,
   even when llama-server had already printed a non-projector cause (an OOM
   such as "cudaMalloc failed: out of memory", an unsupported architecture, or
   a tensor-parallel limit). That masked the real error and told the user to
   update llama.cpp for vision. New _output_has_nonprojector_diagnostic gates
   the signal path: it fires only when no such marker is present, so a bare
   SIGSEGV with no output still retries text-only, but a diagnosed OOM surfaces
   the real error instead of silently dropping vision.

2. _with_flash_attn_off only flipped the first --flash-attn. llama.cpp is
   last-wins, so a leftover enable from extra_args (--flash-attn on, -fa on, or
   the = form) could keep flash attention on and re-crash the retry. It now
   flips every occurrence and returns None only when nothing is flippable.

test_llama_cpp_mmproj_fallback.py and the classification/APU suites: 103 passed.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: pass the text-only retry's exit code to the failure classifier

When the text-only fallback retry itself fails, read its exit code before
_kill_process() clears it and forward it to _classify_llama_start_failure, so an
OS-killed retry surfaces the actionable out-of-memory message instead of the
generic one (matching the primary failure path).

* Studio: scope APU RAM guard to selected GPUs, count MTP drafter, neutral SIGTERM

Three refinements to the startup recovery work in this PR:

- The unified-memory APU RAM guard fired whenever any visible GPU was a
  gfx1150/gfx1151 APU, so on a mixed APU+dGPU host it could refuse a valid
  load placed on the discrete GPU. Scope _amd_apu_wants_unified_memory to the
  selected gpu_indices (physical ids, mapped via CUDA_VISIBLE_DEVICES like
  _is_datacenter_gpu); None still means every visible GPU. Applied to both the
  RAM guard and the GGML_CUDA_ENABLE_UNIFIED_MEMORY env set.
- The RAM guard counted only the main GGUF plus mmproj, so a separate MTP
  drafter (also resident in unified system RAM, even when offloaded to CPU)
  could push the load past the RAM cap and still get OS-killed mid-load. Add
  the drafter weights to the APU RAM total.
- The startup classifier reported SIGTERM (-15) as 'most likely out of memory',
  but SIGTERM is also how an unload/cancel or a supervisor stops the server.
  Keep the OOM wording for SIGKILL (-9, the OOM killer) and report -15
  neutrally.

Tests updated/added accordingly.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: address review on the APU guard and decode-probe ladder

- Map APU physical ids via the active ROCm mask (HIP, then ROCR, then CUDA),
  mirroring _get_gpu_memory, so a HIP_VISIBLE_DEVICES-selected APU is matched.
- Only add the MTP drafter to the APU RAM total when MTP will actually engage,
  so a stale LLAMA_ARG_SPEC_DRAFT_MODEL cannot refuse a non-MTP load.
- After an MTP first-decode hard fault, retry --flash-attn off (keeps MTP)
  before dropping speculative decoding, matching the startup rung.
- Fold the --flash-attn= / -fa= rewrite into one branch.

Tests: tensor-parallel decode-probe assertion updated for the FA-off rung.

* Studio: tighten two comments in the APU guard and RAM preflight

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: refine flash-attn retry and APU RAM guard per review

- _with_flash_attn_off now decides on the effective last-wins value: it returns
  None when FA is already off (no wasted retry), and neutralizes a bare
  --flash-attn / -fa (which llama.cpp reads as on) so the retry cannot re-enable
  it. Length is preserved so downstream index slices stay valid.
- _amd_apu_wants_unified_memory uses 'gpu_indices is not None' so an empty
  selection is respected (not treated as all-visible).
- The APU RAM refusal now checks the base model only (main + mmproj); an
  optional MTP drafter is dropped by the existing MTP-drop fallback rather than
  causing a hard pre-spawn refusal of an otherwise loadable model.

Tests: bare-flag / effective-off / empty-selection / HIP-mask cases added.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-06-18 09:07:25 -07:00

239 lines
4.6 KiB
Text
Raw Permalink Blame History

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*.class
unsloth_compiled_cache/
# Notebook-validator runtime PyPI metadata cache (CI repopulates).
scripts/data/pypi_cache/
# ML artifacts (large files)
feature/
outputs/
exports/
/datasets/
studio/backend/assets/datasets/
unsloth_training_checkpoints/
*.gguf
*.safetensors
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
/lib/
/lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# UV
# Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
#uv.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/latest/usage/project/#working-with-version-control
.pdm.toml
.pdm-python
.pdm-build/
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
.venv_overlay/
.venv_t5/
environment.yaml
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
# Ruff stuff:
.ruff_cache/
.pre-commit-cache/
# PyPI configuration file and IDE/Editors
.pypirc
.vscode
.idea/
.claude/
*.swp
*.swo
# oh-my-codex
.omx/
# Firebase
firebase-debug.log
# Other
resources/
tmp/
**/node_modules/
auth.db
# Tauri local build/generated output
studio/src-tauri/target/
studio/src-tauri/gen/
studio/src-tauri/artifacts/
studio/src-tauri/icons/android/
studio/src-tauri/icons/ios/
studio/src-tauri/icons/128x128@2x.png
studio/src-tauri/icons/64x64.png
studio/src-tauri/icons/Square*Logo.png
studio/src-tauri/icons/StoreLogo.png
studio/src-tauri/icons/squarehq.png
# Local working docs
**/CLAUDE.md
**/claude.md
**/AGENT.md
**/agent.md
docs/canvas-lab-architecture.md
log_rtx.txt
log.txt
setup_leo.sh
server.pid
*.log
# Ignore stray lockfiles; real npm projects opt back in below (npm ci needs them).
package-lock.json
!studio/frontend/package-lock.json
!studio/backend/core/data_recipe/oxc-validator/package-lock.json
!studio/package-lock.json
llama.cpp/
# Stray "~" dir some tools create from a literal ~ TMPDIR; never part of the repo.
/~/
Reference in a new issue View git blame Copy permalink