* fix: disable remote code loading for ai-assist model hint lookup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
- Workers now compute backend_path and venv_t5 locally via Path(__file__)
- Moved .venv_t5 to ~/.unsloth/studio/.venv_t5
- Added ensure_studio_directories() call on server startup
- Expanded CLI studio command into sub-app with setup subcommand
startswith(prefix) could match unrelated split variants whose names
extend the selected file's prefix (e.g. model-Q8_0-v2-00001-of-...).
Now builds an exact regex from the chosen file's base prefix and shard
total so only true siblings are downloaded.
Substring matching (e.g. "Q8_0" in filename) could match superset
variants like "IQ8_0", causing wrong quantizations to be downloaded.
Now uses word-boundary regex for variant matching and discovers split
shards by shared filename prefix rather than treating all variant
matches as shards.
LlamaCppBackend.load_model() only downloaded the first matching GGUF
file. For split models (e.g. 7B Q8_0 with 3 shards), llama-server
needs all shards present. Now collects and downloads all matching files.
Two overlapping /chat/completions requests could both read from the shared
resp_queue, consuming and dropping each other's token events. Replace the
request_id filtering (which silently dropped non-matching messages) with a
threading.Lock that serializes generation — correct for single-GPU inference.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reusing a subprocess after unsloth patches torch internals causes
inspect.getsource() failures when loading a different model type.
Each load now gets a clean Python interpreter.
Replaces cmd_queue-based cancel polling with a shared mp.Event.
Fixes two issues:
- Loading a new model while generating no longer hangs (cancel is instant)
- Subprocess shuts down cleanly after explicit stop generation
Inference now runs in a persistent subprocess, solving the same
transformers version-switching problem that was fixed for training.
The subprocess stays alive between requests (model in GPU memory)
and is only restarted when switching transformers versions.
New files:
- core/inference/worker.py: subprocess entry point with command loop
- core/inference/orchestrator.py: parent-side proxy with same API
Modified:
- core/inference/__init__.py: exports orchestrator as default backend
- routes/inference.py: removed in-process ensure_transformers_version()
Fixes two bugs:
1. Chat template tags (<|im_start|>, <|im_end|>) leaking into output
because /v1/completions treated them as literal text
2. Image hallucination because image_b64 was never passed to llama-server
Now llama-server handles chat templates natively and receives images
as OpenAI-format multimodal content parts for vision models.
Replace Python-side GGUF download with llama-server's native -hf flag for
HuggingFace repos. Add frontend variant picker so users can choose
quantization (Q4_K_M, Q8_0, BF16, etc.) with file sizes. Fix vision
detection via mmproj files instead of hardcoding is_vision=False.
