Studio: keep llama-server discovery from crashing on an access-denied candidate (#6268)

* Studio: keep llama-server discovery from crashing on an access-denied candidate

_find_llama_server_binary probed candidates with Path.is_file(), which raises
PermissionError (WinError 5) when a path exists but is momentarily inaccessible
(antivirus lock, an install replace in flight, an elevated-install ACL),
aborting model validation. Treat a denied-but-present path as the real binary
so discovery returns it; absent paths still skip.

* Retry a transiently locked binary instead of returning a denied path

Returning a still-denied path only moved the PermissionError to the next
is_file() (probe_server_capabilities). Retry briefly so a transient lock
clears and discovery returns an accessible path; on a persistent lock return
nothing rather than a path downstream cannot stat.

* Studio: do not fall back to another llama-server when a pinned one is locked

A denied LLAMA_SERVER_PATH made discovery skip the explicit pin and run a
lower-priority managed or PATH binary, so a load could silently use a stale or
incompatible server. Split the probe into a file/absent/denied status: when the
pinned path exists but stays access-denied, warn and stop rather than falling
back to a different executable.

* Studio: never downgrade past a denied pinned or managed llama-server

Extend the no-fallback rule beyond LLAMA_SERVER_PATH: a present-but-denied
UNSLOTH_LLAMA_CPP_PATH or managed ($STUDIO_HOME/llama.cpp, ~/.unsloth/llama.cpp)
binary now reports temporarily-unavailable instead of silently launching a
lower-priority legacy or PATH server. Shared _scan_pinned/_unavailable helpers;
legacy in-tree and PATH stay genuine fallbacks (a denied candidate there just
continues).

* Studio: let diffusion asset lookup use a locked llama-server path for its dir

DiffusionGemma does not run llama-server; _find_diffusion_assets only needs the
install dir to find the adjacent llama-diffusion-gemma-visual-server. The
no-fallback rule returning None on a transiently locked llama-server therefore
hid an available visual-server and raised 'runner not found'. Add an
include_denied option so diffusion lookup gets the locked path (its dir is all
it needs), while inference keeps the no-denied-path, no-downgrade behavior.

* Studio: report a locked llama-server as temporarily unavailable, not missing

When the pinned/managed binary stays access-denied through the retries, discovery
returns None and load_model raised 'binary not found', a terminal error that
points users at reinstalling rather than retrying a transient AV/install lock.
Reuse include_denied to detect the locked path and raise a distinct
temporarily-unavailable, retry message instead.

* Studio: GGUF preflight treats a locked llama-server as present

The pre-download preflight (and so /api/inference/validate) used the default
discovery, which returns None for a transiently access-denied binary, so it
raised 'binary not found' for a binary that merely needs the lock to clear. Use
include_denied so the existence check counts a locked binary as present; the
load itself still reports a still-locked binary as temporarily unavailable.

studio/backend/core/inference/llama_cpp.py

@@ -1071,7 +1071,7 @@ class LlamaCppBackend:
     # ── Binary discovery ──────────────────────────────────────────
 
     @staticmethod
-    def _find_llama_server_binary() -> Optional[str]:
+    def _find_llama_server_binary(*, include_denied: bool = False) -> Optional[str]:
         """
         Locate the llama-server binary.
 
@@ -1088,28 +1088,70 @@ class LlamaCppBackend:
         """
         binary_name = "llama-server.exe" if sys.platform == "win32" else "llama-server"
 
+        def _file_status(p: Path) -> str:
+            # "file", "absent", or "denied" (exists but stays access-denied
+            # across a short retry: Windows AV/ACL or an install replace in
+            # flight). is_file() raises PermissionError (WinError 5) instead of
+            # returning False for the locked case, so never treat it as missing.
+            for _ in range(5):
+                try:
+                    return "file" if p.is_file() else "absent"
+                except PermissionError:
+                    time.sleep(0.2)
+                except OSError:
+                    return "absent"
+            return "denied"
+
+        def _is_file(p: Path) -> bool:
+            return _file_status(p) == "file"
+
+        def _layout_candidates(d: Path) -> list:
+            # build layouts probed under a llama.cpp dir, highest priority first
+            cands = [d / binary_name, d / "build" / "bin" / binary_name]
+            if sys.platform == "win32":
+                cands.append(d / "build" / "bin" / "Release" / binary_name)
+            return cands
+
+        def _unavailable(p: object) -> None:
+            # a pinned or managed binary that exists but is access-denied: report
+            # it instead of silently downgrading to a lower-priority llama-server
+            logger.warning(
+                f"llama-server at {p} exists but is access-denied (antivirus or "
+                "an in-flight install); not falling back to another binary, "
+                "retry once it is released"
+            )
+            return None
+
+        def _scan_pinned(paths: list):
+            # first existing candidate wins -> (path, None); a present-but-denied
+            # one -> (None, denied_path) so the caller reports it rather than
+            # skipping to a lower-priority location. include_denied returns the
+            # locked path instead: diffusion asset lookup only needs its dir.
+            for p in paths:
+                st = _file_status(p)
+                if st == "file":
+                    return str(p), None
+                if st == "denied":
+                    return (str(p), None) if include_denied else (None, p)
+            return None, None
+
         # 1. Env var: direct path to binary
         env_path = os.environ.get("LLAMA_SERVER_PATH")
-        if env_path and Path(env_path).is_file():
-            return env_path
+        if env_path:
+            hit, locked = _scan_pinned([Path(env_path)])
+            if locked is not None:
+                return _unavailable(locked)
+            if hit:
+                return hit
 
         # 1b. UNSLOTH_LLAMA_CPP_PATH: custom llama.cpp install dir
         custom_llama_cpp = os.environ.get("UNSLOTH_LLAMA_CPP_PATH")
         if custom_llama_cpp:
-            custom_dir = Path(custom_llama_cpp)
-            # Root dir (make builds)
-            root_bin = custom_dir / binary_name
-            if root_bin.is_file():
-                return str(root_bin)
-            # build/bin/ (cmake on Linux)
-            cmake_bin = custom_dir / "build" / "bin" / binary_name
-            if cmake_bin.is_file():
-                return str(cmake_bin)
-            # build/bin/Release/ (cmake on Windows)
-            if sys.platform == "win32":
-                win_bin = custom_dir / "build" / "bin" / "Release" / binary_name
-                if win_bin.is_file():
-                    return str(win_bin)
+            hit, locked = _scan_pinned(_layout_candidates(Path(custom_llama_cpp)))
+            if locked is not None:
+                return _unavailable(locked)
+            if hit:
+                return hit
 
         # 2-4. Match installer layout: env-mode -> $STUDIO_HOME/llama.cpp;
         # default/HOME-redirect -> ~/.unsloth/llama.cpp (sibling of studio).
@@ -1141,31 +1183,18 @@ class LlamaCppBackend:
                 _seen_roots.add(k)
                 _unique_roots.append(r)
         for unsloth_home in _unique_roots:
-            home_root = unsloth_home / binary_name
-            if home_root.is_file():
-                return str(home_root)
-            home_linux = unsloth_home / "build" / "bin" / binary_name
-            if home_linux.is_file():
-                return str(home_linux)
-            if sys.platform == "win32":
-                home_win = unsloth_home / "build" / "bin" / "Release" / binary_name
-                if home_win.is_file():
-                    return str(home_win)
+            hit, locked = _scan_pinned(_layout_candidates(unsloth_home))
+            if locked is not None:
+                return _unavailable(locked)
+            if hit:
+                return hit
 
-        # 5-6. Legacy: in-tree build (older setup.sh / setup.ps1)
+        # 5-6. Legacy: in-tree build (older setup.sh / setup.ps1). A fallback,
+        # so a denied candidate here just continues (no no-fallback halt).
         project_root = Path(__file__).resolve().parents[4]
-        # Root dir (make builds)
-        root_path = project_root / "llama.cpp" / binary_name
-        if root_path.is_file():
-            return str(root_path)
-        # build/bin/ (cmake builds)
-        build_path = project_root / "llama.cpp" / "build" / "bin" / binary_name
-        if build_path.is_file():
-            return str(build_path)
-        if sys.platform == "win32":
-            win_path = project_root / "llama.cpp" / "build" / "bin" / "Release" / binary_name
-            if win_path.is_file():
-                return str(win_path)
+        for p in _layout_candidates(project_root / "llama.cpp"):
+            if _is_file(p):
+                return str(p)
 
         # 7. System PATH
         system_path = shutil.which("llama-server")
@@ -1174,7 +1203,7 @@ class LlamaCppBackend:
 
         # 8. Legacy: extracted to bin/
         bin_path = project_root / "bin" / binary_name
-        if bin_path.is_file():
+        if _is_file(bin_path):
             return str(bin_path)
 
         return None
@@ -2438,7 +2467,9 @@ class LlamaCppBackend:
         visual_bin = os.environ.get("DG_VISUAL_BIN")
         if not visual_bin:
             name = "llama-diffusion-gemma-visual-server" + (".exe" if os.name == "nt" else "")
-            base = self._find_llama_server_binary()
+            # include_denied: a transiently locked llama-server still pins the
+            # install dir so the adjacent visual-server can be found
+            base = self._find_llama_server_binary(include_denied = True)
             if base:
                 base_dir = Path(base).parent
                 for cand in (
@@ -3503,6 +3534,15 @@ class LlamaCppBackend:
                     )
 
             if not binary:
+                # distinguish a transiently locked binary (antivirus / in-flight
+                # install) from a missing one so the user retries, not reinstalls
+                locked = self._find_llama_server_binary(include_denied = True)
+                if locked:
+                    raise RuntimeError(
+                        f"llama-server at {locked} is temporarily unavailable "
+                        "(access-denied; antivirus or an in-flight install). "
+                        "Retry the load once it is released."
+                    )
                 raise RuntimeError(
                     "llama-server binary not found. "
                     "Run setup.sh to build it, install llama.cpp, "

studio/backend/utils/models/model_config.py

@@ -2361,10 +2361,13 @@ class ModelConfig:
             # Does the HF repo contain GGUF files?
             gguf_filename = detect_gguf_model_remote(identifier, hf_token = hf_token)
             if gguf_filename:
-                # Preflight: verify llama-server binary exists before a multi-GB download
+                # Preflight: verify llama-server binary exists before a multi-GB
+                # download. include_denied: a transiently locked binary still
+                # exists (the lock clears long before the download finishes; the
+                # load itself reports a still-locked binary distinctly).
                 from core.inference.llama_cpp import LlamaCppBackend
 
-                if not LlamaCppBackend._find_llama_server_binary():
+                if not LlamaCppBackend._find_llama_server_binary(include_denied = True):
                     raise RuntimeError(
                         "llama-server binary not found — cannot load GGUF models. "
                         "Run setup.sh to build it, or set LLAMA_SERVER_PATH."