Task: Phase 4 + 5 bring-up — NVIDIA driver and llama.cpp

Scope

Bring BigBoy from its current state (Phase 3 complete — filesystems mounted) through Phase 4 (NVIDIA driver) and Phase 5 (llama.cpp built and serving), per planning/STATUS.md. This is a bring-up task, not the auditable release-pipeline work — see CLAUDE.md for why that's explicitly out of scope right now.

Before starting

Read CLAUDE.md at repo root if you haven't already this session.
Read planning/STATUS.md for the current authoritative state of every phase — confirm Phase 3 is actually complete before proceeding.
Read group_vars/bigboy.yml for hardware facts (drive UUIDs, target driver version, GPU architecture) — don't re-derive these.

Phase 4 — NVIDIA driver

Use AlmaLinux's precompiled open kernel module path, not manual DKMS. Confirmed via NVIDIA's own AlmaLinux installation guide (2026-07-17): this is the AlmaLinux-recommended method, works regardless of Secure Boot state, and needs none of the kernel-devel/kernel-headers/DKMS/GCC compilation machinery the kickstart's package list was originally written assuming. "Open" (not proprietary/closed) kernel modules are the right choice on Blackwell-generation cards like the RTX 5060 Ti regardless of precompiled-vs-DKMS.

# Enables the AlmaLinux NVIDIA driver repo + CRB + NVIDIA CUDA repo +
# EPEL in one step
sudo dnf install almalinux-release-nvidia-driver

# Compute-only / headless — no desktop GL/X components, matches this
# hardware's actual role
sudo dnf install nvidia-driver-cuda nvidia-open-kmod

sudo reboot

Confirm nouveau is blacklisted (should already be handled by the kickstart — verify, don't assume).
Do not treat group_vars/bigboy.yml's recorded 595.84 as a version to install for — that was an early planning estimate. The precompiled path installs whatever AlmaLinux's own NVIDIA driver repo currently ships. Record the actual installed version (from nvidia-smi's output) back into group_vars/bigboy.yml once confirmed, replacing the placeholder.
gcc from the kickstart package list is still needed — not for the driver anymore, but for compiling llama.cpp against CUDA in Phase 5. No change needed there.
Done when: nvidia-smi runs cleanly and reports the RTX 5060 Ti.

Phase 5 — llama.cpp

Pinned build tag: b9968. Already confirmed and cloned into the repo — checked against real commit dates (8 days old at time of selection, not tracking master) and reviewed for any fix/regression/ revert pattern in the surrounding range (none found; one relevant fix — llama-batch: fix allowed decreasing pos in a seq — is already included in this tag, not something still pending). Do not re-derive or second-guess this pin; if a newer tag is wanted later, that's a deliberate monthly-cadence decision, not part of this task.
Shallow clone at that tag, build with the standard flags from CLAUDE.md (-DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120).
Model: Ministral 3 14B Instruct, Q4_K_M. This is the confirmed choice for initial bring-up and testing — not a placeholder pending a better pick. Pull via llama.cpp's own Hugging Face integration:
```
llama-server -hf mistralai/Ministral-3-14B-Instruct-2512-GGUF:Q4_K_M \
  -ngl 99 --jinja
```
--jinja is required for this model's chat template/system-prompt handling per its own model card — don't drop it. libcurl-devel must be installed first (confirmed available via dnf from AppStream, per the package recon — not yet installed as of this writing, add it alongside cmake and nginx, also confirmed-available-but-not-yet- installed in that same recon pass).

Root's ~499GB gives plenty of headroom to keep other quants or models around later without needing to commit permanently — this pick is about getting a clean first success, not a final answer for every future use case. Do not use an MXFP4-quantized model regardless of which model is chosen (see CLAUDE.md — known sm_120 build hazard).
Serve via llama-server under systemd:
- Bind 127.0.0.1, not 0.0.0.0
- --api-key set (placeholder value is fine for initial bring-up, but note in the PR/commit that it needs a real value before any external exposure)
- --cont-batching --parallel 4 (or a reasonable default — this isn't the tuning pass, just needs to not be single-request-only)
- nginx reverse proxy in front, following the existing pattern already used for Nextcloud/Forgejo on jahnet — don't invent a new nginx pattern for this
Firewall: only the reverse-proxy port needs opening; the llama-server port itself stays localhost-only.

Success criterion — deliberately minimal for this task

Ministral 3 14B Instruct installed and reachable through llama-server's built-in web UI, producing a coherent chat response — at whatever quality it delivers with zero tuning or refinement. This is the bar for a successful deploy at this stage. Model selection, quantization tradeoffs, and response-quality tuning are all separate, later work — do not block this task on any of that.

Done criteria for this task

nvidia-smi confirms GPU visible and driver loaded
systemctl status llama-server shows active/running
curl http://127.0.0.1:<port>/health (or equivalent) returns healthy
A real chat request through the built-in web UI (or the API directly) returns a coherent response from Ministral 3 14B Instruct — quality as- delivered, no tuning required
b9968 and the Ministral 3 14B Instruct choice recorded in group_vars/bigboy.yml
planning/STATUS.md updated to reflect Phase 4 and 5 as complete

Explicitly out of scope for this task

Do not build the update/promote/prune/publish scripts referenced in CLAUDE.md — that's deferred work, not this task.
Do not build Open WebUI (Phase 8) — llama-server's built-in UI covers the chat-first use case; Phase 8 is likely to be dropped entirely, but that's a separate decision, not part of this task.
Do not modify the Phase 3 filesystem/mount layout — it's already reviewed and complete. If something about it seems to be causing a problem in Phase 4/5, flag it rather than changing it directly.
Do not touch SELinux policy beyond what's needed to get services running (permissive discovery pass if genuinely needed, per CLAUDE.md — but this should be a temporary diagnostic step, not a final state, and not a deep dive into custom policy authoring).

6.5 KiB Raw Blame History