giovannino/bigboy-alma-deploy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
bigboy-alma-deploy/tasks/TASK-phase4-5-bringup.md

4.9 KiB
Raw Blame History

Task: Phase 4 + 5 bring-up — NVIDIA driver and llama.cpp

Scope

Bring BigBoy from its current state (Phase 3 complete — filesystems mounted) through Phase 4 (NVIDIA driver) and Phase 5 (llama.cpp built and serving), per planning/STATUS.md. This is a bring-up task, not the auditable release-pipeline work — see CLAUDE.md for why that's explicitly out of scope right now.

Before starting

  • Read CLAUDE.md at repo root if you haven't already this session.
  • Read planning/STATUS.md for the current authoritative state of every phase — confirm Phase 3 is actually complete before proceeding.
  • Read group_vars/bigboy.yml for hardware facts (drive UUIDs, target driver version, GPU architecture) — don't re-derive these.

Phase 4 — NVIDIA driver

Use AlmaLinux's precompiled open kernel module path, not manual DKMS. Confirmed via NVIDIA's own AlmaLinux installation guide (2026-07-17): this is the AlmaLinux-recommended method, works regardless of Secure Boot state, and needs none of the kernel-devel/kernel-headers/DKMS/GCC compilation machinery the kickstart's package list was originally written assuming. "Open" (not proprietary/closed) kernel modules are the right choice on Blackwell-generation cards like the RTX 5060 Ti regardless of precompiled-vs-DKMS.

# Enables the AlmaLinux NVIDIA driver repo + CRB + NVIDIA CUDA repo +
# EPEL in one step
sudo dnf install almalinux-release-nvidia-driver

# Compute-only / headless — no desktop GL/X components, matches this
# hardware's actual role
sudo dnf install nvidia-driver-cuda nvidia-open-kmod

sudo reboot
  • Confirm nouveau is blacklisted (should already be handled by the kickstart — verify, don't assume).
  • Do not treat group_vars/bigboy.yml's recorded 595.84 as a version to install for — that was an early planning estimate. The precompiled path installs whatever AlmaLinux's own NVIDIA driver repo currently ships. Record the actual installed version (from nvidia-smi's output) back into group_vars/bigboy.yml once confirmed, replacing the placeholder.
  • gcc from the kickstart package list is still needed — not for the driver anymore, but for compiling llama.cpp against CUDA in Phase 5. No change needed there.
  • Done when: nvidia-smi runs cleanly and reports the RTX 5060 Ti.

Phase 5 — llama.cpp

  1. Pick a pinned build tag. Check https://github.com/ggml-org/llama.cpp/releases (or git ls-remote --tags) for a recent, stable-looking tag — do not build against master. Record the chosen tag in group_vars/bigboy.yml once decided.
  2. Shallow clone at that tag, build with the standard flags from CLAUDE.md (-DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120).
  3. Model: pull a Q4_K_M or Q5_K_M GGUF from Bartowski or Unsloth on Hugging Face for initial testing — a 7B-class instruct model is sufficient to validate the pipeline end to end. Do not use an MXFP4-quantized model (see CLAUDE.md — known sm_120 build hazard).
  4. Serve via llama-server under systemd:
    • Bind 127.0.0.1, not 0.0.0.0
    • --api-key set (placeholder value is fine for initial bring-up, but note in the PR/commit that it needs a real value before any external exposure)
    • --cont-batching --parallel 4 (or a reasonable default — this isn't the tuning pass, just needs to not be single-request-only)
    • nginx reverse proxy in front, following the existing pattern already used for Nextcloud/Forgejo on jahnet — don't invent a new nginx pattern for this
  5. Firewall: only the reverse-proxy port needs opening; the llama-server port itself stays localhost-only.

Done criteria for this task

  • nvidia-smi confirms GPU visible and driver loaded
  • systemctl status llama-server shows active/running
  • curl http://127.0.0.1:<port>/health (or equivalent) returns healthy
  • One real inference request through the API returns a coherent response
  • The chosen build tag is recorded in group_vars/bigboy.yml
  • planning/STATUS.md updated to reflect Phase 4 and 5 as complete

Explicitly out of scope for this task

  • Do not build the update/promote/prune/publish scripts referenced in CLAUDE.md — that's deferred work, not this task.
  • Do not build Open WebUI (Phase 8) — llama-server's built-in UI covers the chat-first use case; Phase 8 is likely to be dropped entirely, but that's a separate decision, not part of this task.
  • Do not modify the Phase 3 filesystem/mount layout — it's already reviewed and complete. If something about it seems to be causing a problem in Phase 4/5, flag it rather than changing it directly.
  • Do not touch SELinux policy beyond what's needed to get services running (permissive discovery pass if genuinely needed, per CLAUDE.md — but this should be a temporary diagnostic step, not a final state, and not a deep dive into custom policy authoring).