Update project

CLAUDE.md

@@ -0,0 +1,96 @@
+# CLAUDE.md — bigboy-alma-deploy
+
+This file is read automatically at the start of every Claude Code session in
+this repo. It captures standing decisions so they don't need to be
+re-explained or re-litigated each session. If something here conflicts with
+a task doc you've been handed, the task doc wins for that session — but
+flag the conflict rather than silently picking one.
+
+## What this repo is
+
+Deployment automation for BigBoy — a Ryzen 5 / RTX 5060 Ti (16GB) AI
+inference server, running AlmaLinux 10.2. BigBoy also serves as the
+reference/testbed machine for CE's future RHEL + llama.cpp client
+deployments — meaning decisions made here are expected to generalize, not
+just work once.
+
+## Source of truth for status
+
+`plannng/STATUS.md` is authoritative for what's actually done vs. pending.
+Read it before starting work. Update it when a phase's status changes —
+don't leave it stale.
+
+## Standing technical decisions (do not re-derive or second-guess these)
+
+- **OS**: AlmaLinux 10.2, moved from NixOS due to RTX 5060 Ti driver
+  friction on Nix. RHEL-family chosen fleet-wide for AI-server workloads
+  for compliance/audit reasons (EU AI Act relevance), not just this one box.
+- **Inference engine**: llama.cpp, built from source — deliberately not
+  Ollama. Reasons: full control over CUDA build flags and quantization,
+  built-in web UI via `llama-server` removes the need for a separate
+  Open WebUI layer. Ollama also just wraps llama.cpp's ggml engine
+  underneath on NVIDIA/Linux anyway, so switching back would add a layer
+  of indirection without avoiding the dependency.
+- **GPU target**: `-DCMAKE_CUDA_ARCHITECTURES=120` (Blackwell / sm_120).
+  Requires NVIDIA driver ≥570; target driver is `595.84` (see
+  `group_vars/bigboy.yml`).
+- **Known hazard**: MXFP4-quantized models have open compilation issues on
+  sm_120 as of mid-2026. Stick to standard GGUF quants — Q4_K_M or Q5_K_M —
+  from established quantizers (Bartowski, Unsloth namespaces on Hugging
+  Face). This sidesteps the issue entirely; it is not a performance
+  preference, it's a build-stability one.
+- **Version discipline**: llama.cpp has no semantic versioning — continuous
+  build-tagged releases only. Never build against `master`/HEAD. Every
+  build pins a specific tag, recorded in `group_vars/bigboy.yml`.
+- **Build flags**: standard only — `cmake -B build -DGGML_CUDA=ON
+  -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120`. No exotic
+  tuning flags (e.g. `-DLLAMA_CUDA_MMV_Y`) unless a specific, documented
+  problem requires one. We are already deep in non-standard territory
+  (Blackwell + Alma + sovereignty logging); the inference engine build
+  itself should be as boring and reproducible as possible.
+- **Serving**: `llama-server` under systemd, bound to `127.0.0.1`, fronted
+  by nginx (reuse the existing CE reverse-proxy pattern from jahnet — don't
+  invent a new one). `--api-key` enforced even on LAN-only. `--cont-batching
+  --parallel N` for concurrent chat sessions.
+- **SELinux**: stays enforcing. If early bring-up needs a permissive
+  discovery pass to collect AVC denials via `ausearch`/`audit2allow`,
+  that's a temporary diagnostic state, not a resting state — flip back to
+  enforcing before considering the phase done.
+
+## Style requirements for any script in this repo
+
+Follow the CE OS Bash Style Guide (`Updated_Bash_Style_Guide` in the CE
+project knowledge) for every script:
+- Attribution header (dwarves first, then John A. Hoeven / Claude AI),
+  licence (Unlicense), version, status
+- No `set -e` — every operation checked and logged explicitly
+- Cleanup trap registered before work begins
+- Single confirmation prompt before any system-modifying action
+- No silent failures — every error path logs and either hard-fails or warns
+- Never assume root; invoke `sudo` explicitly for privileged steps only
+
+This repo is a single-purpose ops repo for one Alma server, not part of the
+CEOS multi-distro installer framework — so skip the `ce_env.conf` /
+`pkg_*` / `CE_PRIV` abstraction layer specifically. Use `sudo` directly.
+Everything else in the style guide applies.
+
+## Explicitly deferred — do not build unless a task doc asks for it
+
+An auditable testing/release system for llama.cpp (versioned releases,
+btrfs snapshot safety net, promote/rollback via symlink swap, a Forgejo
+mirror that only ever receives BigBoy-validated tags so client
+deployments never pull raw upstream) was designed in detail on
+2026-07-16, but is **not yet built or integrated into the phase plan**.
+Draft, untested scripts from that design may exist under `scripts/draft/`.
+Do not wire them into the active Ansible roles or treat them as
+representing current repo state — they're a reference for later, not a
+task in progress. Bringing BigBoy up and running is the current priority;
+this gets picked up afterward.
+
+## Scope discipline
+
+Task docs will name a specific phase or task from `STATUS.md`. Do the
+named task. Don't refactor, "improve," or extend adjacent phases that are
+already marked complete or reviewed, even if something nearby looks
+improvable — flag it instead and let it be a deliberate decision, not a
+side effect.

README.md

@@ -1,3 +1,199 @@
 # bigboy-alma-deploy
 
-Refocusing @bigboy AI server to Alma for access to GPU drivers and familiarity.
+**Sovereign AI inference server: RTX 5060 Ti GPU (16GB VRAM), 5-drive btrfs storage on AlmaLinux 10.2**
+
+---
+
+## Quick Links
+
+- **Architecture & Design:** `./plannng/ARCHITECTURE.md`
+- **Deployment Status:** `./plannng/STATUS.md`
+- **Hardware Manifest:** `./plannng/HARDWARE.md`
+- **Sovereignty Policy:** `./plannng/SOVEREIGNTY-POLICY.md`
+- **Claude Code standing context:** `./CLAUDE.md`
+- **Current task handoff:** `./tasks/`
+
+---
+
+## Project Overview
+
+### Goal
+
+Deploy BigBoy as a fully autonomous, idempotent AI inference server running
+AlmaLinux 10.2 with GPU acceleration via NVIDIA RTX 5060 Ti. The system
+serves local models via **llama.cpp** (`llama-server`, which includes its
+own OpenAI-compatible API and web UI) and maintains reproducible deployment
+logs for the RAG corpus. BigBoy also serves as the reference/testbed
+machine for CE's future RHEL + llama.cpp client deployments — decisions
+made here are expected to generalize, not just work once.
+
+### Why AlmaLinux 10?
+
+- RHEL-compatible (10-year support lifecycle)
+- Official NVIDIA precompiled GPU driver support
+- DNF package manager with clear upgrade path
+- Pragmatic middle ground: better stability than NixOS on this hardware
+  (the original NixOS attempt hit unresolved RTX 5060 Ti driver friction),
+  simpler than Debian for this workload
+- RHEL-family chosen fleet-wide for AI-server workloads specifically, for
+  compliance/audit reasons (EU AI Act relevance) — not just a fix for this
+  one box
+
+### Why llama.cpp, not Ollama?
+
+Decided deliberately, not a default: full control over CUDA build flags
+and quantization, and `llama-server`'s built-in web UI removes the need
+for a separate Open WebUI layer entirely. Worth knowing: Ollama itself
+wraps llama.cpp's ggml engine on NVIDIA/Linux under the hood, so it
+wouldn't have avoided this dependency — it would only have added a layer
+of indirection around it. See `CLAUDE.md` for the full standing rationale
+(pinned-tag discipline, standard build flags, known sm_120/MXFP4 hazard).
+
+### Key Constraints
+
+- **GPU VRAM:** 16GB (RTX 5060 Ti) — model selection accordingly
+- **Storage:** 5 drives (1 NVMe + 4 SATA) with btrfs subvolume isolation
+- **Network:** bench LAN (192.168.0.0/24), workbench acts as gateway/DHCP
+  for this segment
+- **Idempotency:** every phase must be re-runnable
+
+---
+
+## Current Status — read this before assuming anything is done
+
+**As of 2026-07-16: BigBoy has no OS installed.** The machine currently
+holds only an abandoned NixOS install attempt. Nothing described as
+"complete" below has been validated on real hardware yet — it means
+*designed, written, and reviewed*, not *running*.
+
+See `./plannng/STATUS.md` for the authoritative, current per-phase state.
+Don't rely on this README for phase-by-phase status — it will drift out
+of date faster than `STATUS.md` is maintained.
+
+### Design-complete (not yet run on hardware)
+
+- Kickstart (`alma10-minimal-bigboy.ks`) — unattended AlmaLinux 10.2
+  minimal install
+- Ansible main playbook scaffold (`ansible/site.yml`, 14-phase
+  orchestration)
+- Phase 3 role (filesystem mounting) — written and reviewed
+- Hardware inventory & UUID manifest (`plannng/HARDWARE.md`)
+
+### Ready to build, once Phase 1–3 actually run on hardware
+
+- Phase 4 (NVIDIA driver)
+- Phase 5 (llama.cpp — build from pinned tag, standard flags, see
+  `CLAUDE.md`)
+
+### Likely dropping
+
+- Phase 8 (Open WebUI) — `llama-server`'s built-in UI covers the
+  chat-first use case; keeping this phase is now a deliberate decision
+  to make, not a default
+
+### Deferred, on purpose
+
+- Phases 6–7, 9–14 (Build Suite, Configuration, Security, Borgmatic,
+  Thermal, Validation, Migration, Observation)
+- An auditable testing/release system for llama.cpp (versioned releases,
+  btrfs snapshot safety net, promote/rollback, a Forgejo mirror that only
+  ever receives BigBoy-validated tags for client deployments) — fully
+  designed 2026-07-16, not yet built. See `CLAUDE.md` for the deferred-work
+  note and `scripts/draft/` if present.
+- Borgmatic backup target (not yet decided)
+- Home LAN migration (waiting on case installation)
+
+---
+
+## Collaborators & Workflow
+
+### John A. Hoeven (@workbench)
+- Architect & operator — makes deployment decisions, edits and pushes
+  from workbench via VS Codium
+
+### Claude Code (@bigboy once installed, or driven from @workbench)
+- Executes scoped tasks against the actual repo — see `CLAUDE.md` for
+  standing context read automatically each session, and `tasks/` for the
+  current scoped handoff
+- Does not decide scope — task docs name exactly what's in and out of
+  bounds for a given session
+
+### Claude Desktop
+- Plans and orchestrates: architecture discussion, design decisions,
+  drafts `CLAUDE.md` and task docs for Claude Code to execute against
+
+---
+
+## Repo Structure
+
+```
+bigboy-alma-deploy/
+├── CLAUDE.md                    # standing context, read by Claude Code every session
+├── README.md                    # this file
+├── alma10-minimal-bigboy.ks     # canonical kickstart — this copy, not plannng/'s
+├── install-phases.txt           # 15-phase (0-14) plan, high level
+├── ansible/
+│   ├── site.yml                 # main playbook
+│   ├── inventory.ini
+│   ├── group_vars/bigboy.yml    # hardware facts: UUIDs, driver version, pinned llama.cpp tag
+│   └── roles/
+│       └── phase-3-filesystems/ # written, reviewed, not yet run on hardware
+├── plannng/                      # (sic — established directory name, not a typo to fix casually)
+│   ├── ARCHITECTURE.md
+│   ├── HARDWARE.md
+│   ├── SOVEREIGNTY-POLICY.md
+│   ├── STATUS.md                # authoritative current status — check this first
+│   ├── alma-*.md                 # Alma-specific reference docs (firewall, package mgmt, nvidia, etc.)
+│   └── alma10-minimal-bigboy.ks  # reference snapshot from planning — root copy is canonical
+├── tasks/
+│   └── TASK-*.md                 # scoped Claude Code handoffs, one per unit of work
+└── scripts/draft/                 # (if present) draft auditable-release-pipeline scripts —
+                                    # unintegrated, see CLAUDE.md before touching
+```
+
+---
+
+## Getting Started
+
+### To review design
+1. Read `./plannng/ARCHITECTURE.md` — design rationale, 14-phase breakdown
+2. Read `./plannng/HARDWARE.md` — component specs, UUIDs, network config
+3. Read `./plannng/STATUS.md` — the real current state, phase by phase
+
+### To pick up work
+1. Check `./plannng/STATUS.md` for what's actually next
+2. Look in `./tasks/` for an existing scoped task doc covering it
+3. If none exists, Claude Desktop drafts one before Claude Code starts —
+   don't hand Claude Code a phase without a task doc scoping it
+
+### Pre-deployment sequence (current priority)
+
+1. Confirm the canonical kickstart (`./alma10-minimal-bigboy.ks`) is
+   correct for the hardware in front of you
+2. Boot BigBoy from install media, run the kickstart (Phase 1)
+3. Let Phase 2 (drive wipe, ~48hr background) complete
+4. Get BigBoy on the network, clone this repo down
+5. Run Phase 3 (filesystem role — already written) via Ansible, either
+   locally on BigBoy or from workbench as the control node
+6. Only then: Phase 4 (NVIDIA driver) and Phase 5 (llama.cpp) — see
+   `./tasks/` for the current scoped handoff covering these
+
+---
+
+## Notes
+
+- `install-phases.txt` documents 15 phases (0–14); this README's status
+  section is a summary, not a replacement for it or for `STATUS.md`.
+- RAG library cross-references from an earlier version of this project
+  (`nvidia-driver-almalinux.md`, `ollama-deployment.md`,
+  `open-webui-deployment.md` under a separate documents library) predate
+  the llama.cpp decision — verify these still exist and are accurate, or
+  need an `llama-cpp-deployment.md` equivalent, before relying on them.
+- All artifacts are version-controlled and pushed to
+  `https://git.jhoeven.net/giovannino/bigboy-alma-deploy`.
+
+---
+
+**Last updated:** 2026-07-16
+**Next phase:** Phase 1 (kickstart install) — nothing is installed on
+BigBoy yet

tasks/TASK-phase4-5-bringup.md

@@ -0,0 +1,76 @@
+# Task: Phase 4 + 5 bring-up — NVIDIA driver and llama.cpp
+
+## Scope
+
+Bring BigBoy from its current state (Phase 3 complete — filesystems mounted)
+through Phase 4 (NVIDIA driver) and Phase 5 (llama.cpp built and serving),
+per `plannng/STATUS.md`. This is a bring-up task, not the auditable
+release-pipeline work — see `CLAUDE.md` for why that's explicitly out of
+scope right now.
+
+## Before starting
+
+- Read `CLAUDE.md` at repo root if you haven't already this session.
+- Read `plannng/STATUS.md` for the current authoritative state of every
+  phase — confirm Phase 3 is actually complete before proceeding.
+- Read `group_vars/bigboy.yml` for hardware facts (drive UUIDs, target
+  driver version, GPU architecture) — don't re-derive these.
+
+## Phase 4 — NVIDIA driver
+
+- Target driver version: see `group_vars/bigboy.yml` (`595.84` at time of
+  writing — confirm this is still current before installing, driver
+  releases move fast).
+- Confirm `nouveau` is blacklisted (should already be handled by the
+  kickstart — verify, don't assume).
+- **Done when**: `nvidia-smi` runs cleanly and reports the RTX 5060 Ti.
+
+## Phase 5 — llama.cpp
+
+1. **Pick a pinned build tag.** Check
+   `https://github.com/ggml-org/llama.cpp/releases` (or `git ls-remote
+   --tags`) for a recent, stable-looking tag — do not build against
+   `master`. Record the chosen tag in `group_vars/bigboy.yml` once decided.
+2. **Shallow clone at that tag**, build with the standard flags from
+   `CLAUDE.md` (`-DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
+   -DCMAKE_CUDA_ARCHITECTURES=120`).
+3. **Model**: pull a Q4_K_M or Q5_K_M GGUF from Bartowski or Unsloth on
+   Hugging Face for initial testing — a 7B-class instruct model is
+   sufficient to validate the pipeline end to end. Do not use an
+   MXFP4-quantized model (see `CLAUDE.md` — known sm_120 build hazard).
+4. **Serve** via `llama-server` under systemd:
+   - Bind `127.0.0.1`, not `0.0.0.0`
+   - `--api-key` set (placeholder value is fine for initial bring-up,
+     but note in the PR/commit that it needs a real value before any
+     external exposure)
+   - `--cont-batching --parallel 4` (or a reasonable default — this isn't
+     the tuning pass, just needs to not be single-request-only)
+   - nginx reverse proxy in front, following the existing pattern already
+     used for Nextcloud/Forgejo on jahnet — don't invent a new nginx
+     pattern for this
+5. **Firewall**: only the reverse-proxy port needs opening; the
+   `llama-server` port itself stays localhost-only.
+
+## Done criteria for this task
+
+- `nvidia-smi` confirms GPU visible and driver loaded
+- `systemctl status llama-server` shows active/running
+- `curl http://127.0.0.1:<port>/health` (or equivalent) returns healthy
+- One real inference request through the API returns a coherent response
+- The chosen build tag is recorded in `group_vars/bigboy.yml`
+- `plannng/STATUS.md` updated to reflect Phase 4 and 5 as complete
+
+## Explicitly out of scope for this task
+
+- Do not build the update/promote/prune/publish scripts referenced in
+  `CLAUDE.md` — that's deferred work, not this task.
+- Do not build Open WebUI (Phase 8) — `llama-server`'s built-in UI covers
+  the chat-first use case; Phase 8 is likely to be dropped entirely, but
+  that's a separate decision, not part of this task.
+- Do not modify the Phase 3 filesystem/mount layout — it's already
+  reviewed and complete. If something about it seems to be causing a
+  problem in Phase 4/5, flag it rather than changing it directly.
+- Do not touch SELinux policy beyond what's needed to get services
+  running (permissive discovery pass if genuinely needed, per
+  `CLAUDE.md` — but this should be a temporary diagnostic step, not a
+  final state, and not a deep dive into custom policy authoring).