diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..56c4ddc --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,96 @@ +# CLAUDE.md — bigboy-alma-deploy + +This file is read automatically at the start of every Claude Code session in +this repo. It captures standing decisions so they don't need to be +re-explained or re-litigated each session. If something here conflicts with +a task doc you've been handed, the task doc wins for that session — but +flag the conflict rather than silently picking one. + +## What this repo is + +Deployment automation for BigBoy — a Ryzen 5 / RTX 5060 Ti (16GB) AI +inference server, running AlmaLinux 10.2. BigBoy also serves as the +reference/testbed machine for CE's future RHEL + llama.cpp client +deployments — meaning decisions made here are expected to generalize, not +just work once. + +## Source of truth for status + +`plannng/STATUS.md` is authoritative for what's actually done vs. pending. +Read it before starting work. Update it when a phase's status changes — +don't leave it stale. + +## Standing technical decisions (do not re-derive or second-guess these) + +- **OS**: AlmaLinux 10.2, moved from NixOS due to RTX 5060 Ti driver + friction on Nix. RHEL-family chosen fleet-wide for AI-server workloads + for compliance/audit reasons (EU AI Act relevance), not just this one box. +- **Inference engine**: llama.cpp, built from source — deliberately not + Ollama. Reasons: full control over CUDA build flags and quantization, + built-in web UI via `llama-server` removes the need for a separate + Open WebUI layer. Ollama also just wraps llama.cpp's ggml engine + underneath on NVIDIA/Linux anyway, so switching back would add a layer + of indirection without avoiding the dependency. +- **GPU target**: `-DCMAKE_CUDA_ARCHITECTURES=120` (Blackwell / sm_120). + Requires NVIDIA driver ≥570; target driver is `595.84` (see + `group_vars/bigboy.yml`). +- **Known hazard**: MXFP4-quantized models have open compilation issues on + sm_120 as of mid-2026. Stick to standard GGUF quants — Q4_K_M or Q5_K_M — + from established quantizers (Bartowski, Unsloth namespaces on Hugging + Face). This sidesteps the issue entirely; it is not a performance + preference, it's a build-stability one. +- **Version discipline**: llama.cpp has no semantic versioning — continuous + build-tagged releases only. Never build against `master`/HEAD. Every + build pins a specific tag, recorded in `group_vars/bigboy.yml`. +- **Build flags**: standard only — `cmake -B build -DGGML_CUDA=ON + -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120`. No exotic + tuning flags (e.g. `-DLLAMA_CUDA_MMV_Y`) unless a specific, documented + problem requires one. We are already deep in non-standard territory + (Blackwell + Alma + sovereignty logging); the inference engine build + itself should be as boring and reproducible as possible. +- **Serving**: `llama-server` under systemd, bound to `127.0.0.1`, fronted + by nginx (reuse the existing CE reverse-proxy pattern from jahnet — don't + invent a new one). `--api-key` enforced even on LAN-only. `--cont-batching + --parallel N` for concurrent chat sessions. +- **SELinux**: stays enforcing. If early bring-up needs a permissive + discovery pass to collect AVC denials via `ausearch`/`audit2allow`, + that's a temporary diagnostic state, not a resting state — flip back to + enforcing before considering the phase done. + +## Style requirements for any script in this repo + +Follow the CE OS Bash Style Guide (`Updated_Bash_Style_Guide` in the CE +project knowledge) for every script: +- Attribution header (dwarves first, then John A. Hoeven / Claude AI), + licence (Unlicense), version, status +- No `set -e` — every operation checked and logged explicitly +- Cleanup trap registered before work begins +- Single confirmation prompt before any system-modifying action +- No silent failures — every error path logs and either hard-fails or warns +- Never assume root; invoke `sudo` explicitly for privileged steps only + +This repo is a single-purpose ops repo for one Alma server, not part of the +CEOS multi-distro installer framework — so skip the `ce_env.conf` / +`pkg_*` / `CE_PRIV` abstraction layer specifically. Use `sudo` directly. +Everything else in the style guide applies. + +## Explicitly deferred — do not build unless a task doc asks for it + +An auditable testing/release system for llama.cpp (versioned releases, +btrfs snapshot safety net, promote/rollback via symlink swap, a Forgejo +mirror that only ever receives BigBoy-validated tags so client +deployments never pull raw upstream) was designed in detail on +2026-07-16, but is **not yet built or integrated into the phase plan**. +Draft, untested scripts from that design may exist under `scripts/draft/`. +Do not wire them into the active Ansible roles or treat them as +representing current repo state — they're a reference for later, not a +task in progress. Bringing BigBoy up and running is the current priority; +this gets picked up afterward. + +## Scope discipline + +Task docs will name a specific phase or task from `STATUS.md`. Do the +named task. Don't refactor, "improve," or extend adjacent phases that are +already marked complete or reviewed, even if something nearby looks +improvable — flag it instead and let it be a deliberate decision, not a +side effect. \ No newline at end of file diff --git a/README.md b/README.md index 53ae58a..88a1e48 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,199 @@ # bigboy-alma-deploy -Refocusing @bigboy AI server to Alma for access to GPU drivers and familiarity. \ No newline at end of file +**Sovereign AI inference server: RTX 5060 Ti GPU (16GB VRAM), 5-drive btrfs storage on AlmaLinux 10.2** + +--- + +## Quick Links + +- **Architecture & Design:** `./plannng/ARCHITECTURE.md` +- **Deployment Status:** `./plannng/STATUS.md` +- **Hardware Manifest:** `./plannng/HARDWARE.md` +- **Sovereignty Policy:** `./plannng/SOVEREIGNTY-POLICY.md` +- **Claude Code standing context:** `./CLAUDE.md` +- **Current task handoff:** `./tasks/` + +--- + +## Project Overview + +### Goal + +Deploy BigBoy as a fully autonomous, idempotent AI inference server running +AlmaLinux 10.2 with GPU acceleration via NVIDIA RTX 5060 Ti. The system +serves local models via **llama.cpp** (`llama-server`, which includes its +own OpenAI-compatible API and web UI) and maintains reproducible deployment +logs for the RAG corpus. BigBoy also serves as the reference/testbed +machine for CE's future RHEL + llama.cpp client deployments — decisions +made here are expected to generalize, not just work once. + +### Why AlmaLinux 10? + +- RHEL-compatible (10-year support lifecycle) +- Official NVIDIA precompiled GPU driver support +- DNF package manager with clear upgrade path +- Pragmatic middle ground: better stability than NixOS on this hardware + (the original NixOS attempt hit unresolved RTX 5060 Ti driver friction), + simpler than Debian for this workload +- RHEL-family chosen fleet-wide for AI-server workloads specifically, for + compliance/audit reasons (EU AI Act relevance) — not just a fix for this + one box + +### Why llama.cpp, not Ollama? + +Decided deliberately, not a default: full control over CUDA build flags +and quantization, and `llama-server`'s built-in web UI removes the need +for a separate Open WebUI layer entirely. Worth knowing: Ollama itself +wraps llama.cpp's ggml engine on NVIDIA/Linux under the hood, so it +wouldn't have avoided this dependency — it would only have added a layer +of indirection around it. See `CLAUDE.md` for the full standing rationale +(pinned-tag discipline, standard build flags, known sm_120/MXFP4 hazard). + +### Key Constraints + +- **GPU VRAM:** 16GB (RTX 5060 Ti) — model selection accordingly +- **Storage:** 5 drives (1 NVMe + 4 SATA) with btrfs subvolume isolation +- **Network:** bench LAN (192.168.0.0/24), workbench acts as gateway/DHCP + for this segment +- **Idempotency:** every phase must be re-runnable + +--- + +## Current Status — read this before assuming anything is done + +**As of 2026-07-16: BigBoy has no OS installed.** The machine currently +holds only an abandoned NixOS install attempt. Nothing described as +"complete" below has been validated on real hardware yet — it means +*designed, written, and reviewed*, not *running*. + +See `./plannng/STATUS.md` for the authoritative, current per-phase state. +Don't rely on this README for phase-by-phase status — it will drift out +of date faster than `STATUS.md` is maintained. + +### Design-complete (not yet run on hardware) + +- Kickstart (`alma10-minimal-bigboy.ks`) — unattended AlmaLinux 10.2 + minimal install +- Ansible main playbook scaffold (`ansible/site.yml`, 14-phase + orchestration) +- Phase 3 role (filesystem mounting) — written and reviewed +- Hardware inventory & UUID manifest (`plannng/HARDWARE.md`) + +### Ready to build, once Phase 1–3 actually run on hardware + +- Phase 4 (NVIDIA driver) +- Phase 5 (llama.cpp — build from pinned tag, standard flags, see + `CLAUDE.md`) + +### Likely dropping + +- Phase 8 (Open WebUI) — `llama-server`'s built-in UI covers the + chat-first use case; keeping this phase is now a deliberate decision + to make, not a default + +### Deferred, on purpose + +- Phases 6–7, 9–14 (Build Suite, Configuration, Security, Borgmatic, + Thermal, Validation, Migration, Observation) +- An auditable testing/release system for llama.cpp (versioned releases, + btrfs snapshot safety net, promote/rollback, a Forgejo mirror that only + ever receives BigBoy-validated tags for client deployments) — fully + designed 2026-07-16, not yet built. See `CLAUDE.md` for the deferred-work + note and `scripts/draft/` if present. +- Borgmatic backup target (not yet decided) +- Home LAN migration (waiting on case installation) + +--- + +## Collaborators & Workflow + +### John A. Hoeven (@workbench) +- Architect & operator — makes deployment decisions, edits and pushes + from workbench via VS Codium + +### Claude Code (@bigboy once installed, or driven from @workbench) +- Executes scoped tasks against the actual repo — see `CLAUDE.md` for + standing context read automatically each session, and `tasks/` for the + current scoped handoff +- Does not decide scope — task docs name exactly what's in and out of + bounds for a given session + +### Claude Desktop +- Plans and orchestrates: architecture discussion, design decisions, + drafts `CLAUDE.md` and task docs for Claude Code to execute against + +--- + +## Repo Structure + +``` +bigboy-alma-deploy/ +├── CLAUDE.md # standing context, read by Claude Code every session +├── README.md # this file +├── alma10-minimal-bigboy.ks # canonical kickstart — this copy, not plannng/'s +├── install-phases.txt # 15-phase (0-14) plan, high level +├── ansible/ +│ ├── site.yml # main playbook +│ ├── inventory.ini +│ ├── group_vars/bigboy.yml # hardware facts: UUIDs, driver version, pinned llama.cpp tag +│ └── roles/ +│ └── phase-3-filesystems/ # written, reviewed, not yet run on hardware +├── plannng/ # (sic — established directory name, not a typo to fix casually) +│ ├── ARCHITECTURE.md +│ ├── HARDWARE.md +│ ├── SOVEREIGNTY-POLICY.md +│ ├── STATUS.md # authoritative current status — check this first +│ ├── alma-*.md # Alma-specific reference docs (firewall, package mgmt, nvidia, etc.) +│ └── alma10-minimal-bigboy.ks # reference snapshot from planning — root copy is canonical +├── tasks/ +│ └── TASK-*.md # scoped Claude Code handoffs, one per unit of work +└── scripts/draft/ # (if present) draft auditable-release-pipeline scripts — + # unintegrated, see CLAUDE.md before touching +``` + +--- + +## Getting Started + +### To review design +1. Read `./plannng/ARCHITECTURE.md` — design rationale, 14-phase breakdown +2. Read `./plannng/HARDWARE.md` — component specs, UUIDs, network config +3. Read `./plannng/STATUS.md` — the real current state, phase by phase + +### To pick up work +1. Check `./plannng/STATUS.md` for what's actually next +2. Look in `./tasks/` for an existing scoped task doc covering it +3. If none exists, Claude Desktop drafts one before Claude Code starts — + don't hand Claude Code a phase without a task doc scoping it + +### Pre-deployment sequence (current priority) + +1. Confirm the canonical kickstart (`./alma10-minimal-bigboy.ks`) is + correct for the hardware in front of you +2. Boot BigBoy from install media, run the kickstart (Phase 1) +3. Let Phase 2 (drive wipe, ~48hr background) complete +4. Get BigBoy on the network, clone this repo down +5. Run Phase 3 (filesystem role — already written) via Ansible, either + locally on BigBoy or from workbench as the control node +6. Only then: Phase 4 (NVIDIA driver) and Phase 5 (llama.cpp) — see + `./tasks/` for the current scoped handoff covering these + +--- + +## Notes + +- `install-phases.txt` documents 15 phases (0–14); this README's status + section is a summary, not a replacement for it or for `STATUS.md`. +- RAG library cross-references from an earlier version of this project + (`nvidia-driver-almalinux.md`, `ollama-deployment.md`, + `open-webui-deployment.md` under a separate documents library) predate + the llama.cpp decision — verify these still exist and are accurate, or + need an `llama-cpp-deployment.md` equivalent, before relying on them. +- All artifacts are version-controlled and pushed to + `https://git.jhoeven.net/giovannino/bigboy-alma-deploy`. + +--- + +**Last updated:** 2026-07-16 +**Next phase:** Phase 1 (kickstart install) — nothing is installed on +BigBoy yet \ No newline at end of file diff --git a/tasks/TASK-phase4-5-bringup.md b/tasks/TASK-phase4-5-bringup.md new file mode 100644 index 0000000..43881e6 --- /dev/null +++ b/tasks/TASK-phase4-5-bringup.md @@ -0,0 +1,76 @@ +# Task: Phase 4 + 5 bring-up — NVIDIA driver and llama.cpp + +## Scope + +Bring BigBoy from its current state (Phase 3 complete — filesystems mounted) +through Phase 4 (NVIDIA driver) and Phase 5 (llama.cpp built and serving), +per `plannng/STATUS.md`. This is a bring-up task, not the auditable +release-pipeline work — see `CLAUDE.md` for why that's explicitly out of +scope right now. + +## Before starting + +- Read `CLAUDE.md` at repo root if you haven't already this session. +- Read `plannng/STATUS.md` for the current authoritative state of every + phase — confirm Phase 3 is actually complete before proceeding. +- Read `group_vars/bigboy.yml` for hardware facts (drive UUIDs, target + driver version, GPU architecture) — don't re-derive these. + +## Phase 4 — NVIDIA driver + +- Target driver version: see `group_vars/bigboy.yml` (`595.84` at time of + writing — confirm this is still current before installing, driver + releases move fast). +- Confirm `nouveau` is blacklisted (should already be handled by the + kickstart — verify, don't assume). +- **Done when**: `nvidia-smi` runs cleanly and reports the RTX 5060 Ti. + +## Phase 5 — llama.cpp + +1. **Pick a pinned build tag.** Check + `https://github.com/ggml-org/llama.cpp/releases` (or `git ls-remote + --tags`) for a recent, stable-looking tag — do not build against + `master`. Record the chosen tag in `group_vars/bigboy.yml` once decided. +2. **Shallow clone at that tag**, build with the standard flags from + `CLAUDE.md` (`-DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release + -DCMAKE_CUDA_ARCHITECTURES=120`). +3. **Model**: pull a Q4_K_M or Q5_K_M GGUF from Bartowski or Unsloth on + Hugging Face for initial testing — a 7B-class instruct model is + sufficient to validate the pipeline end to end. Do not use an + MXFP4-quantized model (see `CLAUDE.md` — known sm_120 build hazard). +4. **Serve** via `llama-server` under systemd: + - Bind `127.0.0.1`, not `0.0.0.0` + - `--api-key` set (placeholder value is fine for initial bring-up, + but note in the PR/commit that it needs a real value before any + external exposure) + - `--cont-batching --parallel 4` (or a reasonable default — this isn't + the tuning pass, just needs to not be single-request-only) + - nginx reverse proxy in front, following the existing pattern already + used for Nextcloud/Forgejo on jahnet — don't invent a new nginx + pattern for this +5. **Firewall**: only the reverse-proxy port needs opening; the + `llama-server` port itself stays localhost-only. + +## Done criteria for this task + +- `nvidia-smi` confirms GPU visible and driver loaded +- `systemctl status llama-server` shows active/running +- `curl http://127.0.0.1:/health` (or equivalent) returns healthy +- One real inference request through the API returns a coherent response +- The chosen build tag is recorded in `group_vars/bigboy.yml` +- `plannng/STATUS.md` updated to reflect Phase 4 and 5 as complete + +## Explicitly out of scope for this task + +- Do not build the update/promote/prune/publish scripts referenced in + `CLAUDE.md` — that's deferred work, not this task. +- Do not build Open WebUI (Phase 8) — `llama-server`'s built-in UI covers + the chat-first use case; Phase 8 is likely to be dropped entirely, but + that's a separate decision, not part of this task. +- Do not modify the Phase 3 filesystem/mount layout — it's already + reviewed and complete. If something about it seems to be causing a + problem in Phase 4/5, flag it rather than changing it directly. +- Do not touch SELinux policy beyond what's needed to get services + running (permissive discovery pass if genuinely needed, per + `CLAUDE.md` — but this should be a temporary diagnostic step, not a + final state, and not a deep dive into custom policy authoring). \ No newline at end of file