bigboy-alma-deploy

Sovereign AI inference server: RTX 5060 Ti GPU (16GB VRAM), 5-drive btrfs storage on AlmaLinux 10.2

---

Quick Links

• Architecture & Design: ./plannng/ARCHITECTURE.md
• Deployment Status: ./plannng/STATUS.md
• Hardware Manifest: ./plannng/HARDWARE.md
• Sovereignty Policy: ./plannng/SOVEREIGNTY-POLICY.md
• Claude Code standing context: ./CLAUDE.md
• Current task handoff: ./tasks/

---

Project Overview

Goal

Deploy BigBoy as a fully autonomous, idempotent AI inference server running AlmaLinux 10.2 with GPU acceleration via NVIDIA RTX 5060 Ti. The system serves local models via llama.cpp (llama-server, which includes its own OpenAI-compatible API and web UI) and maintains reproducible deployment logs for the RAG corpus. BigBoy also serves as the reference/testbed machine for CE's future RHEL + llama.cpp client deployments — decisions made here are expected to generalize, not just work once.

Why AlmaLinux 10?

• RHEL-compatible (10-year support lifecycle)
• Official NVIDIA precompiled GPU driver support
• DNF package manager with clear upgrade path
• Pragmatic middle ground: better stability than NixOS on this hardware (the original NixOS attempt hit unresolved RTX 5060 Ti driver friction), simpler than Debian for this workload
• RHEL-family chosen fleet-wide for AI-server workloads specifically, for compliance/audit reasons (EU AI Act relevance) — not just a fix for this one box

Why llama.cpp, not Ollama?

Decided deliberately, not a default: full control over CUDA build flags and quantization, and llama-server's built-in web UI removes the need for a separate Open WebUI layer entirely. Worth knowing: Ollama itself wraps llama.cpp's ggml engine on NVIDIA/Linux under the hood, so it wouldn't have avoided this dependency — it would only have added a layer of indirection around it. See CLAUDE.md for the full standing rationale (pinned-tag discipline, standard build flags, known sm_120/MXFP4 hazard).

Key Constraints

• GPU VRAM: 16GB (RTX 5060 Ti) — model selection accordingly
• Storage: 5 drives (1 NVMe + 4 SATA) with btrfs subvolume isolation
• Network: bench LAN (192.168.0.0/24), workbench acts as gateway/DHCP for this segment
• Idempotency: every phase must be re-runnable

---

Current Status — read this before assuming anything is done

As of 2026-07-16: BigBoy has no OS installed. The machine currently holds only an abandoned NixOS install attempt. Nothing described as "complete" below has been validated on real hardware yet — it means designed, written, and reviewed, not running.

See ./plannng/STATUS.md for the authoritative, current per-phase state. Don't rely on this README for phase-by-phase status — it will drift out of date faster than STATUS.md is maintained.

Design-complete (not yet run on hardware)

• Kickstart (alma10-minimal-bigboy.ks) — unattended AlmaLinux 10.2 minimal install
• Ansible main playbook scaffold (ansible/site.yml, 14-phase orchestration)
• Phase 3 role (filesystem mounting) — written and reviewed
• Hardware inventory & UUID manifest (plannng/HARDWARE.md)

Ready to build, once Phase 1–3 actually run on hardware

• Phase 4 (NVIDIA driver)
• Phase 5 (llama.cpp — build from pinned tag, standard flags, see CLAUDE.md)

Likely dropping

• Phase 8 (Open WebUI) — llama-server's built-in UI covers the chat-first use case; keeping this phase is now a deliberate decision to make, not a default

Deferred, on purpose

• Phases 6–7, 9–14 (Build Suite, Configuration, Security, Borgmatic, Thermal, Validation, Migration, Observation)
• An auditable testing/release system for llama.cpp (versioned releases, btrfs snapshot safety net, promote/rollback, a Forgejo mirror that only ever receives BigBoy-validated tags for client deployments) — fully designed 2026-07-16, not yet built. See CLAUDE.md for the deferred-work note and scripts/draft/ if present.
• Borgmatic backup target (not yet decided)
• Home LAN migration (waiting on case installation)

---

Collaborators & Workflow

John A. Hoeven (@workbench)

• Architect & operator — makes deployment decisions, edits and pushes from workbench via VS Codium

Claude Code (@bigboy once installed, or driven from @workbench)

• Executes scoped tasks against the actual repo — see CLAUDE.md for standing context read automatically each session, and tasks/ for the current scoped handoff
• Does not decide scope — task docs name exactly what's in and out of bounds for a given session

Claude Desktop

• Plans and orchestrates: architecture discussion, design decisions, drafts CLAUDE.md and task docs for Claude Code to execute against

---

Repo Structure

bigboy-alma-deploy/
├── CLAUDE.md                    # standing context, read by Claude Code every session
├── README.md                    # this file
├── alma10-minimal-bigboy.ks     # canonical kickstart — this copy, not plannng/'s
├── install-phases.txt           # 15-phase (0-14) plan, high level
├── ansible/
│   ├── site.yml                 # main playbook
│   ├── inventory.ini
│   ├── group_vars/bigboy.yml    # hardware facts: UUIDs, driver version, pinned llama.cpp tag
│   └── roles/
│       └── phase-3-filesystems/ # written, reviewed, not yet run on hardware
├── plannng/                      # (sic — established directory name, not a typo to fix casually)
│   ├── ARCHITECTURE.md
│   ├── HARDWARE.md
│   ├── SOVEREIGNTY-POLICY.md
│   ├── STATUS.md                # authoritative current status — check this first
│   ├── alma-*.md                 # Alma-specific reference docs (firewall, package mgmt, nvidia, etc.)
│   └── alma10-minimal-bigboy.ks  # reference snapshot from planning — root copy is canonical
├── tasks/
│   └── TASK-*.md                 # scoped Claude Code handoffs, one per unit of work
└── scripts/draft/                 # (if present) draft auditable-release-pipeline scripts —
                                    # unintegrated, see CLAUDE.md before touching

---

Getting Started

To review design

1. Read ./plannng/ARCHITECTURE.md — design rationale, 14-phase breakdown
2. Read ./plannng/HARDWARE.md — component specs, UUIDs, network config
3. Read ./plannng/STATUS.md — the real current state, phase by phase

To pick up work

1. Check ./plannng/STATUS.md for what's actually next
2. Look in ./tasks/ for an existing scoped task doc covering it
3. If none exists, Claude Desktop drafts one before Claude Code starts — don't hand Claude Code a phase without a task doc scoping it

Pre-deployment sequence (current priority)

1. Confirm the canonical kickstart (./alma10-minimal-bigboy.ks) is correct for the hardware in front of you
2. Boot BigBoy from install media, run the kickstart (Phase 1)
3. Let Phase 2 (drive wipe, ~48hr background) complete
4. Get BigBoy on the network, clone this repo down
5. Run Phase 3 (filesystem role — already written) via Ansible, either locally on BigBoy or from workbench as the control node
6. Only then: Phase 4 (NVIDIA driver) and Phase 5 (llama.cpp) — see ./tasks/ for the current scoped handoff covering these

---

Notes

• install-phases.txt documents 15 phases (0–14); this README's status section is a summary, not a replacement for it or for STATUS.md.
• RAG library cross-references from an earlier version of this project (nvidia-driver-almalinux.md, ollama-deployment.md, open-webui-deployment.md under a separate documents library) predate the llama.cpp decision — verify these still exist and are accurate, or need an llama-cpp-deployment.md equivalent, before relying on them.
• All artifacts are version-controlled and pushed to https://git.jhoeven.net/giovannino/bigboy-alma-deploy.

---

Last updated: 2026-07-16 Next phase: Phase 1 (kickstart install) — nothing is installed on BigBoy yet