bigboy-alma-deploy

Sovereign AI inference server: RTX 5060 Ti GPU (16GB VRAM), 5-drive btrfs storage on AlmaLinux 10.2

---

Quick Links

• Architecture & Design: ./planning/ARCHITECTURE.md
• Deployment Status: ./planning/STATUS.md
• Hardware Manifest: ./planning/HARDWARE.md
• Sovereignty Policy: ./planning/SOVEREIGNTY-POLICY.md
• Claude Code standing context: ./CLAUDE.md
• Current task handoff: ./tasks/

---

Project Overview

Goal

Deploy BigBoy as a fully autonomous, idempotent AI inference server running AlmaLinux 10.2 with GPU acceleration via NVIDIA RTX 5060 Ti. The system serves local models via llama.cpp (llama-server, which includes its own OpenAI-compatible API and web UI) and maintains reproducible deployment logs for the RAG corpus. BigBoy also serves as the reference/testbed machine for CE's future RHEL + llama.cpp client deployments — decisions made here are expected to generalize, not just work once.

Why AlmaLinux 10?

• RHEL-compatible (10-year support lifecycle)
• Official NVIDIA precompiled GPU driver support
• DNF package manager with clear upgrade path
• Pragmatic middle ground: better stability than NixOS on this hardware (the original NixOS attempt hit unresolved RTX 5060 Ti driver friction), simpler than Debian for this workload
• RHEL-family chosen fleet-wide for AI-server workloads specifically, for compliance/audit reasons (EU AI Act relevance) — not just a fix for this one box

Why llama.cpp, not Ollama?

Decided deliberately, not a default: full control over CUDA build flags and quantization, and llama-server's built-in web UI removes the need for a separate Open WebUI layer entirely. Worth knowing: Ollama itself wraps llama.cpp's ggml engine on NVIDIA/Linux under the hood, so it wouldn't have avoided this dependency — it would only have added a layer of indirection around it. See CLAUDE.md for the full standing rationale (pinned-tag discipline, standard build flags, known sm_120/MXFP4 hazard).

Key Constraints

• GPU VRAM: 16GB (RTX 5060 Ti) — model selection accordingly
• Storage: 5 drives (1 NVMe + 4 SATA) with btrfs subvolume isolation
• Network: bench LAN (192.168.0.0/24), workbench acts as gateway/DHCP for this segment
• Idempotency: every phase must be re-runnable

---

Current Status — read this before assuming anything is done

As of 2026-07-18: AlmaLinux 10.2 successfully installed and booted on BigBoy. All five drives confirmed mounted and accessible. Foundation complete; core inference stack (Phase 4–5) ready to build.

See ./planning/STATUS.md for the authoritative, current per-phase state. Don't rely on this README for phase-by-phase status — it will drift out of date faster than STATUS.md is maintained.

Deployed & Verified on Hardware

• Kickstart (alma10-minimal-bigboy.ks) — AlmaLinux 10.2 minimal install successfully executed 2026-07-17
• Phase 3 role (filesystem mounting) — run and verified, all 4 SATA + 1 NVMe drives mounted at intended paths via UUID

Ready to build (Phase 1–3 complete)

• Phase 4 (NVIDIA driver) — precompiled open-kmod via AlmaLinux repos
• Phase 5 (llama.cpp) — build from pinned tag, standard flags, llama-server under systemd, see CLAUDE.md and tasks/TASK-phase4-5-bringup.md

Likely dropping

• Phase 8 (Open WebUI) — llama-server's built-in UI covers the chat-first use case; keeping this phase is now a deliberate decision to make, not a default

Deferred, on purpose

• Phases 6–7, 9–14 (Build Suite, Configuration, Security, Borgmatic, Thermal, Validation, Migration, Observation)
• An auditable testing/release system for llama.cpp (versioned releases, btrfs snapshot safety net, promote/rollback, a Forgejo mirror that only ever receives BigBoy-validated tags for client deployments) — fully designed 2026-07-16, not yet built. See CLAUDE.md for the deferred-work note and scripts/draft/ if present.
• Borgmatic backup target (not yet decided)
• Home LAN migration (waiting on case installation)

---

Collaborators & Workflow

John A. Hoeven (@workbench)

• Architect & operator — makes deployment decisions, edits and pushes from workbench via VS Codium

Claude Code (@bigboy once installed, or driven from @workbench)

• Executes scoped tasks against the actual repo — see CLAUDE.md for standing context read automatically each session, and tasks/ for the current scoped handoff
• Does not decide scope — task docs name exactly what's in and out of bounds for a given session

Claude Desktop

• Plans and orchestrates: architecture discussion, design decisions, drafts CLAUDE.md and task docs for Claude Code to execute against

---

Repo Structure

bigboy-alma-deploy/
├── CLAUDE.md                    # standing context, read by Claude Code every session
├── README.md                    # this file
├── alma10-minimal-bigboy.ks     # canonical kickstart — this copy, not planning/'s
├── install-phases.txt           # 15-phase (0-14) plan, high level
├── ansible/
│   ├── site.yml                 # main playbook
│   ├── inventory.ini
│   ├── group_vars/bigboy.yml    # hardware facts: UUIDs, driver version, pinned llama.cpp tag
│   └── roles/
│       └── phase-3-filesystems/ # written, reviewed, not yet run on hardware
├── planning/                      # renamed from plannng (typo) on 2026-07-17
│   ├── ARCHITECTURE.md
│   ├── HARDWARE.md
│   ├── SOVEREIGNTY-POLICY.md
│   ├── STATUS.md                # authoritative current status — check this first
│   ├── alma-*.md                 # Alma-specific reference docs (firewall, package mgmt, nvidia, etc.)
│   ├── build-journal-20260717.md # detailed build log: drive wipe, install, Phase 3 verification
│   └── alma10-minimal-bigboy.ks  # reference snapshot from planning — root copy is canonical
├── tasks/
│   └── TASK-*.md                 # scoped Claude Code handoffs, one per unit of work
└── scripts/draft/                 # (if present) draft auditable-release-pipeline scripts —
                                    # unintegrated, see CLAUDE.md before touching

---

Getting Started

To review design

1. Read ./planning/ARCHITECTURE.md — design rationale, 14-phase breakdown
2. Read ./planning/HARDWARE.md — component specs, UUIDs, network config
3. Read ./planning/STATUS.md — the real current state, phase by phase

To pick up work

1. Check ./planning/STATUS.md for what's actually next
2. Look in ./tasks/ for an existing scoped task doc covering it
3. If none exists, Claude Desktop drafts one before Claude Code starts — don't hand Claude Code a phase without a task doc scoping it

Next Steps (current priority)

1. Build and deploy Phase 4 (NVIDIA driver) via Ansible or shell commands, verify with nvidia-smi
2. Build and deploy Phase 5 (llama.cpp) — see ./tasks/TASK-phase4-5-bringup.md for the current scoped handoff
3. Verify llama-server running and responding to inference requests
4. Phases 6–14 follow once 4–5 are stable and tested

---

Notes

• install-phases.txt documents 15 phases (0–14); this README's status section is a summary, not a replacement for it or for STATUS.md.
• RAG library cross-references from an earlier version of this project (nvidia-driver-almalinux.md, ollama-deployment.md, open-webui-deployment.md under a separate documents library) predate the llama.cpp decision — verify these still exist and are accurate, or need an llama-cpp-deployment.md equivalent, before relying on them.
• All artifacts are version-controlled and pushed to https://git.jhoeven.net/giovannino/bigboy-alma-deploy.

---

Last updated: 2026-07-18 Current priority: Phase 4 (NVIDIA driver) and Phase 5 (llama.cpp build & serve)