giovannino/bigboy-alma-deploy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
bigboy-alma-deploy/ansible/README.md

15 KiB
Raw Permalink Blame History

BigBoy AlmaLinux 10 Deployment — Ansible Playbook

Idempotent Ansible playbook for configuring BigBoy from a fresh AlmaLinux 10.2 kickstart install through to a running llama.cpp inference server.

Status (2026-07-19): Phases 2 through 5 have all run successfully on real hardware. This isn't a "should work" playbook anymore — it's the one that actually built the working deployment. See ../planning/STATUS.md for the authoritative current state.

Quick Start

Prerequisites

  • BigBoy installed via alma10-minimal-bigboy.ks (Phase 1 — done)
  • SSH key deployed to BigBoy's admin account (see SSH Key Setup below — required before running this playbook, not optional)
  • Ansible 2.20+ on the control machine (workbench)

First run

# From ansible/
ansible-playbook site.yml -i inventory.ini --tags phase-2,phase-3

Re-running a specific phase

ansible-playbook site.yml -i inventory.ini --tags phase-3

Check mode (show what would change, without changing anything)

ansible-playbook site.yml -i inventory.ini --check --tags phase-2,phase-3

Directory Structure

ansible/
├── site.yml                          # Main playbook
├── inventory.ini                     # Hosts + SSH configuration
├── keys/
│   └── workbench.pub                 # gitignored — deploy locally, never commit
├── group_vars/
│   └── bigboy.yml                    # Hardware facts: UUIDs, by-id paths, pinned versions
├── roles/
│   ├── phase-2-system-config/
│   │   └── tasks/main.yml            # DONE — sudoers policy, SSH key, base checks
│   ├── phase-3-filesystems/
│   │   └── tasks/main.yml            # DONE — converts 4 SATA drives xfs -> btrfs
│   ├── phase-4-nvidia-driver/        # not yet a tracked role — Phase 4 was run via
│   │                                   direct shell commands per tasks/TASK-phase4-5-bringup.md,
│   │                                   not Ansible. Worth deciding whether to formalize
│   │                                   into a role for reproducibility.
│   ├── phase-5-llama-cpp/            # same — run manually, not yet an Ansible role
│   └── ... (phases 6-14, not yet scoped)
└── README.md                         # This file

Scoped work-in-progress task handoffs live in ../tasks/ at repo root, not here — TASK-phase2-3.md and TASK-phase4-5-bringup.md are the authoritative descriptions of what each phase actually does; this README is a summary and operational reference, not the source of truth for scope.

Worth knowing: Phase 4 (NVIDIA driver) and Phase 5 (llama.cpp build, model pull, systemd/nginx/firewall setup) were both executed as direct shell commands against BigBoy during the actual deployment session, not through Ansible roles. They worked, and they're real — but they're not currently reproducible by re-running this playbook. If that matters for future deployments (e.g., disaster recovery, or building the next client's box from this same reference), those steps need to become actual Ansible roles at some point — currently tracked as an open item, not done.

Phase Breakdown

Phase 1: Base Install — done

Unattended AlmaLinux 10.2 install via kickstart. See ../alma10-minimal-bigboy.ks.

Phase 2: System Configuration — DONE, verified on hardware

  • Replaced the kickstart's temporary bootstrap NOPASSWD sudoers grant with a real, deliberately-scoped policy
  • Deployed workbench's SSH public key (single key only at this stage — BigBoy sits on the home LAN now; more keys get added later, one at a time, once tested individually)
  • Confirmed hostname, timezone, NTP/chrony, and repo configuration

Real bugs hit and fixed during the actual run: a missing ansible.posix collection (not bundled with the Alpine ansible-core package — ansible-galaxy collection install ansible.posix was required), and a variable-naming mismatch (bigboy_admin_user expected by the role vs. deploy_user actually defined in group_vars) — see CLAUDE.md for the current state of both.

Does not disable password SSH authentication — that's Phase 9, once key-based login is confirmed working over real use, not assumed the moment a key is deployed.

Phase 3: Filesystem Conversion — DONE, verified on hardware

The kickstart formats all five drives as xfs (RHEL10's Anaconda removed btrfs support entirely — it's not a choice, it's what's actually possible at install time). Phase 3 converts the four SATA data drives to real btrfs with subvolumes and zstd compression:

Kickstart mount Drive Final path
/srv/backup-raw BACKUPDRV (Lexar 256GB) /srv/backup
/srv/rag-raw RAGLIB (Samsung EVO 500GB) /srv/rag-library
/srv/ai-raw AILOGS (Samsung EVO 250GB) /srv/ai-logs
/srv/prompt-raw PROMPTLIB (Lexar 256GB) /srv/prompt-library

Root stays xfs, permanently for now. There's no supported in-place xfs-to-btrfs conversion — this would require a full reinstall, planned separately and deliberately deferred until after real-world testing (not before ~a month out from 2026-07-17). Do not attempt to touch root as part of this phase.

Real bugs hit and fixed during the actual run: a device-path targeting bug ({{ item.byid }}1 instead of the correct {{ item.byid }}-part1 suffix — caught when mkfs.btrfs couldn't open the device, no data was touched), a mount-path mismatch specific to the ai-logs drive (the role assumed /srv/ai-logs-raw, the kickstart actually created /srv/ai-raw — caught by manually running lsblk -f after an apparently-successful run and noticing one drive still mounted when the other three weren't), and state: unmounted changed to state: absent so the kickstart's original fstab entries actually get removed, not just the live mounts.

Phase 4: NVIDIA Driver — DONE, run manually (not yet an Ansible role)

AlmaLinux's precompiled open-kmod path (almalinux-release-nvidia-driver → nvidia-driver-cuda nvidia-open-kmod), not manual DKMS. Confirmed as the AlmaLinux-recommended method, Secure Boot compatible, needs none of the kernel-devel/DKMS/compiler machinery DKMS would. Ran clean, no issues — the smoothest phase of the entire deployment.

Phase 5: llama.cpp — DONE, run manually (not yet an Ansible role)

Not Ollama. Built from source, pinned to a specific tag (b9968, recorded in group_vars/bigboy.yml), served via llama-server under systemd behind nginx. Model: Ministral 3 14B Instruct, Q4_K_M — success criterion was a coherent chat response through the built-in web UI, quality as-delivered, no tuning required — met, confirmed via both the web UI and a terminal client (aichat).

Real problems hit and fixed during the actual run:

  • cuda-toolkit (specifically nvcc) is a separate package from the driver runtime installed by Phase 4 — had to be installed separately
  • nvcc lands in a versioned path, not on PATH by default — resolved with a symlink and a /etc/profile.d/ entry
  • An OOM kill during the actual CUDA compile, root-caused via dmesg to unbounded parallel compilation with zero swap configured — fixed with a 16GB NVMe swapfile plus -j2
  • Two rounds of missing baseline utilities (tar, which, then nine more found via a proactive recon scan) — added to the kickstart for future installs

Variables (group_vars/bigboy.yml)

See the file itself for the current, complete set — it's been through a real reconciliation pass (2026-07-19) against everything actually deployed. Key points:

  • IP address: 192.168.1.24 — BigBoy's current, permanent home-LAN address. It ran on the workbench bench LAN for the entire initial deploy first (.240, then .241 after a DHCP reassignment) — both genuinely correct at the time, not mistakes, and noted as history in the file rather than erased.
  • Drive UUIDs: not consumed directly by any role — Phase 3 discovers them at runtime via blkid against the stable by-id paths, since they don't exist until mkfs.btrfs actually runs. The file records the current live UUIDs for reference only.
  • nvidia_driver_version: still a placeholder (CHECK-WITH-nvidia-smi) — the actual value was never captured during the deployment session. Run nvidia-smi and fill this in.
  • API key: deliberately NOT stored in this file — it's committed to a public repo. Needs vault or a gitignored file, same pattern as keys/workbench.pub.

Note on the four SATA drives' filesystem UUIDs: unlike the by-id paths (stable, known ahead of time), the filesystem UUIDs don't exist until Phase 3 runs mkfs.btrfs. Don't hardcode filesystem UUIDs into this file preemptively for a fresh install — Phase 3's role discovers each UUID at runtime and uses that discovered value to write /etc/fstab.

To update: edit this file, re-run the playbook. Variables propagate to every role.

Idempotency

Every task should be safe to re-run:

  • Already-converted drives aren't re-wiped (Phase 3 checks existing filesystem type before acting) — confirmed working correctly on the actual deployment
  • Already-deployed keys aren't duplicated
  • Already-correct sudoers policy isn't rewritten unnecessarily

Principle: running the playbook twice produces the same result as running it once. If a task isn't naturally idempotent, wrap it with an explicit check (stat, command + changed_when) rather than assuming Ansible's module-level idempotency covers everything.

Logging

Phase output is logged to /srv/deployment-log/ on BigBoy (created by the kickstart's %post), one file per phase run:

/srv/deployment-log/
├── kickstart.log                     # from Phase 1's %post
├── phase2-system-config-<date>.log
├── phase3-filesystems-<date>.log
└── ...

ssh john@192.168.1.24 "tail -f /srv/deployment-log/*.log"

Troubleshooting

Phase fails mid-run

  1. Check the phase-specific log under /srv/deployment-log/
  2. Fix manually if needed, or adjust the role
  3. Re-run just that phase: ansible-playbook site.yml --tags phaseN

SSH connection fails

  1. Confirm BigBoy's current IP — 192.168.1.24 on the home LAN as of this writing, but confirm via ansible/inventory.ini rather than assuming it hasn't changed. It's changed twice already during this deployment (bench LAN .240 → .241 → home LAN .24).
  2. Confirm you're connecting as the admin user, not root — root login is locked (rootpw --lock in the kickstart), always was, this isn't something Phase 9 disables later
  3. Confirm the key at keys/workbench.pub locally matches what's actually deployed to BigBoy's authorized_keys

Idempotency broken (task reports "changed" every run)

  • Check changed_when/failed_when logic in the specific task
  • Run with -vv for full task output

Missing ansible.posix collection

Hit during the real Phase 2 run — apk add ansible-core (Alpine's minimal package, deliberately chosen over the full ansible bundle) doesn't include community collections:

ansible-galaxy collection install ansible.posix

Extending (Adding a New Phase)

  1. mkdir -p roles/phaseN-<name>/tasks
  2. Write tasks/main.yml
  3. First, write the task doc at ../tasks/TASK-phaseN-<name>.md scoping exactly what the phase does and doesn't do — per this project's established workflow (Claude Desktop plans/scopes, Claude Code implements against the scoped task, not the other way around)
  4. Add the role to site.yml with an appropriate tag
  5. Run: ansible-playbook site.yml --tags phaseN

Worth doing at some point: Phases 4 and 5 were run as direct shell commands, not Ansible roles. Formalizing them into real roles (following this same pattern) would make the deployment fully reproducible from the repo alone — currently it isn't, for those two phases specifically.

SSH Key Setup (required before running this playbook)

BigBoy ships from the kickstart with password authentication only — root is locked, and no key exists yet. This is the actual chicken-and-egg this playbook's Phase 2 resolves, but the first key push has to happen manually, using password auth, before Ansible can connect via key:

# On workbench — the key already exists at ansible/keys/workbench.pub
# (gitignored, never commit the actual key material)
ssh-copy-id -i ~/.ssh/<your-key>.pub <admin-user>@192.168.1.24

# Verify
ssh -i ~/.ssh/<your-key> <admin-user>@192.168.1.24 "echo connected"

Then update inventory.ini to use ansible_user=<admin-user> (not root) and point ansible_ssh_private_key_file at the correct key.

Password authentication stays enabled even after this — disabling it is explicitly Phase 9's job, not something to do the moment a key works once.

Validation Checklist (Phase 2/3)

ssh john@192.168.1.24

# Sudoers — bootstrap grant gone, real policy in place
sudo -l
ls /etc/sudoers.d/

# Filesystems
df -h /srv/*
mount | grep btrfs

# Root untouched
mount | grep ' / '   # should show xfs, not btrfs

Validation Checklist (Phase 4/5)

nvidia-smi
systemctl status llama-server
systemctl status nginx
curl http://127.0.0.1:8080/health

Known Deferred (not gaps — deliberate)

  • Root filesystem conversion to btrfs — needs a full reinstall (clone/reformat/restore via debusb), planned for ~a month out after real-world testing, not part of this playbook
  • Additional SSH keys (Surface, tinkerpad) — one at a time, only once tested individually
  • Dedicated display-monitor user — for the case-mounted 5" screen relocated from jahnet; needs autologin + btop-only access, not yet built
  • Auditable llama.cpp release pipeline and per-client agent playbooks — both designed in detail, neither built; see CLAUDE.md
  • Phases 4/5 as actual Ansible roles — currently run manually, not automated; a real gap for reproducibility, not deliberately deferred the way the items above are
  • Phases 6–14 — not yet scoped at all

Reference Files

  • ../alma10-minimal-bigboy.ks — kickstart for the unattended OS install
  • ../CLAUDE.md — standing context, read automatically by Claude Code every session
  • ../tasks/TASK-phase2-3.md, ../tasks/TASK-phase4-5-bringup.md — authoritative scope for each phase
  • ../planning/STATUS.md — authoritative current per-phase state
  • ../planning/build-journal-20260717.md — detailed account of Phase 1's install day, including all ten issues found and fixed

Last updated: 2026-07-19 Status: Phase 1 through 5 complete and verified on real hardware. Phase 4/5 currently run manually, not yet formalized as Ansible roles. Phases 6–14 not yet scoped.