giovannino/bigboy-alma-deploy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
bigboy-alma-deploy/ansible/README.md

11 KiB
Raw Blame History

BigBoy AlmaLinux 10 Deployment — Ansible Playbook

Idempotent Ansible playbook for configuring BigBoy from a fresh AlmaLinux 10.2 kickstart install through to a running llama.cpp inference server.

Quick Start

Prerequisites

  • BigBoy installed via alma10-minimal-bigboy.ks (Phase 1 — done)
  • SSH key deployed to BigBoy's admin account (see SSH Key Setup below — required before running this playbook, not optional)
  • Ansible 2.20+ on the control machine (workbench)

First run

# From ansible/
ansible-playbook site.yml -i inventory.ini --tags phase2,phase3

Re-running a specific phase

ansible-playbook site.yml -i inventory.ini --tags phase3

Check mode (show what would change, without changing anything)

ansible-playbook site.yml -i inventory.ini --check --tags phase2,phase3

Directory Structure

ansible/
├── site.yml                          # Main playbook
├── inventory.ini                     # Hosts + SSH configuration
├── keys/
│   └── workbench.pub                 # gitignored — deploy locally, never commit
├── group_vars/
│   └── bigboy.yml                    # Hardware facts: UUIDs, by-id paths, pinned versions
├── roles/
│   ├── phase2-system-config/
│   │   └── tasks/main.yml            # Sudoers policy, SSH key, base checks
│   ├── phase-3-filesystems/
│   │   └── tasks/main.yml            # Convert 4 SATA drives xfs -> btrfs
│   ├── phase-4-nvidia-driver/        # scaffold — see tasks/TASK-phase4-5-bringup.md
│   ├── phase-5-llama-cpp/            # scaffold — see tasks/TASK-phase4-5-bringup.md
│   └── ... (phases 6-14, not yet scoped)
└── README.md                         # This file

Scoped work-in-progress task handoffs live in ../tasks/ at repo root, not here — TASK-phase2-3.md and TASK-phase4-5-bringup.md are the authoritative descriptions of what each phase actually does; this README is a summary and operational reference, not the source of truth for scope.

Phase Breakdown

Phase 1: Base Install — done

Unattended AlmaLinux 10.2 install via kickstart. See ../alma10-minimal-bigboy.ks.

Phase 2: System Configuration — see tasks/TASK-phase2-3.md

  • Replace the kickstart's temporary bootstrap NOPASSWD sudoers grant with a real, deliberately-scoped policy
  • Deploy workbench's SSH public key (single key only at this stage — BigBoy sits on the isolated bench LAN; more keys get added later, one at a time, once on the home LAN)
  • Confirm hostname, timezone, NTP/chrony, and repo configuration

Does not disable password SSH authentication — that's Phase 9, once key-based login is confirmed working over real use, not assumed the moment a key is deployed.

Phase 3: Filesystem Conversion — see tasks/TASK-phase2-3.md

The kickstart formats all five drives as xfs (RHEL10's Anaconda removed btrfs support entirely — it's not a choice, it's what's actually possible at install time). Phase 3 converts the four SATA data drives to real btrfs with subvolumes and zstd compression:

Kickstart mount Drive Final path
/srv/backup-raw BACKUPDRV (Lexar 256GB) /srv/backup
/srv/rag-raw RAGLIB (Samsung EVO 500GB) /srv/rag-library
/srv/ai-raw AILOGS (Samsung EVO 250GB) /srv/ai-logs
/srv/prompt-raw PROMPTLIB (Lexar 256GB) /srv/prompt-library

Root stays xfs, permanently for now. There's no supported in-place xfs-to-btrfs conversion — this would require a full reinstall, planned separately and deliberately deferred until after real-world testing (not before ~a month out). Do not attempt to touch root as part of this phase.

Phase 4: NVIDIA Driver — see tasks/TASK-phase4-5-bringup.md

AlmaLinux's precompiled open-kmod path (almalinux-release-nvidia-driver → nvidia-driver-cuda nvidia-open-kmod), not manual DKMS. Confirmed as the AlmaLinux-recommended method, Secure Boot compatible, needs none of the kernel-devel/DKMS/compiler machinery DKMS would.

Phase 5: llama.cpp — see tasks/TASK-phase4-5-bringup.md

Not Ollama. Built from source, pinned to a specific tag (b9968, recorded in group_vars/bigboy.yml), served via llama-server under systemd behind nginx. Initial model: Ministral 3 14B Instruct, Q4_K_M — success criterion is a coherent chat response through the built-in web UI, quality as-delivered, no tuning required at this stage.

Phases 6–14: not yet scoped

Deferred until 4/5 are stable and tested. No committed design for these yet — don't assume the shape of a future phase from an old draft.

Variables (group_vars/bigboy.yml)

# NVMe (OS + root, stays xfs)
drive_nvme_byid: "nvme-Samsung_SSD_980_500GB_S64DNL0TC07019Z"

# SATA data drives — by-id, NOT sdX (unstable across boots/reorders,
# already proven true once during this deployment)
drive_backup_byid: "ata-Lexar_SSD_NS100_256GB_QM9370R0291730S340"
drive_rag_byid: "ata-Samsung_SSD_870_EVO_500GB_S6PYNL0T620757H"
drive_ai_logs_byid: "ata-Samsung_SSD_870_EVO_250GB_S61WNJ0R458541M"
drive_prompt_byid: "ata-Lexar_SSD_NS100_256GB_QM9370R0290370S340"

# Admin account (matches kickstart's user --name=)
bigboy_admin_user: "<set to actual username>"

# GPU / driver — do NOT treat this as a version to install for; the
# precompiled path installs whatever AlmaLinux's repo currently ships.
# Record the ACTUAL version here after Phase 4 runs, replacing this
# placeholder.
nvidia_driver_version: "<record actual version after Phase 4>"

# llama.cpp — pinned, not tracking master
llama_cpp_tag: "b9968"

# Model — confirmed choice for initial bring-up, not a placeholder
model_repo: "mistralai/Ministral-3-14B-Instruct-2512-GGUF"
model_quant: "Q4_K_M"

Note on the four SATA drives' filesystem UUIDs: unlike the by-id paths above (stable, known ahead of time), the filesystem UUIDs for these drives don't exist yet — they're generated fresh when Phase 3 runs mkfs.btrfs. Don't hardcode filesystem UUIDs into this file preemptively. Phase 3's role discovers each UUID at runtime (via blkid against the by-id path, immediately after formatting) and uses that discovered value to write /etc/fstab — see the role itself for the exact mechanism.

To update: edit this file, re-run the playbook. Variables propagate to every role.

Idempotency

Every task should be safe to re-run:

  • Already-converted drives aren't re-wiped (Phase 3 checks existing filesystem type before acting)
  • Already-deployed keys aren't duplicated
  • Already-correct sudoers policy isn't rewritten unnecessarily

Principle: running the playbook twice produces the same result as running it once. If a task isn't naturally idempotent, wrap it with an explicit check (stat, command + changed_when) rather than assuming Ansible's module-level idempotency covers everything.

Logging

Phase output is logged to /srv/deployment-log/ on BigBoy (created by the kickstart's %post), one file per phase run:

/srv/deployment-log/
├── kickstart.log                     # from Phase 1's %post
├── phase2-system-config-<date>.log
├── phase3-filesystems-<date>.log
└── ...

ssh <admin-user>@192.168.0.240 "tail -f /srv/deployment-log/*.log"

Troubleshooting

Phase fails mid-run

  1. Check the phase-specific log under /srv/deployment-log/
  2. Fix manually if needed, or adjust the role
  3. Re-run just that phase: ansible-playbook site.yml --tags phaseN

SSH connection fails

  1. Confirm BigBoy's current IP — 192.168.0.240 on the bench LAN as of this writing, but confirm via ansible/inventory.ini rather than assuming it hasn't changed
  2. Confirm you're connecting as the admin user, not root — root login is locked (rootpw --lock in the kickstart), always was, this isn't something Phase 9 disables later
  3. Confirm the key at keys/workbench.pub locally matches what's actually deployed to BigBoy's authorized_keys

Idempotency broken (task reports "changed" every run)

  • Check changed_when/failed_when logic in the specific task
  • Run with -vv for full task output

Extending (Adding a New Phase)

  1. mkdir -p roles/phaseN-<name>/tasks
  2. Write tasks/main.yml
  3. First, write the task doc at ../tasks/TASK-phaseN-<name>.md scoping exactly what the phase does and doesn't do — per this project's established workflow (Claude Desktop plans/scopes, Claude Code implements against the scoped task, not the other way around)
  4. Add the role to site.yml with an appropriate tag
  5. Run: ansible-playbook site.yml --tags phaseN

SSH Key Setup (required before running this playbook)

BigBoy ships from the kickstart with password authentication only — root is locked, and no key exists yet. This is the actual chicken-and-egg this playbook's Phase 2 resolves, but the first key push has to happen manually, using password auth, before Ansible can connect via key:

# On workbench — the key already exists at ansible/keys/workbench.pub
# (gitignored, never commit the actual key material)
ssh-copy-id -i ~/.ssh/<your-key>.pub <admin-user>@192.168.0.240

# Verify
ssh -i ~/.ssh/<your-key> <admin-user>@192.168.0.240 "echo connected"

Then update inventory.ini to use ansible_user=<admin-user> (not root) and point ansible_ssh_private_key_file at the correct key.

Password authentication stays enabled even after this — disabling it is explicitly Phase 9's job, not something to do the moment a key works once.

Validation Checklist (Phase 2/3)

ssh <admin-user>@192.168.0.240

# Sudoers — bootstrap grant gone, real policy in place
sudo -l
ls /etc/sudoers.d/

# Filesystems
df -h /srv/*
mount | grep btrfs

# Root untouched
mount | grep ' / '   # should show xfs, not btrfs

Known Deferred (not gaps — deliberate)

  • Root filesystem conversion to btrfs — needs a full reinstall (clone/reformat/restore via debusb), planned for ~a month out after real-world testing, not part of this playbook
  • Additional SSH keys (Surface, tinkerpad) — one at a time, only once BigBoy is reachable on the home LAN
  • Dedicated display-monitor user — for the case-mounted 5" screen relocated from jahnet; needs autologin + btop-only access, not yet built
  • Auditable llama.cpp release pipeline and per-client agent playbooks — both designed in detail, neither built; see CLAUDE.md
  • Phases 6–14 — not yet scoped at all

Reference Files

  • ../alma10-minimal-bigboy.ks — kickstart for the unattended OS install
  • ../CLAUDE.md — standing technical decisions, read automatically by Claude Code every session
  • ../tasks/TASK-phase2-3.md, ../tasks/TASK-phase4-5-bringup.md — authoritative scope for each phase
  • ../planning/STATUS.md — authoritative current per-phase state

Last updated: 2026-07-19 Status: Phase 1 done (real hardware). Phase 2/3 roles being written against scoped task docs. Phase 4/5 scoped, not yet run.