giovannino/bigboy-alma-deploy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
bigboy-alma-deploy/planning/STATUS.md

14 KiB
Raw Permalink Blame History

BigBoy Deployment Status

Real-time status of all deployment phases

Executive Summary

Metric Status
Overall Readiness Foundation + core inference stack complete and verified on real hardware. Phases 6–14 remain, two deliberately deferred.
Phases Complete 1 through 5 (Kickstart, System Config, Filesystems, NVIDIA Driver, llama.cpp)
Real Hardware Test Done — chat confirmed working via web UI and a terminal client, 2026-07-19
Target Go-Live Base AI Server Deployment goal (chat working) — met. Full 14-phase scope continues.

Phase Status Breakdown

Phase 1: Base Install (Kickstart) ✓

Status: ✓ COMPLETE — verified on real hardware, 2026-07-17

What was done:

  • Unattended AlmaLinux 10.2 install via alma10-minimal-bigboy.ks
  • Ten distinct issues found and fixed during install day (see build-journal-20260717.md for the full account): %anaconda section incompatibility, bootloader --location fix, a line-wrap syntax error, --noipv6 correction, btrfs removed from RHEL10's kickstart (switched all partitions to xfs), a cross-repo vim/vim-data mismatch, firewall-cmd's inability to run inside the install-time chroot (replaced with a direct zone-file write), systemctl start/restart being no-ops in that same chroot, and zerombr added for the genuinely-blank drives.
  • Password-auth SSH and a temporary NOPASSWD sudoers bootstrap deliberately left active — both closed out properly in Phase 2.
  • Disk targeting via /dev/disk/by-id, not sdX — proven necessary when drives landed on different letters than expected during the actual install, and correctly resolved anyway.

Phase 2: System Configuration ✓

Status: ✓ COMPLETE — verified on real hardware, 2026-07-19

What was done:

  • Kickstart's temporary bootstrap sudoers grant replaced with a real, deliberately-scoped policy
  • Workbench's SSH public key deployed (single key, by design — see CLAUDE.md for the reasoning on why additional devices' keys wait)
  • Base system confirmed: hostname, timezone, chrony/NTP, all four expected repos enabled

Real bugs found and fixed during this run: missing ansible.posix collection (not bundled with the Alpine ansible-core package), a variable-naming mismatch (bigboy_admin_user vs. the actual deploy_user) — see CLAUDE.md/group_vars/bigboy.yml for the current state of both.

Phase 3: Filesystem Conversion ✓

Status: ✓ COMPLETE — verified on real hardware, 2026-07-19

What was done:

  • The four SATA data drives converted from the kickstart's forced xfs to real btrfs (compress=zstd,noatime), mounted at final paths (/srv/backup, /srv/rag-library, /srv/ai-logs, /srv/prompt-library)
  • Stale kickstart-generated fstab entries removed
  • Root filesystem intentionally NOT converted — stays xfs. No supported in-place xfs-to-btrfs path exists; a full reinstall (clone/ reformat/restore via debusb) is deferred until after real-world testing, not before ~a month out from 2026-07-17. This is a decision, documented in CLAUDE.md, not a gap.

Real bugs found and fixed during this run: a device-path bug ({{ item.byid }}1 instead of the correct {{ item.byid }}-part1 suffix), a mount-path mismatch specific to the ai-logs drive (/srv/ai-logs-raw in the role vs. the kickstart's actual /srv/ai-raw), and state: unmounted changed to state: absent so stale fstab entries actually get removed rather than just the live mount.

Phase 4: NVIDIA GPU Driver ✓

Status: ✓ COMPLETE — verified on real hardware, 2026-07-19

What was done:

  • AlmaLinux's precompiled open-kmod path used (not manual DKMS): almalinux-release-nvidia-driver → nvidia-driver-cuda nvidia-open-kmod, one reboot, nvidia-smi clean on the first attempt
  • The smoothest phase of the entire deployment — no issues encountered

Still needed: the actual installed driver version has not yet been recorded in group_vars/bigboy.yml — run nvidia-smi and update the placeholder there.

Phase 5: llama.cpp ✓

Status: ✓ COMPLETE — verified on real hardware, 2026-07-19

Not Ollama. Built from source, pinned to a specific, vetted tag — decided deliberately, see CLAUDE.md for the full rationale.

What was done:

  • llama.cpp built at pinned tag b9968 (checked against real commit dates and reviewed for fix/regression patterns before selection — standard flags only, -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120)
  • Model: Ministral 3 14B Instruct, Q4_K_M, pulled via llama.cpp's own Hugging Face integration (-hf flag)
  • llama-server running, systemd unit in place, nginx reverse proxy fronting it, firewall opened for the proxy port on both the home LAN and bench LAN ranges, real API key set
  • Chat confirmed working — both through llama-server's built-in web UI (tested from Surface over the home LAN) and through aichat, a terminal client, for LAN segments without GUI access

Real problems found and fixed during this run:

  • cuda-toolkit (specifically nvcc) is a separate package from the driver runtime — not installed by Phase 4, had to be added separately; the plain cuda-toolkit meta-package (not a version-pinned sub-package) was used to stay in sync with whatever driver version Phase 4 actually installed
  • nvcc lands in a versioned path (/usr/local/cuda-13.3/bin), not on PATH by default — a symlink plus a /etc/profile.d/ entry resolved this permanently
  • An OOM kill during the actual CUDA compile (cc1plus/cudafe++ confirmed via dmesg) — root cause was unbounded parallel compilation against memory-hungry CUDA source files with zero swap configured. Fixed with a 16GB NVMe swapfile (persistent via fstab) plus dropping to -j2. zram was separately decided as a complementary fast-swap tier (2026-07-19) and implemented the same day — both are now active simultaneously (zram0 at priority 100, /swapfile at priority -2, confirmed via swapon --show). See ansible/templates/zram-generator.conf for the tracked config.
  • Two rounds of missing baseline utilities discovered mid-deployment (tar, which, then nine more via a proactive utility-recon.sh scan) — all now added to the kickstart's %packages list for future installs.

This is the success criterion set on 2026-07-17 ("base AI Server Deployment... Mistral running, chat working, whatever quality, no refinement required") — met.

Phase 6: Build Suite

Status: 📋 READY TO BUILD — not yet started

Development tools / system utilities beyond what's already in the kickstart. Scope not yet finalized against current (llama.cpp-era) needs — the original scope assumed an Ollama-era package list; revisit before building.

Dependencies: Phase 4 (kernel headers matching running kernel) — done

Phase 7: Configuration

Status: 📋 READY TO BUILD — not yet started

Dotfiles, locale, timezone. Scope largely unchanged from original planning; source for dotfiles still TBD.

Dependencies: None blocking

Phase 8: Open WebUI

Status: ⚠️ LIKELY DROPPING — deliberate decision needed, not a default

llama-server's own built-in web UI already covers the chat-first use case (this is what Phase 5's success was actually verified against). Building a separate Open WebUI layer on top is no longer the default plan — keeping this phase requires an actual reason to want a second UI layer, not inertia from the original Ollama-era plan.

Phase 9: Security Hardening

Status: 📋 DESIGN READY — not yet started

Firewall (firewalld — already partially ahead of schedule, since Phase 5 opened the reverse-proxy port with the same LAN-restriction pattern used elsewhere), SSH hardening (disable password auth, move to key-only — this is the actual trigger condition already flagged in CLAUDE.md and the original kickstart's own changelog), MAC pinning, SELinux review (currently permissive per the kickstart; confirm AVC denial history before flipping to enforcing).

Blocker: key-based login needs to be confirmed working over real, repeated use before disabling password auth — not just "it worked once."

Dependencies: Phase 5 (services running) — done

Phase 10: Borgmatic Backups

Status: 🚫 DEFERRED

Reason: Backup target not yet decided (NAS? USB? Aruba Object Storage, per the sovereignty-logging design discussion?)

Timeline: Decide backup target → implement Phase 10

Phase 11: Thermal Baseline Testing

Status: 📋 READY TO BUILD — not yet started

Original scope referenced llama2:70b/Ollama-specific testing — needs updating to a llama.cpp-appropriate heavy-load test instead. Given the case move to Modcase HYPER (see below), worth doing this after the physical case transition, since cooling characteristics will differ from open-frame testing.

Dependencies: Phase 5 (llama-server running) — done

Phase 12: Full System Validation

Status: 📋 READY TO BUILD — not yet started

Checklist needs updating away from Ollama/Open-WebUI-specific service checks toward llama-server/nginx.

Dependencies: All prior phases — 1 through 5 done, 6-9/11 pending

Phase 13: Home LAN Migration

Status: 🔨 PARTIALLY UNDERWAY — ahead of original schedule

Original plan: deferred until case installation.

Actual state (2026-07-19): BigBoy is already reachable on the home LAN (192.168.1.24, statically reserved on the FritzBox) — this happened ahead of the physical case move, driven by needing to test the chat interface from a device with a GUI, which the bench LAN doesn't have. The physical relocation (case: now Modcase HYPER, not the originally-planned Modcase EVO ITX-2 — printed and ready as of 2026-07-19) is still pending; BigBoy is being assembled into its new case and permanent home over the next few days.

Modcase HYPER licensing note: personal-use licence purchased 2026-07-19. Do not print for resale or client delivery until the commercial tier is purchased.

Phase 14: Observation Period Runbook

Status: 📋 TEMPLATE READY — not yet started

Dependencies: All prior phases complete + operational — not yet at that point (6, 7, 9, 11, 12 still pending)

Cross-Cutting Concerns

Documentation

Item Status
Project docs (README, CLAUDE.md, STATUS.md) 🔨 Being actively reconciled, 2026-07-19
Deployment code (Kickstart, Ansible, inventory) ✓ Current through Phase 5
RAG library references ⚠️ Predate the llama.cpp decision (Ollama/Open-WebUI docs) — verify or replace before relying on them
Systemd/nginx configs for llama-server ❌ Only exist live on BigBoy — not yet tracked as reusable template files in this repo. Real gap: deployment isn't currently reproducible from the repo alone for this piece.

Testing

Item Status
Real hardware test, Phases 1-5 ✓ Done, 2026-07-17 through 2026-07-19
Chat interface confirmed ✓ Web UI + terminal client, 2026-07-19
Idempotency verification ✓ Confirmed for Phase 3 (re-run-safe checks in place)
Integration test (all phases) ⏳ After Phases 6-9/11-12 built

Git & Version Control

Item Status
Forgejo (giovannino/bigboy-alma-deploy) ✓ Active, pushed, current through Phase 5

Critical Path to Go-Live (Full 14-Phase Scope)

Phase 1: Kickstart ✓
    ↓
Phase 2: System Config ✓
    ↓
Phase 3: Filesystems ✓ (root stays xfs, deliberately)
    ↓
Phase 4: GPU Driver ✓
    ↓
Phase 5: llama.cpp ✓ — chat confirmed working — BASE DEPLOYMENT GOAL MET
    ↓
Phase 6-7: Build Suite, Configuration 📋
    ↓
Phase 8: Open WebUI — likely dropping ⚠️
    ↓
Phase 9: Security Hardening 📋
    ↓
Phase 11: Thermal Test (post-case-move) 📋
    ↓
Phase 12: Validation 📋
    ↓
Phase 14: Observation (4 weeks manual) 📋
    ↓
✓ FULL GO-LIVE

Parallel: Phase 10 (backup) — deferred, target undecided
Parallel: Phase 13 (home LAN) — network migration done, physical case move in progress

Blockers & Decisions Needed

Immediate

  • Real gap: firewall rich-rules script now tracked (scripts/configure-llama-firewall.sh); systemd/nginx/zram configs now tracked (ansible/templates/) — confirm all four are actually applied identically on BigBoy vs. what's committed, since the live system was configured first and the templates written to match it afterward
  • Action: record actual NVIDIA driver version in group_vars/bigboy.yml

Medium Term

  • Decision: Phase 8 (Open WebUI) — build or drop? (leaning drop)
  • Blocker: Phase 10 backup target
  • Decision: Phase 9's SELinux review — check AVC denial history before flipping enforcing

Long Term

  • Root filesystem reformat to btrfs — deliberately deferred ~a month from 2026-07-17, after real-world testing
  • The auditable llama.cpp release pipeline and per-client agent playbook pattern — both designed, neither built (see CLAUDE.md)

Success Criteria

✓ Base AI Server Deployment (2026-07-17 goal) — MET, 2026-07-19: llama.cpp built, Ministral 3 14B Instruct running, chat confirmed working through llama-server's web UI, whatever quality it delivers with zero tuning.

Full 14-phase deployment success when:

  • All phases run end-to-end, idempotently
  • llama-server/nginx stable
  • GPU verified healthy under real thermal load (post-case-move)
  • Storage verified healthy (btrfs, no corruption)
  • 4 weeks observation period completed with no critical issues

Last Updated: 2026-07-19 Next Update: After systemd/nginx configs are tracked, or Phase 6-9 implementation begins Prepared By: Claude (Desktop), reconciled against Claude Code's 2026-07-19 repo audit