giovannino/bigboy-alma-deploy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
bigboy-alma-deploy/planning/build-journal-20260717.md

34 KiB
Raw Blame History

BigBoy Build Journal — Alma Deployment

Period covered: 2026-07-16 (Thursday) through 2026-07-19 (Sunday) Status as of the latest entry: Phase 1 through 5 complete. NVIDIA driver working, llama.cpp built and serving Ministral 3 14B Instruct, chat confirmed working via llama-server's web UI and a terminal client.

This is a working log of decisions made and why — not a step-by-step runbook. For the actual procedures, see planning/ and the kickstart file itself. Nothing in this document is sensitive; no hashes, passwords, or credentials appear here even in redacted form.

Repo housekeeping

Before touching hardware, the bigboy-alma-deploy repo needed cleaning up. It had passed through a Forgejo migration and a rename (bigboy-alma → bigboy-alma-deploy), leaving some debris:

  • Duplicate ansible/ trees — one inside the git repo, one sitting outside it in an untracked parent directory. Diffed identical; the untracked copy was archived rather than deleted outright, in case anything in its history mattered later.
  • Two copies of the kickstart file (repo root and plannng/) — diffed down to a single trailing-newline difference. Root copy kept as canonical.
  • A stale README.md — the version in the live repo had been reduced to a one-line stub during the migration, while the outer untracked directory still had the full original README (architecture rationale, collaboration model, timeline). The full version was recovered and merged back in, updated for decisions made since (see below).
  • A stray kickstart/ directory with no clear purpose — investigated, found empty/unused, left alone rather than assumed-safe to delete without checking.

The Forgejo instance itself briefly blocked automated fetching entirely — turned out to be a strict default robots.txt that recent Forgejo versions ship automatically unless the admin supplies a custom one. Fixed with a permissive robots.txt override in Forgejo's custom asset path. Worth knowing: this is unrelated to Forgejo's own REQUIRE_SIGNIN_VIEW setting — one controls whether requests are allowed, the other whether crawlers are invited.

Inference engine: llama.cpp over Ollama

Decided deliberately, revisited twice, and confirmed each time:

  • Why llama.cpp at all: full control over CUDA build flags and quantization choice, plus llama-server's built-in web UI removes the need for a separate Open WebUI layer.
  • Why not Ollama: on NVIDIA/Linux, Ollama runs llama.cpp's own ggml engine underneath it anyway — choosing Ollama wouldn't avoid the dependency, just add a layer of indirection around it, and would give up independent control over exactly which llama.cpp commit is running (Ollama vendors its own snapshot, tied to its own release cycle).
  • The one place Ollama would have genuinely helped: version bookkeeping. Ollama ships proper semver-style tagged releases; llama.cpp has no semantic versioning at all, just a continuous stream of build-tagged commits (170+ in a single month, per upstream activity). That's a real cost of the llama.cpp choice, accepted deliberately rather than overlooked.
  • Known hardware-specific hazard: MXFP4-quantized models have open compilation issues on sm_120 (Blackwell) as of mid-2026. Standard quants — Q4_K_M, Q5_K_M — from established quantizers sidestep this entirely. Noted as a build-stability issue, not just a performance preference.

Conservative-deployment philosophy adopted alongside this: since almost everything publicly documented for local LLM serving assumes Docker + Ollama + n8n, and this deployment is already deviating from that norm on OS choice (Alma instead of the more commonly-documented options) and hardware (bleeding-edge Blackwell GPU), the inference engine build itself should be as boring and standard as possible — pinned tags only, standard CMake flags, no exotic tuning, systemd + nginx rather than anything novel.

Auditable release pipeline — designed, deliberately not built yet

A four-script pipeline was designed for BigBoy's eventual role as CE's reference/testbed machine for RHEL + llama.cpp client deployments:

  • update — build a candidate tag behind a btrfs snapshot safety net, never touching the live service
  • promote — the actual cutover (symlink swap + restart + health verification, with automatic revert on failure); the same mechanism serves as rollback simply by pointing it at an older tag
  • prune — retention/cleanup, generous given 500GB of headroom on the OS NVMe (disk math showed even 12 months of retained builds costs under 20GB)
  • publish — push a tag that's proven itself live on BigBoy onward to a Forgejo mirror, so client deployments elsewhere only ever pull tags that have already been vetted, never raw upstream directly

This was explicitly scoped as future work, once "get BigBoy up and running" stopped being the immediate priority. The four scripts were committed to the repo as drafts, clearly marked unintegrated, rather than either discarded or half-wired into the active phase plan.

A related, smaller correction along the way: installing Ollama does not avoid llama.cpp as a dependency — on NVIDIA/Linux, Ollama uses it as its backend. This reinforced the decision to work with llama.cpp directly rather than through a wrapper that would obscure exactly which version is running.

Claude Code handoff setup

To let Claude Code work on the repo with proper standing context rather than needing everything re-explained each session:

  • CLAUDE.md at repo root — read automatically at the start of every Claude Code session. Captures the standing technical decisions above (llama.cpp choice, build flags, version-pinning discipline, the CE Bash style guide requirements for any script in the repo) and an explicit "don't build this yet" note pointing at the deferred release pipeline.
  • Scoped task docs under tasks/ — narrow, per-unit-of-work handoffs naming exactly what's in scope, what "done" looks like, and — just as important — what's explicitly not in scope for that task, so Code doesn't drift into "helpfully" touching something already settled.

Reality check: BigBoy's actual state

A planning gap surfaced partway through: the phase-by-phase task doc had been drafted assuming Phase 3 (filesystem mounting) was already complete on real hardware. It wasn't — nothing had been installed on BigBoy at all; it still held an abandoned NixOS install attempt, never booted since. This reset the near-term plan:

  • The originally-considered "skip Phase 2/3, prioritize Phase 4/5 for a chat-only deadline" idea became unnecessary once the drive wipe turned out to take minutes, not the ~48 hours originally estimated for spinning disks — these are all SSDs, and actual measured throughput (376–540 MB/s across the four SATA drives) meant the full Phase 1→2→3→4→5 sequence fits comfortably inside the original weekend goal without reordering anything.
  • Target: chat-only functionality live by Sunday night, set on a Thursday afternoon with Phase 1 not yet started.

Drive identification and wipe

Working from a live Debian USB (debusb), all five drives (four SATA data drives + the NixOS-holding NVMe) were identified and wiped:

  • sdX naming rejected as unreliable from the outset — every operation used /dev/disk/by-id/ paths instead, keyed to each drive's actual serial number, immune to reordering across boots or port changes.
  • A cross-check against blkid labels (BACKUPDRV, RAGLIB, AILOGS, PROMPTLIB) confirmed the by-id-to-role mapping before anything was wiped — catching, along the way, that an LVM volume (ce001-vg) visible in the drive list belonged to debusb's own boot drive (a Fanxiang USB-NVMe enclosure), not to BigBoy at all. Worth flagging explicitly: this is the kind of naming collision that's cheap to catch before a wipe and expensive to discover after.
  • All four SATA drives and the OS NVMe wiped clean via dd if=/dev/zero, run detached with nohup/disown (no tmux available on the minimal live image). Each wipe's success was confirmed by its characteristic ending — a No space left on device error paired with a final, matching byte-count summary — rather than assumed from the process simply finishing.

Kickstart evolution

The kickstart file went through several rounds of correction, each driven by a real gap surfaced in conversation:

  1. User account added. The original file locked root entirely with no user account and no path to create one — meaning nothing could actually log in after install. An administrator account was added, deliberately with password authentication (not key-only) for initial bring-up, since not every device in use has an SSH key provisioned yet. This is explicitly a temporary state: Phase 9 (Security Hardening) is responsible for disabling password auth and moving to key-only.

  2. Sudo bootstrap gap identified and closed. Password login alone doesn't grant sudo — and Phase 2 (Ansible), which is what configures the real sudoers policy, itself needs privileged access to do that. A narrowly-scoped, clearly-labeled temporary NOPASSWD exception for the one administrator account closes this gap, with an explicit note in the file that Phase 2's real policy is expected to supersede it — not just coexist with it indefinitely.

  3. Firewall posture reconsidered. The original plan deferred all firewall configuration to Phase 9, which was reasonable when SSH was going to be key-only from first boot. Once password authentication became necessary, that plan created a real gap — password-auth SSH reachable from anywhere that could route to the box, with nothing enforcing otherwise. Firewalld was moved up to being enabled from first boot, with SSH access restricted via rich rules to two specific LAN ranges (home LAN and the workbench bench LAN). Confirmed via a direct check of the router's connected-devices table that WireGuard VPN clients land inside the home LAN range directly, rather than a separate VPN subnet — meaning no third rule was needed to cover remote access.

  4. Disk targeting corrected twice. First pass used sdX device names, inconsistent with the by-id discipline already established for the wipe. Second pass initially considered filesystem UUIDs as the fix — but UUIDs only exist for filesystems that have already been created, which doesn't apply to raw disks about to be partitioned for the first time. Confirmed against Anaconda's own kickstart documentation that --ondisk=/dev/disk/by-id/... is the correct, supported mechanism for exactly this case, and all disk-targeting lines (both SATA drives and the OS NVMe, plus the bootloader's target drive) were switched to it.

  5. Commentary trimmed. The file initially carried extensive inline rationale for each decision — useful during design, but not what a kickstart file is for. Comments were reduced to describing what each block does, with the reasoning kept in conversation history and (via this document) in the planning corpus instead.

Boot media preparation

  • AlmaLinux 10.2 minimal ISO (not the full DVD image, since the kickstart already relies on AlmaLinux's own package mirrors rather than requiring a fully offline install) verified against the official CHECKSUM file and GPG signature before use.
  • Two USB delivery methods prepared in parallel: a FAT32-formatted stick labeled exactly OEMDRV carrying the kickstart at its root as ks.cfg (Anaconda's automatic, no-boot-parameter detection method), with serving the kickstart over HTTP from workbench as the documented fallback if OEMDRV isn't picked up.
  • One practical wrinkle: the OEMDRV USB stick didn't expose a hardware serial number, so it had no /dev/disk/by-id/ entry at all — not every USB device does. Resolved by falling back further, to direct device-path addressing, since the device was unambiguous at the time (only one USB drive attached) and the operation (format once, copy one file, done) didn't need long-term path stability the way the SATA drives' by-id paths do.

State at the end of this entry

  • All five drives wiped and confirmed clean
  • Kickstart finalized: by-id disk targeting, LAN-restricted firewall, password-auth bootstrap with a clearly time-boxed sudo exception, functional-only comments
  • Both boot media prepared (OEMDRV stick + HTTP fallback ready)
  • BigBoy has both USB sticks attached, debusb powered off, ready to boot into the AlmaLinux installer

Next entry picks up with however Phase 1 actually goes.

Entry 2 — 2026-07-17 (Friday): Phase 1 install day

Status at end of this entry: AlmaLinux 10.2 installed and booted successfully on BigBoy. Five drives correctly mounted. Temporary password-auth SSH and NOPASSWD sudo bootstrap both active as designed. Phase 2 (Ansible) not yet run.

This entry is a genuine blow-by-blow, since almost all of it was iterative debugging against real installer failures — ten distinct issues found and fixed across the day, none of them repeats of each other. Kept in full because the specific errors and fixes are exactly what the next person (or the next kickstart-writing session) will want to search for.

First: the drive wipe, actually running

Before the OS install itself, the four SATA drives and the NixOS-holding NVMe were wiped from a live Debian USB (debusb), booted via a Fanxiang NVMe-to-USB enclosure. Confirmed clean via the same No space left on device + matching byte-count signature used in Entry 1. Total time across all five drives: well under the original conservative estimate, consistent with these being SSDs rather than spinning disks.

One environment quirk surfaced here: debusb initially had neither working ethernet nor wifi. Root cause was simply that the interface existed but had never been brought up (ip link set ... up + dhclient) — not a missing driver or firmware issue, once actually checked rather than assumed.

The kickstart's first real boot — and everything that followed

Booting the AlmaLinux 10.2 installer against the kickstart surfaced a sequence of independent failures, each fixed and each confirmed against either live error output or the official pykickstart/NVIDIA documentation before moving to the next attempt:

  1. Unknown kickstart section: %anaconda — this AlmaLinux 10 Anaconda version (a substantially rewritten installer compared to 8/9) doesn't recognize the legacy %anaconda/pwpolicy section. Removed entirely; it was only tuning password-strictness enforcement that was already set to non-strict anyway.

  2. GRUB2 does not support installation to a partition — bootloader --location=partition is a legacy BIOS-era GRUB concept. Fixed to --location=mbr, which is correct even on UEFI/GPT systems despite the name.

  3. Unknown command: --iscrypted — not a real syntax error; the user line's long password-hash value had wrapped across two lines during editing, and kickstart has no line-continuation syntax. The wrapped second line was read as its own, unrecognized top-level command. Fixed by keeping the line unbroken and adding an explicit warning comment against re-wrapping it.

  4. Invalid IPv6 address 'off' — --ipv6=off isn't valid; disabling IPv6 entirely is a separate flag, --noipv6. An easy one to get wrong since the intent ("turn IPv6 off") reads naturally into the wrong syntax.

  5. btrfs partitioning silently unsupported — caught on a documentation review before it could fail live: RHEL10's kickstart removed btrfs support entirely from part/logvol/autopart, and retired the dedicated btrfs command outright. Every partition (root and all four SATA drives) was switched from --fstype=btrfs to --fstype=xfs; Phase 3's Ansible role now carries the job of converting these to real btrfs (with subvolumes) after install, which was already the plan for root specifically, just extended to the SATA drives too.

  6. nothing provides vim-data — a genuine cross-repo version mismatch: vim-data/vim-filesystem live in BaseOS, vim-enhanced/ vim-common live in AppStream, and the two repos hadn't synced the same errata build at the same moment. Not a kickstart bug — an inherent risk whenever packages span multiple independently-updated repos. Fixed pragmatically by dropping vim from %packages (@core's vim-minimal is sufficient for pre-Phase-2 needs; full vim installs cleanly post-install once on the network normally, outside the installer's constrained resolver).

  7. firewall-cmd in %post cannot function — caught on review, before it bit anyone live, and probably the most consequential find of the day: firewalld isn't running inside the kickstart install-time chroot, so firewall-cmd --permanent fails outright ("FirewallD is not running"). Had this shipped, the LAN-restriction rules would have silently never applied — a real risk given password-auth SSH was active. Fixed by writing firewalld's zone XML file directly (/etc/firewalld/zones/public.xml), which needs no running daemon and is read normally the next time firewalld actually starts.

  8. systemctl start/restart sshd in %post — same root cause as #7: systemd isn't running in the chroot, so these are silently ignored. Harmless (enable alone is sufficient, sshd starts normally on the real first boot), but removed to stop the install log claiming something happened that didn't.

  9. A fourth repo entry considered, then deliberately reverted — after the vim mismatch (#6), a matching mirrorlist-based almalinux10-baseos repo entry was drafted to bring BaseOS in line with the other three repos' selection method. On reflection this was walked back before use: it would have created a second source serving the same content as the existing static-URL BaseOS source, risking a genuine duplicate-provider conflict, without actually addressing the real cause (BaseOS and AppStream are simply separate repos that can sync errata at different times, regardless of how either is selected). Worth recording as a real "considered and rejected" decision, not just a fix.

  10. zerombr added — caught on review: since all five drives were genuinely blank (freshly zeroed, no partition table at all), clearpart --initlabel alone is documented as only meaningful in combination with --all, not --none. Without zerombr, Anaconda could plausibly have prompted interactively to ask whether to initialize an unrecognized disk label — which would have silently stalled an unattended install waiting for input that would never come.

A verification failure, separate from all of the above

Partway through, a fix (removing vim) appeared not to have taken effect on a subsequent boot — the exact same version-pinned error recurred. This turned out to be a propagation problem, not a new bug: the corrected file hadn't actually reached the boot media (the OEMDRV stick or HTTP server) before the reboot. Confirmed by inspecting /run/install/ks.cfg directly from the installer's own emergency shell, which showed several fixes missing that had already been made. From that point on, every handoff of the file was paired with a sha256sum check at both the source and the actual served destination, specifically to catch this class of problem rather than trust that a copy step had silently succeeded.

Cross-referencing official documentation directly

Partway through the day, the actual pykickstart command reference and NVIDIA's official AlmaLinux driver installation guide were brought in directly, rather than continuing to debug reactively one live error at a time. This is what surfaced issues #7, #9, and #10 above before they could fail live, and also reshaped the Phase 4 plan:

  • NVIDIA driver installation was originally planned around manual DKMS compilation (hence kernel-devel/kernel-headers/gcc/make in the kickstart's package list). The official guide's precompiled open kernel module path is AlmaLinux's own recommended method — Secure Boot friendly, and needs none of the DKMS/compiler machinery. The kickstart's package list was left unchanged (harmless to keep, and gcc/make are still needed for Phase 5's llama.cpp build regardless) but the Phase 4 task documentation was rewritten around the actual correct commands: dnf install almalinux-release-nvidia-driver, then dnf install nvidia-driver-cuda nvidia-open-kmod for a headless, compute-only install. "Open" kernel modules specifically (not proprietary/closed) are required on Blackwell-generation cards like the RTX 5060 Ti regardless of the DKMS-vs-precompiled question.
  • The driver version recorded in group_vars/bigboy.yml (595.84) was an early planning estimate, not a real target — the precompiled path installs whatever AlmaLinux's NVIDIA driver repo currently ships. Phase 4 is now documented to record the actual installed version after the fact, rather than treating the placeholder as a version to hit.
  • Confirmed which AlmaLinux 10 official repos are enabled (BaseOS, AppStream, CRB) versus which exist but aren't (extras, devel, plus, and several workload-specific repos like HighAvailability/NFV/ RT/SAP/SAPHANA/cloud that are irrelevant to this hardware). Deliberately did not enable extras speculatively — nothing in the current plan needs it, and adding repos without a concrete reason cuts against the same conservative philosophy guiding the rest of this deployment.

Successful boot

The install completed cleanly. lsblk after boot confirmed all five drives mounted at their correct /srv/*-raw paths — notably, on different sdX letters than the original identification pass the night before, which is exactly the scenario the by-id-based disk targeting exists to protect against, and it worked exactly as intended, unprompted.

Passwordless sudo for the administrator account was confirmed present and understood correctly as the deliberate, temporary Phase-2-bootstrap mechanism it was designed to be — not a bug, and not something to leave in place past Phase 2 actually running.

Reflection

Some of today's issues were genuinely discoverable in advance by reading the official documentation carefully before the first boot, rather than after a live failure (the %anaconda section, --location=partition, --ipv6=off, and the btrfs removal all fall in this category). Others were closer to inherently only-discoverable-by-running-it, or specific operational knowledge that isn't obvious from a syntax reference alone (the cross-repo vim mismatch, and the firewall-cmd/systemctl chroot limitations specifically). Worth keeping both categories distinct going forward: "read the docs first" would have caught roughly half of tonight, not all of it.

State at the end of this entry

  • AlmaLinux 10.2 installed and booted on BigBoy
  • All five drives present, correctly identified via by-id, mounted at their intended /srv/*-raw paths (as xfs — btrfs conversion is Phase 3's job, not yet done)
  • SSH reachable via password auth, restricted to the two LAN ranges via the firewalld zone file written at install time
  • Temporary NOPASSWD sudo bootstrap active and confirmed scoped correctly
  • The repo-safe version of the kickstart (credentials stripped back to placeholders, full ten-issue changelog, and an explicitly expanded explanation of the temporary sudo bootstrap) prepared for committing to Forgejo as the durable record of this install
  • Phase 2 (Ansible System Configuration) not yet run
  • Phase 3's filesystem role needs a small but real update before it runs: it currently expects to mount existing btrfs volumes, but what's actually present is xfs; the role needs to wipe and recreate these as real btrfs with subvolumes, not just mount what's there

Next entry picks up with Phase 2.

Entry 3 — 2026-07-19 (Sunday): Phase 2 through 5, and a working chat server

Status at end of this entry: Phase 1 through 5 all complete on real hardware. BigBoy is a functioning AI inference server — NVIDIA driver loaded, llama.cpp built at pinned tag b9968, serving Ministral 3 14B Instruct via llama-server, chat confirmed working both through the built-in web UI and a terminal client. This is the "base AI Server Deployment" success criterion set on 2026-07-17 — met.

Repo reconciliation before touching anything

Before running Phase 2/3, the actual live ansible/site.yml was checked against what had been drafted blind (without seeing the real file). It contained the same root-SSH assumption already known to be wrong (ansible_user: root, ansible_become: false) — and, worse, since play-level vars: takes precedence over inventory-supplied variables in Ansible's resolution order, this would have silently overridden any fix made to inventory.ini, without any visible contradiction. Removed entirely; connection details now live only in inventory.ini. The static roles: list also referenced phase-4-nvidia-driver through phase-14-observation, none of which exist as role directories — Ansible resolves roles at parse time, before tag filtering applies, so this would have failed the play outright regardless of --tags, not just been an unbuilt-but-harmless stub.

The actual first successful Ansible connection

ansible_ssh_private_key_file pointed at id_rsa, but the deployed key was ed25519 — conventionally id_ed25519, a different filename, not just a wrong path. Once corrected, and once BigBoy's actual current IP (.241, not the stale .240 on record) was reflected in inventory.ini, ansible ... -m ping returned a clean pong — the first successful Ansible connection to BigBoy in this project's history.

Phase 2/3 — three real bugs, each caught by actually running it

  1. Missing ansible.posix collection. apk add ansible-core (deliberately chosen over the bundled ansible metapackage) gives the engine but no collections — ansible.posix.authorized_key and ansible.posix.mount both needed ansible-galaxy collection install ansible.posix before anything using them would run.

  2. bigboy_admin_user undefined. The role assumed this variable name; the actual group_vars/bigboy.yml used deploy_user instead. Same person, different name — fixed with a one-line alias (bigboy_admin_user: "{{ deploy_user }}") rather than renaming references throughout the role.

  3. Phase 3's mkfs.btrfs targeted the wrong device path entirely — {{ item.byid }}1 instead of {{ item.byid }}-part1, a bare concatenation error rather than the correct by-id partition suffix convention (already established the first night, during the drive wipe). mkfs.btrfs failed to even open the device — no data was touched, since it errored before writing anything.

  4. A second, related bug specific to one drive: the ai-logs role entry used raw_mount: /srv/ai-logs-raw, but the kickstart actually created /srv/ai-raw (shorter, dropping "logs" — inconsistent with the other three drives' naming, which do match their final names). The unmount task tried to act on a path that was never mounted, correctly reported "nothing to do," and silently left the real mountpoint completely untouched — only caught by manually running lsblk -f after the "successful" run and noticing one drive still mounted when the other three weren't.

  5. Stale fstab entries after conversion. The unmount step used state: unmounted rather than state: absent — unmounting without removing the original kickstart-generated fstab line. Left in place, this would have meant four fstab entries pointing at xfs UUIDs that no longer exist after mkfs.btrfs, a real risk of hanging at next boot waiting for devices that will never appear. Switched to state: absent, which handles both the unmount and the stale-entry cleanup in one step.

Once all four fixes landed, the re-run was clean: all four SATA drives converted to btrfs with zstd compression, mounted at their correct final paths, fstab confirmed accurate, root confirmed still untouched (xfs).

Phase 4 — smooth, genuinely

AlmaLinux's precompiled open-kmod path worked exactly as the NVIDIA documentation described: almalinux-release-nvidia-driver then nvidia-driver-cuda nvidia-open-kmod, one reboot, nvidia-smi clean on the first attempt. The one phase tonight with zero surprises.

Phase 5 — the CUDA toolkit gap, actually hit this time

This was flagged as a real risk before Phase 4/5 even started ("has Phase 4 actually run, and does nvcc --version resolve?") and it played out exactly as anticipated: nvidia-driver-cuda/nvidia-open-kmod install the driver runtime, not the CUDA toolkit — nvcc is a separate package entirely. cmake's configure step failed cleanly with "CUDA Toolkit not found." Fixed by installing the plain cuda-toolkit meta-package (not a specific cuda-toolkit-13-N sub-version, which would risk drifting out of sync with whatever driver version actually got installed) from the same almalinux-nvidia repo the driver bootstrap had already enabled. nvcc itself lands in a versioned path (/usr/local/cuda-13.3/bin) not automatically on PATH — a symlink (/usr/local/cuda → /usr/local/cuda-13.3) plus a /etc/profile.d/ entry resolved this permanently rather than needing a manual export every session.

The OOM kill — a real, if fixable, resource gap

With nvcc resolved, the actual CUDA compile ran into the kernel's OOM killer partway through — cc1plus and cudafe++ both confirmed as victims via dmesg. Root cause: unbounded -j (all 6 cores at once) against CUDA source files, which are known to be memory-hungry to compile due to heavy template instantiation, on a 16GB box with zero swap configured — a gap that traces back to Phase 2/3 never actually implementing the zram_percentage: 50 variable already sitting unused in group_vars/bigboy.yml since the file's original Ollama-era drafting. Fixed pragmatically for tonight with a 16GB NVMe swapfile (not the zram variable — that's still an open reconciliation item, noted but not resolved) plus dropping to -j2. Rebuild resumed from cached objects rather than starting over, and completed cleanly.

First successful chat, and a genuinely capable model choice reconfirmed

llama-server running Ministral 3 14B Instruct at Q4_K_M, served on 127.0.0.1:8080, responded correctly through the browser-based built-in UI. This is the literal success criterion set on 2026-07-17 — met, without any tuning, exactly as scoped.

A TUI chat client, since the bench LAN has no GUI devices

Since BigBoy's actual network segment has no monitor/browser access, aichat (a Rust-based terminal client with configurable OpenAI-compatible endpoints — confirmed current, v0.30.0, actively maintained) was installed to talk to the same running llama-server instance directly, no separate model load. This surfaced two more gaps in the same minimal- install pattern already seen with vim: tar and which were both missing — neither is actually part of GNU coreutils, a common misconception, and neither was explicitly requested in the kickstart's trimmed %packages list. A proactive utility-recon.sh script was written to check a broader baseline of commonly-expected CLI tools in one pass, rather than continuing to discover gaps reactively one at a time. Run against the live system, it found nine further gaps beyond tar/which: bc, bind-utils (dig), bzip2, lsof, nmap-ncat (nc), rsync, strace, traceroute, tree — a reasonable admin/dev baseline (archive tools, network diagnostics, process inspection, debugging, text tools) the kickstart's deliberately trimmed package list simply never anticipated needing. All eleven (plus vim-enhanced, confirmed installable cleanly once on the live network — the earlier vim-data mismatch was purely an installer-time repo sync issue) are now added to the kickstart for future installs from this file.

State at the end of this entry

  • Phases 1 through 5 all complete and verified on real hardware
  • BigBoy is a working AI inference server: NVIDIA driver loaded, llama.cpp built at pinned tag b9968, Ministral 3 14B Instruct serving via llama-server, chat confirmed via both web UI and aichat
  • A 16GB NVMe swapfile is active (/swapfile, persistent via fstab) — not the originally-planned zram approach; that variable in group_vars/bigboy.yml is now a stale, unreconciled leftover worth a deliberate decision later
  • The kickstart has been updated with tar, which, and vim-enhanced added explicitly, and a general-purpose utility recon script exists for checking future installs against a broader baseline
  • Root remains xfs, untouched, as planned — its eventual conversion is still deferred pending real-world testing, per the 2026-07-17 decision
  • Not yet done, deliberately: systemd unit for llama-server (currently running in foreground for testing), nginx reverse proxy, firewall opening for the proxy port, --api-key set to a real value — all still on the Phase 5 task doc's list, not yet executed tonight
  • The auditable llama.cpp release pipeline, per-client agent playbook pattern, and the whole sovereignty-logging design remain exactly as deferred in earlier entries — nothing about tonight's success changes their priority

Next entry picks up with hardening Phase 5 (systemd, nginx, firewall) — or whatever else surfaces first.