From 31564199742bdfd2d27a1b572a72eb46c46f59c3 Mon Sep 17 00:00:00 2001 From: "John A. Hoeven" Date: Sun, 19 Jul 2026 22:24:55 +0200 Subject: [PATCH] Post deploy updates --- CLAUDE.md | 19 -- README.md | 107 +++++++---- ansible/README.md | 210 +++++++++++++-------- planning/STATUS.md | 445 ++++++++++++++++++++++----------------------- 4 files changed, 429 insertions(+), 352 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 9ebd3fa..ed15604 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -112,25 +112,6 @@ built-in web UI, chat confirmed working). No agent roles, no RAG corpus, no ingestion pipeline get built at this stage — that's all downstream of an actual use case being decided, which hasn't happened yet. -**Per-client agent playbook pattern** — not a fixed pipeline to build -once and reuse. CE's own Ambrosiana deployment and coding-assistant setup -serve as proven reference implementations (working examples of agent-role -design, recon checks, ingestion orchestration), but each client -deployment gets its own bespoke agent-role setup based on their actual -use case — captured as a documented, repeatable Ansible playbook for -reinstall recovery and potential future hive federation, not copied -wholesale from CE's own internal setup. The Ansible-docs-as-RAG-corpus -discussion from 2026-07-17 (Phi-4 Mini for license/scraping-permission/ -malicious-content recon, Claude Code for chunking and ingestion -orchestration, Qwen3 0.6B for style-guide formatting) is one example of -the design thinking that goes into building a specific instance of this -pattern — not a template to copy verbatim into a client deploy. **Current -priority for BigBoy specifically is the base AI Server Deployment**: -Phase 4 (NVIDIA driver) → Phase 5 (llama.cpp + Mistral + `llama-server`'s -built-in web UI, chat confirmed working). No agent roles, no RAG corpus, -no ingestion pipeline get built at this stage — that's all downstream of -an actual use case being decided, which hasn't happened yet. - ## Scope discipline Task docs will name a specific phase or task from `STATUS.md`. Do the diff --git a/README.md b/README.md index 514b61a..8b10cbc 100644 --- a/README.md +++ b/README.md @@ -53,15 +53,21 @@ of indirection around it. See `CLAUDE.md` for the full standing rationale - **GPU VRAM:** 16GB (RTX 5060 Ti) — model selection accordingly - **Storage:** 5 drives (1 NVMe + 4 SATA) with btrfs subvolume isolation -- **Network:** bench LAN (192.168.0.0/24), workbench acts as gateway/DHCP - for this segment + on the SATA drives; NVMe root stays xfs (see Current Status) +- **Network:** home LAN (`192.168.1.24`, static reservation) — originally + deployed and tested on the workbench bench LAN (`192.168.0.0/24`), + migrated once chat-interface testing required a GUI-capable device - **Idempotency:** every phase must be re-runnable --- ## Current Status — read this before assuming anything is done -**As of 2026-07-18: AlmaLinux 10.2 successfully installed and booted on BigBoy.** All five drives confirmed mounted and accessible. Foundation complete; core inference stack (Phase 4–5) ready to build. +**As of 2026-07-19: Phases 1 through 5 complete and verified on real +hardware.** BigBoy is a working AI inference server — NVIDIA driver +loaded, llama.cpp built and pinned, Ministral 3 14B Instruct serving via +`llama-server`, chat confirmed working through both the built-in web UI +and a terminal client (`aichat`). See `./planning/STATUS.md` for the authoritative, current per-phase state. Don't rely on this README for phase-by-phase status — it will drift out @@ -69,44 +75,63 @@ of date faster than `STATUS.md` is maintained. ### Deployed & Verified on Hardware -- Kickstart (`alma10-minimal-bigboy.ks`) — AlmaLinux 10.2 minimal install - successfully executed 2026-07-17 -- Phase 3 role (filesystem mounting) — run and verified, all 4 SATA + 1 NVMe - drives mounted at intended paths via UUID - -### Ready to build (Phase 1–3 complete) - -- Phase 4 (NVIDIA driver) — precompiled open-kmod via AlmaLinux repos -- Phase 5 (llama.cpp) — build from pinned tag, standard flags, `llama-server` - under systemd, see `CLAUDE.md` and `tasks/TASK-phase4-5-bringup.md` +- Kickstart (`alma10-minimal-bigboy.ks`) — AlmaLinux 10.2 minimal install, + 2026-07-17 +- Phase 2 (system configuration) — sudoers policy, SSH key, base checks +- Phase 3 (filesystem conversion) — all four SATA drives converted to + real btrfs; root intentionally stays xfs (no supported in-place + conversion path — a full reinstall is deferred ~a month out, after + real-world testing, not a gap) +- Phase 4 (NVIDIA driver) — AlmaLinux's precompiled open-kmod path, + confirmed via `nvidia-smi` +- Phase 5 (llama.cpp) — built at pinned tag `b9968`, serving Ministral 3 + 14B Instruct at Q4_K_M, systemd + nginx + firewall in place, chat + confirmed working ### Likely dropping -- Phase 8 (Open WebUI) — `llama-server`'s built-in UI covers the - chat-first use case; keeping this phase is now a deliberate decision - to make, not a default +- Phase 8 (Open WebUI) — `llama-server`'s built-in UI already covers the + chat-first use case; this is what Phase 5's success was actually + verified against. Keeping this phase now needs an actual reason, not + just inertia from the original plan. ### Deferred, on purpose -- Phases 6–7, 9–14 (Build Suite, Configuration, Security, Borgmatic, - Thermal, Validation, Migration, Observation) +- Phases 6–7, 9, 11–12, 14 — not yet started; see `STATUS.md` for + per-phase detail +- Root filesystem's eventual xfs→btrfs conversion — full reinstall, + deliberately deferred pending real-world testing - An auditable testing/release system for llama.cpp (versioned releases, btrfs snapshot safety net, promote/rollback, a Forgejo mirror that only ever receives BigBoy-validated tags for client deployments) — fully - designed 2026-07-16, not yet built. See `CLAUDE.md` for the deferred-work - note and `scripts/draft/` if present. + designed 2026-07-16, not yet built. See `CLAUDE.md` for the + deferred-work note and `scripts/draft/` if present. +- The per-client agent playbook pattern — same status, see `CLAUDE.md` - Borgmatic backup target (not yet decided) -- Home LAN migration (waiting on case installation) +- Phase 13's physical case move — network migration to the home LAN is + done; the physical relocation into the new Modcase HYPER case is + actively in progress as of 2026-07-19 + +### A known, real gap — not yet closed + +The systemd unit, nginx reverse-proxy config, and firewall rules that +make Phase 5's chat deployment actually work currently exist **only on +BigBoy itself**, configured directly during the live session — not yet +committed to this repo as tracked template files. Until that's fixed, +the deployment isn't fully reproducible from the repo alone. See +`STATUS.md`'s "Immediate" blockers section. --- ## Collaborators & Workflow ### John A. Hoeven (@workbench) + - Architect & operator — makes deployment decisions, edits and pushes from workbench via VS Codium -### Claude Code (@bigboy once installed, or driven from @workbench) +### Claude Code (@bigboy, or driven from @workbench) + - Executes scoped tasks against the actual repo — see `CLAUDE.md` for standing context read automatically each session, and `tasks/` for the current scoped handoff @@ -114,6 +139,7 @@ of date faster than `STATUS.md` is maintained. bounds for a given session ### Claude Desktop + - Plans and orchestrates: architecture discussion, design decisions, drafts `CLAUDE.md` and task docs for Claude Code to execute against @@ -121,7 +147,7 @@ of date faster than `STATUS.md` is maintained. ## Repo Structure -``` +```text bigboy-alma-deploy/ ├── CLAUDE.md # standing context, read by Claude Code every session ├── README.md # this file @@ -132,8 +158,9 @@ bigboy-alma-deploy/ │ ├── inventory.ini │ ├── group_vars/bigboy.yml # hardware facts: UUIDs, driver version, pinned llama.cpp tag │ └── roles/ -│ └── phase-3-filesystems/ # written, reviewed, not yet run on hardware -├── planning/ # renamed from plannng (typo) on 2026-07-17 +│ ├── phase-2-system-config/ # done, verified on hardware +│ └── phase-3-filesystems/ # done, verified on hardware +├── planning/ │ ├── ARCHITECTURE.md │ ├── HARDWARE.md │ ├── SOVEREIGNTY-POLICY.md @@ -143,8 +170,11 @@ bigboy-alma-deploy/ │ └── alma10-minimal-bigboy.ks # reference snapshot from planning — root copy is canonical ├── tasks/ │ └── TASK-*.md # scoped Claude Code handoffs, one per unit of work -└── scripts/draft/ # (if present) draft auditable-release-pipeline scripts — - # unintegrated, see CLAUDE.md before touching +└── scripts/ + ├── package-recon.sh # confirms package availability before finalizing task docs + ├── utility-recon.sh # checks for missing baseline CLI utilities + └── draft/ # draft auditable-release-pipeline scripts — + # unintegrated, see CLAUDE.md before touching ``` --- @@ -152,11 +182,13 @@ bigboy-alma-deploy/ ## Getting Started ### To review design + 1. Read `./planning/ARCHITECTURE.md` — design rationale, 14-phase breakdown 2. Read `./planning/HARDWARE.md` — component specs, UUIDs, network config 3. Read `./planning/STATUS.md` — the real current state, phase by phase ### To pick up work + 1. Check `./planning/STATUS.md` for what's actually next 2. Look in `./tasks/` for an existing scoped task doc covering it 3. If none exists, Claude Desktop drafts one before Claude Code starts — @@ -164,12 +196,15 @@ bigboy-alma-deploy/ ### Next Steps (current priority) -1. Build and deploy Phase 4 (NVIDIA driver) via Ansible or shell commands, - verify with `nvidia-smi` -2. Build and deploy Phase 5 (llama.cpp) — see `./tasks/TASK-phase4-5-bringup.md` - for the current scoped handoff -3. Verify `llama-server` running and responding to inference requests -4. Phases 6–14 follow once 4–5 are stable and tested +1. Commit tracked template files for the systemd unit, nginx config, and + firewall rules currently only configured live on BigBoy +2. Record the actual NVIDIA driver version in `group_vars/bigboy.yml` + (placeholder currently reads `CHECK-WITH-nvidia-smi`) +3. Implement zram alongside the existing swapfile (decision made + 2026-07-19, not yet acted on) +4. Decide Phase 8 (Open WebUI) — build or drop +5. Complete the physical case move to Modcase HYPER, then proceed with + Phases 6–7, 9, 11–12, 14 --- @@ -187,5 +222,7 @@ bigboy-alma-deploy/ --- -**Last updated:** 2026-07-18 -**Current priority:** Phase 4 (NVIDIA driver) and Phase 5 (llama.cpp build & serve) \ No newline at end of file +**Last updated:** 2026-07-19 +**Current priority:** Track the systemd/nginx/firewall configs that +currently only exist on BigBoy itself; complete the case move; decide +Phase 8 \ No newline at end of file diff --git a/ansible/README.md b/ansible/README.md index a97bd79..1591b2f 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -3,6 +3,11 @@ Idempotent Ansible playbook for configuring BigBoy from a fresh AlmaLinux 10.2 kickstart install through to a running llama.cpp inference server. +**Status (2026-07-19): Phases 2 through 5 have all run successfully on +real hardware.** This isn't a "should work" playbook anymore — it's the +one that actually built the working deployment. See `../planning/STATUS.md` +for the authoritative current state. + --- ## Quick Start @@ -18,19 +23,19 @@ Idempotent Ansible playbook for configuring BigBoy from a fresh AlmaLinux ```bash # From ansible/ -ansible-playbook site.yml -i inventory.ini --tags phase2,phase3 +ansible-playbook site.yml -i inventory.ini --tags phase-2,phase-3 ``` ### Re-running a specific phase ```bash -ansible-playbook site.yml -i inventory.ini --tags phase3 +ansible-playbook site.yml -i inventory.ini --tags phase-3 ``` ### Check mode (show what would change, without changing anything) ```bash -ansible-playbook site.yml -i inventory.ini --check --tags phase2,phase3 +ansible-playbook site.yml -i inventory.ini --check --tags phase-2,phase-3 ``` --- @@ -46,12 +51,15 @@ ansible/ ├── group_vars/ │ └── bigboy.yml # Hardware facts: UUIDs, by-id paths, pinned versions ├── roles/ -│ ├── phase2-system-config/ -│ │ └── tasks/main.yml # Sudoers policy, SSH key, base checks +│ ├── phase-2-system-config/ +│ │ └── tasks/main.yml # DONE — sudoers policy, SSH key, base checks │ ├── phase-3-filesystems/ -│ │ └── tasks/main.yml # Convert 4 SATA drives xfs -> btrfs -│ ├── phase-4-nvidia-driver/ # scaffold — see tasks/TASK-phase4-5-bringup.md -│ ├── phase-5-llama-cpp/ # scaffold — see tasks/TASK-phase4-5-bringup.md +│ │ └── tasks/main.yml # DONE — converts 4 SATA drives xfs -> btrfs +│ ├── phase-4-nvidia-driver/ # not yet a tracked role — Phase 4 was run via +│ │ direct shell commands per tasks/TASK-phase4-5-bringup.md, +│ │ not Ansible. Worth deciding whether to formalize +│ │ into a role for reproducibility. +│ ├── phase-5-llama-cpp/ # same — run manually, not yet an Ansible role │ └── ... (phases 6-14, not yet scoped) └── README.md # This file ``` @@ -61,6 +69,16 @@ here — `TASK-phase2-3.md` and `TASK-phase4-5-bringup.md` are the authoritative descriptions of what each phase actually does; this README is a summary and operational reference, not the source of truth for scope. +**Worth knowing:** Phase 4 (NVIDIA driver) and Phase 5 (llama.cpp build, +model pull, systemd/nginx/firewall setup) were both executed as direct +shell commands against BigBoy during the actual deployment session, not +through Ansible roles. They worked, and they're real — but they're not +currently reproducible by re-running this playbook. If that matters for +future deployments (e.g., disaster recovery, or building the next +client's box from this same reference), those steps need to become +actual Ansible roles at some point — currently tracked as an open item, +not done. + --- ## Phase Breakdown @@ -70,20 +88,27 @@ is a summary and operational reference, not the source of truth for scope. Unattended AlmaLinux 10.2 install via kickstart. See `../alma10-minimal-bigboy.ks`. -### Phase 2: System Configuration — see `tasks/TASK-phase2-3.md` +### Phase 2: System Configuration — DONE, verified on hardware -- Replace the kickstart's temporary bootstrap NOPASSWD sudoers grant with - a real, deliberately-scoped policy -- Deploy workbench's SSH public key (single key only at this stage — - BigBoy sits on the isolated bench LAN; more keys get added later, one - at a time, once on the home LAN) -- Confirm hostname, timezone, NTP/chrony, and repo configuration +- Replaced the kickstart's temporary bootstrap NOPASSWD sudoers grant + with a real, deliberately-scoped policy +- Deployed workbench's SSH public key (single key only at this stage — + BigBoy sits on the home LAN now; more keys get added later, one at a + time, once tested individually) +- Confirmed hostname, timezone, NTP/chrony, and repo configuration + +Real bugs hit and fixed during the actual run: a missing +`ansible.posix` collection (not bundled with the Alpine `ansible-core` +package — `ansible-galaxy collection install ansible.posix` was +required), and a variable-naming mismatch (`bigboy_admin_user` expected +by the role vs. `deploy_user` actually defined in `group_vars`) — see +`CLAUDE.md` for the current state of both. **Does not** disable password SSH authentication — that's Phase 9, once key-based login is confirmed working over real use, not assumed the moment a key is deployed. -### Phase 3: Filesystem Conversion — see `tasks/TASK-phase2-3.md` +### Phase 3: Filesystem Conversion — DONE, verified on hardware The kickstart formats all five drives as **xfs** (RHEL10's Anaconda removed btrfs support entirely — it's not a choice, it's what's actually @@ -100,67 +125,79 @@ real btrfs with subvolumes and zstd compression: **Root stays xfs, permanently for now.** There's no supported in-place xfs-to-btrfs conversion — this would require a full reinstall, planned separately and deliberately deferred until after real-world testing (not -before ~a month out). Do not attempt to touch root as part of this phase. +before ~a month out from 2026-07-17). Do not attempt to touch root as +part of this phase. -### Phase 4: NVIDIA Driver — see `tasks/TASK-phase4-5-bringup.md` +Real bugs hit and fixed during the actual run: a device-path targeting +bug (`{{ item.byid }}1` instead of the correct `{{ item.byid }}-part1` +suffix — caught when `mkfs.btrfs` couldn't open the device, no data was +touched), a mount-path mismatch specific to the ai-logs drive +(the role assumed `/srv/ai-logs-raw`, the kickstart actually created +`/srv/ai-raw` — caught by manually running `lsblk -f` after an +apparently-successful run and noticing one drive still mounted when the +other three weren't), and `state: unmounted` changed to `state: absent` +so the kickstart's original fstab entries actually get removed, not just +the live mounts. + +### Phase 4: NVIDIA Driver — DONE, run manually (not yet an Ansible role) AlmaLinux's precompiled open-kmod path (`almalinux-release-nvidia-driver` → `nvidia-driver-cuda nvidia-open-kmod`), not manual DKMS. Confirmed as the AlmaLinux-recommended method, Secure Boot compatible, needs none of -the kernel-devel/DKMS/compiler machinery DKMS would. +the kernel-devel/DKMS/compiler machinery DKMS would. Ran clean, no issues +— the smoothest phase of the entire deployment. -### Phase 5: llama.cpp — see `tasks/TASK-phase4-5-bringup.md` +### Phase 5: llama.cpp — DONE, run manually (not yet an Ansible role) **Not Ollama.** Built from source, pinned to a specific tag (`b9968`, recorded in `group_vars/bigboy.yml`), served via `llama-server` under -systemd behind nginx. Initial model: Ministral 3 14B Instruct, Q4_K_M — -success criterion is a coherent chat response through the built-in web -UI, quality as-delivered, no tuning required at this stage. +systemd behind nginx. Model: Ministral 3 14B Instruct, Q4_K_M — success +criterion was a coherent chat response through the built-in web UI, +quality as-delivered, no tuning required — **met**, confirmed via both +the web UI and a terminal client (`aichat`). -### Phases 6–14: not yet scoped - -Deferred until 4/5 are stable and tested. No committed design for these -yet — don't assume the shape of a future phase from an old draft. +Real problems hit and fixed during the actual run: +- `cuda-toolkit` (specifically `nvcc`) is a separate package from the + driver runtime installed by Phase 4 — had to be installed separately +- `nvcc` lands in a versioned path, not on `PATH` by default — resolved + with a symlink and a `/etc/profile.d/` entry +- An OOM kill during the actual CUDA compile, root-caused via `dmesg` to + unbounded parallel compilation with zero swap configured — fixed with + a 16GB NVMe swapfile plus `-j2` +- Two rounds of missing baseline utilities (`tar`, `which`, then nine + more found via a proactive recon scan) — added to the kickstart for + future installs --- ## Variables (`group_vars/bigboy.yml`) -```yaml -# NVMe (OS + root, stays xfs) -drive_nvme_byid: "nvme-Samsung_SSD_980_500GB_S64DNL0TC07019Z" +See the file itself for the current, complete set — it's been through a +real reconciliation pass (2026-07-19) against everything actually +deployed. Key points: -# SATA data drives — by-id, NOT sdX (unstable across boots/reorders, -# already proven true once during this deployment) -drive_backup_byid: "ata-Lexar_SSD_NS100_256GB_QM9370R0291730S340" -drive_rag_byid: "ata-Samsung_SSD_870_EVO_500GB_S6PYNL0T620757H" -drive_ai_logs_byid: "ata-Samsung_SSD_870_EVO_250GB_S61WNJ0R458541M" -drive_prompt_byid: "ata-Lexar_SSD_NS100_256GB_QM9370R0290370S340" - -# Admin account (matches kickstart's user --name=) -bigboy_admin_user: "" - -# GPU / driver — do NOT treat this as a version to install for; the -# precompiled path installs whatever AlmaLinux's repo currently ships. -# Record the ACTUAL version here after Phase 4 runs, replacing this -# placeholder. -nvidia_driver_version: "" - -# llama.cpp — pinned, not tracking master -llama_cpp_tag: "b9968" - -# Model — confirmed choice for initial bring-up, not a placeholder -model_repo: "mistralai/Ministral-3-14B-Instruct-2512-GGUF" -model_quant: "Q4_K_M" -``` +- **IP address**: `192.168.1.24` — BigBoy's current, permanent home-LAN + address. It ran on the workbench bench LAN for the entire initial + deploy first (`.240`, then `.241` after a DHCP reassignment) — both + genuinely correct at the time, not mistakes, and noted as history in + the file rather than erased. +- **Drive UUIDs**: not consumed directly by any role — Phase 3 discovers + them at runtime via `blkid` against the stable by-id paths, since they + don't exist until `mkfs.btrfs` actually runs. The file records the + current live UUIDs for reference only. +- **`nvidia_driver_version`**: still a placeholder (`CHECK-WITH-nvidia-smi`) + — the actual value was never captured during the deployment session. + Run `nvidia-smi` and fill this in. +- **API key**: deliberately NOT stored in this file — it's committed to + a public repo. Needs vault or a gitignored file, same pattern as + `keys/workbench.pub`. **Note on the four SATA drives' filesystem UUIDs**: unlike the by-id -paths above (stable, known ahead of time), the *filesystem* UUIDs for -these drives don't exist yet — they're generated fresh when Phase 3 runs -`mkfs.btrfs`. Don't hardcode filesystem UUIDs into this file preemptively. -Phase 3's role discovers each UUID at runtime (via `blkid` against the -by-id path, immediately after formatting) and uses that discovered value -to write `/etc/fstab` — see the role itself for the exact mechanism. +paths (stable, known ahead of time), the *filesystem* UUIDs don't exist +until Phase 3 runs `mkfs.btrfs`. Don't hardcode filesystem UUIDs into +this file preemptively for a fresh install — Phase 3's role discovers +each UUID at runtime and uses that discovered value to write +`/etc/fstab`. **To update:** edit this file, re-run the playbook. Variables propagate to every role. @@ -171,7 +208,8 @@ to every role. Every task should be safe to re-run: - Already-converted drives aren't re-wiped (Phase 3 checks existing - filesystem type before acting) + filesystem type before acting) — confirmed working correctly on the + actual deployment - Already-deployed keys aren't duplicated - Already-correct sudoers policy isn't rewritten unnecessarily @@ -196,7 +234,7 @@ the kickstart's `%post`), one file per phase run: ``` ```bash -ssh @192.168.0.240 "tail -f /srv/deployment-log/*.log" +ssh john@192.168.1.24 "tail -f /srv/deployment-log/*.log" ``` --- @@ -211,9 +249,10 @@ ssh @192.168.0.240 "tail -f /srv/deployment-log/*.log" ### SSH connection fails -1. Confirm BigBoy's current IP — `192.168.0.240` on the bench LAN as of +1. Confirm BigBoy's current IP — `192.168.1.24` on the home LAN as of this writing, but confirm via `ansible/inventory.ini` rather than - assuming it hasn't changed + assuming it hasn't changed. It's changed twice already during this + deployment (bench LAN `.240` → `.241` → home LAN `.24`). 2. Confirm you're connecting as the admin user, **not root** — root login is locked (`rootpw --lock` in the kickstart), always was, this isn't something Phase 9 disables later @@ -225,6 +264,15 @@ ssh @192.168.0.240 "tail -f /srv/deployment-log/*.log" - Check `changed_when`/`failed_when` logic in the specific task - Run with `-vv` for full task output +### Missing `ansible.posix` collection + +Hit during the real Phase 2 run — `apk add ansible-core` (Alpine's +minimal package, deliberately chosen over the full `ansible` bundle) +doesn't include community collections: +```bash +ansible-galaxy collection install ansible.posix +``` + --- ## Extending (Adding a New Phase) @@ -238,6 +286,11 @@ ssh @192.168.0.240 "tail -f /srv/deployment-log/*.log" 4. Add the role to `site.yml` with an appropriate tag 5. Run: `ansible-playbook site.yml --tags phaseN` +**Worth doing at some point**: Phases 4 and 5 were run as direct shell +commands, not Ansible roles. Formalizing them into real roles (following +this same pattern) would make the deployment fully reproducible from the +repo alone — currently it isn't, for those two phases specifically. + --- ## SSH Key Setup (required before running this playbook) @@ -250,10 +303,10 @@ manually, using password auth, before Ansible can connect via key: ```bash # On workbench — the key already exists at ansible/keys/workbench.pub # (gitignored, never commit the actual key material) -ssh-copy-id -i ~/.ssh/.pub @192.168.0.240 +ssh-copy-id -i ~/.ssh/.pub @192.168.1.24 # Verify -ssh -i ~/.ssh/ @192.168.0.240 "echo connected" +ssh -i ~/.ssh/ @192.168.1.24 "echo connected" ``` Then update `inventory.ini` to use `ansible_user=` (not @@ -268,7 +321,7 @@ once. ## Validation Checklist (Phase 2/3) ```bash -ssh @192.168.0.240 +ssh john@192.168.1.24 # Sudoers — bootstrap grant gone, real policy in place sudo -l @@ -282,6 +335,15 @@ mount | grep btrfs mount | grep ' / ' # should show xfs, not btrfs ``` +## Validation Checklist (Phase 4/5) + +```bash +nvidia-smi +systemctl status llama-server +systemctl status nginx +curl http://127.0.0.1:8080/health +``` + --- ## Known Deferred (not gaps — deliberate) @@ -290,12 +352,15 @@ mount | grep ' / ' # should show xfs, not btrfs (clone/reformat/restore via `debusb`), planned for ~a month out after real-world testing, not part of this playbook - **Additional SSH keys** (Surface, tinkerpad) — one at a time, only - once BigBoy is reachable on the home LAN + once tested individually - **Dedicated display-monitor user** — for the case-mounted 5" screen relocated from jahnet; needs autologin + `btop`-only access, not yet built - **Auditable llama.cpp release pipeline** and **per-client agent playbooks** — both designed in detail, neither built; see `CLAUDE.md` +- **Phases 4/5 as actual Ansible roles** — currently run manually, not + automated; a real gap for reproducibility, not deliberately deferred + the way the items above are - **Phases 6–14** — not yet scoped at all --- @@ -303,14 +368,17 @@ mount | grep ' / ' # should show xfs, not btrfs ## Reference Files - `../alma10-minimal-bigboy.ks` — kickstart for the unattended OS install -- `../CLAUDE.md` — standing technical decisions, read automatically by - Claude Code every session +- `../CLAUDE.md` — standing context, read automatically by Claude Code + every session - `../tasks/TASK-phase2-3.md`, `../tasks/TASK-phase4-5-bringup.md` — authoritative scope for each phase - `../planning/STATUS.md` — authoritative current per-phase state +- `../planning/build-journal-20260717.md` — detailed account of Phase 1's + install day, including all ten issues found and fixed --- **Last updated:** 2026-07-19 -**Status:** Phase 1 done (real hardware). Phase 2/3 roles being written -against scoped task docs. Phase 4/5 scoped, not yet run. +**Status:** Phase 1 through 5 complete and verified on real hardware. +Phase 4/5 currently run manually, not yet formalized as Ansible roles. +Phases 6–14 not yet scoped. \ No newline at end of file diff --git a/planning/STATUS.md b/planning/STATUS.md index 4f7faa8..0571ea7 100644 --- a/planning/STATUS.md +++ b/planning/STATUS.md @@ -1,6 +1,6 @@ # BigBoy Deployment Status -**Real-time status of all 14 deployment phases** +**Real-time status of all deployment phases** --- @@ -8,172 +8,190 @@ | Metric | Status | |--------|--------| -| **Overall Readiness** | 40% — Foundation complete, core phases buildable | -| **Phases Complete** | 1 of 14 (Phase 3) | -| **Phases Buildable** | 3 of 14 (Phases 4-5 + 8 optional) | -| **Documentation** | Complete for all 14 phases in RAG library | -| **Real Hardware Test** | Pending (awaiting BigBoy installation) | -| **Target Go-Live** | Q3 2026 | +| **Overall Readiness** | Foundation + core inference stack complete and verified on real hardware. Phases 6–14 remain, two deliberately deferred. | +| **Phases Complete** | 1 through 5 (Kickstart, System Config, Filesystems, NVIDIA Driver, llama.cpp) | +| **Real Hardware Test** | **Done** — chat confirmed working via web UI and a terminal client, 2026-07-19 | +| **Target Go-Live** | Base AI Server Deployment goal (chat working) — met. Full 14-phase scope continues. | --- ## Phase Status Breakdown -### Phase 3: Filesystem Validation ✓ +### Phase 1: Base Install (Kickstart) ✓ -**Status:** ✓ **COMPLETE** +**Status:** ✓ **COMPLETE — verified on real hardware, 2026-07-17** -**What Was Done:** -- Ansible role created: `/home/john/projects/bigboy-setup/ansible/roles/phase-3-filesystems/` -- Task: Mount 4 SATA drives via UUID + subvolume name -- Compression: zstd (level 3 general, level 1 for logs) -- Mount options: noatime, nofail -- Output: Logs to `/srv/deployment-log/phase-03-filesystems-*.log` +**What was done:** +- Unattended AlmaLinux 10.2 install via `alma10-minimal-bigboy.ks` +- Ten distinct issues found and fixed during install day (see + `build-journal-20260717.md` for the full account): `%anaconda` section + incompatibility, `bootloader --location` fix, a line-wrap syntax error, + `--noipv6` correction, btrfs removed from RHEL10's kickstart (switched + all partitions to xfs), a cross-repo `vim`/`vim-data` mismatch, + `firewall-cmd`'s inability to run inside the install-time chroot + (replaced with a direct zone-file write), `systemctl start/restart` + being no-ops in that same chroot, and `zerombr` added for the + genuinely-blank drives. +- Password-auth SSH and a temporary NOPASSWD sudoers bootstrap + deliberately left active — both closed out properly in Phase 2. +- Disk targeting via `/dev/disk/by-id`, not `sdX` — proven necessary when + drives landed on different letters than expected during the actual + install, and correctly resolved anyway. -**Testing:** -- Logic reviewed ✓ -- Idempotent mount module used ✓ -- Error handling via blocks ✓ +### Phase 2: System Configuration ✓ -**Status:** Ready for real hardware test +**Status:** ✓ **COMPLETE — verified on real hardware, 2026-07-19** ---- +**What was done:** +- Kickstart's temporary bootstrap sudoers grant replaced with a real, + deliberately-scoped policy +- Workbench's SSH public key deployed (single key, by design — see + `CLAUDE.md` for the reasoning on why additional devices' keys wait) +- Base system confirmed: hostname, timezone, chrony/NTP, all four + expected repos enabled -### Phase 4: NVIDIA GPU Driver 🔨 +**Real bugs found and fixed during this run:** missing `ansible.posix` +collection (not bundled with the Alpine `ansible-core` package), a +variable-naming mismatch (`bigboy_admin_user` vs. the actual +`deploy_user`) — see `CLAUDE.md`/`group_vars/bigboy.yml` for the current +state of both. -**Status:** 🔨 **READY TO BUILD** +### Phase 3: Filesystem Conversion ✓ -**What's Done:** -- RAG documentation complete: `/home/john/documents/library/rag/use-case/nvidia-driver-almalinux.md` -- Includes precompiled method (recommended) + DKMS fallback -- RTX 5060 Ti specifics documented -- Environment variables, troubleshooting covered +**Status:** ✓ **COMPLETE — verified on real hardware, 2026-07-19** -**What's Needed:** -- Ansible role scaffold exists at `/home/john/projects/bigboy-setup/ansible/roles/phase-4-nvidia-driver/` -- Task file to fill: `tasks/main.yml` -- Use nvidia-driver-almalinux.md as reference -- ~20 lines YAML (enable repos, install packages, verify) +**What was done:** +- The four SATA data drives converted from the kickstart's forced xfs to + real btrfs (`compress=zstd,noatime`), mounted at final paths + (`/srv/backup`, `/srv/rag-library`, `/srv/ai-logs`, + `/srv/prompt-library`) +- Stale kickstart-generated fstab entries removed +- **Root filesystem intentionally NOT converted — stays xfs.** No + supported in-place xfs-to-btrfs path exists; a full reinstall (clone/ + reformat/restore via `debusb`) is deferred until after real-world + testing, not before ~a month out from 2026-07-17. This is a decision, + documented in `CLAUDE.md`, not a gap. -**Estimated Effort:** 30 minutes to code + test +**Real bugs found and fixed during this run:** a device-path bug +(`{{ item.byid }}1` instead of the correct `{{ item.byid }}-part1` +suffix), a mount-path mismatch specific to the ai-logs drive +(`/srv/ai-logs-raw` in the role vs. the kickstart's actual +`/srv/ai-raw`), and `state: unmounted` changed to `state: absent` so +stale fstab entries actually get removed rather than just the live +mount. -**Blocker:** None — ready to implement +### Phase 4: NVIDIA GPU Driver ✓ -**Dependencies:** -- Phase 3 (filesystems) — logged to `/srv/deployment-log/` +**Status:** ✓ **COMPLETE — verified on real hardware, 2026-07-19** ---- +**What was done:** +- AlmaLinux's precompiled open-kmod path used (not manual DKMS): + `almalinux-release-nvidia-driver` → `nvidia-driver-cuda + nvidia-open-kmod`, one reboot, `nvidia-smi` clean on the first attempt +- The smoothest phase of the entire deployment — no issues encountered -### Phase 5: Ollama Installation 🔨 +**Still needed:** the actual installed driver version has not yet been +recorded in `group_vars/bigboy.yml` — run `nvidia-smi` and update the +placeholder there. -**Status:** 🔨 **READY TO BUILD** +### Phase 5: llama.cpp ✓ -**What's Done:** -- RAG documentation complete: `/home/john/documents/library/rag/use-case/ollama-deployment.md` -- Includes installation (pip + systemd), GPU configuration, CLI commands, REST API -- RTX 5060 Ti tuning (VRAM limits, keep-alive, parallel requests) -- Model selection guide (mistral:7b recommended for 4GB) +**Status:** ✓ **COMPLETE — verified on real hardware, 2026-07-19** -**What's Needed:** -- Ansible role scaffold exists at `/home/john/projects/bigboy-setup/ansible/roles/phase-5-ollama/` -- Task file to fill: `tasks/main.yml` -- Use ollama-deployment.md as reference -- ~30 lines YAML (install, systemd service, pull test model, verify API) +**Not Ollama.** Built from source, pinned to a specific, vetted tag — +decided deliberately, see `CLAUDE.md` for the full rationale. -**Estimated Effort:** 45 minutes to code + test +**What was done:** +- llama.cpp built at pinned tag `b9968` (checked against real commit + dates and reviewed for fix/regression patterns before selection — + standard flags only, `-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120`) +- Model: Ministral 3 14B Instruct, Q4_K_M, pulled via llama.cpp's own + Hugging Face integration (`-hf` flag) +- `llama-server` running, systemd unit in place, nginx reverse proxy + fronting it, firewall opened for the proxy port on both the home LAN + and bench LAN ranges, real API key set +- **Chat confirmed working** — both through `llama-server`'s built-in + web UI (tested from Surface over the home LAN) and through `aichat`, a + terminal client, for LAN segments without GUI access -**Blocker:** None — ready to implement +**Real problems found and fixed during this run:** +- `cuda-toolkit` (specifically `nvcc`) is a separate package from the + driver runtime — not installed by Phase 4, had to be added separately; + the plain `cuda-toolkit` meta-package (not a version-pinned + sub-package) was used to stay in sync with whatever driver version + Phase 4 actually installed +- `nvcc` lands in a versioned path (`/usr/local/cuda-13.3/bin`), not on + `PATH` by default — a symlink plus a `/etc/profile.d/` entry resolved + this permanently +- An OOM kill during the actual CUDA compile (`cc1plus`/`cudafe++` + confirmed via `dmesg`) — root cause was unbounded parallel compilation + against memory-hungry CUDA source files with zero swap configured. + Fixed with a 16GB NVMe swapfile (persistent via fstab) plus dropping to + `-j2`. zram was separately decided as a complementary fast-swap tier + (2026-07-19) — **not yet implemented**, tracked as open work. +- Two rounds of missing baseline utilities discovered mid-deployment + (`tar`, `which`, then nine more via a proactive `utility-recon.sh` + scan) — all now added to the kickstart's `%packages` list for future + installs. -**Dependencies:** -- Phase 4 (GPU driver) - -**Note:** Phase 8 (Open WebUI) is optional enhancement to Phase 5 +**This is the success criterion set on 2026-07-17** ("base AI Server +Deployment... Mistral running, chat working, whatever quality, no +refinement required") — **met.** --- ### Phase 6: Build Suite -**Status:** 📋 **READY TO BUILD** +**Status:** 📋 **READY TO BUILD** — not yet started -**What's Done:** -- RAG documentation: `/home/john/documents/library/rag/use-case/dnf-package-management.md` -- Group list: `@Development Tools` -- Individual packages: kernel-headers, kernel-devel, btrfs-progs, smartmontools, nvtop +Development tools / system utilities beyond what's already in the +kickstart. Scope not yet finalized against current (llama.cpp-era) needs +— the original scope assumed an Ollama-era package list; revisit before +building. -**What's Needed:** -- Ansible role: `roles/phase-6-build-suite/tasks/main.yml` -- Simple dnf group install + packages -- ~15 lines YAML - -**Estimated Effort:** 15 minutes - -**Dependencies:** Phase 4 (kernel headers need matching kernel) +**Dependencies:** Phase 4 (kernel headers matching running kernel) — done --- ### Phase 7: Configuration -**Status:** 📋 **READY TO BUILD** +**Status:** 📋 **READY TO BUILD** — not yet started -**What's Done:** -- Scope defined: dotfiles, locale, timezone -- RAG docs available (Ansible modules reference) +Dotfiles, locale, timezone. Scope largely unchanged from original +planning; source for dotfiles still TBD. -**What's Needed:** -- Ansible role: `roles/phase-7-configuration/tasks/main.yml` -- Copy tmux.conf, vimrc, bash profile (source TBD) -- Set locale, timezone via ansible.builtin.lineinfile + timedatectl -- ~20 lines YAML - -**Estimated Effort:** 20 minutes - -**Blocker:** Source for dotfiles (use CE OS defaults or create new) - -**Dependencies:** None +**Dependencies:** None blocking --- -### Phase 8: Open WebUI (Optional) +### Phase 8: Open WebUI -**Status:** 📋 **READY TO BUILD** +**Status:** ⚠️ **LIKELY DROPPING** — deliberate decision needed, not a default -**What's Done:** -- RAG documentation complete: `/home/john/documents/library/rag/use-case/open-webui-deployment.md` -- Installation (pip + systemd), Ollama integration, features, troubleshooting - -**What's Needed:** -- Ansible role: `roles/phase-8-open-webui/tasks/main.yml` -- Python 3.11 install, pip install open-webui, systemd service, wait for API -- ~25 lines YAML - -**Estimated Effort:** 30 minutes - -**Status:** Optional (nice-to-have UI; Ollama CLI + API sufficient without it) - -**Dependencies:** Phase 5 (Ollama must be running) +`llama-server`'s own built-in web UI already covers the chat-first use +case (this is what Phase 5's success was actually verified against). +Building a separate Open WebUI layer on top is no longer the default +plan — keeping this phase requires an actual reason to want a second UI +layer, not inertia from the original Ollama-era plan. --- ### Phase 9: Security Hardening -**Status:** 📋 **DESIGN READY** +**Status:** 📋 **DESIGN READY** — not yet started -**What's Done:** -- Tasks defined: firewall (firewalld), SSH hardening, MAC pinning, SELinux review -- Open WebUI RAG doc includes security section +Firewall (firewalld — already partially ahead of schedule, since Phase 5 +opened the reverse-proxy port with the same LAN-restriction pattern used +elsewhere), SSH hardening (**disable password auth, move to key-only** — +this is the actual trigger condition already flagged in `CLAUDE.md` and +the original kickstart's own changelog), MAC pinning, SELinux review +(currently permissive per the kickstart; confirm AVC denial history +before flipping to enforcing). -**What's Needed:** -- Ansible role: `roles/phase-9-security/tasks/main.yml` -- firewall-cmd to open SSH 22, WebUI 8080, Cockpit 9090; Ollama 11434 localhost-only -- sshd config: disable password, PermitRootLogin=no -- MAC pinning via nmcli or network config -- ~40 lines YAML +**Blocker:** key-based login needs to be confirmed working over real, +repeated use before disabling password auth — not just "it worked once." -**Estimated Effort:** 1 hour (firewall rules need care) - -**Blocker:** SSH key setup must be complete before disabling password auth - -**Dependencies:** Phase 5 (services running) +**Dependencies:** Phase 5 (services running) — done --- @@ -181,11 +199,8 @@ **Status:** 🚫 **DEFERRED** -**Reason:** Backup target not yet decided (NAS? USB? Aruba?) - -**What's Ready:** Packages (borgbackup, borgmatic), retention policy framework - -**Prerequisite Decision:** Where to backup? NFS mount? Local USB? +**Reason:** Backup target not yet decided (NAS? USB? Aruba Object +Storage, per the sovereignty-logging design discussion?) **Timeline:** Decide backup target → implement Phase 10 @@ -193,82 +208,56 @@ ### Phase 11: Thermal Baseline Testing -**Status:** 📋 **READY TO BUILD** +**Status:** 📋 **READY TO BUILD** — not yet started -**What's Done:** -- Tasks defined: memtest86, Ollama inference test, log temps +Original scope referenced `llama2:70b`/Ollama-specific testing — needs +updating to a llama.cpp-appropriate heavy-load test instead. Given the +case move to Modcase HYPER (see below), worth doing this *after* the +physical case transition, since cooling characteristics will differ from +open-frame testing. -**What's Needed:** -- Ansible role: `roles/phase-11-thermal-testing/tasks/main.yml` -- Install memtest86, run with timeout -- Start Ollama with large model (llama2:70b if VRAM allows, or heavy context) -- Monitor with nvidia-smi in background -- Log temps to `/srv/deployment-log/phase-11-thermal-*.log` -- ~30 lines YAML - -**Estimated Effort:** 45 minutes - -**Dependencies:** Phase 5 (Ollama running) - -**Test Duration:** ~30 minutes (memtest + inference) +**Dependencies:** Phase 5 (llama-server running) — done --- ### Phase 12: Full System Validation -**Status:** 📋 **READY TO BUILD** +**Status:** 📋 **READY TO BUILD** — not yet started -**What's Done:** -- Validation checklist defined +Checklist needs updating away from Ollama/Open-WebUI-specific service +checks toward `llama-server`/nginx. -**What's Needed:** -- Ansible role: `roles/phase-12-validation/tasks/main.yml` -- Check services: systemctl status ollama open-webui firewalld sshd -- Verify GPU: nvidia-smi check -- Verify storage: btrfs filesystem usage / -- Verify logs: grep errors /srv/deployment-log/*.log -- Generate report -- ~25 lines YAML - -**Estimated Effort:** 30 minutes - -**Dependencies:** All prior phases +**Dependencies:** All prior phases — 1 through 5 done, 6-9/11 pending --- ### Phase 13: Home LAN Migration -**Status:** 🚫 **DEFERRED** +**Status:** 🔨 **PARTIALLY UNDERWAY** — ahead of original schedule -**Reason:** Waiting for Modcase EVO ITX-2 case installation +**Original plan:** deferred until case installation. -**What's Needed:** -- Ansible role: `roles/phase-13-home-migration/tasks/main.yml` -- Static IP assignment (if not via DHCP reservation) -- DNS configuration (if needed) -- Network documentation -- Bench → home transition runbook +**Actual state (2026-07-19):** BigBoy is already reachable on the home +LAN (`192.168.1.24`, statically reserved on the FritzBox) — this +happened ahead of the physical case move, driven by needing to test the +chat interface from a device with a GUI, which the bench LAN doesn't +have. The **physical relocation** (case: now **Modcase HYPER**, not the +originally-planned Modcase EVO ITX-2 — printed and ready as of +2026-07-19) is still pending; BigBoy is being assembled into its new +case and permanent home over the next few days. -**Timeline:** After case installed +**Modcase HYPER licensing note:** personal-use licence purchased +2026-07-19. Do not print for resale or client delivery until the +commercial tier is purchased. --- ### Phase 14: Observation Period Runbook -**Status:** 📋 **TEMPLATE READY** +**Status:** 📋 **TEMPLATE READY** — not yet started -**What's Done:** -- Routine defined: daily temp checks, disk usage, service status, log review -- Duration: 4 weeks post-deployment - -**What's Needed:** -- Runbook document: daily/weekly checklist -- Ansible role (optional): periodic health check job -- Or manual execution per runbook - -**Estimated Effort:** 20 minutes (runbook), 0 (if manual) - -**Dependencies:** All prior phases complete + operational +**Dependencies:** All prior phases complete + operational — not yet at +that point (6, 7, 9, 11, 12 still pending) --- @@ -278,101 +267,103 @@ | Item | Status | |------|--------| -| RAG library (DNF, NVIDIA, Ollama, Open WebUI, btrfs, Ansible) | ✓ Complete | -| Deployment code (Kickstart, Ansible, inventory) | ✓ Complete (Phase 3) | -| Project docs (README, ARCHITECTURE, STATUS, HARDWARE) | 🔨 Current | -| Phase 4-5 Ansible role code | 📋 Buildable | +| Project docs (README, CLAUDE.md, STATUS.md) | 🔨 Being actively reconciled, 2026-07-19 | +| Deployment code (Kickstart, Ansible, inventory) | ✓ Current through Phase 5 | +| RAG library references | ⚠️ Predate the llama.cpp decision (Ollama/Open-WebUI docs) — verify or replace before relying on them | +| Systemd/nginx configs for llama-server | ❌ Only exist live on BigBoy — not yet tracked as reusable template files in this repo. Real gap: deployment isn't currently reproducible from the repo alone for this piece. | ### Testing | Item | Status | |------|--------| -| Logical review (code review) | ✓ Phase 3 done | -| Real hardware test | 🚫 Pending BigBoy installation | -| Idempotency verification | ✓ Expected for all phases | -| Integration test (all 14 phases) | ⏳ After Phase 4-5 built | +| Real hardware test, Phases 1-5 | ✓ **Done**, 2026-07-17 through 2026-07-19 | +| Chat interface confirmed | ✓ Web UI + terminal client, 2026-07-19 | +| Idempotency verification | ✓ Confirmed for Phase 3 (re-run-safe checks in place) | +| Integration test (all phases) | ⏳ After Phases 6-9/11-12 built | ### Git & Version Control | Item | Status | |------|--------| -| Local /projects/bigboy-setup/ | ✓ Active | -| Local /projects/bigboy-alma/ (docs) | ✓ Active | -| Forgejo (giovannino/bigboy-alma-deploy) | ⏳ Push after hardware test | +| Forgejo (giovannino/bigboy-alma-deploy) | ✓ **Active, pushed, current through Phase 5** | --- -## Critical Path to Go-Live +## Critical Path to Go-Live (Full 14-Phase Scope) ``` -Phase 3: Filesystem ✓ +Phase 1: Kickstart ✓ ↓ -Phase 4: GPU Driver (30 min build) 🔨 +Phase 2: System Config ✓ ↓ -Phase 5: Ollama (45 min build) 🔨 +Phase 3: Filesystems ✓ (root stays xfs, deliberately) ↓ -Real Hardware Test (hours) +Phase 4: GPU Driver ✓ ↓ -Phase 6-8: Build, Config, WebUI (1-2 hours) 📋 +Phase 5: llama.cpp ✓ — chat confirmed working — BASE DEPLOYMENT GOAL MET ↓ -Phase 9: Security (1 hour) 📋 +Phase 6-7: Build Suite, Configuration 📋 ↓ -Phase 11: Thermal Test (30 min) 📋 +Phase 8: Open WebUI — likely dropping ⚠️ ↓ -Phase 12: Validation (30 min) 📋 +Phase 9: Security Hardening 📋 + ↓ +Phase 11: Thermal Test (post-case-move) 📋 + ↓ +Phase 12: Validation 📋 ↓ Phase 14: Observation (4 weeks manual) 📋 ↓ -✓ GO-LIVE +✓ FULL GO-LIVE -Parallel: Phase 10 (backup) — deferred until target decided -Parallel: Phase 13 (home migration) — deferred until case installed +Parallel: Phase 10 (backup) — deferred, target undecided +Parallel: Phase 13 (home LAN) — network migration done, physical case move in progress ``` -**Estimate:** -- Implementation (Phases 4-8): 2-3 hours -- First hardware test: 4+ hours (includes troubleshooting) -- Phases 9-12: 2-3 hours -- Total time to operational: 8-12 hours (compressed schedule) -- Observation period: 4 weeks - --- ## Blockers & Decisions Needed -### Immediate (Next Week) -- **Decision:** Build Phase 4-5 roles? → Yes/No/Wait for hardware? -- **Action:** Confirm BigBoy hardware available for testing +### Immediate +- **Real gap:** systemd unit, nginx config, and firewall rules for + `llama-server` exist only on BigBoy — need tracked template files in + the repo for actual reproducibility +- **Action:** record actual NVIDIA driver version in `group_vars/bigboy.yml` +- **Action:** implement zram alongside the existing swapfile (decision + already made 2026-07-19, not yet acted on) -### Medium Term (Before Go-Live) -- **Blocker:** Phase 10 backup target (NAS? USB? Remote?) -- **Decision:** Open WebUI required or optional? (Phase 8) -- **Decision:** Home LAN migration scope (Phase 13) +### Medium Term +- **Decision:** Phase 8 (Open WebUI) — build or drop? (leaning drop) +- **Blocker:** Phase 10 backup target +- **Decision:** Phase 9's SELinux review — check AVC denial history + before flipping enforcing -### Long Term (Post-Deployment) -- **Decision:** Borgmatic schedule, retention policy -- **Decision:** Phase 14 observation period (manual or automated health checks?) +### Long Term +- Root filesystem reformat to btrfs — deliberately deferred ~a month + from 2026-07-17, after real-world testing +- The auditable llama.cpp release pipeline and per-client agent + playbook pattern — both designed, neither built (see `CLAUDE.md`) --- ## Success Criteria -✓ **Phase Complete** when: -- Code written & tested logically -- Real hardware executes without errors -- Output logged to `/srv/deployment-log/` -- Logs reviewed & no blocking issues found -- Phase can be re-run idempotently +✓ **Base AI Server Deployment** (2026-07-17 goal) — **MET, 2026-07-19**: +llama.cpp built, Ministral 3 14B Instruct running, chat confirmed working +through `llama-server`'s web UI, whatever quality it delivers with zero +tuning. -✓ **Deployment Success** when: -- All 14 phases run end-to-end without manual intervention -- Services stable (Ollama, Open WebUI responding) -- GPU verified healthy (nvidia-smi + thermal test) -- Storage verified healthy (btrfs filesystem usage, no corruption) +**Full 14-phase deployment success** when: +- All phases run end-to-end, idempotently +- `llama-server`/nginx stable +- GPU verified healthy under real thermal load (post-case-move) +- Storage verified healthy (btrfs, no corruption) - 4 weeks observation period completed with no critical issues --- -**Last Updated:** 2026-06-27 -**Next Update:** After Phase 4-5 implementation or real hardware test -**Prepared By:** Claude Code (@workbench) +**Last Updated:** 2026-07-19 +**Next Update:** After systemd/nginx configs are tracked, or Phase 6-9 +implementation begins +**Prepared By:** Claude (Desktop), reconciled against Claude Code's +2026-07-19 repo audit \ No newline at end of file