Post deploy updates
This commit is contained in:
parent
c1a347e579
commit
3156419974
GPG key ID:
306E507219506D4E
4 changed files with 455 additions and 378 deletions
19
CLAUDE.md
19
CLAUDE.md
|
|
@ -112,25 +112,6 @@ built-in web UI, chat confirmed working). No agent roles, no RAG corpus,
|
|||
no ingestion pipeline get built at this stage — that's all downstream of
|
||||
an actual use case being decided, which hasn't happened yet.
|
||||
|
||||
**Per-client agent playbook pattern** — not a fixed pipeline to build
|
||||
once and reuse. CE's own Ambrosiana deployment and coding-assistant setup
|
||||
serve as proven reference implementations (working examples of agent-role
|
||||
design, recon checks, ingestion orchestration), but each client
|
||||
deployment gets its own bespoke agent-role setup based on their actual
|
||||
use case — captured as a documented, repeatable Ansible playbook for
|
||||
reinstall recovery and potential future hive federation, not copied
|
||||
wholesale from CE's own internal setup. The Ansible-docs-as-RAG-corpus
|
||||
discussion from 2026-07-17 (Phi-4 Mini for license/scraping-permission/
|
||||
malicious-content recon, Claude Code for chunking and ingestion
|
||||
orchestration, Qwen3 0.6B for style-guide formatting) is one example of
|
||||
the design thinking that goes into building a specific instance of this
|
||||
pattern — not a template to copy verbatim into a client deploy. **Current
|
||||
priority for BigBoy specifically is the base AI Server Deployment**:
|
||||
Phase 4 (NVIDIA driver) → Phase 5 (llama.cpp + Mistral + `llama-server`'s
|
||||
built-in web UI, chat confirmed working). No agent roles, no RAG corpus,
|
||||
no ingestion pipeline get built at this stage — that's all downstream of
|
||||
an actual use case being decided, which hasn't happened yet.
|
||||
|
||||
## Scope discipline
|
||||
|
||||
Task docs will name a specific phase or task from `STATUS.md`. Do the
|
||||
|
|
|
|||
105
README.md
105
README.md
|
|
@ -53,15 +53,21 @@ of indirection around it. See `CLAUDE.md` for the full standing rationale
|
|||
|
||||
- **GPU VRAM:** 16GB (RTX 5060 Ti) — model selection accordingly
|
||||
- **Storage:** 5 drives (1 NVMe + 4 SATA) with btrfs subvolume isolation
|
||||
- **Network:** bench LAN (192.168.0.0/24), workbench acts as gateway/DHCP
|
||||
for this segment
|
||||
on the SATA drives; NVMe root stays xfs (see Current Status)
|
||||
- **Network:** home LAN (`192.168.1.24`, static reservation) — originally
|
||||
deployed and tested on the workbench bench LAN (`192.168.0.0/24`),
|
||||
migrated once chat-interface testing required a GUI-capable device
|
||||
- **Idempotency:** every phase must be re-runnable
|
||||
|
||||
---
|
||||
|
||||
## Current Status — read this before assuming anything is done
|
||||
|
||||
**As of 2026-07-18: AlmaLinux 10.2 successfully installed and booted on BigBoy.** All five drives confirmed mounted and accessible. Foundation complete; core inference stack (Phase 4–5) ready to build.
|
||||
**As of 2026-07-19: Phases 1 through 5 complete and verified on real
|
||||
hardware.** BigBoy is a working AI inference server — NVIDIA driver
|
||||
loaded, llama.cpp built and pinned, Ministral 3 14B Instruct serving via
|
||||
`llama-server`, chat confirmed working through both the built-in web UI
|
||||
and a terminal client (`aichat`).
|
||||
|
||||
See `./planning/STATUS.md` for the authoritative, current per-phase state.
|
||||
Don't rely on this README for phase-by-phase status — it will drift out
|
||||
|
|
@ -69,44 +75,63 @@ of date faster than `STATUS.md` is maintained.
|
|||
|
||||
### Deployed & Verified on Hardware
|
||||
|
||||
- Kickstart (`alma10-minimal-bigboy.ks`) — AlmaLinux 10.2 minimal install
|
||||
successfully executed 2026-07-17
|
||||
- Phase 3 role (filesystem mounting) — run and verified, all 4 SATA + 1 NVMe
|
||||
drives mounted at intended paths via UUID
|
||||
|
||||
### Ready to build (Phase 1–3 complete)
|
||||
|
||||
- Phase 4 (NVIDIA driver) — precompiled open-kmod via AlmaLinux repos
|
||||
- Phase 5 (llama.cpp) — build from pinned tag, standard flags, `llama-server`
|
||||
under systemd, see `CLAUDE.md` and `tasks/TASK-phase4-5-bringup.md`
|
||||
- Kickstart (`alma10-minimal-bigboy.ks`) — AlmaLinux 10.2 minimal install,
|
||||
2026-07-17
|
||||
- Phase 2 (system configuration) — sudoers policy, SSH key, base checks
|
||||
- Phase 3 (filesystem conversion) — all four SATA drives converted to
|
||||
real btrfs; root intentionally stays xfs (no supported in-place
|
||||
conversion path — a full reinstall is deferred ~a month out, after
|
||||
real-world testing, not a gap)
|
||||
- Phase 4 (NVIDIA driver) — AlmaLinux's precompiled open-kmod path,
|
||||
confirmed via `nvidia-smi`
|
||||
- Phase 5 (llama.cpp) — built at pinned tag `b9968`, serving Ministral 3
|
||||
14B Instruct at Q4_K_M, systemd + nginx + firewall in place, chat
|
||||
confirmed working
|
||||
|
||||
### Likely dropping
|
||||
|
||||
- Phase 8 (Open WebUI) — `llama-server`'s built-in UI covers the
|
||||
chat-first use case; keeping this phase is now a deliberate decision
|
||||
to make, not a default
|
||||
- Phase 8 (Open WebUI) — `llama-server`'s built-in UI already covers the
|
||||
chat-first use case; this is what Phase 5's success was actually
|
||||
verified against. Keeping this phase now needs an actual reason, not
|
||||
just inertia from the original plan.
|
||||
|
||||
### Deferred, on purpose
|
||||
|
||||
- Phases 6–7, 9–14 (Build Suite, Configuration, Security, Borgmatic,
|
||||
Thermal, Validation, Migration, Observation)
|
||||
- Phases 6–7, 9, 11–12, 14 — not yet started; see `STATUS.md` for
|
||||
per-phase detail
|
||||
- Root filesystem's eventual xfs→btrfs conversion — full reinstall,
|
||||
deliberately deferred pending real-world testing
|
||||
- An auditable testing/release system for llama.cpp (versioned releases,
|
||||
btrfs snapshot safety net, promote/rollback, a Forgejo mirror that only
|
||||
ever receives BigBoy-validated tags for client deployments) — fully
|
||||
designed 2026-07-16, not yet built. See `CLAUDE.md` for the deferred-work
|
||||
note and `scripts/draft/` if present.
|
||||
designed 2026-07-16, not yet built. See `CLAUDE.md` for the
|
||||
deferred-work note and `scripts/draft/` if present.
|
||||
- The per-client agent playbook pattern — same status, see `CLAUDE.md`
|
||||
- Borgmatic backup target (not yet decided)
|
||||
- Home LAN migration (waiting on case installation)
|
||||
- Phase 13's physical case move — network migration to the home LAN is
|
||||
done; the physical relocation into the new Modcase HYPER case is
|
||||
actively in progress as of 2026-07-19
|
||||
|
||||
### A known, real gap — not yet closed
|
||||
|
||||
The systemd unit, nginx reverse-proxy config, and firewall rules that
|
||||
make Phase 5's chat deployment actually work currently exist **only on
|
||||
BigBoy itself**, configured directly during the live session — not yet
|
||||
committed to this repo as tracked template files. Until that's fixed,
|
||||
the deployment isn't fully reproducible from the repo alone. See
|
||||
`STATUS.md`'s "Immediate" blockers section.
|
||||
|
||||
---
|
||||
|
||||
## Collaborators & Workflow
|
||||
|
||||
### John A. Hoeven (@workbench)
|
||||
|
||||
- Architect & operator — makes deployment decisions, edits and pushes
|
||||
from workbench via VS Codium
|
||||
|
||||
### Claude Code (@bigboy once installed, or driven from @workbench)
|
||||
### Claude Code (@bigboy, or driven from @workbench)
|
||||
|
||||
- Executes scoped tasks against the actual repo — see `CLAUDE.md` for
|
||||
standing context read automatically each session, and `tasks/` for the
|
||||
current scoped handoff
|
||||
|
|
@ -114,6 +139,7 @@ of date faster than `STATUS.md` is maintained.
|
|||
bounds for a given session
|
||||
|
||||
### Claude Desktop
|
||||
|
||||
- Plans and orchestrates: architecture discussion, design decisions,
|
||||
drafts `CLAUDE.md` and task docs for Claude Code to execute against
|
||||
|
||||
|
|
@ -121,7 +147,7 @@ of date faster than `STATUS.md` is maintained.
|
|||
|
||||
## Repo Structure
|
||||
|
||||
```
|
||||
```text
|
||||
bigboy-alma-deploy/
|
||||
├── CLAUDE.md # standing context, read by Claude Code every session
|
||||
├── README.md # this file
|
||||
|
|
@ -132,8 +158,9 @@ bigboy-alma-deploy/
|
|||
│ ├── inventory.ini
|
||||
│ ├── group_vars/bigboy.yml # hardware facts: UUIDs, driver version, pinned llama.cpp tag
|
||||
│ └── roles/
|
||||
│ └── phase-3-filesystems/ # written, reviewed, not yet run on hardware
|
||||
├── planning/ # renamed from plannng (typo) on 2026-07-17
|
||||
│ ├── phase-2-system-config/ # done, verified on hardware
|
||||
│ └── phase-3-filesystems/ # done, verified on hardware
|
||||
├── planning/
|
||||
│ ├── ARCHITECTURE.md
|
||||
│ ├── HARDWARE.md
|
||||
│ ├── SOVEREIGNTY-POLICY.md
|
||||
|
|
@ -143,7 +170,10 @@ bigboy-alma-deploy/
|
|||
│ └── alma10-minimal-bigboy.ks # reference snapshot from planning — root copy is canonical
|
||||
├── tasks/
|
||||
│ └── TASK-*.md # scoped Claude Code handoffs, one per unit of work
|
||||
└── scripts/draft/ # (if present) draft auditable-release-pipeline scripts —
|
||||
└── scripts/
|
||||
├── package-recon.sh # confirms package availability before finalizing task docs
|
||||
├── utility-recon.sh # checks for missing baseline CLI utilities
|
||||
└── draft/ # draft auditable-release-pipeline scripts —
|
||||
# unintegrated, see CLAUDE.md before touching
|
||||
```
|
||||
|
||||
|
|
@ -152,11 +182,13 @@ bigboy-alma-deploy/
|
|||
## Getting Started
|
||||
|
||||
### To review design
|
||||
|
||||
1. Read `./planning/ARCHITECTURE.md` — design rationale, 14-phase breakdown
|
||||
2. Read `./planning/HARDWARE.md` — component specs, UUIDs, network config
|
||||
3. Read `./planning/STATUS.md` — the real current state, phase by phase
|
||||
|
||||
### To pick up work
|
||||
|
||||
1. Check `./planning/STATUS.md` for what's actually next
|
||||
2. Look in `./tasks/` for an existing scoped task doc covering it
|
||||
3. If none exists, Claude Desktop drafts one before Claude Code starts —
|
||||
|
|
@ -164,12 +196,15 @@ bigboy-alma-deploy/
|
|||
|
||||
### Next Steps (current priority)
|
||||
|
||||
1. Build and deploy Phase 4 (NVIDIA driver) via Ansible or shell commands,
|
||||
verify with `nvidia-smi`
|
||||
2. Build and deploy Phase 5 (llama.cpp) — see `./tasks/TASK-phase4-5-bringup.md`
|
||||
for the current scoped handoff
|
||||
3. Verify `llama-server` running and responding to inference requests
|
||||
4. Phases 6–14 follow once 4–5 are stable and tested
|
||||
1. Commit tracked template files for the systemd unit, nginx config, and
|
||||
firewall rules currently only configured live on BigBoy
|
||||
2. Record the actual NVIDIA driver version in `group_vars/bigboy.yml`
|
||||
(placeholder currently reads `CHECK-WITH-nvidia-smi`)
|
||||
3. Implement zram alongside the existing swapfile (decision made
|
||||
2026-07-19, not yet acted on)
|
||||
4. Decide Phase 8 (Open WebUI) — build or drop
|
||||
5. Complete the physical case move to Modcase HYPER, then proceed with
|
||||
Phases 6–7, 9, 11–12, 14
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -187,5 +222,7 @@ bigboy-alma-deploy/
|
|||
|
||||
---
|
||||
|
||||
**Last updated:** 2026-07-18
|
||||
**Current priority:** Phase 4 (NVIDIA driver) and Phase 5 (llama.cpp build & serve)
|
||||
**Last updated:** 2026-07-19
|
||||
**Current priority:** Track the systemd/nginx/firewall configs that
|
||||
currently only exist on BigBoy itself; complete the case move; decide
|
||||
Phase 8
|
||||
|
|
@ -3,6 +3,11 @@
|
|||
Idempotent Ansible playbook for configuring BigBoy from a fresh AlmaLinux
|
||||
10.2 kickstart install through to a running llama.cpp inference server.
|
||||
|
||||
**Status (2026-07-19): Phases 2 through 5 have all run successfully on
|
||||
real hardware.** This isn't a "should work" playbook anymore — it's the
|
||||
one that actually built the working deployment. See `../planning/STATUS.md`
|
||||
for the authoritative current state.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
|
@ -18,19 +23,19 @@ Idempotent Ansible playbook for configuring BigBoy from a fresh AlmaLinux
|
|||
|
||||
```bash
|
||||
# From ansible/
|
||||
ansible-playbook site.yml -i inventory.ini --tags phase2,phase3
|
||||
ansible-playbook site.yml -i inventory.ini --tags phase-2,phase-3
|
||||
```
|
||||
|
||||
### Re-running a specific phase
|
||||
|
||||
```bash
|
||||
ansible-playbook site.yml -i inventory.ini --tags phase3
|
||||
ansible-playbook site.yml -i inventory.ini --tags phase-3
|
||||
```
|
||||
|
||||
### Check mode (show what would change, without changing anything)
|
||||
|
||||
```bash
|
||||
ansible-playbook site.yml -i inventory.ini --check --tags phase2,phase3
|
||||
ansible-playbook site.yml -i inventory.ini --check --tags phase-2,phase-3
|
||||
```
|
||||
|
||||
---
|
||||
|
|
@ -46,12 +51,15 @@ ansible/
|
|||
├── group_vars/
|
||||
│ └── bigboy.yml # Hardware facts: UUIDs, by-id paths, pinned versions
|
||||
├── roles/
|
||||
│ ├── phase2-system-config/
|
||||
│ │ └── tasks/main.yml # Sudoers policy, SSH key, base checks
|
||||
│ ├── phase-2-system-config/
|
||||
│ │ └── tasks/main.yml # DONE — sudoers policy, SSH key, base checks
|
||||
│ ├── phase-3-filesystems/
|
||||
│ │ └── tasks/main.yml # Convert 4 SATA drives xfs -> btrfs
|
||||
│ ├── phase-4-nvidia-driver/ # scaffold — see tasks/TASK-phase4-5-bringup.md
|
||||
│ ├── phase-5-llama-cpp/ # scaffold — see tasks/TASK-phase4-5-bringup.md
|
||||
│ │ └── tasks/main.yml # DONE — converts 4 SATA drives xfs -> btrfs
|
||||
│ ├── phase-4-nvidia-driver/ # not yet a tracked role — Phase 4 was run via
|
||||
│ │ direct shell commands per tasks/TASK-phase4-5-bringup.md,
|
||||
│ │ not Ansible. Worth deciding whether to formalize
|
||||
│ │ into a role for reproducibility.
|
||||
│ ├── phase-5-llama-cpp/ # same — run manually, not yet an Ansible role
|
||||
│ └── ... (phases 6-14, not yet scoped)
|
||||
└── README.md # This file
|
||||
```
|
||||
|
|
@ -61,6 +69,16 @@ here — `TASK-phase2-3.md` and `TASK-phase4-5-bringup.md` are the
|
|||
authoritative descriptions of what each phase actually does; this README
|
||||
is a summary and operational reference, not the source of truth for scope.
|
||||
|
||||
**Worth knowing:** Phase 4 (NVIDIA driver) and Phase 5 (llama.cpp build,
|
||||
model pull, systemd/nginx/firewall setup) were both executed as direct
|
||||
shell commands against BigBoy during the actual deployment session, not
|
||||
through Ansible roles. They worked, and they're real — but they're not
|
||||
currently reproducible by re-running this playbook. If that matters for
|
||||
future deployments (e.g., disaster recovery, or building the next
|
||||
client's box from this same reference), those steps need to become
|
||||
actual Ansible roles at some point — currently tracked as an open item,
|
||||
not done.
|
||||
|
||||
---
|
||||
|
||||
## Phase Breakdown
|
||||
|
|
@ -70,20 +88,27 @@ is a summary and operational reference, not the source of truth for scope.
|
|||
Unattended AlmaLinux 10.2 install via kickstart. See
|
||||
`../alma10-minimal-bigboy.ks`.
|
||||
|
||||
### Phase 2: System Configuration — see `tasks/TASK-phase2-3.md`
|
||||
### Phase 2: System Configuration — DONE, verified on hardware
|
||||
|
||||
- Replace the kickstart's temporary bootstrap NOPASSWD sudoers grant with
|
||||
a real, deliberately-scoped policy
|
||||
- Deploy workbench's SSH public key (single key only at this stage —
|
||||
BigBoy sits on the isolated bench LAN; more keys get added later, one
|
||||
at a time, once on the home LAN)
|
||||
- Confirm hostname, timezone, NTP/chrony, and repo configuration
|
||||
- Replaced the kickstart's temporary bootstrap NOPASSWD sudoers grant
|
||||
with a real, deliberately-scoped policy
|
||||
- Deployed workbench's SSH public key (single key only at this stage —
|
||||
BigBoy sits on the home LAN now; more keys get added later, one at a
|
||||
time, once tested individually)
|
||||
- Confirmed hostname, timezone, NTP/chrony, and repo configuration
|
||||
|
||||
Real bugs hit and fixed during the actual run: a missing
|
||||
`ansible.posix` collection (not bundled with the Alpine `ansible-core`
|
||||
package — `ansible-galaxy collection install ansible.posix` was
|
||||
required), and a variable-naming mismatch (`bigboy_admin_user` expected
|
||||
by the role vs. `deploy_user` actually defined in `group_vars`) — see
|
||||
`CLAUDE.md` for the current state of both.
|
||||
|
||||
**Does not** disable password SSH authentication — that's Phase 9, once
|
||||
key-based login is confirmed working over real use, not assumed the
|
||||
moment a key is deployed.
|
||||
|
||||
### Phase 3: Filesystem Conversion — see `tasks/TASK-phase2-3.md`
|
||||
### Phase 3: Filesystem Conversion — DONE, verified on hardware
|
||||
|
||||
The kickstart formats all five drives as **xfs** (RHEL10's Anaconda
|
||||
removed btrfs support entirely — it's not a choice, it's what's actually
|
||||
|
|
@ -100,67 +125,79 @@ real btrfs with subvolumes and zstd compression:
|
|||
**Root stays xfs, permanently for now.** There's no supported in-place
|
||||
xfs-to-btrfs conversion — this would require a full reinstall, planned
|
||||
separately and deliberately deferred until after real-world testing (not
|
||||
before ~a month out). Do not attempt to touch root as part of this phase.
|
||||
before ~a month out from 2026-07-17). Do not attempt to touch root as
|
||||
part of this phase.
|
||||
|
||||
### Phase 4: NVIDIA Driver — see `tasks/TASK-phase4-5-bringup.md`
|
||||
Real bugs hit and fixed during the actual run: a device-path targeting
|
||||
bug (`{{ item.byid }}1` instead of the correct `{{ item.byid }}-part1`
|
||||
suffix — caught when `mkfs.btrfs` couldn't open the device, no data was
|
||||
touched), a mount-path mismatch specific to the ai-logs drive
|
||||
(the role assumed `/srv/ai-logs-raw`, the kickstart actually created
|
||||
`/srv/ai-raw` — caught by manually running `lsblk -f` after an
|
||||
apparently-successful run and noticing one drive still mounted when the
|
||||
other three weren't), and `state: unmounted` changed to `state: absent`
|
||||
so the kickstart's original fstab entries actually get removed, not just
|
||||
the live mounts.
|
||||
|
||||
### Phase 4: NVIDIA Driver — DONE, run manually (not yet an Ansible role)
|
||||
|
||||
AlmaLinux's precompiled open-kmod path (`almalinux-release-nvidia-driver`
|
||||
→ `nvidia-driver-cuda nvidia-open-kmod`), not manual DKMS. Confirmed as
|
||||
the AlmaLinux-recommended method, Secure Boot compatible, needs none of
|
||||
the kernel-devel/DKMS/compiler machinery DKMS would.
|
||||
the kernel-devel/DKMS/compiler machinery DKMS would. Ran clean, no issues
|
||||
— the smoothest phase of the entire deployment.
|
||||
|
||||
### Phase 5: llama.cpp — see `tasks/TASK-phase4-5-bringup.md`
|
||||
### Phase 5: llama.cpp — DONE, run manually (not yet an Ansible role)
|
||||
|
||||
**Not Ollama.** Built from source, pinned to a specific tag (`b9968`,
|
||||
recorded in `group_vars/bigboy.yml`), served via `llama-server` under
|
||||
systemd behind nginx. Initial model: Ministral 3 14B Instruct, Q4_K_M —
|
||||
success criterion is a coherent chat response through the built-in web
|
||||
UI, quality as-delivered, no tuning required at this stage.
|
||||
systemd behind nginx. Model: Ministral 3 14B Instruct, Q4_K_M — success
|
||||
criterion was a coherent chat response through the built-in web UI,
|
||||
quality as-delivered, no tuning required — **met**, confirmed via both
|
||||
the web UI and a terminal client (`aichat`).
|
||||
|
||||
### Phases 6–14: not yet scoped
|
||||
|
||||
Deferred until 4/5 are stable and tested. No committed design for these
|
||||
yet — don't assume the shape of a future phase from an old draft.
|
||||
Real problems hit and fixed during the actual run:
|
||||
- `cuda-toolkit` (specifically `nvcc`) is a separate package from the
|
||||
driver runtime installed by Phase 4 — had to be installed separately
|
||||
- `nvcc` lands in a versioned path, not on `PATH` by default — resolved
|
||||
with a symlink and a `/etc/profile.d/` entry
|
||||
- An OOM kill during the actual CUDA compile, root-caused via `dmesg` to
|
||||
unbounded parallel compilation with zero swap configured — fixed with
|
||||
a 16GB NVMe swapfile plus `-j2`
|
||||
- Two rounds of missing baseline utilities (`tar`, `which`, then nine
|
||||
more found via a proactive recon scan) — added to the kickstart for
|
||||
future installs
|
||||
|
||||
---
|
||||
|
||||
## Variables (`group_vars/bigboy.yml`)
|
||||
|
||||
```yaml
|
||||
# NVMe (OS + root, stays xfs)
|
||||
drive_nvme_byid: "nvme-Samsung_SSD_980_500GB_S64DNL0TC07019Z"
|
||||
See the file itself for the current, complete set — it's been through a
|
||||
real reconciliation pass (2026-07-19) against everything actually
|
||||
deployed. Key points:
|
||||
|
||||
# SATA data drives — by-id, NOT sdX (unstable across boots/reorders,
|
||||
# already proven true once during this deployment)
|
||||
drive_backup_byid: "ata-Lexar_SSD_NS100_256GB_QM9370R0291730S340"
|
||||
drive_rag_byid: "ata-Samsung_SSD_870_EVO_500GB_S6PYNL0T620757H"
|
||||
drive_ai_logs_byid: "ata-Samsung_SSD_870_EVO_250GB_S61WNJ0R458541M"
|
||||
drive_prompt_byid: "ata-Lexar_SSD_NS100_256GB_QM9370R0290370S340"
|
||||
|
||||
# Admin account (matches kickstart's user --name=)
|
||||
bigboy_admin_user: "<set to actual username>"
|
||||
|
||||
# GPU / driver — do NOT treat this as a version to install for; the
|
||||
# precompiled path installs whatever AlmaLinux's repo currently ships.
|
||||
# Record the ACTUAL version here after Phase 4 runs, replacing this
|
||||
# placeholder.
|
||||
nvidia_driver_version: "<record actual version after Phase 4>"
|
||||
|
||||
# llama.cpp — pinned, not tracking master
|
||||
llama_cpp_tag: "b9968"
|
||||
|
||||
# Model — confirmed choice for initial bring-up, not a placeholder
|
||||
model_repo: "mistralai/Ministral-3-14B-Instruct-2512-GGUF"
|
||||
model_quant: "Q4_K_M"
|
||||
```
|
||||
- **IP address**: `192.168.1.24` — BigBoy's current, permanent home-LAN
|
||||
address. It ran on the workbench bench LAN for the entire initial
|
||||
deploy first (`.240`, then `.241` after a DHCP reassignment) — both
|
||||
genuinely correct at the time, not mistakes, and noted as history in
|
||||
the file rather than erased.
|
||||
- **Drive UUIDs**: not consumed directly by any role — Phase 3 discovers
|
||||
them at runtime via `blkid` against the stable by-id paths, since they
|
||||
don't exist until `mkfs.btrfs` actually runs. The file records the
|
||||
current live UUIDs for reference only.
|
||||
- **`nvidia_driver_version`**: still a placeholder (`CHECK-WITH-nvidia-smi`)
|
||||
— the actual value was never captured during the deployment session.
|
||||
Run `nvidia-smi` and fill this in.
|
||||
- **API key**: deliberately NOT stored in this file — it's committed to
|
||||
a public repo. Needs vault or a gitignored file, same pattern as
|
||||
`keys/workbench.pub`.
|
||||
|
||||
**Note on the four SATA drives' filesystem UUIDs**: unlike the by-id
|
||||
paths above (stable, known ahead of time), the *filesystem* UUIDs for
|
||||
these drives don't exist yet — they're generated fresh when Phase 3 runs
|
||||
`mkfs.btrfs`. Don't hardcode filesystem UUIDs into this file preemptively.
|
||||
Phase 3's role discovers each UUID at runtime (via `blkid` against the
|
||||
by-id path, immediately after formatting) and uses that discovered value
|
||||
to write `/etc/fstab` — see the role itself for the exact mechanism.
|
||||
paths (stable, known ahead of time), the *filesystem* UUIDs don't exist
|
||||
until Phase 3 runs `mkfs.btrfs`. Don't hardcode filesystem UUIDs into
|
||||
this file preemptively for a fresh install — Phase 3's role discovers
|
||||
each UUID at runtime and uses that discovered value to write
|
||||
`/etc/fstab`.
|
||||
|
||||
**To update:** edit this file, re-run the playbook. Variables propagate
|
||||
to every role.
|
||||
|
|
@ -171,7 +208,8 @@ to every role.
|
|||
|
||||
Every task should be safe to re-run:
|
||||
- Already-converted drives aren't re-wiped (Phase 3 checks existing
|
||||
filesystem type before acting)
|
||||
filesystem type before acting) — confirmed working correctly on the
|
||||
actual deployment
|
||||
- Already-deployed keys aren't duplicated
|
||||
- Already-correct sudoers policy isn't rewritten unnecessarily
|
||||
|
||||
|
|
@ -196,7 +234,7 @@ the kickstart's `%post`), one file per phase run:
|
|||
```
|
||||
|
||||
```bash
|
||||
ssh <admin-user>@192.168.0.240 "tail -f /srv/deployment-log/*.log"
|
||||
ssh john@192.168.1.24 "tail -f /srv/deployment-log/*.log"
|
||||
```
|
||||
|
||||
---
|
||||
|
|
@ -211,9 +249,10 @@ ssh <admin-user>@192.168.0.240 "tail -f /srv/deployment-log/*.log"
|
|||
|
||||
### SSH connection fails
|
||||
|
||||
1. Confirm BigBoy's current IP — `192.168.0.240` on the bench LAN as of
|
||||
1. Confirm BigBoy's current IP — `192.168.1.24` on the home LAN as of
|
||||
this writing, but confirm via `ansible/inventory.ini` rather than
|
||||
assuming it hasn't changed
|
||||
assuming it hasn't changed. It's changed twice already during this
|
||||
deployment (bench LAN `.240` → `.241` → home LAN `.24`).
|
||||
2. Confirm you're connecting as the admin user, **not root** — root login
|
||||
is locked (`rootpw --lock` in the kickstart), always was, this isn't
|
||||
something Phase 9 disables later
|
||||
|
|
@ -225,6 +264,15 @@ ssh <admin-user>@192.168.0.240 "tail -f /srv/deployment-log/*.log"
|
|||
- Check `changed_when`/`failed_when` logic in the specific task
|
||||
- Run with `-vv` for full task output
|
||||
|
||||
### Missing `ansible.posix` collection
|
||||
|
||||
Hit during the real Phase 2 run — `apk add ansible-core` (Alpine's
|
||||
minimal package, deliberately chosen over the full `ansible` bundle)
|
||||
doesn't include community collections:
|
||||
```bash
|
||||
ansible-galaxy collection install ansible.posix
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Extending (Adding a New Phase)
|
||||
|
|
@ -238,6 +286,11 @@ ssh <admin-user>@192.168.0.240 "tail -f /srv/deployment-log/*.log"
|
|||
4. Add the role to `site.yml` with an appropriate tag
|
||||
5. Run: `ansible-playbook site.yml --tags phaseN`
|
||||
|
||||
**Worth doing at some point**: Phases 4 and 5 were run as direct shell
|
||||
commands, not Ansible roles. Formalizing them into real roles (following
|
||||
this same pattern) would make the deployment fully reproducible from the
|
||||
repo alone — currently it isn't, for those two phases specifically.
|
||||
|
||||
---
|
||||
|
||||
## SSH Key Setup (required before running this playbook)
|
||||
|
|
@ -250,10 +303,10 @@ manually, using password auth, before Ansible can connect via key:
|
|||
```bash
|
||||
# On workbench — the key already exists at ansible/keys/workbench.pub
|
||||
# (gitignored, never commit the actual key material)
|
||||
ssh-copy-id -i ~/.ssh/<your-key>.pub <admin-user>@192.168.0.240
|
||||
ssh-copy-id -i ~/.ssh/<your-key>.pub <admin-user>@192.168.1.24
|
||||
|
||||
# Verify
|
||||
ssh -i ~/.ssh/<your-key> <admin-user>@192.168.0.240 "echo connected"
|
||||
ssh -i ~/.ssh/<your-key> <admin-user>@192.168.1.24 "echo connected"
|
||||
```
|
||||
|
||||
Then update `inventory.ini` to use `ansible_user=<admin-user>` (not
|
||||
|
|
@ -268,7 +321,7 @@ once.
|
|||
## Validation Checklist (Phase 2/3)
|
||||
|
||||
```bash
|
||||
ssh <admin-user>@192.168.0.240
|
||||
ssh john@192.168.1.24
|
||||
|
||||
# Sudoers — bootstrap grant gone, real policy in place
|
||||
sudo -l
|
||||
|
|
@ -282,6 +335,15 @@ mount | grep btrfs
|
|||
mount | grep ' / ' # should show xfs, not btrfs
|
||||
```
|
||||
|
||||
## Validation Checklist (Phase 4/5)
|
||||
|
||||
```bash
|
||||
nvidia-smi
|
||||
systemctl status llama-server
|
||||
systemctl status nginx
|
||||
curl http://127.0.0.1:8080/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Known Deferred (not gaps — deliberate)
|
||||
|
|
@ -290,12 +352,15 @@ mount | grep ' / ' # should show xfs, not btrfs
|
|||
(clone/reformat/restore via `debusb`), planned for ~a month out after
|
||||
real-world testing, not part of this playbook
|
||||
- **Additional SSH keys** (Surface, tinkerpad) — one at a time, only
|
||||
once BigBoy is reachable on the home LAN
|
||||
once tested individually
|
||||
- **Dedicated display-monitor user** — for the case-mounted 5" screen
|
||||
relocated from jahnet; needs autologin + `btop`-only access, not yet
|
||||
built
|
||||
- **Auditable llama.cpp release pipeline** and **per-client agent
|
||||
playbooks** — both designed in detail, neither built; see `CLAUDE.md`
|
||||
- **Phases 4/5 as actual Ansible roles** — currently run manually, not
|
||||
automated; a real gap for reproducibility, not deliberately deferred
|
||||
the way the items above are
|
||||
- **Phases 6–14** — not yet scoped at all
|
||||
|
||||
---
|
||||
|
|
@ -303,14 +368,17 @@ mount | grep ' / ' # should show xfs, not btrfs
|
|||
## Reference Files
|
||||
|
||||
- `../alma10-minimal-bigboy.ks` — kickstart for the unattended OS install
|
||||
- `../CLAUDE.md` — standing technical decisions, read automatically by
|
||||
Claude Code every session
|
||||
- `../CLAUDE.md` — standing context, read automatically by Claude Code
|
||||
every session
|
||||
- `../tasks/TASK-phase2-3.md`, `../tasks/TASK-phase4-5-bringup.md` —
|
||||
authoritative scope for each phase
|
||||
- `../planning/STATUS.md` — authoritative current per-phase state
|
||||
- `../planning/build-journal-20260717.md` — detailed account of Phase 1's
|
||||
install day, including all ten issues found and fixed
|
||||
|
||||
---
|
||||
|
||||
**Last updated:** 2026-07-19
|
||||
**Status:** Phase 1 done (real hardware). Phase 2/3 roles being written
|
||||
against scoped task docs. Phase 4/5 scoped, not yet run.
|
||||
**Status:** Phase 1 through 5 complete and verified on real hardware.
|
||||
Phase 4/5 currently run manually, not yet formalized as Ansible roles.
|
||||
Phases 6–14 not yet scoped.
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
# BigBoy Deployment Status
|
||||
|
||||
**Real-time status of all 14 deployment phases**
|
||||
**Real-time status of all deployment phases**
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -8,172 +8,190 @@
|
|||
|
||||
| Metric | Status |
|
||||
|--------|--------|
|
||||
| **Overall Readiness** | 40% — Foundation complete, core phases buildable |
|
||||
| **Phases Complete** | 1 of 14 (Phase 3) |
|
||||
| **Phases Buildable** | 3 of 14 (Phases 4-5 + 8 optional) |
|
||||
| **Documentation** | Complete for all 14 phases in RAG library |
|
||||
| **Real Hardware Test** | Pending (awaiting BigBoy installation) |
|
||||
| **Target Go-Live** | Q3 2026 |
|
||||
| **Overall Readiness** | Foundation + core inference stack complete and verified on real hardware. Phases 6–14 remain, two deliberately deferred. |
|
||||
| **Phases Complete** | 1 through 5 (Kickstart, System Config, Filesystems, NVIDIA Driver, llama.cpp) |
|
||||
| **Real Hardware Test** | **Done** — chat confirmed working via web UI and a terminal client, 2026-07-19 |
|
||||
| **Target Go-Live** | Base AI Server Deployment goal (chat working) — met. Full 14-phase scope continues. |
|
||||
|
||||
---
|
||||
|
||||
## Phase Status Breakdown
|
||||
|
||||
### Phase 3: Filesystem Validation ✓
|
||||
### Phase 1: Base Install (Kickstart) ✓
|
||||
|
||||
**Status:** ✓ **COMPLETE**
|
||||
**Status:** ✓ **COMPLETE — verified on real hardware, 2026-07-17**
|
||||
|
||||
**What Was Done:**
|
||||
- Ansible role created: `/home/john/projects/bigboy-setup/ansible/roles/phase-3-filesystems/`
|
||||
- Task: Mount 4 SATA drives via UUID + subvolume name
|
||||
- Compression: zstd (level 3 general, level 1 for logs)
|
||||
- Mount options: noatime, nofail
|
||||
- Output: Logs to `/srv/deployment-log/phase-03-filesystems-*.log`
|
||||
**What was done:**
|
||||
- Unattended AlmaLinux 10.2 install via `alma10-minimal-bigboy.ks`
|
||||
- Ten distinct issues found and fixed during install day (see
|
||||
`build-journal-20260717.md` for the full account): `%anaconda` section
|
||||
incompatibility, `bootloader --location` fix, a line-wrap syntax error,
|
||||
`--noipv6` correction, btrfs removed from RHEL10's kickstart (switched
|
||||
all partitions to xfs), a cross-repo `vim`/`vim-data` mismatch,
|
||||
`firewall-cmd`'s inability to run inside the install-time chroot
|
||||
(replaced with a direct zone-file write), `systemctl start/restart`
|
||||
being no-ops in that same chroot, and `zerombr` added for the
|
||||
genuinely-blank drives.
|
||||
- Password-auth SSH and a temporary NOPASSWD sudoers bootstrap
|
||||
deliberately left active — both closed out properly in Phase 2.
|
||||
- Disk targeting via `/dev/disk/by-id`, not `sdX` — proven necessary when
|
||||
drives landed on different letters than expected during the actual
|
||||
install, and correctly resolved anyway.
|
||||
|
||||
**Testing:**
|
||||
- Logic reviewed ✓
|
||||
- Idempotent mount module used ✓
|
||||
- Error handling via blocks ✓
|
||||
### Phase 2: System Configuration ✓
|
||||
|
||||
**Status:** Ready for real hardware test
|
||||
**Status:** ✓ **COMPLETE — verified on real hardware, 2026-07-19**
|
||||
|
||||
---
|
||||
**What was done:**
|
||||
- Kickstart's temporary bootstrap sudoers grant replaced with a real,
|
||||
deliberately-scoped policy
|
||||
- Workbench's SSH public key deployed (single key, by design — see
|
||||
`CLAUDE.md` for the reasoning on why additional devices' keys wait)
|
||||
- Base system confirmed: hostname, timezone, chrony/NTP, all four
|
||||
expected repos enabled
|
||||
|
||||
### Phase 4: NVIDIA GPU Driver 🔨
|
||||
**Real bugs found and fixed during this run:** missing `ansible.posix`
|
||||
collection (not bundled with the Alpine `ansible-core` package), a
|
||||
variable-naming mismatch (`bigboy_admin_user` vs. the actual
|
||||
`deploy_user`) — see `CLAUDE.md`/`group_vars/bigboy.yml` for the current
|
||||
state of both.
|
||||
|
||||
**Status:** 🔨 **READY TO BUILD**
|
||||
### Phase 3: Filesystem Conversion ✓
|
||||
|
||||
**What's Done:**
|
||||
- RAG documentation complete: `/home/john/documents/library/rag/use-case/nvidia-driver-almalinux.md`
|
||||
- Includes precompiled method (recommended) + DKMS fallback
|
||||
- RTX 5060 Ti specifics documented
|
||||
- Environment variables, troubleshooting covered
|
||||
**Status:** ✓ **COMPLETE — verified on real hardware, 2026-07-19**
|
||||
|
||||
**What's Needed:**
|
||||
- Ansible role scaffold exists at `/home/john/projects/bigboy-setup/ansible/roles/phase-4-nvidia-driver/`
|
||||
- Task file to fill: `tasks/main.yml`
|
||||
- Use nvidia-driver-almalinux.md as reference
|
||||
- ~20 lines YAML (enable repos, install packages, verify)
|
||||
**What was done:**
|
||||
- The four SATA data drives converted from the kickstart's forced xfs to
|
||||
real btrfs (`compress=zstd,noatime`), mounted at final paths
|
||||
(`/srv/backup`, `/srv/rag-library`, `/srv/ai-logs`,
|
||||
`/srv/prompt-library`)
|
||||
- Stale kickstart-generated fstab entries removed
|
||||
- **Root filesystem intentionally NOT converted — stays xfs.** No
|
||||
supported in-place xfs-to-btrfs path exists; a full reinstall (clone/
|
||||
reformat/restore via `debusb`) is deferred until after real-world
|
||||
testing, not before ~a month out from 2026-07-17. This is a decision,
|
||||
documented in `CLAUDE.md`, not a gap.
|
||||
|
||||
**Estimated Effort:** 30 minutes to code + test
|
||||
**Real bugs found and fixed during this run:** a device-path bug
|
||||
(`{{ item.byid }}1` instead of the correct `{{ item.byid }}-part1`
|
||||
suffix), a mount-path mismatch specific to the ai-logs drive
|
||||
(`/srv/ai-logs-raw` in the role vs. the kickstart's actual
|
||||
`/srv/ai-raw`), and `state: unmounted` changed to `state: absent` so
|
||||
stale fstab entries actually get removed rather than just the live
|
||||
mount.
|
||||
|
||||
**Blocker:** None — ready to implement
|
||||
### Phase 4: NVIDIA GPU Driver ✓
|
||||
|
||||
**Dependencies:**
|
||||
- Phase 3 (filesystems) — logged to `/srv/deployment-log/`
|
||||
**Status:** ✓ **COMPLETE — verified on real hardware, 2026-07-19**
|
||||
|
||||
---
|
||||
**What was done:**
|
||||
- AlmaLinux's precompiled open-kmod path used (not manual DKMS):
|
||||
`almalinux-release-nvidia-driver` → `nvidia-driver-cuda
|
||||
nvidia-open-kmod`, one reboot, `nvidia-smi` clean on the first attempt
|
||||
- The smoothest phase of the entire deployment — no issues encountered
|
||||
|
||||
### Phase 5: Ollama Installation 🔨
|
||||
**Still needed:** the actual installed driver version has not yet been
|
||||
recorded in `group_vars/bigboy.yml` — run `nvidia-smi` and update the
|
||||
placeholder there.
|
||||
|
||||
**Status:** 🔨 **READY TO BUILD**
|
||||
### Phase 5: llama.cpp ✓
|
||||
|
||||
**What's Done:**
|
||||
- RAG documentation complete: `/home/john/documents/library/rag/use-case/ollama-deployment.md`
|
||||
- Includes installation (pip + systemd), GPU configuration, CLI commands, REST API
|
||||
- RTX 5060 Ti tuning (VRAM limits, keep-alive, parallel requests)
|
||||
- Model selection guide (mistral:7b recommended for 4GB)
|
||||
**Status:** ✓ **COMPLETE — verified on real hardware, 2026-07-19**
|
||||
|
||||
**What's Needed:**
|
||||
- Ansible role scaffold exists at `/home/john/projects/bigboy-setup/ansible/roles/phase-5-ollama/`
|
||||
- Task file to fill: `tasks/main.yml`
|
||||
- Use ollama-deployment.md as reference
|
||||
- ~30 lines YAML (install, systemd service, pull test model, verify API)
|
||||
**Not Ollama.** Built from source, pinned to a specific, vetted tag —
|
||||
decided deliberately, see `CLAUDE.md` for the full rationale.
|
||||
|
||||
**Estimated Effort:** 45 minutes to code + test
|
||||
**What was done:**
|
||||
- llama.cpp built at pinned tag `b9968` (checked against real commit
|
||||
dates and reviewed for fix/regression patterns before selection —
|
||||
standard flags only, `-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120`)
|
||||
- Model: Ministral 3 14B Instruct, Q4_K_M, pulled via llama.cpp's own
|
||||
Hugging Face integration (`-hf` flag)
|
||||
- `llama-server` running, systemd unit in place, nginx reverse proxy
|
||||
fronting it, firewall opened for the proxy port on both the home LAN
|
||||
and bench LAN ranges, real API key set
|
||||
- **Chat confirmed working** — both through `llama-server`'s built-in
|
||||
web UI (tested from Surface over the home LAN) and through `aichat`, a
|
||||
terminal client, for LAN segments without GUI access
|
||||
|
||||
**Blocker:** None — ready to implement
|
||||
**Real problems found and fixed during this run:**
|
||||
- `cuda-toolkit` (specifically `nvcc`) is a separate package from the
|
||||
driver runtime — not installed by Phase 4, had to be added separately;
|
||||
the plain `cuda-toolkit` meta-package (not a version-pinned
|
||||
sub-package) was used to stay in sync with whatever driver version
|
||||
Phase 4 actually installed
|
||||
- `nvcc` lands in a versioned path (`/usr/local/cuda-13.3/bin`), not on
|
||||
`PATH` by default — a symlink plus a `/etc/profile.d/` entry resolved
|
||||
this permanently
|
||||
- An OOM kill during the actual CUDA compile (`cc1plus`/`cudafe++`
|
||||
confirmed via `dmesg`) — root cause was unbounded parallel compilation
|
||||
against memory-hungry CUDA source files with zero swap configured.
|
||||
Fixed with a 16GB NVMe swapfile (persistent via fstab) plus dropping to
|
||||
`-j2`. zram was separately decided as a complementary fast-swap tier
|
||||
(2026-07-19) — **not yet implemented**, tracked as open work.
|
||||
- Two rounds of missing baseline utilities discovered mid-deployment
|
||||
(`tar`, `which`, then nine more via a proactive `utility-recon.sh`
|
||||
scan) — all now added to the kickstart's `%packages` list for future
|
||||
installs.
|
||||
|
||||
**Dependencies:**
|
||||
- Phase 4 (GPU driver)
|
||||
|
||||
**Note:** Phase 8 (Open WebUI) is optional enhancement to Phase 5
|
||||
**This is the success criterion set on 2026-07-17** ("base AI Server
|
||||
Deployment... Mistral running, chat working, whatever quality, no
|
||||
refinement required") — **met.**
|
||||
|
||||
---
|
||||
|
||||
### Phase 6: Build Suite
|
||||
|
||||
**Status:** 📋 **READY TO BUILD**
|
||||
**Status:** 📋 **READY TO BUILD** — not yet started
|
||||
|
||||
**What's Done:**
|
||||
- RAG documentation: `/home/john/documents/library/rag/use-case/dnf-package-management.md`
|
||||
- Group list: `@Development Tools`
|
||||
- Individual packages: kernel-headers, kernel-devel, btrfs-progs, smartmontools, nvtop
|
||||
Development tools / system utilities beyond what's already in the
|
||||
kickstart. Scope not yet finalized against current (llama.cpp-era) needs
|
||||
— the original scope assumed an Ollama-era package list; revisit before
|
||||
building.
|
||||
|
||||
**What's Needed:**
|
||||
- Ansible role: `roles/phase-6-build-suite/tasks/main.yml`
|
||||
- Simple dnf group install + packages
|
||||
- ~15 lines YAML
|
||||
|
||||
**Estimated Effort:** 15 minutes
|
||||
|
||||
**Dependencies:** Phase 4 (kernel headers need matching kernel)
|
||||
**Dependencies:** Phase 4 (kernel headers matching running kernel) — done
|
||||
|
||||
---
|
||||
|
||||
### Phase 7: Configuration
|
||||
|
||||
**Status:** 📋 **READY TO BUILD**
|
||||
**Status:** 📋 **READY TO BUILD** — not yet started
|
||||
|
||||
**What's Done:**
|
||||
- Scope defined: dotfiles, locale, timezone
|
||||
- RAG docs available (Ansible modules reference)
|
||||
Dotfiles, locale, timezone. Scope largely unchanged from original
|
||||
planning; source for dotfiles still TBD.
|
||||
|
||||
**What's Needed:**
|
||||
- Ansible role: `roles/phase-7-configuration/tasks/main.yml`
|
||||
- Copy tmux.conf, vimrc, bash profile (source TBD)
|
||||
- Set locale, timezone via ansible.builtin.lineinfile + timedatectl
|
||||
- ~20 lines YAML
|
||||
|
||||
**Estimated Effort:** 20 minutes
|
||||
|
||||
**Blocker:** Source for dotfiles (use CE OS defaults or create new)
|
||||
|
||||
**Dependencies:** None
|
||||
**Dependencies:** None blocking
|
||||
|
||||
---
|
||||
|
||||
### Phase 8: Open WebUI (Optional)
|
||||
### Phase 8: Open WebUI
|
||||
|
||||
**Status:** 📋 **READY TO BUILD**
|
||||
**Status:** ⚠️ **LIKELY DROPPING** — deliberate decision needed, not a default
|
||||
|
||||
**What's Done:**
|
||||
- RAG documentation complete: `/home/john/documents/library/rag/use-case/open-webui-deployment.md`
|
||||
- Installation (pip + systemd), Ollama integration, features, troubleshooting
|
||||
|
||||
**What's Needed:**
|
||||
- Ansible role: `roles/phase-8-open-webui/tasks/main.yml`
|
||||
- Python 3.11 install, pip install open-webui, systemd service, wait for API
|
||||
- ~25 lines YAML
|
||||
|
||||
**Estimated Effort:** 30 minutes
|
||||
|
||||
**Status:** Optional (nice-to-have UI; Ollama CLI + API sufficient without it)
|
||||
|
||||
**Dependencies:** Phase 5 (Ollama must be running)
|
||||
`llama-server`'s own built-in web UI already covers the chat-first use
|
||||
case (this is what Phase 5's success was actually verified against).
|
||||
Building a separate Open WebUI layer on top is no longer the default
|
||||
plan — keeping this phase requires an actual reason to want a second UI
|
||||
layer, not inertia from the original Ollama-era plan.
|
||||
|
||||
---
|
||||
|
||||
### Phase 9: Security Hardening
|
||||
|
||||
**Status:** 📋 **DESIGN READY**
|
||||
**Status:** 📋 **DESIGN READY** — not yet started
|
||||
|
||||
**What's Done:**
|
||||
- Tasks defined: firewall (firewalld), SSH hardening, MAC pinning, SELinux review
|
||||
- Open WebUI RAG doc includes security section
|
||||
Firewall (firewalld — already partially ahead of schedule, since Phase 5
|
||||
opened the reverse-proxy port with the same LAN-restriction pattern used
|
||||
elsewhere), SSH hardening (**disable password auth, move to key-only** —
|
||||
this is the actual trigger condition already flagged in `CLAUDE.md` and
|
||||
the original kickstart's own changelog), MAC pinning, SELinux review
|
||||
(currently permissive per the kickstart; confirm AVC denial history
|
||||
before flipping to enforcing).
|
||||
|
||||
**What's Needed:**
|
||||
- Ansible role: `roles/phase-9-security/tasks/main.yml`
|
||||
- firewall-cmd to open SSH 22, WebUI 8080, Cockpit 9090; Ollama 11434 localhost-only
|
||||
- sshd config: disable password, PermitRootLogin=no
|
||||
- MAC pinning via nmcli or network config
|
||||
- ~40 lines YAML
|
||||
**Blocker:** key-based login needs to be confirmed working over real,
|
||||
repeated use before disabling password auth — not just "it worked once."
|
||||
|
||||
**Estimated Effort:** 1 hour (firewall rules need care)
|
||||
|
||||
**Blocker:** SSH key setup must be complete before disabling password auth
|
||||
|
||||
**Dependencies:** Phase 5 (services running)
|
||||
**Dependencies:** Phase 5 (services running) — done
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -181,11 +199,8 @@
|
|||
|
||||
**Status:** 🚫 **DEFERRED**
|
||||
|
||||
**Reason:** Backup target not yet decided (NAS? USB? Aruba?)
|
||||
|
||||
**What's Ready:** Packages (borgbackup, borgmatic), retention policy framework
|
||||
|
||||
**Prerequisite Decision:** Where to backup? NFS mount? Local USB?
|
||||
**Reason:** Backup target not yet decided (NAS? USB? Aruba Object
|
||||
Storage, per the sovereignty-logging design discussion?)
|
||||
|
||||
**Timeline:** Decide backup target → implement Phase 10
|
||||
|
||||
|
|
@ -193,82 +208,56 @@
|
|||
|
||||
### Phase 11: Thermal Baseline Testing
|
||||
|
||||
**Status:** 📋 **READY TO BUILD**
|
||||
**Status:** 📋 **READY TO BUILD** — not yet started
|
||||
|
||||
**What's Done:**
|
||||
- Tasks defined: memtest86, Ollama inference test, log temps
|
||||
Original scope referenced `llama2:70b`/Ollama-specific testing — needs
|
||||
updating to a llama.cpp-appropriate heavy-load test instead. Given the
|
||||
case move to Modcase HYPER (see below), worth doing this *after* the
|
||||
physical case transition, since cooling characteristics will differ from
|
||||
open-frame testing.
|
||||
|
||||
**What's Needed:**
|
||||
- Ansible role: `roles/phase-11-thermal-testing/tasks/main.yml`
|
||||
- Install memtest86, run with timeout
|
||||
- Start Ollama with large model (llama2:70b if VRAM allows, or heavy context)
|
||||
- Monitor with nvidia-smi in background
|
||||
- Log temps to `/srv/deployment-log/phase-11-thermal-*.log`
|
||||
- ~30 lines YAML
|
||||
|
||||
**Estimated Effort:** 45 minutes
|
||||
|
||||
**Dependencies:** Phase 5 (Ollama running)
|
||||
|
||||
**Test Duration:** ~30 minutes (memtest + inference)
|
||||
**Dependencies:** Phase 5 (llama-server running) — done
|
||||
|
||||
---
|
||||
|
||||
### Phase 12: Full System Validation
|
||||
|
||||
**Status:** 📋 **READY TO BUILD**
|
||||
**Status:** 📋 **READY TO BUILD** — not yet started
|
||||
|
||||
**What's Done:**
|
||||
- Validation checklist defined
|
||||
Checklist needs updating away from Ollama/Open-WebUI-specific service
|
||||
checks toward `llama-server`/nginx.
|
||||
|
||||
**What's Needed:**
|
||||
- Ansible role: `roles/phase-12-validation/tasks/main.yml`
|
||||
- Check services: systemctl status ollama open-webui firewalld sshd
|
||||
- Verify GPU: nvidia-smi check
|
||||
- Verify storage: btrfs filesystem usage /
|
||||
- Verify logs: grep errors /srv/deployment-log/*.log
|
||||
- Generate report
|
||||
- ~25 lines YAML
|
||||
|
||||
**Estimated Effort:** 30 minutes
|
||||
|
||||
**Dependencies:** All prior phases
|
||||
**Dependencies:** All prior phases — 1 through 5 done, 6-9/11 pending
|
||||
|
||||
---
|
||||
|
||||
### Phase 13: Home LAN Migration
|
||||
|
||||
**Status:** 🚫 **DEFERRED**
|
||||
**Status:** 🔨 **PARTIALLY UNDERWAY** — ahead of original schedule
|
||||
|
||||
**Reason:** Waiting for Modcase EVO ITX-2 case installation
|
||||
**Original plan:** deferred until case installation.
|
||||
|
||||
**What's Needed:**
|
||||
- Ansible role: `roles/phase-13-home-migration/tasks/main.yml`
|
||||
- Static IP assignment (if not via DHCP reservation)
|
||||
- DNS configuration (if needed)
|
||||
- Network documentation
|
||||
- Bench → home transition runbook
|
||||
**Actual state (2026-07-19):** BigBoy is already reachable on the home
|
||||
LAN (`192.168.1.24`, statically reserved on the FritzBox) — this
|
||||
happened ahead of the physical case move, driven by needing to test the
|
||||
chat interface from a device with a GUI, which the bench LAN doesn't
|
||||
have. The **physical relocation** (case: now **Modcase HYPER**, not the
|
||||
originally-planned Modcase EVO ITX-2 — printed and ready as of
|
||||
2026-07-19) is still pending; BigBoy is being assembled into its new
|
||||
case and permanent home over the next few days.
|
||||
|
||||
**Timeline:** After case installed
|
||||
**Modcase HYPER licensing note:** personal-use licence purchased
|
||||
2026-07-19. Do not print for resale or client delivery until the
|
||||
commercial tier is purchased.
|
||||
|
||||
---
|
||||
|
||||
### Phase 14: Observation Period Runbook
|
||||
|
||||
**Status:** 📋 **TEMPLATE READY**
|
||||
**Status:** 📋 **TEMPLATE READY** — not yet started
|
||||
|
||||
**What's Done:**
|
||||
- Routine defined: daily temp checks, disk usage, service status, log review
|
||||
- Duration: 4 weeks post-deployment
|
||||
|
||||
**What's Needed:**
|
||||
- Runbook document: daily/weekly checklist
|
||||
- Ansible role (optional): periodic health check job
|
||||
- Or manual execution per runbook
|
||||
|
||||
**Estimated Effort:** 20 minutes (runbook), 0 (if manual)
|
||||
|
||||
**Dependencies:** All prior phases complete + operational
|
||||
**Dependencies:** All prior phases complete + operational — not yet at
|
||||
that point (6, 7, 9, 11, 12 still pending)
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -278,101 +267,103 @@
|
|||
|
||||
| Item | Status |
|
||||
|------|--------|
|
||||
| RAG library (DNF, NVIDIA, Ollama, Open WebUI, btrfs, Ansible) | ✓ Complete |
|
||||
| Deployment code (Kickstart, Ansible, inventory) | ✓ Complete (Phase 3) |
|
||||
| Project docs (README, ARCHITECTURE, STATUS, HARDWARE) | 🔨 Current |
|
||||
| Phase 4-5 Ansible role code | 📋 Buildable |
|
||||
| Project docs (README, CLAUDE.md, STATUS.md) | 🔨 Being actively reconciled, 2026-07-19 |
|
||||
| Deployment code (Kickstart, Ansible, inventory) | ✓ Current through Phase 5 |
|
||||
| RAG library references | ⚠️ Predate the llama.cpp decision (Ollama/Open-WebUI docs) — verify or replace before relying on them |
|
||||
| Systemd/nginx configs for llama-server | ❌ Only exist live on BigBoy — not yet tracked as reusable template files in this repo. Real gap: deployment isn't currently reproducible from the repo alone for this piece. |
|
||||
|
||||
### Testing
|
||||
|
||||
| Item | Status |
|
||||
|------|--------|
|
||||
| Logical review (code review) | ✓ Phase 3 done |
|
||||
| Real hardware test | 🚫 Pending BigBoy installation |
|
||||
| Idempotency verification | ✓ Expected for all phases |
|
||||
| Integration test (all 14 phases) | ⏳ After Phase 4-5 built |
|
||||
| Real hardware test, Phases 1-5 | ✓ **Done**, 2026-07-17 through 2026-07-19 |
|
||||
| Chat interface confirmed | ✓ Web UI + terminal client, 2026-07-19 |
|
||||
| Idempotency verification | ✓ Confirmed for Phase 3 (re-run-safe checks in place) |
|
||||
| Integration test (all phases) | ⏳ After Phases 6-9/11-12 built |
|
||||
|
||||
### Git & Version Control
|
||||
|
||||
| Item | Status |
|
||||
|------|--------|
|
||||
| Local /projects/bigboy-setup/ | ✓ Active |
|
||||
| Local /projects/bigboy-alma/ (docs) | ✓ Active |
|
||||
| Forgejo (giovannino/bigboy-alma-deploy) | ⏳ Push after hardware test |
|
||||
| Forgejo (giovannino/bigboy-alma-deploy) | ✓ **Active, pushed, current through Phase 5** |
|
||||
|
||||
---
|
||||
|
||||
## Critical Path to Go-Live
|
||||
## Critical Path to Go-Live (Full 14-Phase Scope)
|
||||
|
||||
```
|
||||
Phase 3: Filesystem ✓
|
||||
Phase 1: Kickstart ✓
|
||||
↓
|
||||
Phase 4: GPU Driver (30 min build) 🔨
|
||||
Phase 2: System Config ✓
|
||||
↓
|
||||
Phase 5: Ollama (45 min build) 🔨
|
||||
Phase 3: Filesystems ✓ (root stays xfs, deliberately)
|
||||
↓
|
||||
Real Hardware Test (hours)
|
||||
Phase 4: GPU Driver ✓
|
||||
↓
|
||||
Phase 6-8: Build, Config, WebUI (1-2 hours) 📋
|
||||
Phase 5: llama.cpp ✓ — chat confirmed working — BASE DEPLOYMENT GOAL MET
|
||||
↓
|
||||
Phase 9: Security (1 hour) 📋
|
||||
Phase 6-7: Build Suite, Configuration 📋
|
||||
↓
|
||||
Phase 11: Thermal Test (30 min) 📋
|
||||
Phase 8: Open WebUI — likely dropping ⚠️
|
||||
↓
|
||||
Phase 12: Validation (30 min) 📋
|
||||
Phase 9: Security Hardening 📋
|
||||
↓
|
||||
Phase 11: Thermal Test (post-case-move) 📋
|
||||
↓
|
||||
Phase 12: Validation 📋
|
||||
↓
|
||||
Phase 14: Observation (4 weeks manual) 📋
|
||||
↓
|
||||
✓ GO-LIVE
|
||||
✓ FULL GO-LIVE
|
||||
|
||||
Parallel: Phase 10 (backup) — deferred until target decided
|
||||
Parallel: Phase 13 (home migration) — deferred until case installed
|
||||
Parallel: Phase 10 (backup) — deferred, target undecided
|
||||
Parallel: Phase 13 (home LAN) — network migration done, physical case move in progress
|
||||
```
|
||||
|
||||
**Estimate:**
|
||||
- Implementation (Phases 4-8): 2-3 hours
|
||||
- First hardware test: 4+ hours (includes troubleshooting)
|
||||
- Phases 9-12: 2-3 hours
|
||||
- Total time to operational: 8-12 hours (compressed schedule)
|
||||
- Observation period: 4 weeks
|
||||
|
||||
---
|
||||
|
||||
## Blockers & Decisions Needed
|
||||
|
||||
### Immediate (Next Week)
|
||||
- **Decision:** Build Phase 4-5 roles? → Yes/No/Wait for hardware?
|
||||
- **Action:** Confirm BigBoy hardware available for testing
|
||||
### Immediate
|
||||
- **Real gap:** systemd unit, nginx config, and firewall rules for
|
||||
`llama-server` exist only on BigBoy — need tracked template files in
|
||||
the repo for actual reproducibility
|
||||
- **Action:** record actual NVIDIA driver version in `group_vars/bigboy.yml`
|
||||
- **Action:** implement zram alongside the existing swapfile (decision
|
||||
already made 2026-07-19, not yet acted on)
|
||||
|
||||
### Medium Term (Before Go-Live)
|
||||
- **Blocker:** Phase 10 backup target (NAS? USB? Remote?)
|
||||
- **Decision:** Open WebUI required or optional? (Phase 8)
|
||||
- **Decision:** Home LAN migration scope (Phase 13)
|
||||
### Medium Term
|
||||
- **Decision:** Phase 8 (Open WebUI) — build or drop? (leaning drop)
|
||||
- **Blocker:** Phase 10 backup target
|
||||
- **Decision:** Phase 9's SELinux review — check AVC denial history
|
||||
before flipping enforcing
|
||||
|
||||
### Long Term (Post-Deployment)
|
||||
- **Decision:** Borgmatic schedule, retention policy
|
||||
- **Decision:** Phase 14 observation period (manual or automated health checks?)
|
||||
### Long Term
|
||||
- Root filesystem reformat to btrfs — deliberately deferred ~a month
|
||||
from 2026-07-17, after real-world testing
|
||||
- The auditable llama.cpp release pipeline and per-client agent
|
||||
playbook pattern — both designed, neither built (see `CLAUDE.md`)
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
✓ **Phase Complete** when:
|
||||
- Code written & tested logically
|
||||
- Real hardware executes without errors
|
||||
- Output logged to `/srv/deployment-log/`
|
||||
- Logs reviewed & no blocking issues found
|
||||
- Phase can be re-run idempotently
|
||||
✓ **Base AI Server Deployment** (2026-07-17 goal) — **MET, 2026-07-19**:
|
||||
llama.cpp built, Ministral 3 14B Instruct running, chat confirmed working
|
||||
through `llama-server`'s web UI, whatever quality it delivers with zero
|
||||
tuning.
|
||||
|
||||
✓ **Deployment Success** when:
|
||||
- All 14 phases run end-to-end without manual intervention
|
||||
- Services stable (Ollama, Open WebUI responding)
|
||||
- GPU verified healthy (nvidia-smi + thermal test)
|
||||
- Storage verified healthy (btrfs filesystem usage, no corruption)
|
||||
**Full 14-phase deployment success** when:
|
||||
- All phases run end-to-end, idempotently
|
||||
- `llama-server`/nginx stable
|
||||
- GPU verified healthy under real thermal load (post-case-move)
|
||||
- Storage verified healthy (btrfs, no corruption)
|
||||
- 4 weeks observation period completed with no critical issues
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-06-27
|
||||
**Next Update:** After Phase 4-5 implementation or real hardware test
|
||||
**Prepared By:** Claude Code (@workbench)
|
||||
**Last Updated:** 2026-07-19
|
||||
**Next Update:** After systemd/nginx configs are tracked, or Phase 6-9
|
||||
implementation begins
|
||||
**Prepared By:** Claude (Desktop), reconciled against Claude Code's
|
||||
2026-07-19 repo audit
|
||||
Loading…
Add table
Add a link
Reference in a new issue