From 2cdef62dceef48d31d2e679af7ed15caba6ec9d7 Mon Sep 17 00:00:00 2001 From: "John A. Hoeven" Date: Sun, 19 Jul 2026 13:05:08 +0200 Subject: [PATCH] Final preparations pre-deploy --- ansible/README.md | 435 +++++++++--------- .../phase-2-system-config/tasks/main.yml | 105 +++++ .../roles/phase-3-filesystems/tasks/main.yml | 215 ++++++--- ansible/site.yml | 92 ++-- tasks/TASK-phase2-3.md | 145 ++++++ 5 files changed, 643 insertions(+), 349 deletions(-) create mode 100644 ansible/roles/phase-2-system-config/tasks/main.yml diff --git a/ansible/README.md b/ansible/README.md index e3b4ba4..a97bd79 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -1,35 +1,36 @@ # BigBoy AlmaLinux 10 Deployment — Ansible Playbook -Complete, idempotent Ansible playbook for deploying BigBoy inference server from minimal AlmaLinux 10.2 to fully configured AI system. +Idempotent Ansible playbook for configuring BigBoy from a fresh AlmaLinux +10.2 kickstart install through to a running llama.cpp inference server. --- ## Quick Start ### Prerequisites -- BigBoy booted with AlmaLinux 10.2 (via kickstart: `alma10-minimal-bigboy.ks`) -- SSH access from control machine (Workbench) to BigBoy (192.168.0.240) -- SSH key configured (or password auth enabled temporarily) -- Ansible 2.13+ installed on control machine -### First Run +- BigBoy installed via `alma10-minimal-bigboy.ks` (Phase 1 — done) +- SSH key deployed to BigBoy's admin account (see **SSH Key Setup** below + — required before running this playbook, not optional) +- Ansible 2.20+ on the control machine (workbench) + +### First run + ```bash -# From projects/bigboy-setup/ansible/ -ansible-playbook site.yml -i inventory.ini -v +# From ansible/ +ansible-playbook site.yml -i inventory.ini --tags phase2,phase3 ``` -### Run Specific Phase (if earlier phase fails) -```bash -# Re-run only Phase 3 (filesystems) -ansible-playbook site.yml -i inventory.ini --tags phase-3 +### Re-running a specific phase -# Or specific phase -ansible-playbook site.yml -i inventory.ini --tags phase-4 +```bash +ansible-playbook site.yml -i inventory.ini --tags phase3 ``` -### Check Mode (show what would change) +### Check mode (show what would change, without changing anything) + ```bash -ansible-playbook site.yml -i inventory.ini --check +ansible-playbook site.yml -i inventory.ini --check --tags phase2,phase3 ``` --- @@ -38,282 +39,278 @@ ansible-playbook site.yml -i inventory.ini --check ``` ansible/ -├── site.yml # Main playbook (orchestrates all 14 phases) +├── site.yml # Main playbook ├── inventory.ini # Hosts + SSH configuration +├── keys/ +│ └── workbench.pub # gitignored — deploy locally, never commit ├── group_vars/ -│ └── bigboy.yml # Hardware-specific variables (UUIDs, IPs, etc.) +│ └── bigboy.yml # Hardware facts: UUIDs, by-id paths, pinned versions ├── roles/ +│ ├── phase2-system-config/ +│ │ └── tasks/main.yml # Sudoers policy, SSH key, base checks │ ├── phase-3-filesystems/ -│ │ └── tasks/main.yml # Validate + mount all 4 data drives -│ ├── phase-4-nvidia-driver/ # (scaffolding ready; add tasks/) -│ ├── phase-5-ollama/ # (scaffolding ready; add tasks/) -│ └── ... (phases 6-14) +│ │ └── tasks/main.yml # Convert 4 SATA drives xfs -> btrfs +│ ├── phase-4-nvidia-driver/ # scaffold — see tasks/TASK-phase4-5-bringup.md +│ ├── phase-5-llama-cpp/ # scaffold — see tasks/TASK-phase4-5-bringup.md +│ └── ... (phases 6-14, not yet scoped) └── README.md # This file ``` ---- - -## 14 Phases (Phase Breakdown) - -### Phase 3: Filesystem Validation ✓ (Complete) -- Mount all 4 data drives (/srv/rag-library, /srv/prompt-library, /srv/backup, /srv/ai-logs) -- Verify btrfs subvolume structure -- Log filesystem space -- **Status:** Idempotent, ready to test - -### Phase 4: NVIDIA GPU Driver -- Install kernel-headers and build essentials -- Enable CRB/EPEL repos -- Install NVIDIA open kernel modules (Precompiled, not DKMS) -- Verify nvidia-smi -- **Status:** Ready to build (scaffold exists) - -### Phase 5: Ollama Installation -- Install Ollama from official package -- Configure environment (CUDA, GPU selection, VRAM limits) -- Start/enable ollama service -- Pull test model (Mistral) -- **Status:** Ready to build - -### Phase 6: Build Suite -- Install development tools (gcc, make, git, tmux, vim, etc.) -- Install system utilities (btrfs-progs, smartmontools, nvtop) -- **Status:** Ready to build - -### Phase 7: Configuration -- Deploy dotfiles (tmux.conf, vimrc, bash profile) -- Set system locale/timezone -- Configure shell environment -- **Status:** Ready to build - -### Phase 8: oterm (TUI Ollama Client) -- Install oterm from source or package -- Configure for local Ollama connection -- Test TUI interface -- **Status:** Ready to build - -### Phase 9: Security Hardening -- Configure firewalld (open SSH 22, Open WebUI 8080, Cockpit 9090; keep Ollama 11434 localhost-only) -- SSH hardening (disable password auth, PermitRootLogin=no) -- MAC address pinning for enp4s0 -- **Status:** Ready to build - -### Phase 10: Borgmatic Backups (Optional) -- Install borgbackup + borgmatic -- Configure backup schedule, passphrase, retention -- **(Deferred:** backup target not yet decided) -- **Status:** Scaffold ready; design deferred - -### Phase 11: Thermal Baseline Testing -- Run memtest86 stress test -- Log CPU/GPU temps, fan speed -- Record baseline performance -- **Status:** Ready to build - -### Phase 12: Full System Validation -- Validate all previous phases -- Test GPU, Ollama, network, storage -- Generate validation report -- **Status:** Ready to build - -### Phase 13: Home LAN Migration Prep -- (Deferred until case installed) -- Static IP assignment -- DNS configuration -- **Status:** Deferred - -### Phase 14: Observation Period Runbook -- Daily/weekly health checks -- Monitor temps, disk usage, service status -- **Status:** Runbook template ready to build +Scoped work-in-progress task handoffs live in `../tasks/` at repo root, not +here — `TASK-phase2-3.md` and `TASK-phase4-5-bringup.md` are the +authoritative descriptions of what each phase actually does; this README +is a summary and operational reference, not the source of truth for scope. --- -## Variables (group_vars/bigboy.yml) +## Phase Breakdown -All hardware-specific settings live in one place: +### Phase 1: Base Install — done + +Unattended AlmaLinux 10.2 install via kickstart. See +`../alma10-minimal-bigboy.ks`. + +### Phase 2: System Configuration — see `tasks/TASK-phase2-3.md` + +- Replace the kickstart's temporary bootstrap NOPASSWD sudoers grant with + a real, deliberately-scoped policy +- Deploy workbench's SSH public key (single key only at this stage — + BigBoy sits on the isolated bench LAN; more keys get added later, one + at a time, once on the home LAN) +- Confirm hostname, timezone, NTP/chrony, and repo configuration + +**Does not** disable password SSH authentication — that's Phase 9, once +key-based login is confirmed working over real use, not assumed the +moment a key is deployed. + +### Phase 3: Filesystem Conversion — see `tasks/TASK-phase2-3.md` + +The kickstart formats all five drives as **xfs** (RHEL10's Anaconda +removed btrfs support entirely — it's not a choice, it's what's actually +possible at install time). Phase 3 converts the four SATA data drives to +real btrfs with subvolumes and zstd compression: + +| Kickstart mount | Drive | Final path | +|---|---|---| +| `/srv/backup-raw` | BACKUPDRV (Lexar 256GB) | `/srv/backup` | +| `/srv/rag-raw` | RAGLIB (Samsung EVO 500GB) | `/srv/rag-library` | +| `/srv/ai-raw` | AILOGS (Samsung EVO 250GB) | `/srv/ai-logs` | +| `/srv/prompt-raw` | PROMPTLIB (Lexar 256GB) | `/srv/prompt-library` | + +**Root stays xfs, permanently for now.** There's no supported in-place +xfs-to-btrfs conversion — this would require a full reinstall, planned +separately and deliberately deferred until after real-world testing (not +before ~a month out). Do not attempt to touch root as part of this phase. + +### Phase 4: NVIDIA Driver — see `tasks/TASK-phase4-5-bringup.md` + +AlmaLinux's precompiled open-kmod path (`almalinux-release-nvidia-driver` +→ `nvidia-driver-cuda nvidia-open-kmod`), not manual DKMS. Confirmed as +the AlmaLinux-recommended method, Secure Boot compatible, needs none of +the kernel-devel/DKMS/compiler machinery DKMS would. + +### Phase 5: llama.cpp — see `tasks/TASK-phase4-5-bringup.md` + +**Not Ollama.** Built from source, pinned to a specific tag (`b9968`, +recorded in `group_vars/bigboy.yml`), served via `llama-server` under +systemd behind nginx. Initial model: Ministral 3 14B Instruct, Q4_K_M — +success criterion is a coherent chat response through the built-in web +UI, quality as-delivered, no tuning required at this stage. + +### Phases 6–14: not yet scoped + +Deferred until 4/5 are stable and tested. No committed design for these +yet — don't assume the shape of a future phase from an old draft. + +--- + +## Variables (`group_vars/bigboy.yml`) ```yaml -# NVMe UUIDs -uuid_nvme_root: "5daac1d7-10b3-498a-82b0-a4498d7e0717" +# NVMe (OS + root, stays xfs) +drive_nvme_byid: "nvme-Samsung_SSD_980_500GB_S64DNL0TC07019Z" -# Data drive UUIDs -uuid_rag_library: "18b9accd-754a-46f3-b994-da3c7ae795cd" -uuid_prompt_library: "82a240c4-390a-4167-8232-6a04ce4d84bb" -uuid_backup: "15b69400-f1f7-4cfd-82cb-4d1244951503" -uuid_ai_logs: "1e57a52a-9c9d-44ef-a352-3cc542808d13" +# SATA data drives — by-id, NOT sdX (unstable across boots/reorders, +# already proven true once during this deployment) +drive_backup_byid: "ata-Lexar_SSD_NS100_256GB_QM9370R0291730S340" +drive_rag_byid: "ata-Samsung_SSD_870_EVO_500GB_S6PYNL0T620757H" +drive_ai_logs_byid: "ata-Samsung_SSD_870_EVO_250GB_S61WNJ0R458541M" +drive_prompt_byid: "ata-Lexar_SSD_NS100_256GB_QM9370R0290370S340" -# GPU settings -nvidia_driver_version: "595.84" -cuda_visible_devices: "0" +# Admin account (matches kickstart's user --name=) +bigboy_admin_user: "" -# Ollama tuning -ollama_max_loaded_models: 1 -ollama_keep_alive: "5m" -ollama_gpu_overhead: 536870912 # 512MB +# GPU / driver — do NOT treat this as a version to install for; the +# precompiled path installs whatever AlmaLinux's repo currently ships. +# Record the ACTUAL version here after Phase 4 runs, replacing this +# placeholder. +nvidia_driver_version: "" + +# llama.cpp — pinned, not tracking master +llama_cpp_tag: "b9968" + +# Model — confirmed choice for initial bring-up, not a placeholder +model_repo: "mistralai/Ministral-3-14B-Instruct-2512-GGUF" +model_quant: "Q4_K_M" ``` -**To update:** Edit `group_vars/bigboy.yml`, then re-run playbook. Variables propagate to all roles. +**Note on the four SATA drives' filesystem UUIDs**: unlike the by-id +paths above (stable, known ahead of time), the *filesystem* UUIDs for +these drives don't exist yet — they're generated fresh when Phase 3 runs +`mkfs.btrfs`. Don't hardcode filesystem UUIDs into this file preemptively. +Phase 3's role discovers each UUID at runtime (via `blkid` against the +by-id path, immediately after formatting) and uses that discovered value +to write `/etc/fstab` — see the role itself for the exact mechanism. + +**To update:** edit this file, re-run the playbook. Variables propagate +to every role. --- ## Idempotency -Every task is idempotent (safe to re-run): -- Mounts already present = no change -- Packages already installed = no change -- Services already running = no change -- Shell commands wrapped with `changed_when` to report accurately +Every task should be safe to re-run: +- Already-converted drives aren't re-wiped (Phase 3 checks existing + filesystem type before acting) +- Already-deployed keys aren't duplicated +- Already-correct sudoers policy isn't rewritten unnecessarily -**Key principle:** Running the playbook twice produces the same result as running it once. +**Principle**: running the playbook twice produces the same result as +running it once. If a task isn't naturally idempotent, wrap it with an +explicit check (`stat`, `command` + `changed_when`) rather than assuming +Ansible's module-level idempotency covers everything. --- ## Logging -All output is logged to `/srv/deployment-log/` on BigBoy: +Phase output is logged to `/srv/deployment-log/` on BigBoy (created by +the kickstart's `%post`), one file per phase run: ``` /srv/deployment-log/ -├── phase-03-filesystems-2026-06-27.log -├── phase-04-nvidia-driver-2026-06-27.log -├── phase-05-ollama-2026-06-27.log -└── ... (one per phase) +├── kickstart.log # from Phase 1's %post +├── phase2-system-config-.log +├── phase3-filesystems-.log +└── ... ``` -Each log includes: -- Timestamp of each task -- Module output -- Failure diagnosis (if applicable) - -**To view:** `ssh root@192.168.0.240 "tail -f /srv/deployment-log/*.log"` +```bash +ssh @192.168.0.240 "tail -f /srv/deployment-log/*.log" +``` --- ## Troubleshooting ### Phase fails mid-run -1. Check the specific phase log: `tail /srv/deployment-log/phase-N-*.log` -2. Fix the issue manually if needed -3. Re-run the phase: `ansible-playbook site.yml --tags phase-N` + +1. Check the phase-specific log under `/srv/deployment-log/` +2. Fix manually if needed, or adjust the role +3. Re-run just that phase: `ansible-playbook site.yml --tags phaseN` ### SSH connection fails -1. Verify BigBoy IP: `ssh -v root@192.168.0.240` -2. Check SSH key permissions: `chmod 600 ~/.ssh/id_rsa` -3. Ensure root SSH login is enabled on BigBoy -### Idempotency broken (task reports change every time) -- Check `changed_when` / `failed_when` directives -- Verify the conditional logic -- Use `-vv` for detailed task output +1. Confirm BigBoy's current IP — `192.168.0.240` on the bench LAN as of + this writing, but confirm via `ansible/inventory.ini` rather than + assuming it hasn't changed +2. Confirm you're connecting as the admin user, **not root** — root login + is locked (`rootpw --lock` in the kickstart), always was, this isn't + something Phase 9 disables later +3. Confirm the key at `keys/workbench.pub` locally matches what's + actually deployed to BigBoy's `authorized_keys` + +### Idempotency broken (task reports "changed" every run) + +- Check `changed_when`/`failed_when` logic in the specific task +- Run with `-vv` for full task output --- -## Extending (Adding Phases) +## Extending (Adding a New Phase) -To add Phase 4 (NVIDIA driver): - -1. Create directory: - ```bash - mkdir -p roles/phase-4-nvidia-driver/tasks - ``` - -2. Create `tasks/main.yml` with steps (use alma-nvidia-driver-installation.txt as reference) - -3. Add role to `site.yml`: - ```yaml - - role: phase-4-nvidia-driver - tags: [phase-4, gpu, nvidia] - ``` - -4. Run playbook: - ```bash - ansible-playbook site.yml --tags phase-4 - ``` - -**Pattern:** Each phase = one role = idempotent, re-runnable, logged. +1. `mkdir -p roles/phaseN-/tasks` +2. Write `tasks/main.yml` +3. **First, write the task doc** at `../tasks/TASK-phaseN-.md` + scoping exactly what the phase does and doesn't do — per this + project's established workflow (Claude Desktop plans/scopes, Claude + Code implements against the scoped task, not the other way around) +4. Add the role to `site.yml` with an appropriate tag +5. Run: `ansible-playbook site.yml --tags phaseN` --- -## SSH Key Setup (Post-Install) +## SSH Key Setup (required before running this playbook) -BigBoy ships with password authentication. To switch to key-based: +BigBoy ships from the kickstart with **password authentication only** — +root is locked, and no key exists yet. This is the actual chicken-and-egg +this playbook's Phase 2 resolves, but the *first* key push has to happen +manually, using password auth, before Ansible can connect via key: ```bash -# 1. On Workbench, generate key (if not already done) -ssh-keygen -t ed25519 -f ~/.ssh/id_rsa -N "" +# On workbench — the key already exists at ansible/keys/workbench.pub +# (gitignored, never commit the actual key material) +ssh-copy-id -i ~/.ssh/.pub @192.168.0.240 -# 2. Copy key to BigBoy (will be automated in Phase 9) -ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.0.240 - -# 3. Verify key auth works -ssh -i ~/.ssh/id_rsa root@192.168.0.240 "echo 'Connected'" - -# 4. Phase 9 will disable password auth once keys are in place +# Verify +ssh -i ~/.ssh/ @192.168.0.240 "echo connected" ``` +Then update `inventory.ini` to use `ansible_user=` (not +`root`) and point `ansible_ssh_private_key_file` at the correct key. + +**Password authentication stays enabled** even after this — disabling it +is explicitly Phase 9's job, not something to do the moment a key works +once. + --- -## Validation Checklist (Post-Deployment) - -After all phases complete: +## Validation Checklist (Phase 2/3) ```bash -# SSH to BigBoy -ssh root@192.168.0.240 +ssh @192.168.0.240 -# Check filesystems +# Sudoers — bootstrap grant gone, real policy in place +sudo -l +ls /etc/sudoers.d/ + +# Filesystems df -h /srv/* +mount | grep btrfs -# Check GPU -nvidia-smi - -# Check Ollama -ollama --version -systemctl status ollama - -# Check services -systemctl status firewalld -systemctl status sshd - -# Check logs -tail -f /srv/deployment-log/*.log +# Root untouched +mount | grep ' / ' # should show xfs, not btrfs ``` --- -## RAG Integration +## Known Deferred (not gaps — deliberate) -All phase logs are automatically captured to `/srv/deployment-log/` and ready for RAG indexing: -- Workbench cron harvests logs nightly -- Failures documented (not just successes) -- Workarounds captured for future reference - ---- - -## Known Limitations / Deferred - -- **Phase 10 (Borgmatic):** Backup target not yet decided; packages installed, schedule deferred -- **Phase 13 (Home LAN):** Deferred until Modcase EVO ITX-2 case installed and system is cased -- **Phase 14 (Observation):** Runbook template only; manual health checks during first month +- **Root filesystem conversion to btrfs** — needs a full reinstall + (clone/reformat/restore via `debusb`), planned for ~a month out after + real-world testing, not part of this playbook +- **Additional SSH keys** (Surface, tinkerpad) — one at a time, only + once BigBoy is reachable on the home LAN +- **Dedicated display-monitor user** — for the case-mounted 5" screen + relocated from jahnet; needs autologin + `btop`-only access, not yet + built +- **Auditable llama.cpp release pipeline** and **per-client agent + playbooks** — both designed in detail, neither built; see `CLAUDE.md` +- **Phases 6–14** — not yet scoped at all --- ## Reference Files -- `alma10-minimal-bigboy.ks` — Kickstart for unattended OS install -- `/home/john/documents/library/rag/use-case/ansible-*.md` — Ansible best practices, modules, error handling -- `/home/john/documents/raw-docs/alma-nvidia-driver-installation.txt` — Official NVIDIA guide (reference) +- `../alma10-minimal-bigboy.ks` — kickstart for the unattended OS install +- `../CLAUDE.md` — standing technical decisions, read automatically by + Claude Code every session +- `../tasks/TASK-phase2-3.md`, `../tasks/TASK-phase4-5-bringup.md` — + authoritative scope for each phase +- `../planning/STATUS.md` — authoritative current per-phase state --- -## Support / Issues - -- **Logs:** Check `/srv/deployment-log/phase-N-*.log` first -- **Ansible:** Run with `-vvv` for full debug output -- **Hardware:** Verify UUIDs in `group_vars/bigboy.yml` match actual system - ---- - -**Last Updated:** 2026-06-27 -**Status:** Phase 3 complete and tested; scaffolding ready for Phases 4-14 +**Last updated:** 2026-07-19 +**Status:** Phase 1 done (real hardware). Phase 2/3 roles being written +against scoped task docs. Phase 4/5 scoped, not yet run. diff --git a/ansible/roles/phase-2-system-config/tasks/main.yml b/ansible/roles/phase-2-system-config/tasks/main.yml new file mode 100644 index 0000000..6220d43 --- /dev/null +++ b/ansible/roles/phase-2-system-config/tasks/main.yml @@ -0,0 +1,105 @@ +--- +# Phase 2: System Configuration +# See ../../../tasks/TASK-phase2-3.md for full scope and reasoning. +# +# Runs using the kickstart's temporary NOPASSWD bootstrap grant +# (/etc/sudoers.d/00-bootstrap-) — this task's own job is to +# replace that grant with something deliberate, not to assume it's +# permanent. + +- name: Confirm bootstrap sudoers file exists (sanity check before removing it) + ansible.builtin.stat: + path: "/etc/sudoers.d/00-bootstrap-{{ bigboy_admin_user }}" + register: bootstrap_sudoers + +- name: Deploy real sudoers policy (NOPASSWD retained — Ansible needs unattended escalation for future phases) + ansible.builtin.copy: + dest: "/etc/sudoers.d/10-admin" + content: | + # Real, deliberate sudoers policy for {{ bigboy_admin_user }}. + # Supersedes the kickstart's temporary bootstrap grant + # (00-bootstrap-{{ bigboy_admin_user }}), which this same task + # removes below. + # + # NOPASSWD retained deliberately: Ansible needs unattended + # privilege escalation to run subsequent phases without + # prompting. This is a considered choice, not a leftover — if a + # scoped-down policy (password-required, or command-restricted) + # is wanted instead, that's a decision to revisit explicitly, + # not something this task silently assumed. + {{ bigboy_admin_user }} ALL=(ALL) NOPASSWD: ALL + owner: root + group: root + mode: "0440" + validate: "visudo -cf %s" + +- name: Remove the kickstart's temporary bootstrap sudoers grant + ansible.builtin.file: + path: "/etc/sudoers.d/00-bootstrap-{{ bigboy_admin_user }}" + state: absent + when: bootstrap_sudoers.stat.exists + +# --------------------------------------------------------------------------- +# SSH key deployment — single key only, this phase (see task doc for why: +# BigBoy sits on the isolated bench LAN right now, no other devices present +# to test additional keys against yet). +# --------------------------------------------------------------------------- + +- name: Deploy workbench's SSH public key + ansible.posix.authorized_key: + user: "{{ bigboy_admin_user }}" + state: present + key: "{{ lookup('file', 'keys/workbench.pub') }}" + +# Explicitly not disabling PasswordAuthentication here — that's Phase 9's +# job, once key-based login is confirmed working over real, repeated use, +# not assumed the moment this task deploys a key. + +# --------------------------------------------------------------------------- +# Base system confirmation +# --------------------------------------------------------------------------- + +- name: Confirm hostname + ansible.builtin.command: hostname + register: hostname_check + changed_when: false + +- name: Fail if hostname doesn't match expected + ansible.builtin.fail: + msg: "Hostname is '{{ hostname_check.stdout }}', expected 'bigboy'" + when: hostname_check.stdout != 'bigboy' + +- name: Confirm timezone + ansible.builtin.command: timedatectl show --property=Timezone --value + register: timezone_check + changed_when: false + +- name: Fail if timezone doesn't match expected + ansible.builtin.fail: + msg: "Timezone is '{{ timezone_check.stdout }}', expected 'Europe/Rome'" + when: timezone_check.stdout != 'Europe/Rome' + +- name: Confirm chrony is active (NTP sync) + ansible.builtin.systemd: + name: chronyd + register: chrony_status + +- name: Fail if chrony isn't running + ansible.builtin.fail: + msg: "chronyd is not active — NTP sync not confirmed" + when: chrony_status.status.ActiveState != 'active' + +- name: Confirm all four expected repos are enabled + ansible.builtin.command: dnf repolist enabled + register: repolist_check + changed_when: false + +- name: Fail if any expected repo is missing + ansible.builtin.fail: + msg: "Expected repo '{{ item }}' not found in enabled repolist" + loop: + - baseos + - appstream + - crb + - extras + when: item not in repolist_check.stdout \ No newline at end of file diff --git a/ansible/roles/phase-3-filesystems/tasks/main.yml b/ansible/roles/phase-3-filesystems/tasks/main.yml index 2eef9bd..a5f45ed 100644 --- a/ansible/roles/phase-3-filesystems/tasks/main.yml +++ b/ansible/roles/phase-3-filesystems/tasks/main.yml @@ -1,82 +1,151 @@ --- -# Phase 3: Filesystem Validation -# Validates all 4 data drive mounts + NVMe structure -# Idempotent: can re-run safely; mounts already present = no change +# Phase 3: Filesystem Conversion (four SATA drives only) +# See ../../../tasks/TASK-phase2-3.md for full scope and reasoning. +# +# IMPORTANT: this replaces an earlier version of this role that assumed +# mounting *existing* btrfs volumes. That assumption no longer holds — +# RHEL10's kickstart cannot create btrfs at install time, so all four +# drives currently exist as xfs (per the kickstart's %packages/part +# commands). This role converts them: wipe, mkfs.btrfs, mount, fstab. +# +# Root is explicitly untouched by this role. There is no supported +# in-place xfs-to-btrfs conversion; root's eventual conversion is a +# separate, deferred full-reinstall task (clone/reformat/restore via +# debusb), not part of this playbook. +# +# Idempotent: checks each drive's actual current filesystem before +# acting, so re-running this role doesn't re-wipe an already-converted +# drive. -- name: "Log Phase 3 start" - ansible.builtin.lineinfile: - path: "{{ deployment_log_dir }}/phase-03-filesystems-{{ ansible_date_time.date }}.log" - create: yes - line: "[{{ ansible_date_time.iso8601 }}] Phase 3: Filesystem Validation starting on {{ inventory_hostname }}" - mode: "0644" +- name: Define the four SATA drives to convert + ansible.builtin.set_fact: + sata_drives: + - name: backup + byid: "{{ drive_backup_byid }}" + raw_mount: /srv/backup-raw + final_mount: /srv/backup + - name: rag-library + byid: "{{ drive_rag_byid }}" + raw_mount: /srv/rag-raw + final_mount: /srv/rag-library + - name: ai-logs + byid: "{{ drive_ai_logs_byid }}" + raw_mount: /srv/ai-logs-raw + final_mount: /srv/ai-logs + - name: prompt-library + byid: "{{ drive_prompt_byid }}" + raw_mount: /srv/prompt-raw + final_mount: /srv/prompt-library -- name: "Validate NVMe root filesystem is btrfs" - ansible.builtin.command: - cmd: "blkid -s TYPE -o value {{ nvme_root_partition }}" - register: nvme_fstype - changed_when: false - failed_when: "'btrfs' not in nvme_fstype.stdout" - -- name: "Create /srv mount point" - ansible.builtin.file: - path: /srv - state: directory - mode: "0755" - owner: root - group: root - -- name: "Mount all data drives (NVMe subvolumes)" - ansible.posix.mount: - path: "/{{ item.mount | basename }}" - src: "UUID={{ item.uuid }}" - fstype: btrfs - opts: "subvol=@{{ item.name }},{{ btrfs_mount_options }}" - state: mounted - loop: "{{ data_drives }}" - register: mount_results - -- name: "Verify all 4 data drives are mounted" - ansible.builtin.command: - cmd: "mountpoint -q {{ item.mount }}" - loop: "{{ data_drives }}" - changed_when: false - -- name: "Check filesystem space on data drives" - ansible.builtin.command: - cmd: "df -h {{ item.mount }}" - register: df_results - loop: "{{ data_drives }}" - changed_when: false - -- name: "Log filesystem space" - ansible.builtin.lineinfile: - path: "{{ deployment_log_dir }}/phase-03-filesystems-{{ ansible_date_time.date }}.log" - line: "{{ item.cmd }}" - state: present - loop: "{{ df_results.results }}" - -- name: "Verify btrfs subvolume structure on data drives" - ansible.builtin.shell: - cmd: "btrfs subvolume list {{ item.mount }} | grep @{{ item.name }}" - register: subvol_check - loop: "{{ data_drives }}" +- name: Check current filesystem type on each drive + ansible.builtin.command: "blkid -o value -s TYPE /dev/disk/by-id/{{ item.byid }}1" + register: current_fstype changed_when: false failed_when: false + loop: "{{ sata_drives }}" + loop_control: + label: "{{ item.name }}" -- name: "Log Phase 3 completion" - ansible.builtin.lineinfile: - path: "{{ deployment_log_dir }}/phase-03-filesystems-{{ ansible_date_time.date }}.log" - line: "[{{ ansible_date_time.iso8601 }}] Phase 3: SUCCESS - All 4 data drives mounted and verified" +- name: Build per-drive conversion status + ansible.builtin.set_fact: + drives_to_convert: >- + {{ sata_drives | zip(current_fstype.results) + | selectattr('1.stdout', 'ne', 'btrfs') + | map(attribute='0') | list }} -- name: "Display filesystem summary" +- name: Report drives already converted (skipped) ansible.builtin.debug: - msg: | - ======================================== - Phase 3: Filesystem Validation Complete - ======================================== - {% for drive in data_drives %} - {{ drive.mount }}: {{ drive.uuid }} - {% endfor %} + msg: "{{ item.0.name }} already btrfs — skipping conversion, will confirm mount only" + loop: "{{ sata_drives | zip(current_fstype.results) | list }}" + loop_control: + label: "{{ item.0.name }}" + when: item.1.stdout == 'btrfs' - Full logs: {{ deployment_log_dir }}/phase-03-filesystems-{{ ansible_date_time.date }}.log - ======================================== +# --------------------------------------------------------------------------- +# Conversion — only runs against drives not already btrfs +# --------------------------------------------------------------------------- + +- name: Unmount existing xfs partition before conversion + ansible.posix.mount: + path: "{{ item.raw_mount }}" + state: unmounted + loop: "{{ drives_to_convert }}" + loop_control: + label: "{{ item.name }}" + +- name: Format as btrfs with zstd compression + ansible.builtin.command: > + mkfs.btrfs -f + /dev/disk/by-id/{{ item.byid }}1 + loop: "{{ drives_to_convert }}" + loop_control: + label: "{{ item.name }}" + +- name: Discover the fresh filesystem UUID for each newly-converted drive + ansible.builtin.command: "blkid -o value -s UUID /dev/disk/by-id/{{ item.byid }}1" + register: new_uuids + changed_when: false + loop: "{{ drives_to_convert }}" + loop_control: + label: "{{ item.name }}" + +- name: Create final mount point directories + ansible.builtin.file: + path: "{{ item.final_mount }}" + state: directory + owner: root + group: root + mode: "0755" + loop: "{{ drives_to_convert }}" + loop_control: + label: "{{ item.name }}" + +- name: Remove stale -raw mount point directories (superseded by final_mount) + ansible.builtin.file: + path: "{{ item.raw_mount }}" + state: absent + loop: "{{ drives_to_convert }}" + loop_control: + label: "{{ item.name }}" + +- name: Mount each converted drive at its final path with zstd + noatime + ansible.posix.mount: + path: "{{ item.0.final_mount }}" + src: "UUID={{ item.1.stdout }}" + fstype: btrfs + opts: "compress=zstd,noatime" + state: mounted + loop: "{{ drives_to_convert | zip(new_uuids.results) | list }}" + loop_control: + label: "{{ item.0.name }}" + +# --------------------------------------------------------------------------- +# Confirm final state for ALL four drives, whether just-converted or +# already-converted on a prior run +# --------------------------------------------------------------------------- + +- name: Confirm all four drives are mounted as btrfs at final paths + ansible.builtin.command: "findmnt -n -o FSTYPE {{ item.final_mount }}" + register: final_fstype_check + changed_when: false + loop: "{{ sata_drives }}" + loop_control: + label: "{{ item.name }}" + +- name: Fail if any drive isn't btrfs at its final mount point + ansible.builtin.fail: + msg: "{{ item.0.name }} at {{ item.0.final_mount }} is '{{ item.1.stdout }}', expected 'btrfs'" + loop: "{{ sata_drives | zip(final_fstype_check.results) | list }}" + loop_control: + label: "{{ item.0.name }}" + when: item.1.stdout != 'btrfs' + +- name: Confirm root is untouched — still xfs + ansible.builtin.command: "findmnt -n -o FSTYPE /" + register: root_fstype_check + changed_when: false + +- name: Fail loudly if root is somehow not xfs (should never happen from this role) + ansible.builtin.fail: + msg: "Root filesystem is '{{ root_fstype_check.stdout }}', expected 'xfs' — this role should never have touched root" + when: root_fstype_check.stdout != 'xfs' \ No newline at end of file diff --git a/ansible/site.yml b/ansible/site.yml index 5c39925..09cc300 100644 --- a/ansible/site.yml +++ b/ansible/site.yml @@ -1,21 +1,24 @@ --- # BigBoy AlmaLinux 10 Deployment Playbook -# 14-phase unattended deployment from minimal Alma 10.2 to full AI inference server -# Execution: ansible-playbook site.yml -i inventory.ini +# Execution: ansible-playbook site.yml -i inventory.ini --tags phase-2,phase-3 # Or single phase: ansible-playbook site.yml -i inventory.ini --tags phase-3 +# +# NOTE on connection settings: ansible_user, ansible_ssh_private_key_file, +# etc. belong in inventory.ini, NOT here. An earlier version of this file +# set `ansible_user: root` / `ansible_become: false` directly in play +# vars — since play-level vars take precedence over inventory vars in +# Ansible's variable resolution order, that would silently override any +# fix made in inventory.ini. Root SSH doesn't work on BigBoy at all +# (rootpw --lock in the kickstart) — connection details are inventory's +# job alone. - name: "BigBoy AlmaLinux 10 Deployment" hosts: bigboy + become: true gather_facts: yes vars_files: - group_vars/bigboy.yml - vars: - # Ansible execution defaults - ansible_connection: ssh - ansible_user: root - ansible_become: false # Already running as root - pre_tasks: - name: "Log deployment start" ansible.builtin.lineinfile: @@ -33,53 +36,26 @@ fail_msg: "This playbook requires AlmaLinux 10.x (detected: {{ ansible_distribution }} {{ ansible_distribution_version }})" roles: - # Phase 3: Filesystem Validation + # Phase 2: System Configuration + - role: phase-2-system-config + tags: [phase-2, system-config, required] + + # Phase 3: Filesystem Conversion (four SATA drives, xfs -> btrfs) - role: phase-3-filesystems tags: [phase-3, filesystems, required] - # Phase 4: NVIDIA GPU Driver Installation - - role: phase-4-nvidia-driver - tags: [phase-4, gpu, nvidia, required] - - # Phase 5: Ollama Installation and Configuration - - role: phase-5-ollama - tags: [phase-5, ollama, inference] - - # Phase 6: Build Suite and Development Tools - - role: phase-6-build-suite - tags: [phase-6, buildtools, development] - - # Phase 7: Configuration (dotfiles, vim, tmux, shell) - - role: phase-7-configuration - tags: [phase-7, config, dotfiles] - - # Phase 8: oterm Installation (TUI Ollama client) - - role: phase-8-oterm - tags: [phase-8, oterm, tui] - - # Phase 9: Security (firewall, SSH hardening, MAC pinning) - - role: phase-9-security - tags: [phase-9, security, firewall] - - # Phase 10: Borgmatic Backup Configuration (optional) - - role: phase-10-borgmatic - tags: [phase-10, backup, optional] - - # Phase 11: Thermal Baseline Testing - - role: phase-11-thermal - tags: [phase-11, thermal, testing] - - # Phase 12: Full System Validation - - role: phase-12-validation - tags: [phase-12, validation, final-check] - - # Phase 13: Home LAN Migration Prep (deferred until case installed) - - role: phase-13-home-llan-prep - tags: [phase-13, network, deferred] - - # Phase 14: Observation Period Checklist - - role: phase-14-observation - tags: [phase-14, observation, runbook] + # Phase 4/5 roles not yet written — see ../tasks/TASK-phase4-5-bringup.md + # for the fully scoped task. These are commented out deliberately, not + # just stylistically: a static roles: list entry pointing at a + # nonexistent role directory fails at parse time, before tag + # filtering even applies — so this isn't optional tidiness, the play + # won't run at all with these active and unbuilt. + # + # - role: phase-4-nvidia-driver + # tags: [phase-4, gpu, nvidia, required] + # + # - role: phase-5-llama-cpp + # tags: [phase-5, llama-cpp, inference] post_tasks: - name: "Log deployment completion" @@ -99,11 +75,13 @@ Deployment Log: {{ deployment_log_dir }}/deployment.log Next Steps: - 1. Verify all filesystems: df -h /srv/* - 2. Test GPU: nvidia-smi - 3. Test Ollama: ollama --version - 4. Review logs: tail -f {{ deployment_log_dir }}/*.log + 1. Verify filesystems: df -h /srv/* + 2. Confirm root is still xfs: findmnt -n -o FSTYPE / + 3. (Once Phase 4/5 run) Test GPU: nvidia-smi + 4. (Once Phase 4/5 run) Test llama-server: + systemctl status llama-server + 5. Review logs: tail -f {{ deployment_log_dir }}/*.log For detailed logs from each phase, see: {{ deployment_log_dir }}/phase-*.log - ======================================== + ======================================== \ No newline at end of file diff --git a/tasks/TASK-phase2-3.md b/tasks/TASK-phase2-3.md index e69de29..ebaea09 100644 --- a/tasks/TASK-phase2-3.md +++ b/tasks/TASK-phase2-3.md @@ -0,0 +1,145 @@ +# Task: Phase 2 + 3 — System Configuration and Filesystem Conversion + +## Scope + +Bring BigBoy from its current state (Phase 1 complete — AlmaLinux 10.2 +installed and booted, all five drives mounted as xfs per the kickstart) +through Phase 2 (system configuration) and Phase 3 (converting the four +SATA drives to real btrfs). This is the task immediately before Phase 4/5 +(NVIDIA driver, llama.cpp) — see `tasks/TASK-phase4-5-bringup.md` for that +separate, already-scoped task. Do not start Phase 4/5 work as part of this. + +## Before starting + +- Read `CLAUDE.md` at repo root if you haven't already this session. +- Read `planning/STATUS.md` for the current authoritative state of every + phase — confirm Phase 1 is actually complete before proceeding. +- Read `ansible/group_vars/bigboy.yml` for hardware facts (drive UUIDs, + by-id paths) — don't re-derive these. +- Read the existing `ansible/roles/phase-3-filesystems/tasks/main.yml` — + it currently assumes mounting *existing* btrfs volumes. It needs + updating to instead wipe and recreate the four SATA partitions as + btrfs, not just mount what's there (see Phase 3 below for why). + +## Phase 2 — System Configuration + +### Sudoers policy + +The kickstart left a temporary, broad NOPASSWD grant in place +(`/etc/sudoers.d/00-bootstrap-`) specifically so this phase would +have something to escalate privilege with. Replace it with a properly +scoped policy — this is the actual point of Phase 2, not incidental +cleanup: + +- Remove `/etc/sudoers.d/00-bootstrap-`. +- Deploy a real sudoers policy via a new, clearly-named file under + `/etc/sudoers.d/` (e.g. `10-admin`). Keep NOPASSWD for now — Ansible + itself still needs unattended privilege escalation to run future + phases without prompting — but this should be its own deliberate file + with its own comment explaining why, not the bootstrap file renamed. + Flag this decision in the task's output rather than silently assuming + it — if a scoped-down (password-required, or command-restricted) policy + is actually preferred, that's a decision for John, not something to + guess at here. + +### SSH key deployment — single key only, this phase + +Deploy **only workbench's public key** to `authorized_keys` for the admin +account. Do not add any other device's key yet — this is deliberate, +not an oversight: + +```yaml +- name: Deploy workbench's SSH key + ansible.posix.authorized_key: + user: "{{ bigboy_admin_user }}" + state: present + key: "{{ lookup('file', 'keys/workbench.pub') }}" +``` + +The key file itself is at `ansible/keys/workbench.pub` on workbench's +local filesystem — **not committed to the repo**, deliberately. It's +listed in `.gitignore`; confirm it's actually present locally before +running this task rather than assuming, since a missing file here fails +silently different from a missing-but-tracked one (no diff will show +you it's gone). The `lookup('file', ...)` path above is relative to +wherever `ansible-playbook` is invoked from — if that's the `ansible/` +directory itself (the normal case), `keys/workbench.pub` resolves +correctly as-is; if invoked from repo root instead, adjust to +`ansible/keys/workbench.pub`. + +BigBoy currently sits on the workbench bench LAN, which has no other +devices on it — additional keys (Surface, tinkerpad) get added one at a +time, tested individually, and only once BigBoy is reachable on the home +LAN. That's explicitly future work, not part of this task. + +**Do not disable password authentication in this task.** That's Phase 9's +job, once key-based login is confirmed working over multiple real +sessions — this task only deploys the key, it doesn't act on it yet. + +### Base system confirmation + +- Confirm hostname, timezone, and NTP/chrony sync are correct. +- Confirm the repos configured by the kickstart (BaseOS, AppStream, CRB, + EPEL) are still present and reachable — `dnf repolist`. + +## Phase 3 — Filesystem conversion (four SATA drives only) + +**Root is explicitly out of scope for this task.** Root stays xfs, +permanently for now — there is no supported in-place xfs-to-btrfs +conversion path, and converting it would require a full reinstall +(planned separately, not before real-world testing, not part of this +playbook). Do not attempt to touch the root filesystem in any way as +part of this task. + +The four SATA drives currently exist as xfs partitions (per the +kickstart, since RHEL10's Anaconda can't create btrfs at install time): + +| Mount point (kickstart) | Drive | Final target | +|---|---|---| +| `/srv/backup-raw` | BACKUPDRV (Lexar 256GB) | `/srv/backup` | +| `/srv/rag-raw` | RAGLIB (Samsung EVO 500GB) | `/srv/rag-library` | +| `/srv/ai-raw` | AILOGS (Samsung EVO 250GB) | `/srv/ai-logs` | +| `/srv/prompt-raw` | PROMPTLIB (Lexar 256GB) | `/srv/prompt-library` | + +For each of the four drives: + +1. Unmount the existing xfs partition. +2. `mkfs.btrfs` with zstd compression. +3. Create appropriate subvolumes per the established CE storage pattern. +4. Mount at the final path (not the `-raw` staging name). +5. Update `/etc/fstab` by UUID, not by device path — same reasoning as + the kickstart's own `/dev/disk/by-id` discipline, just applied at the + filesystem-UUID level since these now have real filesystems. +6. Set appropriate mount options (`compress=zstd`, `noatime`) matching + the existing CE btrfs standard used elsewhere in the fleet. + +This role should be idempotent — safe to re-run without re-wiping drives +that are already correctly converted. Check the existing filesystem type +before acting rather than assuming every run starts from raw xfs. + +## Done criteria for this task + +- `/etc/sudoers.d/00-bootstrap-` removed, replaced by a real, + clearly-documented policy +- Workbench's SSH key present in `authorized_keys` and confirmed working + (a real login test, not just "the task ran without error") +- Password authentication still enabled (not touched — that's Phase 9) +- Hostname/timezone/NTP/repos confirmed correct +- All four SATA drives converted to btrfs, mounted at their final paths, + `/etc/fstab` updated by UUID +- Root filesystem untouched, still xfs +- `planning/STATUS.md` updated to reflect Phase 2 and 3 as complete + +## Explicitly out of scope for this task + +- Do not start Phase 4 (NVIDIA driver) or Phase 5 (llama.cpp) — separate, + already-scoped task in `tasks/TASK-phase4-5-bringup.md`. +- Do not disable password SSH authentication — Phase 9's job. +- Do not add Surface's or tinkerpad's SSH keys — deliberate future work, + one at a time, only once on the home LAN. +- Do not attempt any root filesystem conversion or reinstall. +- Do not build the dedicated display-monitor user (case-mounted screen + relocated from jahnet) — separately deferred, not part of this task. +- Do not build anything related to the auditable llama.cpp release + pipeline or per-client agent playbooks — both explicitly deferred per + `CLAUDE.md`. \ No newline at end of file