giovannino/bigboy-alma-deploy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
bigboy-alma-deploy/ansible
History
John A. Hoeven f96101e4cd
 Add project files
2026-07-15 11:58:58 +02:00
..
group_vars Add project files 2026-07-15 11:58:58 +02:00
roles/phase-3-filesystems/tasks Add project files 2026-07-15 11:58:58 +02:00
inventory.ini Add project files 2026-07-15 11:58:58 +02:00
README.md Add project files 2026-07-15 11:58:58 +02:00
site.yml Add project files 2026-07-15 11:58:58 +02:00

README.md

BigBoy AlmaLinux 10 Deployment — Ansible Playbook

Complete, idempotent Ansible playbook for deploying BigBoy inference server from minimal AlmaLinux 10.2 to fully configured AI system.

Quick Start

Prerequisites

  • BigBoy booted with AlmaLinux 10.2 (via kickstart: alma10-minimal-bigboy.ks)
  • SSH access from control machine (Workbench) to BigBoy (192.168.0.240)
  • SSH key configured (or password auth enabled temporarily)
  • Ansible 2.13+ installed on control machine

First Run

# From projects/bigboy-setup/ansible/
ansible-playbook site.yml -i inventory.ini -v

Run Specific Phase (if earlier phase fails)

# Re-run only Phase 3 (filesystems)
ansible-playbook site.yml -i inventory.ini --tags phase-3

# Or specific phase
ansible-playbook site.yml -i inventory.ini --tags phase-4

Check Mode (show what would change)

ansible-playbook site.yml -i inventory.ini --check

Directory Structure

ansible/
├── site.yml                          # Main playbook (orchestrates all 14 phases)
├── inventory.ini                     # Hosts + SSH configuration
├── group_vars/
│   └── bigboy.yml                    # Hardware-specific variables (UUIDs, IPs, etc.)
├── roles/
│   ├── phase-3-filesystems/
│   │   └── tasks/main.yml            # Validate + mount all 4 data drives
│   ├── phase-4-nvidia-driver/        # (scaffolding ready; add tasks/)
│   ├── phase-5-ollama/               # (scaffolding ready; add tasks/)
│   └── ... (phases 6-14)
└── README.md                         # This file

14 Phases (Phase Breakdown)

Phase 3: Filesystem Validation ✓ (Complete)

  • Mount all 4 data drives (/srv/rag-library, /srv/prompt-library, /srv/backup, /srv/ai-logs)
  • Verify btrfs subvolume structure
  • Log filesystem space
  • Status: Idempotent, ready to test

Phase 4: NVIDIA GPU Driver

  • Install kernel-headers and build essentials
  • Enable CRB/EPEL repos
  • Install NVIDIA open kernel modules (Precompiled, not DKMS)
  • Verify nvidia-smi
  • Status: Ready to build (scaffold exists)

Phase 5: Ollama Installation

  • Install Ollama from official package
  • Configure environment (CUDA, GPU selection, VRAM limits)
  • Start/enable ollama service
  • Pull test model (Mistral)
  • Status: Ready to build

Phase 6: Build Suite

  • Install development tools (gcc, make, git, tmux, vim, etc.)
  • Install system utilities (btrfs-progs, smartmontools, nvtop)
  • Status: Ready to build

Phase 7: Configuration

  • Deploy dotfiles (tmux.conf, vimrc, bash profile)
  • Set system locale/timezone
  • Configure shell environment
  • Status: Ready to build

Phase 8: oterm (TUI Ollama Client)

  • Install oterm from source or package
  • Configure for local Ollama connection
  • Test TUI interface
  • Status: Ready to build

Phase 9: Security Hardening

  • Configure firewalld (open SSH 22, Open WebUI 8080, Cockpit 9090; keep Ollama 11434 localhost-only)
  • SSH hardening (disable password auth, PermitRootLogin=no)
  • MAC address pinning for enp4s0
  • Status: Ready to build

Phase 10: Borgmatic Backups (Optional)

  • Install borgbackup + borgmatic
  • Configure backup schedule, passphrase, retention
  • (Deferred: backup target not yet decided)
  • Status: Scaffold ready; design deferred

Phase 11: Thermal Baseline Testing

  • Run memtest86 stress test
  • Log CPU/GPU temps, fan speed
  • Record baseline performance
  • Status: Ready to build

Phase 12: Full System Validation

  • Validate all previous phases
  • Test GPU, Ollama, network, storage
  • Generate validation report
  • Status: Ready to build

Phase 13: Home LAN Migration Prep

  • (Deferred until case installed)
  • Static IP assignment
  • DNS configuration
  • Status: Deferred

Phase 14: Observation Period Runbook

  • Daily/weekly health checks
  • Monitor temps, disk usage, service status
  • Status: Runbook template ready to build

Variables (group_vars/bigboy.yml)

All hardware-specific settings live in one place:

# NVMe UUIDs
uuid_nvme_root: "5daac1d7-10b3-498a-82b0-a4498d7e0717"

# Data drive UUIDs
uuid_rag_library: "18b9accd-754a-46f3-b994-da3c7ae795cd"
uuid_prompt_library: "82a240c4-390a-4167-8232-6a04ce4d84bb"
uuid_backup: "15b69400-f1f7-4cfd-82cb-4d1244951503"
uuid_ai_logs: "1e57a52a-9c9d-44ef-a352-3cc542808d13"

# GPU settings
nvidia_driver_version: "595.84"
cuda_visible_devices: "0"

# Ollama tuning
ollama_max_loaded_models: 1
ollama_keep_alive: "5m"
ollama_gpu_overhead: 536870912  # 512MB

To update: Edit group_vars/bigboy.yml, then re-run playbook. Variables propagate to all roles.

Idempotency

Every task is idempotent (safe to re-run):

  • Mounts already present = no change
  • Packages already installed = no change
  • Services already running = no change
  • Shell commands wrapped with changed_when to report accurately

Key principle: Running the playbook twice produces the same result as running it once.

Logging

All output is logged to /srv/deployment-log/ on BigBoy:

/srv/deployment-log/
├── phase-03-filesystems-2026-06-27.log
├── phase-04-nvidia-driver-2026-06-27.log
├── phase-05-ollama-2026-06-27.log
└── ... (one per phase)

Each log includes:

  • Timestamp of each task
  • Module output
  • Failure diagnosis (if applicable)

To view: ssh root@192.168.0.240 "tail -f /srv/deployment-log/*.log"

Troubleshooting

Phase fails mid-run

  1. Check the specific phase log: tail /srv/deployment-log/phase-N-*.log
  2. Fix the issue manually if needed
  3. Re-run the phase: ansible-playbook site.yml --tags phase-N

SSH connection fails

  1. Verify BigBoy IP: ssh -v root@192.168.0.240
  2. Check SSH key permissions: chmod 600 ~/.ssh/id_rsa
  3. Ensure root SSH login is enabled on BigBoy

Idempotency broken (task reports change every time)

  • Check changed_when / failed_when directives
  • Verify the conditional logic
  • Use -vv for detailed task output

Extending (Adding Phases)

To add Phase 4 (NVIDIA driver):

  1. Create directory:

    mkdir -p roles/phase-4-nvidia-driver/tasks

  2. Create tasks/main.yml with steps (use alma-nvidia-driver-installation.txt as reference)

  3. Add role to site.yml:

    - role: phase-4-nvidia-driver
  tags: [phase-4, gpu, nvidia]

  4. Run playbook:

    ansible-playbook site.yml --tags phase-4

Pattern: Each phase = one role = idempotent, re-runnable, logged.

SSH Key Setup (Post-Install)

BigBoy ships with password authentication. To switch to key-based:

# 1. On Workbench, generate key (if not already done)
ssh-keygen -t ed25519 -f ~/.ssh/id_rsa -N ""

# 2. Copy key to BigBoy (will be automated in Phase 9)
ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.0.240

# 3. Verify key auth works
ssh -i ~/.ssh/id_rsa root@192.168.0.240 "echo 'Connected'"

# 4. Phase 9 will disable password auth once keys are in place

Validation Checklist (Post-Deployment)

After all phases complete:

# SSH to BigBoy
ssh root@192.168.0.240

# Check filesystems
df -h /srv/*

# Check GPU
nvidia-smi

# Check Ollama
ollama --version
systemctl status ollama

# Check services
systemctl status firewalld
systemctl status sshd

# Check logs
tail -f /srv/deployment-log/*.log

RAG Integration

All phase logs are automatically captured to /srv/deployment-log/ and ready for RAG indexing:

  • Workbench cron harvests logs nightly
  • Failures documented (not just successes)
  • Workarounds captured for future reference

Known Limitations / Deferred

  • Phase 10 (Borgmatic): Backup target not yet decided; packages installed, schedule deferred
  • Phase 13 (Home LAN): Deferred until Modcase EVO ITX-2 case installed and system is cased
  • Phase 14 (Observation): Runbook template only; manual health checks during first month

Reference Files

  • alma10-minimal-bigboy.ks — Kickstart for unattended OS install
  • /home/john/documents/library/rag/use-case/ansible-*.md — Ansible best practices, modules, error handling
  • /home/john/documents/raw-docs/alma-nvidia-driver-installation.txt — Official NVIDIA guide (reference)

Support / Issues

  • Logs: Check /srv/deployment-log/phase-N-*.log first
  • Ansible: Run with -vvv for full debug output
  • Hardware: Verify UUIDs in group_vars/bigboy.yml match actual system

Last Updated: 2026-06-27 Status: Phase 3 complete and tested; scaffolding ready for Phases 4-14