Updates and repo cleanup
This commit is contained in:
parent
71dd3c9e23
commit
2308251255
GPG key ID:
306E507219506D4E
13 changed files with 117 additions and 4219 deletions
171
CLAUDE.md
171
CLAUDE.md
|
|
@ -1,103 +1,96 @@
|
|||
# Task: Phase 4 + 5 bring-up — NVIDIA driver and llama.cpp
|
||||
# CLAUDE.md — bigboy-alma-deploy
|
||||
|
||||
## Scope
|
||||
This file is read automatically at the start of every Claude Code session in
|
||||
this repo. It captures standing decisions so they don't need to be
|
||||
re-explained or re-litigated each session. If something here conflicts with
|
||||
a task doc you've been handed, the task doc wins for that session — but
|
||||
flag the conflict rather than silently picking one.
|
||||
|
||||
Bring BigBoy from its current state (Phase 3 complete — filesystems mounted)
|
||||
through Phase 4 (NVIDIA driver) and Phase 5 (llama.cpp built and serving),
|
||||
per `planning/STATUS.md`. This is a bring-up task, not the auditable
|
||||
release-pipeline work — see `CLAUDE.md` for why that's explicitly out of
|
||||
scope right now.
|
||||
## What this repo is
|
||||
|
||||
## Before starting
|
||||
Deployment automation for BigBoy — a Ryzen 5 / RTX 5060 Ti (16GB) AI
|
||||
inference server, running AlmaLinux 10.2. BigBoy also serves as the
|
||||
reference/testbed machine for CE's future RHEL + llama.cpp client
|
||||
deployments — meaning decisions made here are expected to generalize, not
|
||||
just work once.
|
||||
|
||||
- Read `CLAUDE.md` at repo root if you haven't already this session.
|
||||
- Read `planning/STATUS.md` for the current authoritative state of every
|
||||
phase — confirm Phase 3 is actually complete before proceeding.
|
||||
- Read `group_vars/bigboy.yml` for hardware facts (drive UUIDs, target
|
||||
driver version, GPU architecture) — don't re-derive these.
|
||||
## Source of truth for status
|
||||
|
||||
## Phase 4 — NVIDIA driver
|
||||
`planning/STATUS.md` is authoritative for what's actually done vs. pending.
|
||||
Read it before starting work. Update it when a phase's status changes —
|
||||
don't leave it stale.
|
||||
|
||||
Use AlmaLinux's **precompiled open kernel module** path, not manual DKMS.
|
||||
Confirmed via NVIDIA's own AlmaLinux installation guide (2026-07-17): this
|
||||
is the AlmaLinux-recommended method, works regardless of Secure Boot
|
||||
state, and needs none of the kernel-devel/kernel-headers/DKMS/GCC
|
||||
compilation machinery the kickstart's package list was originally
|
||||
written assuming. "Open" (not proprietary/closed) kernel modules are the
|
||||
right choice on Blackwell-generation cards like the RTX 5060 Ti
|
||||
regardless of precompiled-vs-DKMS.
|
||||
## Standing technical decisions (do not re-derive or second-guess these)
|
||||
|
||||
```bash
|
||||
# Enables the AlmaLinux NVIDIA driver repo + CRB + NVIDIA CUDA repo +
|
||||
# EPEL in one step
|
||||
sudo dnf install almalinux-release-nvidia-driver
|
||||
- **OS**: AlmaLinux 10.2, moved from NixOS due to RTX 5060 Ti driver
|
||||
friction on Nix. RHEL-family chosen fleet-wide for AI-server workloads
|
||||
for compliance/audit reasons (EU AI Act relevance), not just this one box.
|
||||
- **Inference engine**: llama.cpp, built from source — deliberately not
|
||||
Ollama. Reasons: full control over CUDA build flags and quantization,
|
||||
built-in web UI via `llama-server` removes the need for a separate
|
||||
Open WebUI layer. Ollama also just wraps llama.cpp's ggml engine
|
||||
underneath on NVIDIA/Linux anyway, so switching back would add a layer
|
||||
of indirection without avoiding the dependency.
|
||||
- **GPU target**: `-DCMAKE_CUDA_ARCHITECTURES=120` (Blackwell / sm_120).
|
||||
Requires NVIDIA driver ≥570; target driver is `595.84` (see
|
||||
`group_vars/bigboy.yml`).
|
||||
- **Known hazard**: MXFP4-quantized models have open compilation issues on
|
||||
sm_120 as of mid-2026. Stick to standard GGUF quants — Q4_K_M or Q5_K_M —
|
||||
from established quantizers (Bartowski, Unsloth namespaces on Hugging
|
||||
Face). This sidesteps the issue entirely; it is not a performance
|
||||
preference, it's a build-stability one.
|
||||
- **Version discipline**: llama.cpp has no semantic versioning — continuous
|
||||
build-tagged releases only. Never build against `master`/HEAD. Every
|
||||
build pins a specific tag, recorded in `group_vars/bigboy.yml`.
|
||||
- **Build flags**: standard only — `cmake -B build -DGGML_CUDA=ON
|
||||
-DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120`. No exotic
|
||||
tuning flags (e.g. `-DLLAMA_CUDA_MMV_Y`) unless a specific, documented
|
||||
problem requires one. We are already deep in non-standard territory
|
||||
(Blackwell + Alma + sovereignty logging); the inference engine build
|
||||
itself should be as boring and reproducible as possible.
|
||||
- **Serving**: `llama-server` under systemd, bound to `127.0.0.1`, fronted
|
||||
by nginx (reuse the existing CE reverse-proxy pattern from jahnet — don't
|
||||
invent a new one). `--api-key` enforced even on LAN-only. `--cont-batching
|
||||
--parallel N` for concurrent chat sessions.
|
||||
- **SELinux**: stays enforcing. If early bring-up needs a permissive
|
||||
discovery pass to collect AVC denials via `ausearch`/`audit2allow`,
|
||||
that's a temporary diagnostic state, not a resting state — flip back to
|
||||
enforcing before considering the phase done.
|
||||
|
||||
# Compute-only / headless — no desktop GL/X components, matches this
|
||||
# hardware's actual role
|
||||
sudo dnf install nvidia-driver-cuda nvidia-open-kmod
|
||||
## Style requirements for any script in this repo
|
||||
|
||||
sudo reboot
|
||||
```
|
||||
Follow the CE OS Bash Style Guide (`Updated_Bash_Style_Guide` in the CE
|
||||
project knowledge) for every script:
|
||||
- Attribution header (dwarves first, then John A. Hoeven / Claude AI),
|
||||
licence (Unlicense), version, status
|
||||
- No `set -e` — every operation checked and logged explicitly
|
||||
- Cleanup trap registered before work begins
|
||||
- Single confirmation prompt before any system-modifying action
|
||||
- No silent failures — every error path logs and either hard-fails or warns
|
||||
- Never assume root; invoke `sudo` explicitly for privileged steps only
|
||||
|
||||
- Confirm `nouveau` is blacklisted (should already be handled by the
|
||||
kickstart — verify, don't assume).
|
||||
- **Do not treat `group_vars/bigboy.yml`'s recorded `595.84` as a version
|
||||
to install for** — that was an early planning estimate. The precompiled
|
||||
path installs whatever AlmaLinux's own NVIDIA driver repo currently
|
||||
ships. Record the *actual* installed version (from `nvidia-smi`'s
|
||||
output) back into `group_vars/bigboy.yml` once confirmed, replacing the
|
||||
placeholder.
|
||||
- `gcc` from the kickstart package list is still needed — not for the
|
||||
driver anymore, but for compiling llama.cpp against CUDA in Phase 5.
|
||||
No change needed there.
|
||||
- **Done when**: `nvidia-smi` runs cleanly and reports the RTX 5060 Ti.
|
||||
This repo is a single-purpose ops repo for one Alma server, not part of the
|
||||
CEOS multi-distro installer framework — so skip the `ce_env.conf` /
|
||||
`pkg_*` / `CE_PRIV` abstraction layer specifically. Use `sudo` directly.
|
||||
Everything else in the style guide applies.
|
||||
|
||||
## Phase 5 — llama.cpp
|
||||
## Explicitly deferred — do not build unless a task doc asks for it
|
||||
|
||||
1. **Pick a pinned build tag.** Check
|
||||
`https://github.com/ggml-org/llama.cpp/releases` (or `git ls-remote
|
||||
--tags`) for a recent, stable-looking tag — do not build against
|
||||
`master`. Record the chosen tag in `group_vars/bigboy.yml` once decided.
|
||||
2. **Shallow clone at that tag**, build with the standard flags from
|
||||
`CLAUDE.md` (`-DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
|
||||
-DCMAKE_CUDA_ARCHITECTURES=120`).
|
||||
3. **Model**: pull a Q4_K_M or Q5_K_M GGUF from Bartowski or Unsloth on
|
||||
Hugging Face for initial testing — a 7B-class instruct model is
|
||||
sufficient to validate the pipeline end to end. Do not use an
|
||||
MXFP4-quantized model (see `CLAUDE.md` — known sm_120 build hazard).
|
||||
4. **Serve** via `llama-server` under systemd:
|
||||
- Bind `127.0.0.1`, not `0.0.0.0`
|
||||
- `--api-key` set (placeholder value is fine for initial bring-up,
|
||||
but note in the PR/commit that it needs a real value before any
|
||||
external exposure)
|
||||
- `--cont-batching --parallel 4` (or a reasonable default — this isn't
|
||||
the tuning pass, just needs to not be single-request-only)
|
||||
- nginx reverse proxy in front, following the existing pattern already
|
||||
used for Nextcloud/Forgejo on jahnet — don't invent a new nginx
|
||||
pattern for this
|
||||
5. **Firewall**: only the reverse-proxy port needs opening; the
|
||||
`llama-server` port itself stays localhost-only.
|
||||
An auditable testing/release system for llama.cpp (versioned releases,
|
||||
btrfs snapshot safety net, promote/rollback via symlink swap, a Forgejo
|
||||
mirror that only ever receives BigBoy-validated tags so client
|
||||
deployments never pull raw upstream) was designed in detail on
|
||||
2026-07-16, but is **not yet built or integrated into the phase plan**.
|
||||
Draft, untested scripts from that design may exist under `scripts/draft/`.
|
||||
Do not wire them into the active Ansible roles or treat them as
|
||||
representing current repo state — they're a reference for later, not a
|
||||
task in progress. Bringing BigBoy up and running is the current priority;
|
||||
this gets picked up afterward.
|
||||
|
||||
## Done criteria for this task
|
||||
## Scope discipline
|
||||
|
||||
- `nvidia-smi` confirms GPU visible and driver loaded
|
||||
- `systemctl status llama-server` shows active/running
|
||||
- `curl http://127.0.0.1:<port>/health` (or equivalent) returns healthy
|
||||
- One real inference request through the API returns a coherent response
|
||||
- The chosen build tag is recorded in `group_vars/bigboy.yml`
|
||||
- `planning/STATUS.md` updated to reflect Phase 4 and 5 as complete
|
||||
|
||||
## Explicitly out of scope for this task
|
||||
|
||||
- Do not build the update/promote/prune/publish scripts referenced in
|
||||
`CLAUDE.md` — that's deferred work, not this task.
|
||||
- Do not build Open WebUI (Phase 8) — `llama-server`'s built-in UI covers
|
||||
the chat-first use case; Phase 8 is likely to be dropped entirely, but
|
||||
that's a separate decision, not part of this task.
|
||||
- Do not modify the Phase 3 filesystem/mount layout — it's already
|
||||
reviewed and complete. If something about it seems to be causing a
|
||||
problem in Phase 4/5, flag it rather than changing it directly.
|
||||
- Do not touch SELinux policy beyond what's needed to get services
|
||||
running (permissive discovery pass if genuinely needed, per
|
||||
`CLAUDE.md` — but this should be a temporary diagnostic step, not a
|
||||
final state, and not a deep dive into custom policy authoring).
|
||||
Task docs will name a specific phase or task from `STATUS.md`. Do the
|
||||
named task. Don't refactor, "improve," or extend adjacent phases that are
|
||||
already marked complete or reviewed, even if something nearby looks
|
||||
improvable — flag it instead and let it be a deliberate decision, not a
|
||||
side effect.
|
||||
Loading…
Add table
Add a link
Reference in a new issue