giovannino/bigboy-alma-deploy

Fork 0

John A. Hoeven f8dec92246

Update ks and phase 4-5

2026-07-17 16:09:35 +02:00

6.5 KiB

Raw Permalink Blame History

AI Services on AlmaLinux — Ollama, Open WebUI, Cockpit for BigBoy

Reference for installing and configuring inference/management services on AlmaLinux 10.2

Service Architecture

Service	Port	Purpose	Runs As
Ollama	11434	Local inference engine (localhost only)	ollama (system user)
Open WebUI	8080	Web interface for Ollama (LAN-facing)	open-webui (system user)
Cockpit	9090	System management dashboard (LAN-facing)	cockpit (systemd managed)

Installation

Ollama

Install from official repository:

curl -fsSL https://ollama.ai/install.sh | sh

This:

Downloads and installs the ollama binary
Creates ollama system user
Installs systemd service unit
Enables and starts the service

Verify installation:

ollama --version
systemctl status ollama

Pull a model (after GPU driver is confirmed working):

ollama pull mistral                # Mistral Small 3.1 7B (smallest, fastest)
# Or for larger model:
ollama pull mistral:24b            # 24B variant if VRAM available

Open WebUI

Install from package (if available in repos):

sudo dnf install -y open-webui

If not in repos, install via Python/pip (alternative):

sudo dnf install -y python3 python3-pip
pip install --user open-webui

Or: Run as a container (preferred for isolation):

sudo dnf install -y podman
podman run -d --name open-webui \
  -p 8080:8080 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Verify it's running:

systemctl status open-webui
# Or if containerized:
podman ps | grep open-webui

Test connection:

curl http://localhost:8080
# Should return HTML (the Open WebUI frontend)

Cockpit

Install:

sudo dnf install -y cockpit cockpit-podman cockpit-pcp

Enable and start:

sudo systemctl enable cockpit.socket
sudo systemctl start cockpit.socket

Verify it's listening:

sudo ss -tlnp | grep 9090
# Expected: tcp LISTEN ... :9090 ... cockpit

Access via browser (from another machine on LAN):

https://192.168.0.240:9090
# Login as john (with sudo privileges)

Configuration

Ollama — Environment Variables

Location: /etc/default/ollama (create if doesn't exist)

sudo cat > /etc/default/ollama <<'EOF'
# Ollama configuration for BigBoy

# GPU/CUDA settings
CUDA_VISIBLE_DEVICES=0           # Use GPU 0 (the RTX 5060 Ti)
OLLAMA_MAX_LOADED_MODELS=1       # Never load multiple models into VRAM simultaneously
OLLAMA_KEEP_ALIVE=5m             # Unload model after 5 minutes of inactivity

# Memory management
OLLAMA_MAX_QUEUE=4               # Queue up to 4 requests, don't reject
OLLAMA_FLASH_ATTENTION=1         # Use flash attention (lower peak VRAM)
OLLAMA_GPU_OVERHEAD=536870912    # Reserve 512MB explicitly for GPU overhead

# Listening
OLLAMA_HOST=127.0.0.1:11434      # Localhost only (Open WebUI is the LAN interface)

# Logging (optional)
OLLAMA_DEBUG=0                   # Set to 1 for verbose debug logs
EOF

Reload after editing:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Open WebUI — Configuration

Connect to local Ollama:

Open WebUI web interface → Settings → Backend
  OLLAMA_API_BASE_URL: http://localhost:11434

Optional: Persist configuration in environment:

sudo cat >> /etc/default/open-webui <<'EOF'
OLLAMA_BASE_URL=http://127.0.0.1:11434
EOF

Cockpit — HTTPS Certificate

Cockpit requires HTTPS. Certificate is auto-generated on first start:

ls -la /etc/cockpit/ws-certs.d/
# Should see auto-generated certificate

On first connection, browser will warn about self-signed cert — accept it.

Optional: Use a real certificate (not needed for internal use):

# Place your certificate and key in /etc/cockpit/ws-certs.d/
sudo cp your-cert.crt /etc/cockpit/ws-certs.d/
sudo cp your-key.key /etc/cockpit/ws-certs.d/
sudo systemctl restart cockpit

Service Management

Check status of all three:

systemctl status ollama
systemctl status open-webui      # Or podman ps if containerized
systemctl status cockpit

Start/stop/restart:

sudo systemctl start ollama
sudo systemctl stop ollama
sudo systemctl restart ollama

# Same for open-webui and cockpit

Enable on boot:

sudo systemctl enable ollama
sudo systemctl enable open-webui
sudo systemctl enable cockpit

View logs:

journalctl -u ollama -f           # Follow Ollama logs
journalctl -u open-webui -f
journalctl -u cockpit -f

Verification Checklist

Run after all three services are installed and started:

Ollama is running and accessible:

curl http://localhost:11434/api/version
# Expected: {"version": "x.x.x"}

GPU is recognized by Ollama:

ollama list
# Should show any pulled models

Open WebUI can connect to Ollama:

curl http://localhost:8080
# Should return HTML (not a connection error)

Cockpit is listening on 9090:
```
sudo ss -tlnp | grep 9090
```

All ports are firewalled correctly (see alma-firewall.md):

sudo firewall-cmd --list-ports
# Should show 8080/tcp, 9090/tcp (11434 NOT exposed to LAN)

Troubleshooting

Ollama won't start:

Check NVIDIA driver is loaded: nvidia-smi
Check CUDA environment: ollama --version and nvidia-smi
View error logs: journalctl -u ollama -e

Open WebUI can't connect to Ollama:

Verify Ollama is listening: curl http://localhost:11434/api/version
Check Open WebUI logs: journalctl -u open-webui -e
Verify firewall isn't blocking localhost (it shouldn't): sudo firewall-cmd --list-all

Cockpit won't open in browser:

Verify it's listening: sudo ss -tlnp | grep 9090
Try https:// not http:// (HTTPS required)
Check firewall allows 9090 from your client IP

GPU memory exhaustion:

Set OLLAMA_MAX_LOADED_MODELS=1 to avoid multiple models in VRAM
Reduce OLLAMA_KEEP_ALIVE to unload faster
Monitor GPU memory: nvidia-smi or watch -n 1 nvidia-smi

Next Steps

Verify all three services are running (see Verification Checklist)
Configure firewall rules (see alma-firewall.md)
Test inference: ollama pull mistral && ollama run mistral "Hello"
Access Open WebUI from another machine: http://192.168.0.240:8080
Log into Cockpit: https://192.168.0.240:9090

6.5 KiB Raw Permalink Blame History