biblioteca-ce-ambrosiana/ambrosiana-refinement-synthesis.md

Ambrosiana RAG Refinement Synthesis

Project: Full-scale implementation of Ambrosiana's RAG hive on @Workbench
Period: 2026-06-28 through 2026-07-18
Status: Design complete; deployment commencing 2026-06-29
Authors: Claude (Code) + John Hoeven (OHIOD)
Licence: Unlicense (open-source exemplar)

---

Executive Summary

Ambrosiana was designed as a sophisticated RAG (Retrieval-Augmented Generation) knowledge system for BCEA device but encountered OS difficulties. This project retrofits the entire Ambrosiana vision to @Workbench as the operational implementation.

Key deliverables:

• Refined RAG styleguide optimized for Qwen 0.6b (and all model sizes)
• Agent pipeline with 5 specialized system users orchestrating document synthesis, validation, and indexing
• RIS (Retrieval Information System) providing hierarchical discoverability of 1000+ docs
• Validator enforcing precision and completeness automatically

Target: >95% first-pass document quality through deterministic validation at ingest time.

---

The Problem

@Workbench had a functional RAG library (554MB rag.db) but lacked:

1. Styleguide precision — existing docs were large, bundled, cross-referential
2. Generation pipeline — no orchestrated path from source → validated doc
3. Atomic structure — docs covered multiple concepts, defeating retrieval ranking
4. Discoverability — no way to navigate 1000+ docs by category/type
5. Validation — no deterministic checks before docs entered the library

Ambrosiana's design addressed all of this, but was never operationalized.

---

Solution: Three Layers

Layer 1: Precision Styleguide (Deliverables 0–3)

Eight Irreducible Principles for RAG Documents:

1. Self-Containment — no doc dependencies; every doc reads alone
2. Explicit Enumeration — all cases listed; no "typical" or "depends"
3. No Vague Language — forbidden: might, could, depends, usually, consider, etc.
4. Quantification — "~5-15% CPU" not "moderate"; always measurable
5. Action-First — command/procedure immediately; background last
6. Source Attribution — every doc declares origin (GPL-2.0+, upstream-doc, etc.)
7. Platform Specificity — "Alpine 3.19+ musl" not generic "Linux"
8. No Interpretation — facts from source only; [UNVERIFIED] for gaps

Why: Qwen 0.6b cannot infer, reason around ambiguity, or complete patterns. Precision in docs forces clarity that frontier models inherit automatically.

Format: Atomic documents — one concept per doc, 200–400 tokens, complete + standalone.

Storage: Unlimited atomicity = 500GB–2TB for comprehensive corpus (users budget accordingly).

Layer 2: Agent Pipeline (Handoff 8)

Five Specialized Agents Orchestrate Ingest:

Step 1: agent-scraper
  Fetches source from approved URLs
  → ~/tmp/ (sensitive) or Gitea repo (public)

Step 2: agent-vetter
  Scans for PPIA (personal/privacy attributes)
  Anomaly detection, safety checks
  → Pass/Flag/Quarantine

Step 3: agent-librarian ← KEY AGENT
  Reads source + generation.txt prompt
  Qwen 0.6b generates atomic doc
  cleanup.txt post-processes
  → Draft doc in ~/staging/

Step 4: VALIDATOR (deterministic)
  Checks: structure, precision, completeness, licensing
  Exit codes: 0 (PASS), 1 (FAIL), 2 (WARN)
  → Approved/ or Review/ or Halted

Step 5: agent-commit
  OHIOD approval gate
  Constructs git commit from handoff chain
  → Commits to ce-rag-library

Step 6: agent-indexer (webhook-triggered)
  Reindexes library by RIS code
  Updates SQLite index
  → Query-ready

Agent-librarian home:

/home/agent-librarian/
├── docs/
│   └── rag_doc_styleguide_atomic.md
├── prompts/
│   ├── generation.txt (Qwen instructions)
│   └── cleanup.txt (post-process fixes)
├── validator.py
└── staging/
    ├── approved/ (PASS)
    ├── review/   (WARN)
    └── [draft docs before validation]

Generation prompts are explicit instruction trees:

• No inference; no interpretation
• Examples of passing/failing output
• Cleanup prompt fixes common issues before validation

Layer 3: RIS (Retrieval Information System) (Handoff 9)

Hierarchical taxonomy enabling discovery at scale.

Code format: RIS:[Category][Subcategory]-[Topic]

• Example: RIS:2b-002 = Category 2 (Tools), Subcategory b (BTRFS), Topic 002 (Mount)

9 Major Categories:

• 0: Foundational / General
• 1: Systems / Infrastructure
• 2: Tools / Applications
• 3: Workflows / Procedures
• 4: Concepts / Theory
• 5: Reference / Specs
• 6: Error Handling / Troubleshooting
• 7: Compliance / Policy
• 8: Testing / Validation

Integration:

• Every atomic doc includes RIS metadata: **RIS:** 2b-002
• Validator enforces RIS format + valid categories
• SQLite index (ris-index.sqlite3) catalogs by RIS code
• Qwen can retrieve: "all BTRFS docs" = SELECT * WHERE ris_code LIKE '2b%'

Scalability:

• Flat structure (no reindexing when adding categories)
• Subcategories expand: 1a, 1b, 1c, ... → no retroactive changes
• Topics 001–009 reserved; 010–099 for subdivisions; 100+ for major sections

---

Design Philosophy

Quality First, Storage Second

Traditional doc systems optimize storage:

• Bundle related concepts (saves space)
• Create cross-references (reduces duplication)
• Design for size economy

Ambrosiana optimizes for model usability + knowledge quality:

• Atomic docs: one concept per doc (no bundling)
• No cross-references: each doc complete + standalone
• Unlimited proliferation: 10,000 small docs > 1000 large docs
• Storage is user budget (500GB–2TB typical)

Why: Retrieval ranking improves (tight keyword density), Qwen gets complete context (no "see also"), frontier models apply knowledge directly (no interpretation needed).

Qwen's Constraints = Frontier Benefit

Designing for Qwen 0.6b's limitations (no inference, no ambiguity recovery) produces docs that:

• Frontier models use without reasoning overhead
• Scale to any model size (Mistral, Claude inherit clarity)
• Remain deterministically validatable

GPL is Freedom, Not Restriction

GPL-2.0+ is Tier 1 (freely redistributable):

• Man pages, kernel docs, GNU tools → freely ingestible
• Compliance = footer attribution only (no special restrictions)
• Corpus inherits GPL if any GPL-licensed doc present
• This is correct and intended (open-source distribution model)

---

Implementation Schedule

Phase 1: Agent Provisioning & Pilot (2026-06-29 through 2026-07-01)

Day 1 (Provisioning):

• Create 5 agent users (librarian, vetter, scraper, commit, indexer)
• Deploy home directories + styleguide files
• Set up permissions (doas/sudo)
• Configure OpenSMTPD for agent mail

Day 2 (Pilot):

• Classify 8–10 BTRFS docs as atomic (Deliverable 2 format)
• Add RIS codes + metadata
• Run through agent-librarian generation + validation
• Measure pass rate (target >95%)

Day 3 (Integration):

• Commit pilot docs to ce-rag-library
• Index by RIS code
• Verify query performance

Phase 2: Full Corpus Rollout (2026-07-07 through 2026-07-18)

• Rechunk 100+ existing workbench docs into atomic docs
• Classify each with RIS codes
• Apply precision principles (Deliverable 0A)
• Run validator; iterate on failures
• Commit to ce-rag-library in batches
• Measure BM25 retrieval improvement

Phase 3+: Ongoing (After 2026-07-18)

• Add new subcategories to RIS as needed (no reindexing)
• Continuous ingest from approved sources
• Agent-librarian as operational pipeline
• Quarterly corpus audits for precision violations

---

Success Metrics

Metric	Target	Measurement
First-pass validator pass rate	>95%	Phase 1 pilot: 8–10 docs
RIS categories utilized	5+	Phase 1: use at least 2b, 6b, 3c
Validator violations caught	Deterministic	No hallucinations after cleanup
Query ranking improvement	+30%	BM25 scores on atomized corpus
Docs per category	10–50	No single category >500 docs
Ingest time per doc	<5 min	End-to-end pipeline

---

Key Files & Locations

Deployed to @Workbench

Component	Path	Owner	Status
RIS catalogue	/home/agent-indexer/docs/ris-catalogue.md	agent-indexer	Deploy
Styleguide	/home/agent-librarian/docs/rag_doc_styleguide_atomic.md	agent-librarian	Deploy
Generation prompt	/home/agent-librarian/prompts/generation.txt	agent-librarian	Deploy
Cleanup prompt	/home/agent-librarian/prompts/cleanup.txt	agent-librarian	Deploy
Validator	/home/agent-librarian/validator.py	agent-librarian	Deploy
Indexer script	/home/agent-indexer/bin/index-rag.sh	agent-indexer	Deploy
Orchestrator	/usr/local/bin/ce-ragdoc-ingest	root	Deploy
RIS index (SQLite)	/var/lib/rag/ris-index.sqlite3	agent-indexer	Generated

In Ambrosiana Repo (This Project)

Deliverable	File	Type
Principles	deliverables/00a-precision-principles.md	Design
Chunking	deliverables/00b-chunking-strategy.md	Design
Storage	deliverables/00c-storage-budget.md	Design
Rules	deliverables/01-precision-rules.md	Spec
Format	deliverables/02-format-specification.md	Spec
Prompts	deliverables/03-generation-prompts.md	Spec
Validator	deliverables/04-validator-implementation.py	Code
Test Run	deliverables/05-test-run-report.md	Template
Handoffs	handoff-[1-9].md	Process docs

---

Open Questions / Future Refinement

1. Reorg opportunities: Should RIS categories merge/split after Phase 1? (Defer to after pilot.)
2. LLM-as-judge: Should quality validator use frontier model for fact-checking? (Phase 2 enhancement.)
3. Semantic reranking: Add semantic similarity layer over BM25? (Phase 3 research.)
4. Cascade integration: How to invoke Mistral/Claude for complex queries? (Depends on Piccard integration.)
5. Versioning: Archive old doc versions in RIS? (Post-Phase-1 decision.)

---

Why This Matters

Ambrosiana is the open-source RAG exemplar — a reference implementation of:

• Small-model-first design (Qwen → Mistral → Claude)
• Privacy-first knowledge architecture
• Deterministic validation at ingest time
• Hierarchical discovery at scale
• Attribution-driven licensing (GPL is feature, not restriction)

By implementing it fully on @Workbench, we prove it works operationally and can distribute as the canonical reference for the broader open-source community.

---

Credits

Design & Architecture: John Hoeven (OHIOD)
Implementation & Orchestration: Claude (Code)
Agent Infrastructure: CE OS Team
Licensing Philosophy: Stallman's Four Freedoms

Ethical Framework: CE AI Ethics (OHIOD autonomy, transparency, scope routing)

---

Status: Ready for Phase 1 deployment.
Date: 2026-06-28
Next: Begin agent provisioning (2026-06-29)

---

This document describes Ambrosiana: a privacy-first, quality-optimized RAG system designed for small models, immediately usable by frontier models, and distributed as open-source exemplar.

Implemented on @Workbench as lieutenant to BCEA's original vision.

---

Co-authored by: Claude Sonnet 4.6 noreply@anthropic.com
CE Attribution Standards: Creative work of humans (OHIOD design), synthesized with AI assistance (Claude implementation).