Add man pages styleguide ai and reorganize directory structure

man-pages-styleguide/man-pages-styleguide-ai.md

@@ -0,0 +1,218 @@
+# CE Man Page Reformatting Styleguide — AI Corpus (v1.0)
+
+## 0. Purpose and scope
+
+Governs reformatting of individual man pages (Sections 1–8) into the
+CE RAG corpus for AI consumption: microSLM retrieval (Qwen 0.6B) and
+frontier reasoning over retrieval (Claude).
+
+Out of scope: enthusiast-tutorial transformation of man pages. Separate
+future project, separate voice rules.
+
+Sibling document (not yet authored): OHIOD-readable variant. Same rule
+body (Sections 1–8 below), Section 9 (Voice) replaced only. Authored
+after this version passes beta testing.
+
+---
+
+## 1. File boundary
+
+One source document = one file.
+
+| Source type | File unit |
+|---|---|
+| Individual man page | One page = one file |
+| Multi-chapter book (ProGit, Tao of Tmux) | One chapter = one file |
+
+Rationale: file boundary = smallest unit upstream treats as independently
+coherent and independently revised.
+
+Naming: `[source]-[identifier]-rag.md`
+Examples: `man-pages-conventions-v1-rag.md`, `progit-v2-ch03-branching-in-git-rag.md`
+
+---
+
+## 2. Chunk mechanism
+
+One file, many chunks. Chunk = database retrieval unit (SQLite FTS5/BM25
++ embeddings via `agent-indexer`), not a filesystem unit.
+
+Default: chunk boundary = header boundary. One `##`/`###` heading = one
+atomic unit = one chunk.
+
+Override: explicit delimiter comment where heading structure and
+atomicity disagree (a single heading spans two content kinds).
+
+```
+<!-- CHUNK_START --> ... <!-- CHUNK_END -->
+```
+
+Atomicity rule: a chunk boundary falls wherever source content changes
+kind (rule vs. example vs. table vs. exception list), not at a word count.
+
+---
+
+## 3. Format
+
+Markdown body. YAML frontmatter per chunk (`---`-delimited, standard
+frontmatter convention).
+
+Rejected: JSON (escaping fragility on quoted code/chars), YAML-as-whole-
+document (whitespace/colon fragility on prose), epub (pagination format,
+requires unpacking before RAG use).
+
+```markdown
+---
+ris: unassigned
+source: man-pages(7) > STYLE GUIDE > Preferred terms
+kind: rule-table
+scope: exhaustive
+authority: normative
+related: [fcntl(2), open(2)]
+---
+
+[chunk body]
+```
+
+### Frontmatter fields
+
+| Field | Values | Purpose |
+|---|---|---|
+| `ris` | code \| `unassigned` | Rosetta Indexing System pointer. Retrofit pending category scheme decision. |
+| `source` | breadcrumb string | Document > section path |
+| `kind` | `definition` \| `reference` \| `rule-table` \| `prose` \| `example` \| `pointer` \| `manifest` | Content shape — governs retrieval/formatting treatment |
+| `scope` | `exhaustive` \| `illustrative` | Is this list/example complete, or a sample? Primary anti-hallucination signal for frontier. |
+| `authority` | `normative` \| `editorial` | Verbatim upstream rule vs. AIB restructuring |
+| `related` | list of `name(section)` | Cross-references, write-once. Resolved to corpus pointers at ingestion/re-index time by `agent-librarian`/`agent-indexer`, not authored as resolved. |
+
+---
+
+## 4. Granularity guidance
+
+No hard token cap. Constraint is Section 2 (one kind of information per
+chunk), not length.
+
+Soft flag: chunks exceeding ~150–200 words warrant a second look — likely
+signals a chunk that should have split at a content-kind boundary.
+
+---
+
+## 5. Table-conversion rule
+
+**Trigger:** 3+ instances of a consistent-field mapping (item→rule,
+category→treatment, term→form), regardless of source presentation
+(prose, list, or already tabular).
+
+**Canonical shapes** (reuse, do not invent per-table schemas):
+
+| Shape | Columns |
+|---|---|
+| Mapping | `Item \| Rule \| Notes` |
+| Applicability | `Category \| Applies to \| Description` |
+
+**Scope preservation:** table conversion must not upgrade an open-ended
+source list ("...and so on") to `scope: exhaustive`. Tabularizing an
+illustrative list keeps `scope: illustrative`.
+
+**Never tabularize:** sequential or conditional content — content
+encoding order-dependence or a decision tree (when to use `.SS` vs. a
+custom heading; alphabetization rules for ERRORS). Table format strips
+the relationship that is the actual content.
+
+---
+
+## 6. Per-section transform table
+
+Applies uniformly to all 20 canonical man-page sections. Every routine
+runs against every source page unconditionally; absence is a normal
+return value, not an error condition.
+
+| Section | KIND | Transform |
+|---|---|---|
+| NAME | `definition` | Verbatim, single chunk, never split. `scope: exhaustive`. |
+| SYNOPSIS | `reference` | Code-fenced, structure preserved literally. One chunk per feature-test-macro variant if applicable. `scope: exhaustive`. |
+| CONFIGURATION | `prose` / `rule-table` | Prose by default; table only if 3+ configurable items (Section 5 rule). |
+| DESCRIPTION | mixed, split by content | Split at topic shift → `prose` chunks. Embedded flag/option enumerations → separate `rule-table` chunk. |
+| OPTIONS | `rule-table` | `Option \| Effect \| Notes`. Split into self-contained option groups if exceeding granularity guidance. |
+| EXIT STATUS | `rule-table` | `Value \| Meaning`. `scope: exhaustive`. |
+| RETURN VALUE | `prose` / `rule-table` | Contrastive statement default; table for 3+ cases. |
+| ERRORS | `rule-table` | `Errno \| Condition \| Notes`. Preserve duplicate errno rows per distinct condition — do not deduplicate by name. Alphabetical order. `authority: normative`. |
+| ENVIRONMENT | `prose` / `rule-table` | Threshold per Section 5. |
+| FILES | `prose` / `rule-table` | Threshold per Section 5. |
+| VERSIONS | `rule-table` | `Version \| Change`, chronological row order. |
+| ATTRIBUTES | `rule-table` | `Interface \| Attribute \| Value`. `authority: normative`, `scope: exhaustive`. |
+| CONFORMING TO | `prose` | Short list, period-terminated. Table only if per-standard caveats reach 3+. |
+| NOTES | `prose` | Split at topic shift. Named subsections (Linux Notes, Glibc Notes) = separate chunks. |
+| BUGS | `reference-list` / `rule-table` | List if short; table if 3+ consistent bug→status entries. |
+| EXAMPLES | `example` | Code + explanation never split across chunks. Shell session logs preserve bold-input/plain-output distinction explicitly. `scope: illustrative` always. `authority` per source of the example. |
+| AUTHORS | `pointer` | `AUTHORS: See man {section} {page}.` `scope: exhaustive`, `authority: editorial`. |
+| REPORTING BUGS | `pointer` | Same template. |
+| COPYRIGHT | `pointer` | Same template. |
+| SEE ALSO | *(not a body chunk)* | Populates `related:` frontmatter field directly from source. Zero transformation judgment. |
+
+---
+
+## 7. Manifest chunk
+
+One per document. Not a per-section stub. Records section presence/
+absence as a single fact set.
+
+```
+Sections present: NAME, SYNOPSIS, DESCRIPTION, OPTIONS, EXIT STATUS, SEE ALSO
+Sections absent from source: CONFIGURATION, RETURN VALUE, ERRORS, ENVIRONMENT,
+FILES, VERSIONS, ATTRIBUTES, CONFORMING TO, NOTES, BUGS, EXAMPLES, AUTHORS,
+REPORTING BUGS, COPYRIGHT
+```
+
+`kind: manifest`, `scope: exhaustive`. Replaces the need for absent-
+section stubs (10–14 near-zero-value chunks per page at 1/14th the cost).
+
+---
+
+## 8. Pipeline shape
+
+**Scan (deterministic, no model):** regex/parser match against the
+canonical 20-name list. Produces manifest data. `.SH` markers are
+unambiguous; no reasoning required for standard headings.
+
+**Orchestrator (7B-tier reasoning, e.g. Mistral):** routing decisions
+only — custom/nonstandard headings, threshold judgment on borderline
+table-conversion cases, splitting mixed-kind sections (DESCRIPTION with
+embedded tables). Styleguide (this document) is its reference-of-truth.
+Not used for section presence/absence — that's the deterministic scan's
+job.
+
+**Extraction (single agent, 20 prompt-library entries):** one
+deterministic extractor agent; each of the 20 sections is a prompt
+template in `/var/lib/cervello-elettrico/prompts/`, not a separate agent
+identity. System-user isolation governs privilege boundaries (filesystem/
+network scope), not per-task identity — all 20 transforms share an
+identical permission profile, so one agent suffices.
+
+**Retrieval (Qwen 0.6B, drone-tier):** consumes finished chunks only.
+Not involved in scan, routing, or extraction — those require more context
+and reasoning depth than 0.6B provides.
+
+---
+
+## 9. Voice (AI-corpus variant — swap point for OHIOD sibling)
+
+AI-primary. Human-legible as a byproduct of being plain markdown, not
+as a design goal. Non-binary text is the only legibility floor; no
+tradeoff toward friendlier phrasing at any point.
+
+- No rhetorical framing ("it's worth noting," "as mentioned above")
+- No tutorial voice — match man-pages(7)'s own register, don't soften it
+- `authority: editorial` chunks match `authority: normative` chunks in
+  density; the frontmatter field carries the provenance distinction, not tone
+- Definitions: bare contrastive statements, not explanations
+- Pointer-stub sections: template only, zero padding
+  (`AUTHORS: See man 7 man-pages.` — not "not reproduced in this corpus")
+
+---
+
+## Version history
+
+| Version | Date | Notes |
+|---|---|---|
+| v1.0 | July 2026 | Initial AI-corpus styleguide. Derived from man-pages(7) reformatting design session. Pending: RIS retrofit, beta testing, OHIOD sibling. |