docs: bootstrap specs ground truth (#5794)

* docs(specs): restore bootstrap after dev rewrite

* docs(specs): remove runtime inventory snapshot

* docs(specs): reconcile current dev truth

* docs(specs): document scheduled task actions as an owner-attribution source

Owner Attribution covered cookie, bearer-token and internal-loopback
requests. Scheduled task actions are a fourth source and behave
differently: _execute_action passes owner=task.owner off the stored
ScheduledTask row, so no request and no resolved principal are in
flight, and route-level require_user() never runs.

Webhook triggers are the sharp case. They are unauthenticated by
design with the token as the only credential and execute under the
stored task.owner.

Paths cite routes/task/task_routes.py, the canonical location after
the task subpackage move (#6081); routes/task_routes.py on current dev
is the backward-compat shim.

* docs(specs): add chained tasks to the trigger list, refresh dev stamp

Review feedback from RaresKeY on the previous commit.

"Every trigger path" was too broad: success-chained tasks are another
path into _execute_action. Added them with their own citation, and
noted that chaining additionally requires the target task to share
task.owner and rejects cycles, which is stricter than the trigger-side
checks. Softened the lead-in to "these trigger paths".

Line 56 still pointed at routes/task_routes.py for webhook credential
validation. That path is the backward-compat shim on current dev after
the task subpackage move (#6081); repointed to the canonical
routes/task/task_routes.py.

Stamp moved to dev@2a6b09b. Inspection backing that bump was scoped:
every file path cited in this spec was mechanically checked to resolve
on 2a6b09b, and every file:line in the Owner Attribution additions was
read against it. Behavioral claims elsewhere in the file were not
re-audited.

* docs(specs): correct SECURE_COOKIES description to match current behavior

Third of the stale details RaresKeY enumerated. The cookie section
described SECURE_COOKIES as purely opt-in, which stopped being true.

_secure_cookie() (routes/auth_routes.py:89) treats an explicit true or
false as authoritative and derives the Secure attribute from the
request otherwise, including when the variable is unset and when
docker-compose injects it present-but-empty. Either the connection
scheme or the first X-Forwarded-Proto hop being https is enough.

* docs(specs): refresh current dev truth

---------

Co-authored-by: StressTestor <212606152+StressTestor@users.noreply.github.com>

.env.example

@@ -76,12 +76,24 @@ SEARXNG_INSTANCE=http://localhost:8080
 # Change this if another local service already uses 7000 (macOS AirPlay often does).
 # APP_PORT=7000
 
+# Optional HTTP address advertised in companion/mobile pairing codes. Set this
+# when Docker would otherwise advertise a container address or loopback. Use a
+# LAN or Tailscale IPv4 address, a single-label hostname, or an mDNS *.local
+# name that the phone can reach. HTTPS and public hostnames are not supported
+# by the current companion client. Do not include credentials, a path, query,
+# or fragment.
+# COMPANION_BASE_URL=http://192.168.1.50:7000
+
 # Development-only auth bypass for loopback requests.
 # Keep false for Docker, LAN, reverse proxy, and any shared deployment.
 # LOCALHOST_BYPASS=false
 
-# Mark session cookies Secure. Set true when Odysseus is served through HTTPS
+# Mark session cookies Secure. Left unset, this follows the request scheme:
-# by a trusted reverse proxy or private access gateway.
+# an HTTPS login gets a Secure cookie, a plain-HTTP one does not. Set true to
+# force it on, or false to force it off while you still serve plain HTTP.
+# Upgrading: this used to default to false. Drop a leftover SECURE_COOKIES=false
+# from your .env unless you still need that escape hatch — it keeps HTTPS logins
+# on a non-Secure cookie.
 # SECURE_COOKIES=true
 
 # Optional: pre-seed the first admin password during setup.
@@ -151,6 +163,21 @@ SEARXNG_INSTANCE=http://localhost:8080
 # Local HTTP setups may use the callback URL inferred by the application.
 # GOOGLE_OAUTH_REDIRECT_URI=https://your-domain.com/api/email/oauth/google/callback
 
+# Origin the MCP OAuth callback is sent back to, for remote (Streamable HTTP)
+# MCP servers that register it dynamically. Defaults to http://localhost:$APP_PORT,
+# which is right only when you reach Odysseus directly on that port. Set it for
+# HTTPS, reverse-proxy, hosted, and Docker installs — inside the container the
+# app always listens on 7000 and cannot see the host port map, so the default is
+# wrong there whenever APP_PORT is not 7000.
+#
+# Not for Google MCP servers. Those use Desktop App credentials, and Google only
+# accepts loopback redirect URIs for that client type, so a public origin here is
+# rejected with redirect_uri_mismatch. Leave it unset for a Google-only install:
+# the loopback default is what Google wants, and remote users finish through the
+# paste-back page, which never has to load the redirect.
+# https://developers.google.com/identity/protocols/oauth2/native-app
+# OAUTH_REDIRECT_BASE_URL=https://your-domain.com
+
 # ============================================================
 # Misc
 # ============================================================

.gitattributes

@@ -15,6 +15,13 @@ docker/entrypoint.sh text eol=lf
 *.cmd text eol=crlf
 *.bat text eol=crlf
 
+# Vendored third-party bundles in static/lib/ are published minified artifacts
+# and must stay byte-identical to what npm ships — stripping trailing whitespace
+# to satisfy `git diff --check` would desync them from the upstream release. Turn
+# the whitespace check off for that tree instead, and keep the bundles out of
+# GitHub's language statistics.
+static/lib/** -whitespace linguist-vendored
+
 # Binary assets — never normalize.
 *.png  binary
 *.jpg  binary

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -26,6 +26,18 @@ body:
         - label: I am running the latest code from the `dev` branch (the default branch you get on clone, where fixes land first) and the bug still reproduces there. Please `git pull` the latest `dev` before filing.
           required: true
 
+  - type: input
+    id: revision
+    attributes:
+      label: Odysseus Revision
+      description: |
+        From the repository root (on the host when using Docker), run
+        `git show -s --abbrev=12 --format='%h (%cs)' HEAD`
+        and paste the output exactly.
+      placeholder: "1fef4929cf1d (2026-08-11)"
+    validations:
+      required: true
+
   - type: dropdown
     id: install-method
     attributes:

.github/pull_request_template.md

@@ -28,6 +28,7 @@ Fixes #
 - [ ] This PR targets `dev`
 - [ ] My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
 - [ ] I actually ran the app (`docker compose up` or `uvicorn app:app`) and verified the change works end-to-end. Type-checks and unit tests are not enough.
+- [ ] I did not run the app/runtime validation and stated that gap in **How to Test**. Leave this unchecked when the app-run box above is checked.
 
 ## How to Test
 

.github/scripts/check-issue-description.js

@@ -41,6 +41,14 @@ module.exports = async ({ github, context, core }) => {
       break;
 
     case 'bug': {
+      const revisionText = section('Odysseus Revision');
+      if (!/^[0-9a-f]{12} \(\d{4}-\d{2}-\d{2}\)$/i.test(revisionText)) {
+        failures.push(
+          '**Odysseus Revision** — paste the 12-character commit SHA and date, ' +
+          'for example `1fef4929cf1d (2026-08-11)`',
+        );
+      }
+
       if (!section('Install Method')) {
         failures.push('**Install Method** — select how you installed Odysseus');
       }

.github/scripts/check-pr-description.js

@@ -21,11 +21,11 @@ module.exports = async ({ github, context, core }) => {
     return strip(m?.[0].replace(new RegExp(`#+\\s+${heading}`, 'i'), '') ?? '');
   }
 
-  const problems = [];
+  const descriptionProblems = [];
 
   // 1. Summary must be filled in.
   if (section('Summary').length < 20) {
-    problems.push('**Summary** is empty or too short — describe what changed and why.');
+    descriptionProblems.push('**Summary** is empty or too short — describe what changed and why.');
   }
 
   // 2. Linked Issue must reference a real issue. Accept a bare #NNN, a closing
@@ -34,18 +34,18 @@ module.exports = async ({ github, context, core }) => {
   const linkedSection = section('Linked Issue');
   const hasIssueRef = /#\d+\b/.test(linkedSection) || /\/issues\/\d+/.test(linkedSection);
   if (!linkedSection || !hasIssueRef) {
-    problems.push('**Linked Issue** — add a reference like `Fixes #NNN`, a bare `#NNN`, or a link to the issue.');
+    descriptionProblems.push('**Linked Issue** — add a reference like `Fixes #NNN`, a bare `#NNN`, or a link to the issue.');
   }
 
   // 3. At least one Type of Change box must be checked.
   const typeBlock = body.match(/##\s+Type of Change[\s\S]*?(?=\n##\s|$)/i)?.[0] ?? '';
   if (!/- \[x\]/i.test(typeBlock)) {
-    problems.push('**Type of Change** — check at least one box.');
+    descriptionProblems.push('**Type of Change** — check at least one box.');
   }
 
   // 4. Duplicate-search checklist item must be checked.
   if (!/- \[x\] I searched/i.test(body)) {
-    problems.push('**Checklist** — check the duplicate-search box to confirm you searched existing issues and PRs.');
+    descriptionProblems.push('**Checklist** — check the duplicate-search box to confirm you searched existing issues and PRs.');
   }
 
   // 5. How to Test must contain enough real detail for a reviewer to act on.
@@ -53,7 +53,83 @@ module.exports = async ({ github, context, core }) => {
   //    code block — so we only require non-trivial content, not a specific shape.
   const howTo = section('How to Test');
   if (howTo.length < 30) {
-    problems.push('**How to Test** — explain how a reviewer can verify this change. Numbered steps, the commands you ran, or a short code block all work — give a sentence or two of real detail (not just "tested locally").');
+    descriptionProblems.push('**How to Test** — explain how a reviewer can verify this change. Numbered steps, the commands you ran, or a short code block all work — give a sentence or two of real detail (not just "tested locally").');
+  }
+
+  // Classify paths from GitHub's API. This workflow runs in the privileged base
+  // context, so it must never check out or execute code from the PR branch.
+  const changedFiles = await github.paginate(github.rest.pulls.listFiles, {
+    owner, repo, pull_number: prNum, per_page: 100,
+  });
+  const changedPaths = changedFiles.map(file => file.filename);
+
+  function isUiSensitivePath(filename) {
+    const path = filename.toLowerCase();
+    return path.startsWith('static/')
+      || path.startsWith('templates/')
+      || /\.(?:html?|css|svg)$/.test(path);
+  }
+
+  function isDocsOnlyPath(filename) {
+    const path = filename.toLowerCase();
+    return /\.(?:md|mdx|rst|adoc|txt)$/.test(path)
+      || (path.startsWith('docs/') && !isUiSensitivePath(path));
+  }
+
+  function isRuntimeSensitivePath(filename) {
+    const path = filename.toLowerCase();
+    if (isUiSensitivePath(path)) return false;
+    if (path.startsWith('tests/') || path.startsWith('.github/')) return false;
+    return /^(?:app\.py|routes\/|services\/|src\/|core\/|mcp_servers\/|scripts\/|docker\/)/.test(path)
+      || /^(?:dockerfile|docker-compose.*\.ya?ml|requirements(?:-optional)?\.txt|pyproject\.toml|setup\.py)$/.test(path)
+      || /\.(?:py|sh|ps1|bat)$/.test(path);
+  }
+
+  let classification = 'tooling';
+  if (changedPaths.some(isUiSensitivePath)) {
+    classification = 'UI-sensitive';
+  } else if (changedPaths.some(isRuntimeSensitivePath)) {
+    classification = 'backend/runtime';
+  } else if (changedPaths.length > 0 && changedPaths.every(isDocsOnlyPath)) {
+    classification = 'docs-only';
+  }
+
+  const appRan = /- \[x\]\s+I actually ran the app\b/i.test(body);
+  const appNotRun = /- \[x\]\s+I did not run the app\/runtime validation\b/i.test(body);
+  // Anchor on the wording, not the template's emphasis: a ticked box the author
+  // retyped without the surrounding ** renders identically on the PR page, so
+  // treating it as unchecked is invisible from their side. Matches the two
+  // attestations above, which already ignore formatting.
+  const screenshotChecked = /- \[x\]\s+[*_]{0,2}Screenshot or short clip[*_]{0,2}/i.test(body);
+  const screenshotSection = section('Screenshots / clips');
+  const hasVisualEvidence = /!\[[^\]]*\]\([^)]+\)|<(?:img|video|source)\b[^>]*(?:src|href)=|https?:\/\/[^\s)]+/i.test(screenshotSection);
+  const evidenceGaps = [];
+  let needsRuntimeValidation = false;
+  let needsVisualEvidence = false;
+
+  if (classification === 'backend/runtime' || classification === 'UI-sensitive') {
+    if (appRan && appNotRun) {
+      needsRuntimeValidation = true;
+      evidenceGaps.push('The app-run and explicit not-run boxes are both checked. Select the one state that is true.');
+    } else if (!appRan) {
+      needsRuntimeValidation = true;
+      if (appNotRun) {
+        evidenceGaps.push('The author explicitly reports that app/runtime validation was not performed.');
+      } else {
+        evidenceGaps.push('App/runtime validation is not author-attested. Check the run box only after running it, or check the explicit not-run box and describe the gap.');
+      }
+    }
+  }
+
+  if (classification === 'UI-sensitive') {
+    if (!screenshotChecked) {
+      needsVisualEvidence = true;
+      evidenceGaps.push('The screenshot/clip checkbox is not checked for this UI-sensitive change.');
+    }
+    if (!hasVisualEvidence) {
+      needsVisualEvidence = true;
+      evidenceGaps.push('The Screenshots / clips section does not contain an actual attachment or link.');
+    }
   }
 
   // ── Comment ──────────────────────────────────────────────────────────────
@@ -62,22 +138,43 @@ module.exports = async ({ github, context, core }) => {
   });
   const existing = comments.find(c => (c.body ?? '').includes(MARKER));
 
-  if (problems.length === 0) {
+  if (descriptionProblems.length === 0 && evidenceGaps.length === 0) {
     if (existing) {
       await github.rest.issues.deleteComment({ owner, repo, comment_id: existing.id });
     }
   } else {
-    const commentBody = [
+    const commentLines = [MARKER];
-      MARKER,
+    if (descriptionProblems.length > 0) {
-      '⚠️ **PR description — action needed**',
+      commentLines.push(
-      '',
+        '⚠️ **PR description — action needed**',
-      'The following required sections are missing or incomplete. Please update the PR description to address them:',
+        '',
-      '',
+        'The following required sections are missing or incomplete. Please update the PR description to address them:',
-      problems.map(p => `- ${p}`).join('\n'),
+        '',
+        descriptionProblems.map(problem => `- ${problem}`).join('\n'),
+      );
+    } else {
+      commentLines.push(
+        '⚠️ **PR description is complete; validation evidence is still outstanding**',
+        '',
+        `Changed-file classification: **${classification}**.`,
+      );
+    }
+    if (evidenceGaps.length > 0) {
+      commentLines.push(
+        '',
+        '**Author-reported runtime / visual state**',
+        '',
+        evidenceGaps.map(gap => `- ${gap}`).join('\n'),
+        '',
+        'Checkboxes are author attestations. GitHub Actions results remain the execution evidence for CI; this check does not prove that a local command ran.',
+      );
+    }
+    commentLines.push(
       '',
       '---',
-      '_This comment is deleted automatically once all sections are complete._',
+      '_This comment updates automatically when the description or changed files change._',
-    ].join('\n');
+    );
+    const commentBody = commentLines.join('\n');
 
     if (existing) {
       await github.rest.issues.updateComment({ owner, repo, comment_id: existing.id, body: commentBody });
@@ -97,34 +194,47 @@ module.exports = async ({ github, context, core }) => {
       return true;
     } catch (e) {
       if (e.status === 404) return false;
+      if (e.status === 403) {
+        core.warning(`Could not inspect label "${name}" — token lacks label read access; skipping.`);
+        return false;
+      }
       throw e;
     }
   }
 
-  async function swapLabel(num, add, remove) {
+  async function setLabel(name, wanted) {
-    if (await labelExists(add)) {
+    if (wanted && await labelExists(name)) {
       try {
-        await github.rest.issues.addLabels({ owner, repo, issue_number: num, labels: [add] });
+        await github.rest.issues.addLabels({ owner, repo, issue_number: prNum, labels: [name] });
       } catch (e) {
         // Fail soft on a token that can't write labels so a label permission
         // problem never masks the actual description verdict.
-        if (e.status !== 403) throw e;
+        if (e.status !== 403 && e.status !== 404) throw e;
-        core.warning(`Could not add "${add}" — token lacks label write here; skipping.`);
+        core.warning(`Could not add "${name}" — label is unavailable or the token lacks label write access; skipping.`);
       }
+    } else if (wanted) {
+      core.warning(`Label "${name}" does not exist in the repo — skipping. Create it once to enable labelling.`);
     } else {
-      core.warning(`Label "${add}" does not exist in the repo — skipping. Create it once to enable labelling.`);
+      try {
-    }
+        await github.rest.issues.removeLabel({ owner, repo, issue_number: prNum, name });
-    try {
+      } catch (e) {
-      await github.rest.issues.removeLabel({ owner, repo, issue_number: num, name: remove });
+        if (e.status !== 404 && e.status !== 410 && e.status !== 403) throw e;
-    } catch (e) {
+      }
-      if (e.status !== 404 && e.status !== 410 && e.status !== 403) throw e;
     }
   }
 
-  if (problems.length === 0) {
+  const descriptionComplete = descriptionProblems.length === 0;
-    await swapLabel(prNum, 'ready for review', 'needs work');
+  const evidenceComplete = evidenceGaps.length === 0;
-  } else {
+  const isDraft = Boolean(context.payload.pull_request.draft);
-    await swapLabel(prNum, 'needs work', 'ready for review');
+  await setLabel(
-    core.setFailed(`PR description has ${problems.length} issue(s) — see bot comment for details.`);
+    'ready for review',
+    descriptionComplete && evidenceComplete && !isDraft,
+  );
+  await setLabel('needs work', !descriptionComplete);
+  await setLabel('needs runtime validation', needsRuntimeValidation);
+  await setLabel('needs visual evidence', needsVisualEvidence);
+
+  if (!descriptionComplete) {
+    core.setFailed(`PR description has ${descriptionProblems.length} issue(s) — see bot comment for details.`);
   }
 };

.github/workflows/ci.yml

@@ -21,7 +21,7 @@ jobs:
     runs-on: ubuntu-latest
     continue-on-error: true
     steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           fetch-depth: 0
           persist-credentials: false
@@ -73,10 +73,10 @@ jobs:
     name: Python syntax (compileall)
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           persist-credentials: false
-      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6.2.0
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97  # v7.0.0
         with:
           python-version: "3.11"
       # Byte-compile sources — catches syntax errors without installing deps.
@@ -86,10 +86,10 @@ jobs:
     name: JS syntax (node --check)
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           persist-credentials: false
-      - uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e  # v6.4.0
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020  # v7.0.0
         with:
           node-version: "20"
       # Syntax-check our own JS (skip vendored libs in static/lib).
@@ -105,7 +105,7 @@ jobs:
     runs-on: ubuntu-latest
     # Make Python test validation authoritative for the configured scope.
     steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           fetch-depth: 0
           persist-credentials: false
@@ -132,7 +132,7 @@ jobs:
             echo "docs_only=false" >> "$GITHUB_OUTPUT"
           fi
 
-      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6.2.0
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97  # v7.0.0
         if: steps.docs-check.outputs.docs_only != 'true'
         with:
           python-version: "3.11"

.github/workflows/codeql.yml

@@ -27,15 +27,15 @@ jobs:
         language: [actions, javascript-typescript, python]
     steps:
       - name: Checkout
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           persist-credentials: false
       - name: Initialize CodeQL
-        uses: github/codeql-action/init@8aad20d150bbac5944a9f9d289da16a4b0d87c1e  # v4.36.2
+        uses: github/codeql-action/init@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd  # v4.37.7
         with:
           languages: ${{ matrix.language }}
           build-mode: none
       - name: Perform CodeQL Analysis
-        uses: github/codeql-action/analyze@8aad20d150bbac5944a9f9d289da16a4b0d87c1e  # v4.36.2
+        uses: github/codeql-action/analyze@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd  # v4.37.7
         with:
           category: "/language:${{ matrix.language }}"

.github/workflows/container-scan.yml

@@ -37,12 +37,12 @@ jobs:
       contents: read
     steps:
       - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           persist-credentials: false
 
       - name: Lint Dockerfile
-        uses: hadolint/hadolint-action@2332a7b74a6de0dda2e2221d575162eba76ba5e5  # v3.3.0
+        uses: hadolint/hadolint-action@2a66e89f53d0771bb131a7fa31f3136336094aa6  # v3.4.0
         with:
           dockerfile: Dockerfile
           # DL3008: pinning apt package versions is impractical on a -slim base

.github/workflows/container-trivy.yml

@@ -52,17 +52,17 @@ jobs:
       contents: read
     steps:
       - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           persist-credentials: false
 
       - name: Set up Buildx
-        uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5  # v4.1.0
+        uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e  # v4.3.0
 
       # Build without pushing so a broken Dockerfile is caught here, and the
       # exact image we ship is what gets scanned.
       - name: Build image
-        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf  # v7.2.0
+        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a  # v7.3.0
         with:
           context: .
           push: false
@@ -93,15 +93,15 @@ jobs:
       security-events: write  # upload SARIF to the Security tab
     steps:
       - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           persist-credentials: false
 
       - name: Set up Buildx
-        uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5  # v4.1.0
+        uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e  # v4.3.0
 
       - name: Build image
-        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf  # v7.2.0
+        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a  # v7.3.0
         with:
           context: .
           push: false
@@ -119,7 +119,7 @@ jobs:
           TRIVY_DB_REPOSITORY: ghcr.io/aquasecurity/trivy-db:2
 
       - name: Upload Trivy results
-        uses: github/codeql-action/upload-sarif@8aad20d150bbac5944a9f9d289da16a4b0d87c1e  # v4.36.2
+        uses: github/codeql-action/upload-sarif@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd  # v4.37.7
         with:
           sarif_file: trivy-results.sarif
           category: trivy-image

.github/workflows/dependency-review.yml

@@ -36,7 +36,7 @@ jobs:
       contents: read
     steps:
       - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           persist-credentials: false
 
@@ -55,12 +55,12 @@ jobs:
       contents: read
     steps:
       - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           persist-credentials: false
 
       - name: Set up Python
-        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6.2.0
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97  # v7.0.0
         with:
           python-version: '3.12'
 

.github/workflows/docker-publish.yml

@@ -45,20 +45,20 @@ jobs:
             arch: arm64
             runner: ubuntu-24.04-arm
     steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           persist-credentials: false
       - name: Set up Buildx
-        uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5  # v4.1.0
+        uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e  # v4.3.0
       - name: Log in to GHCR
-        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee  # v4.2.0
+        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f  # v4.6.0
         with:
           registry: ${{ env.REGISTRY }}
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
       - name: Build and push by digest
         id: build
-        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf  # v7.2.0
+        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a  # v7.3.0
         with:
           context: .
           platforms: ${{ matrix.platform }}
@@ -86,7 +86,7 @@ jobs:
       contents: read
       packages: write
     steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           persist-credentials: false
       - name: Read APP_VERSION + short sha
@@ -103,16 +103,16 @@ jobs:
           pattern: digest-*
           merge-multiple: true
       - name: Set up Buildx
-        uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5  # v4.1.0
+        uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e  # v4.3.0
       - name: Log in to GHCR
-        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee  # v4.2.0
+        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f  # v4.6.0
         with:
           registry: ${{ env.REGISTRY }}
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
       - name: Compute tags
         id: meta
-        uses: docker/metadata-action@80c7e94dd9b9319bd5eb7a0e0fe9291e23a2a2e9  # v6.1.0
+        uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302  # v6.2.0
         with:
           images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
           tags: |

.github/workflows/issue-description-check.yml

@@ -14,7 +14,7 @@ jobs:
     # Skip bots (Dependabot, release-drafter, etc.)
     if: ${{ github.event.issue.user.type != 'Bot' }}
     steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           sparse-checkout: .github/scripts
           persist-credentials: false

.github/workflows/pr-description-check.yml

@@ -5,7 +5,11 @@ on:
   # works on fork PRs. Safe here: the checkout pins to the base branch (no fork
   # code runs) and the scripts only read context.payload and call the GitHub API.
   pull_request_target:  # zizmor: ignore[dangerous-triggers]
-    types: [opened, edited, synchronize, reopened, ready_for_review]
+    types: [opened, edited, synchronize, reopened, ready_for_review, converted_to_draft]
+
+concurrency:
+  group: pr-description-${{ github.event.pull_request.number }}
+  cancel-in-progress: true
 
 # Default-deny at the workflow level; each job opts into only the scopes it needs.
 # Note: modifying a PR's labels/comments needs pull-requests:write even though the
@@ -23,7 +27,7 @@ jobs:
     # Skip bots: they open PRs programmatically and have their own process.
     if: github.event.pull_request.user.type != 'Bot'
     steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           ref: ${{ github.base_ref }}
           sparse-checkout: .github/scripts
@@ -59,12 +63,14 @@ jobs:
 
   check-mergeable:
     name: Flag unmergeable PRs
+    needs: check-description
     runs-on: ubuntu-latest
     permissions:
       pull-requests: write
       issues: write
-    # Skip bots: they open PRs programmatically and have their own process.
+    # Run after description validation failures, but never from an obsolete
-    if: github.event.pull_request.user.type != 'Bot'
+    # workflow run canceled by a newer PR event.
+    if: ${{ !cancelled() && github.event.pull_request.user.type != 'Bot' }}
     steps:
       - uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3  # v9.0.0
         with:

.github/workflows/secret-scan.yml

@@ -35,7 +35,7 @@ jobs:
       contents: read
     steps:
       - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           # Full history so a secret committed in an earlier commit (and later
           # deleted) is still caught -- deletion does not remove it from Git.

.github/workflows/workflow-security.yml

@@ -36,7 +36,7 @@ jobs:
       contents: read
     steps:
       - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           persist-credentials: false
 
@@ -61,12 +61,12 @@ jobs:
       contents: read
     steps:
       - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
           persist-credentials: false
 
       - name: Set up Python
-        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6.2.0
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97  # v7.0.0
         with:
           python-version: '3.12'
 

ACKNOWLEDGMENTS.md

@@ -65,6 +65,16 @@ Vendored in `static/lib/` and served directly:
 | [jsPDF](https://github.com/parallax/jsPDF) (bundled in html2pdf) | PDF generation | MIT |
 | [html2canvas](https://github.com/niklasvh/html2canvas) (bundled in html2pdf) | DOM → canvas rasterization | MIT |
 | [node-qrcode](https://github.com/soldair/node-qrcode) (`qrcode.min.js`) | QR-code rendering (2FA setup) | MIT |
+| [KaTeX](https://github.com/KaTeX/KaTeX) v0.16.22 (`katex/katex.min.{js,css}` + `katex/fonts/*.woff2`) | Math typesetting | MIT ([`licenses/KaTeX-MIT-LICENSE.txt`](licenses/KaTeX-MIT-LICENSE.txt)) |
+| [Mermaid](https://github.com/mermaid-js/mermaid) v11.16.1 (`mermaid.min.js`) | Diagrams from text | MIT ([`licenses/Mermaid-MIT-LICENSE.txt`](licenses/Mermaid-MIT-LICENSE.txt)) |
+
+KaTeX and Mermaid are loaded on first use by `static/js/markdown.js` rather than
+from `index.html`, so a session that renders no math and no diagram never fetches
+either. Only the `.woff2` KaTeX fonts are shipped, matching `static/fonts/`; the
+`.woff` and `.ttf` variants its stylesheet also lists are never requested by a
+browser that supports `woff2`. The bundles are the published npm artifacts,
+unmodified — `.gitattributes` turns the whitespace check off for `static/lib/`
+so they can stay byte-identical to upstream.
 
 ## Front-end libraries loaded at runtime (CDN)
 
@@ -72,8 +82,6 @@ Referenced from `cdn.jsdelivr.net` / `cdnjs.cloudflare.com` at runtime — not v
 
 | Library | Purpose | License |
 |---|---|---|
-| [KaTeX](https://github.com/KaTeX/KaTeX) 0.16.22 | Math typesetting | MIT |
-| [Mermaid](https://github.com/mermaid-js/mermaid) 11 | Diagrams from text | MIT |
 | [Pyodide](https://github.com/pyodide/pyodide) 0.27.5 | In-browser Python runtime | MPL-2.0 |
 | [PDFObject](https://github.com/pipwerks/PDFObject) 2.1.1 | Inline PDF embedding | MIT |
 

README.md

@@ -59,15 +59,20 @@ Help is welcome. The best entry points are fresh-install testing, provider setup
 
 ## Security
 
-Odysseus is a self-hosted workspace with powerful local tools. Keep auth enabled, keep private data out of Git, and do not expose raw model/service ports publicly. Deployment details are in the [setup guide](docs/setup.md#security-notes).
+Odysseus is a self-hosted workspace with powerful local tools. Keep auth enabled, keep private data out of Git, and do not expose raw model/service ports publicly.
+
+- Keep `AUTH_ENABLED=true` for any network-accessible deployment.
+- Keep `LOCALHOST_BYPASS=false` outside local development.
+
+Deployment details are in the [setup guide](docs/setup.md#security-notes).
 
 ## Star History
 
-<a href="https://www.star-history.com/?repos=odysseus-dev%2Fodysseus&type=date&legend=top-left">
+<a href="https://star-history.dera.page/#odysseus-dev/odysseus&type=date&legend=top-left">
  <picture>
-   <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=odysseus-dev/odysseus&type=date&theme=dark&legend=top-left" />
+   <source media="(prefers-color-scheme: dark)" srcset="https://star-history.dera.page/svg?repos=odysseus-dev/odysseus&type=date&theme=dark&legend=top-left" />
-   <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=odysseus-dev/odysseus&type=date&legend=top-left" />
+   <source media="(prefers-color-scheme: light)" srcset="https://star-history.dera.page/svg?repos=odysseus-dev/odysseus&type=date&legend=top-left" />
-   <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=odysseus-dev/odysseus&type=date&legend=top-left" />
+   <img alt="Star History Chart" src="https://star-history.dera.page/svg?repos=odysseus-dev/odysseus&type=date&legend=top-left" />
  </picture>
 </a>
 

SECURITY.md

@@ -10,7 +10,7 @@ Security fixes are handled on the default branch until formal releases are cut.
 
 - Keep `AUTH_ENABLED=true` for any network-accessible deployment.
 - Keep `LOCALHOST_BYPASS=false` outside local development.
-- Set `SECURE_COOKIES=true` when Odysseus is served through HTTPS by a trusted reverse proxy or private access gateway.
+- Leave `SECURE_COOKIES` unset unless you need to override it: session cookies are marked `Secure` whenever the request arrives over HTTPS. Set `SECURE_COOKIES=true` to force it on (for a proxy Odysseus cannot see the scheme of), or `SECURE_COOKIES=false` to force it off while you still serve plain HTTP alongside HTTPS.
 - Use HTTPS when exposing the app beyond localhost.
 - Put the authenticated Odysseus web/API entrypoint behind a trusted reverse proxy or private access layer such as Cloudflare Access, Tailscale, or a VPN.
 - Keep ChromaDB, SearXNG, ntfy, Ollama, vLLM, llama.cpp, databases, and raw model/provider APIs internal-only.

THREAT_MODEL.md

@@ -37,7 +37,7 @@ Non-admin defaults are in `core/auth.py:DEFAULT_PRIVILEGES`. Tool enforcement is
 
 - **Sessions:** bcrypt passwords, 7-day session tokens stored atomically in `data/sessions.json` via `core/atomic_io.py`.
 - **2FA:** TOTP with 8 single-use backup codes. Verified after password check, before session issuance.
-- **Reserved usernames:** `internal-tool`, `api`, `demo`, `system` cannot be registered or renamed into. Defined in `core/auth.py:RESERVED_USERNAMES`.
+- **Reserved usernames:** request sentinels and the Default/Local storage owner cannot be registered or renamed into. Defined in `core/auth.py:RESERVED_USERNAMES`.
   - `internal-tool` is security-critical: `core/middleware.py:require_admin` treats any request where `request.state.current_user == "internal-tool"` as the in-process tool loopback and grants admin unconditionally. A real account with that name would silently pass every `require_admin` check.
 - **Orphan sessions:** `validate_token` re-checks that the user record still exists on every call. A deleted user's cookie is dropped on next request rather than continuing to authenticate.
 

app.py

@@ -67,7 +67,13 @@ from core.constants import (
     REQUEST_TIMEOUT, OPENAI_API_KEY, AUTH_FILE,
 )
 from core.database import SessionLocal, ApiToken
-from core.middleware import SecurityHeadersMiddleware, is_cors_preflight
+from core.middleware import (
+    SecurityHeadersMiddleware,
+    get_application_route_path,
+    is_cors_preflight,
+    path_is_route_or_child,
+    with_asgi_root_path,
+)
 from core.auth import AuthManager, normalize_known_username
 from core.exceptions import (
     SessionNotFoundError, InvalidFileUploadError,
@@ -78,6 +84,7 @@ import bcrypt as _bcrypt
 
 from src.app_helpers import abs_join, serve_html_with_nonce
 from src.generated_images import GENERATED_IMAGE_HEADERS, resolve_generated_image_path
+from src.owner_identity import auth_disabled
 from starlette.responses import RedirectResponse
 
 # ========= LOGGING =========
@@ -248,7 +255,7 @@ from routes.auth_routes import setup_auth_routes, SESSION_COOKIE
 
 auth_manager = AuthManager()
 app.state.auth_manager = auth_manager
-AUTH_ENABLED = os.getenv("AUTH_ENABLED", "true").lower() != "false"
+AUTH_ENABLED = not auth_disabled()
 LOCALHOST_BYPASS = os.getenv("LOCALHOST_BYPASS", "false").lower() == "true"
 if LOCALHOST_BYPASS:
     logger.warning("LOCALHOST_BYPASS is enabled, loopback requests bypass authentication. Do not expose this instance to a network.")
@@ -284,7 +291,7 @@ if AUTH_ENABLED:
     def _is_auth_exempt(path: str) -> bool:
         if path in AUTH_EXEMPT_EXACT:
             return True
-        if any(path.startswith(p) for p in AUTH_EXEMPT_PREFIXES):
+        if any(path_is_route_or_child(path, p) for p in AUTH_EXEMPT_PREFIXES):
             return True
         return any(p.match(path) for p in AUTH_EXEMPT_PATTERNS)
 
@@ -355,7 +362,7 @@ if AUTH_ENABLED:
 
     class AuthMiddleware(BaseHTTPMiddleware):
         async def dispatch(self, request: Request, call_next):
-            path = request.url.path
+            path = get_application_route_path(request.scope)
             # A genuine CORS preflight (OPTIONS + Access-Control-Request-Method)
             # carries no credentials by design and must reach CORSMiddleware to be
             # answered. AuthMiddleware is the outermost middleware, so gating the
@@ -399,7 +406,10 @@ if AUTH_ENABLED:
             if not auth_manager.is_configured:
                 # No users yet — redirect to login for first-time setup
                 if not path.startswith("/api/"):
-                    return RedirectResponse(url="/login", status_code=302)
+                    return RedirectResponse(
+                        url=with_asgi_root_path(request.scope, "/login"),
+                        status_code=302,
+                    )
                 return JSONResponse(status_code=401, content={"error": "Setup required"})
 
             # --- Bearer token auth (API tokens for external integrations) ---
@@ -461,7 +471,10 @@ if AUTH_ENABLED:
             if not auth_manager.validate_token(token):
                 if path.startswith("/api/"):
                     return JSONResponse(status_code=401, content={"error": "Not authenticated"})
-                return RedirectResponse(url="/login", status_code=302)
+                return RedirectResponse(
+                    url=with_asgi_root_path(request.scope, "/login"),
+                    status_code=302,
+                )
 
             # Attach current username to request state for downstream routes
             request.state.current_user = auth_manager.get_username_for_token(token)
@@ -630,13 +643,24 @@ app.include_router(auth_router)
 
 @app.post("/api/activity/heartbeat")
 async def activity_heartbeat():
-    from src.interactive_gate import mark_browser_activity
+    from src.interactive_gate import (
+        mark_browser_activity,
+        maybe_stop_background_tasks_for_heartbeat,
+    )
+
     await mark_browser_activity()
+
     async def _stop_background():
         try:
-            await task_scheduler.stop_background_tasks_for_foreground(reason="browser heartbeat")
+            await maybe_stop_background_tasks_for_heartbeat(
+                task_scheduler.stop_background_tasks_for_foreground
+            )
         except Exception:
-            logging.getLogger("app.foreground_gate").debug("heartbeat task stop failed", exc_info=True)
+            logging.getLogger("app.foreground_gate").debug(
+                "heartbeat task stop failed",
+                exc_info=True,
+            )
+
     asyncio.create_task(_stop_background())
     return {"ok": True}
 
@@ -760,7 +784,7 @@ from src.task_scheduler import TaskScheduler
 task_scheduler = TaskScheduler(session_manager)
 from src.event_bus import set_task_scheduler
 set_task_scheduler(task_scheduler)
-from routes.task_routes import setup_task_routes
+from routes.task.task_routes import setup_task_routes
 app.include_router(setup_task_routes(task_scheduler))
 
 from routes.assistant_routes import setup_assistant_routes

build-macos-app.sh

@@ -73,6 +73,10 @@ cat > "$APP/Contents/MacOS/$APP_NAME.tmpl" <<'LAUNCHER'
 INSTALL_DIR="__INSTALL_DIR__"
 PORT="__PORT__"
 URL="http://127.0.0.1:${PORT}"
+# uvicorn is started with --port below, but APP_PORT is what the app itself
+# reads when it needs to build a URL for this instance (internal_api_base(),
+# companion pairing, the MCP OAuth callback), so export it as well.
+export APP_PORT="$PORT"
 export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH"
 
 UVICORN="$INSTALL_DIR/venv/bin/uvicorn"

companion/pairing.py

@@ -6,11 +6,14 @@ units so the route layer stays thin and the logic is directly testable.
 
 from __future__ import annotations
 
+import ipaddress
 import json
 import os
+import re
 import secrets
 import socket
 import uuid
+from urllib.parse import urlsplit
 
 import bcrypt
 
@@ -20,6 +23,102 @@ PAIRING_VERSION = 1
 COMPANION_SCOPE = "chat"
 
 
+_COMPANION_IPV4_NETWORKS = tuple(
+    ipaddress.ip_network(cidr)
+    for cidr in (
+        "10.0.0.0/8",
+        "100.64.0.0/10",
+        "127.0.0.0/8",
+        "169.254.0.0/16",
+        "172.16.0.0/12",
+        "192.168.0.0/16",
+    )
+)
+_DNS_LABEL_RE = re.compile(r"[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\Z")
+
+
+def _valid_companion_client_host(host: str) -> bool:
+    """Match the host forms supported by the current v1 Expo client."""
+    if not host or len(host) > 253 or not host.isascii() or "%" in host:
+        return False
+
+    try:
+        address = ipaddress.ip_address(host)
+    except ValueError:
+        labels = host.split(".")
+        if any(not _DNS_LABEL_RE.fullmatch(label) for label in labels):
+            return False
+        if any(label.startswith("xn--") for label in labels):
+            return False
+        # WHATWG URL parsers treat a decimal or ``0x`` single-label hostname
+        # as an IPv4 number even though Python's strict ``ipaddress`` parser
+        # rejects that spelling.  The v1 client interpolates this host back
+        # into a URL, so accepting e.g. ``134744072`` would make the phone send
+        # its bearer token to public 8.8.8.8.  Keep DNS labels unambiguous.
+        if len(labels) == 1 and (
+            labels[0].isdigit()
+            or re.fullmatch(r"0x[0-9a-f]*", labels[0]) is not None
+        ):
+            return False
+        return len(labels) == 1 or (len(labels) >= 2 and labels[-1] == "local")
+
+    return isinstance(address, ipaddress.IPv4Address) and any(
+        address in network for network in _COMPANION_IPV4_NETWORKS
+    )
+
+
+def parse_companion_base_url(value: str) -> tuple[str, int]:
+    """Validate a v1 companion address and return its legacy (host, port).
+
+    The deployed client understands only HTTP plus a LAN-style host and port.
+    Reject anything outside that exact contract instead of advertising a URL
+    the client would reject, downgrade, or interpret differently.
+    """
+    if not isinstance(value, str) or not value:
+        raise ValueError("COMPANION_BASE_URL must be a canonical HTTP LAN origin")
+    if not value.isascii():
+        raise ValueError("COMPANION_BASE_URL must contain only ASCII characters")
+    if any(
+        ord(char) <= 32 or ord(char) == 127 or char in {"\\", "%"}
+        for char in value
+    ):
+        raise ValueError(
+            "COMPANION_BASE_URL contains a forbidden character"
+        )
+
+    try:
+        parsed = urlsplit(value)
+        port = parsed.port
+    except ValueError as exc:
+        raise ValueError("COMPANION_BASE_URL must be a valid HTTP LAN origin") from exc
+
+    host = parsed.hostname
+    if parsed.scheme.lower() != "http" or not parsed.netloc or not host:
+        raise ValueError("COMPANION_BASE_URL must be a canonical HTTP LAN origin")
+    if parsed.username is not None or parsed.password is not None:
+        raise ValueError("COMPANION_BASE_URL must not contain credentials")
+    if parsed.path or parsed.query or parsed.fragment:
+        raise ValueError("COMPANION_BASE_URL must not contain a path, query, or fragment")
+    if port is not None and not 1 <= port <= 65535:
+        raise ValueError("COMPANION_BASE_URL port must be between 1 and 65535")
+    if not _valid_companion_client_host(host):
+        raise ValueError("COMPANION_BASE_URL host is not supported by companion v1")
+
+    netloc = f"{host}:{port}" if port is not None else host
+    origin = f"http://{netloc}"
+    if value != origin:
+        raise ValueError("COMPANION_BASE_URL must be a canonical HTTP LAN origin")
+    return host, port or 80
+
+
+def configured_companion_origin() -> tuple[str, int] | None:
+    """Return the validated operator-configured v1 address, if any."""
+    value = os.environ.get("COMPANION_BASE_URL")
+    if value is None or value == "":
+        return None
+    return parse_companion_base_url(value)
+
+
 def default_port() -> int:
     """Best guess at the port the server is reachable on. Callers that know the
     real request port should pass it explicitly."""

companion/routes.py

@@ -23,7 +23,7 @@ from fastapi import APIRouter, HTTPException, Request
 from fastapi.responses import HTMLResponse
 
 from core.middleware import require_admin
-from src.auth_helpers import get_current_user
+from src.auth_helpers import _auth_disabled, get_current_user
 
 from companion import pairing as _pairing
 
@@ -113,8 +113,9 @@ def setup_companion_routes() -> APIRouter:
         The stock /api/models route scopes to get_current_user, which for a
         bearer token is the sandboxed pseudo-user "api" (owns nothing). Here we
         scope to the token's real owner instead, plus legacy null-owner shared
-        rows -- the same rule as owner_filter. Read-only; never returns api_key
+        rows -- the same rule as owner_filter. Explicit auth-disabled mode keeps
-        material.
+        the stock route's single-user all-endpoints view. Read-only; never
+        returns api_key material.
         """
         require_models_scope(request)
         import json as _json
@@ -123,6 +124,11 @@ def setup_companion_routes() -> APIRouter:
         from src.endpoint_resolver import build_chat_url
 
         owner = token_owner(request)
+        single_user_mode = (
+            owner is None
+            and not getattr(request.state, "api_token", False)
+            and _auth_disabled()
+        )
         out = []
         db = SessionLocal()
         try:
@@ -133,7 +139,7 @@ def setup_companion_routes() -> APIRouter:
             if owner:
                 q = q.filter((ModelEndpoint.owner == owner) | (ModelEndpoint.owner == None))  # noqa: E711
             for ep in q.all():
-                if not owner_can_see(ep.owner, owner):
+                if not single_user_mode and not owner_can_see(ep.owner, owner):
                     continue
                 try:
                     model_ids = _json.loads(ep.cached_models) if ep.cached_models else []
@@ -194,19 +200,27 @@ def setup_companion_routes() -> APIRouter:
         the code works immediately, no restart. `?format=json` returns the
         payload for an in-app pairing screen."""
         require_admin(request)
+        try:
+            configured_origin = _pairing.configured_companion_origin()
+        except ValueError as exc:
+            raise HTTPException(500, str(exc)) from None
         owner = get_current_user(request)
         invalidate = getattr(request.app.state, "invalidate_token_cache", None)
         token_id, raw_token = mint_pairing_token(owner, invalidate)
 
-        hosts = _pairing.lan_ip_candidates()
+        if configured_origin:
-        host = hosts[0] if hosts else "127.0.0.1"
+            host, port = configured_origin
-        port = request.url.port or _pairing.default_port()
+            hosts = [host]
+        else:
+            hosts = _pairing.lan_ip_candidates()
+            host = hosts[0] if hosts else "127.0.0.1"
+            port = request.url.port or _pairing.default_port()
         payload = _pairing.pairing_payload(host, port, raw_token)
         qr = _pairing.pairing_qr_png_data_uri(payload)
         qr_ok = bool(qr and qr.startswith("data:image/png;base64,"))
 
         if (request.query_params.get("format") or "").lower() == "json":
-            return {
+            response = {
                 "host": host,
                 "port": port,
                 "token": raw_token,
@@ -215,6 +229,7 @@ def setup_companion_routes() -> APIRouter:
                 "payload": payload,
                 "qr": qr if qr_ok else None,
             }
+            return response
 
         import json as _json
         payload_json = _json.dumps(payload, separators=(",", ":"))

core/atomic_io.py

@@ -30,11 +30,20 @@ def atomic_write_json(path: str, data: Any, *, indent: Optional[int] = None) ->
     """
     os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
     tmp = f"{path}.tmp.{uuid.uuid4().hex}"
-    with open(tmp, "w", encoding="utf-8") as f:
+
-        json.dump(data, f, indent=indent)
+    try:
-        f.flush()
+        with open(tmp, "w", encoding="utf-8") as f:
-        os.fsync(f.fileno())
+            json.dump(data, f, indent=indent)
-    os.replace(tmp, path)
+            f.flush()
+            os.fsync(f.fileno())
+        os.replace(tmp, path)
+    finally:
+        # Directly unlink to avoid a check-then-act race condition.
+        # Swallows FileNotFoundError (on success path) and other cleanup OSErrors.
+        try:
+            os.unlink(tmp)
+        except OSError:
+            pass
 
 
 def atomic_write_text(path: str, text: str) -> None:
@@ -42,8 +51,17 @@ def atomic_write_text(path: str, text: str) -> None:
         raise TypeError("atomic_write_text expects a string")
     os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
     tmp = f"{path}.tmp.{uuid.uuid4().hex}"
-    with open(tmp, "w", encoding="utf-8") as f:
+
-        f.write(text)
+    try:
-        f.flush()
+        with open(tmp, "w", encoding="utf-8") as f:
-        os.fsync(f.fileno())
+            f.write(text)
-    os.replace(tmp, path)
+            f.flush()
+            os.fsync(f.fileno())
+        os.replace(tmp, path)
+    finally:
+        # Directly unlink to avoid a check-then-act race condition.
+        # Swallows FileNotFoundError (on success path) and other cleanup OSErrors.
+        try:
+            os.unlink(tmp)
+        except OSError:
+            pass

core/auth.py

@@ -20,7 +20,6 @@ logger = logging.getLogger(__name__)
 
 
 from core.atomic_io import atomic_write_json as _atomic_write_json  # noqa: E402
-from core.middleware import INTERNAL_TOOL_USER  # noqa: E402
 
 DEFAULT_PRIVILEGES = {
     "can_use_agent": True,
@@ -49,24 +48,18 @@ ADMIN_PRIVILEGES["allowed_models_restricted"] = False
 ADMIN_PRIVILEGES["block_all_models"] = False
 
 from src.constants import AUTH_FILE, PASSWORD_MIN_LENGTH
+from src.owner_identity import RESERVED_AUTH_USERNAMES
 DEFAULT_AUTH_PATH = AUTH_FILE
 TOKEN_TTL = 60 * 60 * 24 * 7  # 7 days
 
-# Usernames the auth + middleware layer reserve as internal "synthetic owner"
+# Usernames the auth + middleware layer reserves for request sentinels and
-# sentinels; they must never belong to a real account. The most dangerous is
+# internal storage owners; they must never belong to a real login account.
-# "internal-tool": `core.middleware.require_admin` treats any request whose
+# "internal-tool" is the most dangerous because `core.middleware.require_admin`
-# `current_user == "internal-tool"` as the in-process tool loopback and grants
+# treats it as the in-process tool loopback. "api" collides with bearer-token
-# admin, and because the cookie auth path sets `current_user` to the raw
+# attribution. "demo"/"system" are synthetic owners already special-cased by
-# username, an account literally named "internal-tool" would be silently
+# scheduler/assistant/research paths. The Default/Local owner is a storage
-# treated as an admin by every `require_admin`-gated route. "api" collides with
+# bucket for explicit auth-disabled no-login mode, not a login username.
-# the bearer-token owner-attribution sentinel. "demo"/"system" round out the
+RESERVED_USERNAMES = frozenset(RESERVED_AUTH_USERNAMES)
-# synthetic-owner set the rest of the codebase already special-cases (see
-# `_SYNTHETIC_OWNERS` in routes/assistant_routes.py and the matching guards in
-# src/task_scheduler.py / routes/research_routes.py) — a real account with one
-# of those names would be denied an assistant and inconsistently owner-scoped.
-# Refuse to create or rename into any of them so the sentinels can't be
-# impersonated. (Keep this in sync with that synthetic-owner set.)
-RESERVED_USERNAMES = frozenset({INTERNAL_TOOL_USER, "api", "demo", "system"})
 
 
 def normalize_known_username(users: Dict[str, Any], username: str | None) -> Optional[str]:

core/database.py

@@ -1491,8 +1491,25 @@ def _migrate_assign_legacy_owner():
             with open(prefs_path, "r", encoding="utf-8") as f:
                 prefs = _json.load(f)
             if "_users" not in prefs and prefs:
-                # Flat format → nest under admin user
+                # Flat format → nest ordinary preferences under the admin
-                new_prefs = {"_users": {admin_user: prefs}}
+                # user. Foreground fallback is an explicit per-owner opt-in,
+                # so auth-disabled consent must remain inert at the flat root
+                # rather than becoming consent for the first named owner.
+                foreground_keys = {
+                    "foreground_fallback_enabled",
+                    "foreground_model_fallbacks",
+                }
+                named_prefs = {
+                    key: value
+                    for key, value in prefs.items()
+                    if key not in foreground_keys
+                }
+                new_prefs = {
+                    key: prefs[key]
+                    for key in foreground_keys
+                    if key in prefs
+                }
+                new_prefs["_users"] = {admin_user: named_prefs}
                 with open(prefs_path, "w", encoding="utf-8") as f:
                     _json.dump(new_prefs, f, indent=2)
                 logger.info(f"Migrated user_prefs.json to per-user format under '{admin_user}'")

core/middleware.py

@@ -3,10 +3,14 @@
 
 import os
 import secrets
+from collections.abc import Mapping
 
 from fastapi import HTTPException, Request
 from starlette.middleware.base import BaseHTTPMiddleware
 from starlette.responses import Response
+from starlette.routing import get_route_path
+
+from src.owner_identity import INTERNAL_TOOL_USER, auth_disabled
 
 
 # Per-process token that lets the in-app tool layer hit admin-gated
@@ -15,8 +19,30 @@ from starlette.responses import Response
 # same value from this module. Never persisted or exposed externally.
 INTERNAL_TOOL_TOKEN = os.environ.get("ODYSSEUS_INTERNAL_TOKEN") or secrets.token_hex(32)
 INTERNAL_TOOL_HEADER = "X-Odysseus-Internal-Token"
-# Pseudo-username on in-process tool-loopback requests; require_admin trusts it and it is reserved.
+
-INTERNAL_TOOL_USER = "internal-tool"
+
+def get_application_route_path(scope: Mapping[str, object]) -> str:
+    """Return the application-relative path used by Starlette routing.
+
+    Uvicorn prefixes ``scope["path"]`` with a configured ASGI ``root_path``;
+    Starlette removes that prefix before matching routes. Middleware policy
+    must use the same path form or a deployment prefix can change which policy
+    applies to an otherwise unchanged application route.
+    """
+    return get_route_path(scope)
+
+
+def with_asgi_root_path(scope: Mapping[str, object], path: str) -> str:
+    """Prefix an application path for a client-facing redirect target."""
+    root_path = scope.get("root_path", "")
+    if not isinstance(root_path, str) or not root_path:
+        return path
+    return f"{root_path.rstrip('/')}{path}"
+
+
+def path_is_route_or_child(path: str, prefix: str) -> bool:
+    """Return whether ``path`` is exactly ``prefix`` or below that route."""
+    return path == prefix or path.startswith(prefix + "/")
 
 
 def is_cors_preflight(method: str, headers) -> bool:
@@ -47,7 +73,7 @@ def require_admin(request: Request):
         pass
 
     auth_mgr = getattr(request.app.state, "auth_manager", None)
-    if os.getenv("AUTH_ENABLED", "true").lower() == "false":
+    if auth_disabled():
         return
     if not auth_mgr or not auth_mgr.is_configured:
         raise HTTPException(403, "Admin only")

core/models.py

@@ -8,6 +8,11 @@ These are simple datacontainers. All persistence is handled by SessionManager.
 from dataclasses import dataclass
 from typing import Dict, List, Any, Optional, TYPE_CHECKING
 
+from src.tool_approval_scopes import (
+    CHAT_SESSION_APPROVAL_CONTEXT_MARKER,
+    CHAT_SESSION_APPROVAL_DECISION,
+)
+
 if TYPE_CHECKING:
     from .session_manager import SessionManager
 
@@ -31,6 +36,35 @@ set_session_manager = set_session_manager_instance
 get_session_manager = get_session_manager_instance
 
 
+def _history_grants_chat_session_approval(
+    history: List["ChatMessage"],
+    session_id: str,
+) -> bool:
+    """Return whether this exact chat has a resolved session-scope grant."""
+
+    expected_session = str(session_id or "")
+    if not expected_session:
+        return False
+    for message in reversed(history or []):
+        metadata = getattr(message, "metadata", None)
+        if not isinstance(metadata, dict):
+            continue
+        tool_events = metadata.get("tool_events")
+        if not isinstance(tool_events, list):
+            continue
+        for event in reversed(tool_events):
+            ask_user = event.get("ask_user") if isinstance(event, dict) else None
+            if not isinstance(ask_user, dict):
+                continue
+            if (
+                ask_user.get("kind") == "tool_approval"
+                and ask_user.get("resolved") == CHAT_SESSION_APPROVAL_DECISION
+                and str(ask_user.get("session_id") or "") == expected_session
+            ):
+                return True
+    return False
+
+
 @dataclass
 class ChatMessage:
     """A single chat message."""
@@ -116,11 +150,27 @@ class Session:
         the model. Display/history-load paths use the raw ``history`` and are
         unaffected.
         """
-        return [
+        messages = [
             msg.to_dict()
             for msg in self.history
             if (msg.metadata or {}).get("source") != "slash"
         ]
+        if not _history_grants_chat_session_approval(self.history, self.id):
+            return messages
+
+        # Keep the grant close to the latest user request so route-neutral
+        # compaction/trimming preserves it. Copy the metadata instead of
+        # mutating the durable transcript object.
+        for index in range(len(messages) - 1, -1, -1):
+            if messages[index].get("role") != "user":
+                continue
+            message = dict(messages[index])
+            metadata = dict(message.get("metadata") or {})
+            metadata[CHAT_SESSION_APPROVAL_CONTEXT_MARKER] = True
+            message["metadata"] = metadata
+            messages[index] = message
+            break
+        return messages
 
     def get(self, key: str, default=None):
         """Dict-like access for compatibility."""

core/session_manager.py

@@ -14,6 +14,8 @@ import logging
 from datetime import datetime, timezone, timedelta
 from typing import Dict, Optional
 
+from sqlalchemy import func
+
 from .database import Session as DbSession, ChatMessage as DbChatMessage, Document as DbDocument, SessionLocal, utcnow_naive
 from .models import Session, ChatMessage
 from src.attachment_refs import persistable_message_content
@@ -92,14 +94,28 @@ class SessionManager:
         try:
             db_sessions = db.query(DbSession).filter(
                 DbSession.archived == False,
-                DbSession.message_count > 0,
+                DbSession.messages.any(),
             ).order_by(DbSession.last_accessed.desc()).limit(100).all()
 
+            # message_count is derived metadata and can drift after interrupted
+            # or legacy writes. Count only the bounded discovery set so startup
+            # remains metadata-only while lazy hydration sees an authoritative
+            # positive count for every discovered non-empty session.
+            message_counts = {}
+            if db_sessions:
+                message_counts = dict(
+                    db.query(DbChatMessage.session_id, func.count(DbChatMessage.id))
+                    .filter(DbChatMessage.session_id.in_([row.id for row in db_sessions]))
+                    .group_by(DbChatMessage.session_id)
+                    .all()
+                )
+
             loaded_count = 0
             for db_session in db_sessions:
                 try:
                     session = self._db_to_session_meta(db_session)
                     if session is not None:
+                        session.message_count = message_counts[db_session.id]
                         self.sessions[db_session.id] = session
                         loaded_count += 1
                 except Exception as e:

docker-compose.gpu-amd.yml

@@ -46,10 +46,11 @@ services:
       - DATABASE_URL=${DATABASE_URL:-sqlite:///./data/app.db}
       - AUTH_ENABLED=${AUTH_ENABLED:-true}
       - LOCALHOST_BYPASS=${LOCALHOST_BYPASS:-false}
+      - COMPANION_BASE_URL=${COMPANION_BASE_URL:-}
       - ODYSSEUS_ADMIN_USER=${ODYSSEUS_ADMIN_USER:-admin}
       - ODYSSEUS_ADMIN_PASSWORD=${ODYSSEUS_ADMIN_PASSWORD:-}
       - ALLOWED_ORIGINS=${ALLOWED_ORIGINS:-http://localhost,http://127.0.0.1}
-      - SECURE_COOKIES=${SECURE_COOKIES:-false}
+      - SECURE_COOKIES=${SECURE_COOKIES:-}
       - EMBEDDING_URL=${EMBEDDING_URL:-}
       - EMBEDDING_MODEL=${EMBEDDING_MODEL:-}
       - EMBEDDING_API_KEY=${EMBEDDING_API_KEY:-}
@@ -74,6 +75,11 @@ services:
       - GOOGLE_OAUTH_CLIENT_ID=${GOOGLE_OAUTH_CLIENT_ID:-}
       - GOOGLE_OAUTH_CLIENT_SECRET=${GOOGLE_OAUTH_CLIENT_SECRET:-}
       - GOOGLE_OAUTH_REDIRECT_URI=${GOOGLE_OAUTH_REDIRECT_URI:-}
+      # Externally reachable origin for MCP OAuth callbacks. The container
+      # always listens on 7000 and cannot see the host port map above, so
+      # remote MCP OAuth needs this set whenever the browser reaches
+      # Odysseus on anything other than http://localhost:7000.
+      - OAUTH_REDIRECT_BASE_URL=${OAUTH_REDIRECT_BASE_URL:-}
       - TAVILY_API_KEY=${TAVILY_API_KEY:-}
       - SERPER_API_KEY=${SERPER_API_KEY:-}
       # PUID / PGID — the user/group the container drops to before
@@ -129,12 +135,17 @@ services:
           fi
           sed "s|__SEARXNG_SECRET__|$$secret|g" /tmp/searxng-settings.yml.template > /etc/searxng/settings.yml
         fi
+        # Advisory: a settings file the migration cannot parse or rewrite must
+        # not be what stops searxng from booting. It explains itself on stderr
+        # and we carry on, letting searxng report anything genuinely wrong.
+        /usr/local/searxng/.venv/bin/python /tmp/migrate-searxng-settings.py /etc/searxng/settings.yml || true
         exec /usr/local/searxng/entrypoint.sh
     ports:
       - "127.0.0.1:8080:8080"
     volumes:
       - searxng-data:/etc/searxng
       - ./config/searxng/settings.yml:/tmp/searxng-settings.yml.template:ro,z
+      - ./scripts/migrate_searxng_settings.py:/tmp/migrate-searxng-settings.py:ro,z
     environment:
       - SEARXNG_BASE_URL=http://localhost:8080/
       - SEARXNG_SECRET=${SEARXNG_SECRET:-}

docker-compose.gpu-nvidia.yml

@@ -45,10 +45,11 @@ services:
       - DATABASE_URL=${DATABASE_URL:-sqlite:///./data/app.db}
       - AUTH_ENABLED=${AUTH_ENABLED:-true}
       - LOCALHOST_BYPASS=${LOCALHOST_BYPASS:-false}
+      - COMPANION_BASE_URL=${COMPANION_BASE_URL:-}
       - ODYSSEUS_ADMIN_USER=${ODYSSEUS_ADMIN_USER:-admin}
       - ODYSSEUS_ADMIN_PASSWORD=${ODYSSEUS_ADMIN_PASSWORD:-}
       - ALLOWED_ORIGINS=${ALLOWED_ORIGINS:-http://localhost,http://127.0.0.1}
-      - SECURE_COOKIES=${SECURE_COOKIES:-false}
+      - SECURE_COOKIES=${SECURE_COOKIES:-}
       - EMBEDDING_URL=${EMBEDDING_URL:-}
       - EMBEDDING_MODEL=${EMBEDDING_MODEL:-}
       - EMBEDDING_API_KEY=${EMBEDDING_API_KEY:-}
@@ -73,6 +74,11 @@ services:
       - GOOGLE_OAUTH_CLIENT_ID=${GOOGLE_OAUTH_CLIENT_ID:-}
       - GOOGLE_OAUTH_CLIENT_SECRET=${GOOGLE_OAUTH_CLIENT_SECRET:-}
       - GOOGLE_OAUTH_REDIRECT_URI=${GOOGLE_OAUTH_REDIRECT_URI:-}
+      # Externally reachable origin for MCP OAuth callbacks. The container
+      # always listens on 7000 and cannot see the host port map above, so
+      # remote MCP OAuth needs this set whenever the browser reaches
+      # Odysseus on anything other than http://localhost:7000.
+      - OAUTH_REDIRECT_BASE_URL=${OAUTH_REDIRECT_BASE_URL:-}
       - TAVILY_API_KEY=${TAVILY_API_KEY:-}
       - SERPER_API_KEY=${SERPER_API_KEY:-}
       # PUID / PGID — the user/group the container drops to before
@@ -132,12 +138,17 @@ services:
           fi
           sed "s|__SEARXNG_SECRET__|$$secret|g" /tmp/searxng-settings.yml.template > /etc/searxng/settings.yml
         fi
+        # Advisory: a settings file the migration cannot parse or rewrite must
+        # not be what stops searxng from booting. It explains itself on stderr
+        # and we carry on, letting searxng report anything genuinely wrong.
+        /usr/local/searxng/.venv/bin/python /tmp/migrate-searxng-settings.py /etc/searxng/settings.yml || true
         exec /usr/local/searxng/entrypoint.sh
     ports:
       - "127.0.0.1:8080:8080"
     volumes:
       - searxng-data:/etc/searxng
       - ./config/searxng/settings.yml:/tmp/searxng-settings.yml.template:ro,z
+      - ./scripts/migrate_searxng_settings.py:/tmp/migrate-searxng-settings.py:ro,z
     environment:
       - SEARXNG_BASE_URL=http://localhost:8080/
       - SEARXNG_SECRET=${SEARXNG_SECRET:-}

docker-compose.yml

@@ -34,10 +34,11 @@ services:
       - DATABASE_URL=${DATABASE_URL:-sqlite:///./data/app.db}
       - AUTH_ENABLED=${AUTH_ENABLED:-true}
       - LOCALHOST_BYPASS=${LOCALHOST_BYPASS:-false}
+      - COMPANION_BASE_URL=${COMPANION_BASE_URL:-}
       - ODYSSEUS_ADMIN_USER=${ODYSSEUS_ADMIN_USER:-admin}
       - ODYSSEUS_ADMIN_PASSWORD=${ODYSSEUS_ADMIN_PASSWORD:-}
       - ALLOWED_ORIGINS=${ALLOWED_ORIGINS:-http://localhost,http://127.0.0.1}
-      - SECURE_COOKIES=${SECURE_COOKIES:-false}
+      - SECURE_COOKIES=${SECURE_COOKIES:-}
       - EMBEDDING_URL=${EMBEDDING_URL:-}
       - EMBEDDING_MODEL=${EMBEDDING_MODEL:-}
       - EMBEDDING_API_KEY=${EMBEDDING_API_KEY:-}
@@ -62,6 +63,11 @@ services:
       - GOOGLE_OAUTH_CLIENT_ID=${GOOGLE_OAUTH_CLIENT_ID:-}
       - GOOGLE_OAUTH_CLIENT_SECRET=${GOOGLE_OAUTH_CLIENT_SECRET:-}
       - GOOGLE_OAUTH_REDIRECT_URI=${GOOGLE_OAUTH_REDIRECT_URI:-}
+      # Externally reachable origin for MCP OAuth callbacks. The container
+      # always listens on 7000 and cannot see the host port map above, so
+      # remote MCP OAuth needs this set whenever the browser reaches
+      # Odysseus on anything other than http://localhost:7000.
+      - OAUTH_REDIRECT_BASE_URL=${OAUTH_REDIRECT_BASE_URL:-}
       - TAVILY_API_KEY=${TAVILY_API_KEY:-}
       - SERPER_API_KEY=${SERPER_API_KEY:-}
       # PUID / PGID — the user/group the container drops to before
@@ -110,12 +116,17 @@ services:
           fi
           sed "s|__SEARXNG_SECRET__|$$secret|g" /tmp/searxng-settings.yml.template > /etc/searxng/settings.yml
         fi
+        # Advisory: a settings file the migration cannot parse or rewrite must
+        # not be what stops searxng from booting. It explains itself on stderr
+        # and we carry on, letting searxng report anything genuinely wrong.
+        /usr/local/searxng/.venv/bin/python /tmp/migrate-searxng-settings.py /etc/searxng/settings.yml || true
         exec /usr/local/searxng/entrypoint.sh
     ports:
       - "127.0.0.1:8080:8080"
     volumes:
       - searxng-data:/etc/searxng
       - ./config/searxng/settings.yml:/tmp/searxng-settings.yml.template:ro,z
+      - ./scripts/migrate_searxng_settings.py:/tmp/migrate-searxng-settings.py:ro,z
     environment:
       - SEARXNG_BASE_URL=http://localhost:8080/
       - SEARXNG_SECRET=${SEARXNG_SECRET:-}

docs/setup.md

@@ -441,10 +441,19 @@ A grab-bag of small gotchas that otherwise turn into long debugging sessions.
 | Package | Feature unlocked |
 |---------|-----------------|
 | `faster-whisper` | Local speech-to-text (microphone -> text) via the "local" STT provider. |
+| `kokoro`, `soundfile` | Local Kokoro-82M text-to-speech on a CUDA GPU. The pinned Kokoro release supports Odysseus installs on Python 3.11-3.12; these packages are intentionally skipped on Python 3.13+ (including the Python 3.14 container image). |
 | `ddgs` | DuckDuckGo as a search provider option. |
 | `PyMuPDF` | PDF page rendering in the side viewer panel and form-filling. (Note: AGPL-3.0) |
 | `markitdown` | Office/EPUB document text extraction (converts .docx/.xlsx/.pptx/.xls/.epub to Markdown). |
 
+Install the optional set only when you need these features:
+
+```bash
+pip install -r requirements-optional.txt
+```
+
+The default Docker image currently uses Python 3.14, while Kokoro 0.9.4 declares Python `>=3.10,<3.13`. Odysseus itself continues to support Python 3.11+, but this pinned optional local-TTS feature requires a native Python 3.11 or 3.12 environment. Kokoro declares `torch`, but the local provider only activates when that torch build has CUDA and a GPU is visible; install the CUDA build appropriate for your host. Browser and configured endpoint TTS remain available on Python 3.13+ and in the container image.
+
 ### Faster, reproducible installs with uv (optional)
 [uv](https://docs.astral.sh/uv/) works as a drop-in replacement for the
 venv + pip steps in the native install guides, no project changes are needed but this change results in faster installs along with a lockfile for reproducible environments. After [installing `uv`](https://docs.astral.sh/uv/getting-started/installation/), use:
@@ -475,7 +484,7 @@ Odysseus is a self-hosted workspace with powerful local tools: shell access, fil
 
 - Keep `AUTH_ENABLED=true` for any network-accessible deployment.
 - Keep `LOCALHOST_BYPASS=false` outside local development.
-- Use `SECURE_COOKIES=true` when Odysseus is served through HTTPS by a trusted reverse proxy or private access gateway.
+- Leave `SECURE_COOKIES` unset unless you need to override it: session cookies are marked `Secure` whenever the request arrives over HTTPS. Use `SECURE_COOKIES=true` to force it on for a proxy whose scheme Odysseus cannot see, or `SECURE_COOKIES=false` to force it off while you still serve plain HTTP alongside HTTPS.
 - Do not expose it directly to the public internet without HTTPS and a trusted reverse proxy or private access layer.
 - Keep `.env`, `data/`, `logs/`, databases, uploads, generated media, backups, auth/session files, API keys, and model/provider tokens out of Git and private shares. They are ignored by default.
 - Review `data/auth.json` after first boot: disable open signup unless you intentionally want it, make only your own account admin, and keep demo/test accounts non-admin.
@@ -486,6 +495,14 @@ Odysseus is a self-hosted workspace with powerful local tools: shell access, fil
 - Keep ChromaDB, SearXNG, ntfy, Ollama, vLLM, llama.cpp, databases, and raw model/provider APIs internal-only. Expose only the authenticated Odysseus web/API entrypoint through your trusted proxy or private access layer.
 - Before publishing a fork, run `git status --short` and confirm no private files from `.env`, `data/`, `logs/`, uploads, backups, or local databases are staged.
 
+> **Upgrading an existing install:** `SECURE_COOKIES` used to default to
+> `false`, so an install set up before scheme derivation may still carry
+> `SECURE_COOKIES=false` in its own `.env`. That explicit value stays
+> authoritative, so HTTPS logins keep getting a non-`Secure` session cookie.
+> Pulling this change updates the tracked Compose files, but nothing rewrites
+> your `.env` — drop the line from it unless you deliberately serve plain HTTP
+> alongside HTTPS and want the escape hatch.
+
 ### Private or proxied deployments
 Odysseus serves plain HTTP on its app port. Docker Compose binds Odysseus and the bundled services to `127.0.0.1` by default, so a typical production/private setup is:
 
@@ -494,9 +511,162 @@ Odysseus serves plain HTTP on its app port. Docker Compose binds Odysseus and th
 3. Put the authenticated Odysseus web/API entrypoint behind that layer.
 4. Keep raw service and model ports internal-only.
 
-Cloudflare Access, Tailscale, Caddy, nginx, and Traefik can all fit this pattern; none are required by Odysseus. If your access layer reaches Odysseus on the same host, proxy to `http://127.0.0.1:7000` and keep `AUTH_ENABLED=true`, `LOCALHOST_BYPASS=false`, and `SECURE_COOKIES=true`.
+Cloudflare Access, Tailscale, Caddy, nginx, and Traefik can all fit this pattern; none are required by Odysseus. If your access layer reaches Odysseus on the same host, proxy to `http://127.0.0.1:7000` and keep `AUTH_ENABLED=true` and `LOCALHOST_BYPASS=false`. Any proxy that forwards `X-Forwarded-Proto: https` gets `Secure` session cookies without configuration, so `SECURE_COOKIES` only needs setting when you want to override that — force it on for a proxy that forwards no scheme at all, or off while you still serve plain HTTP.
 `ALLOWED_ORIGINS` lists exact permitted origins for cross-origin browser/API clients; ordinary same-origin reverse-proxy access usually does not need a special CORS entry.
 
+#### Faster over the network: HTTP/2
+
+The frontend is raw ES modules with no bundler, so a page load is a few hundred
+small same-origin requests. Over HTTP/1.1 browsers typically allow only a small
+number of concurrent connections per host (commonly around six), so many of
+those requests are serialized across multiple round trips. On localhost that
+costs almost nothing. Over a LAN, VPN, or remote link it can become a major
+part of load time, especially as latency increases.
+
+HTTP/2 multiplexes them onto one connection and the serialisation disappears.
+Odysseus needs no changes for this — uvicorn keeps speaking HTTP/1.1 on
+loopback and the proxy speaks HTTP/2 to the browser. Mainstream browsers
+negotiate HTTP/2 for normal web pages over TLS; they do not use the cleartext
+h2c mode here, so browser-facing HTTP/2 requires a certificate. The
+`--ssl-certfile` route in *HTTPS + LAN/Tailscale exposure* above gives you
+HTTPS but not HTTP/2 — uvicorn does not speak it.
+
+**1. Install Caddy.** See the [install docs](https://caddyserver.com/docs/install)
+for your platform; on macOS, `brew install caddy`.
+
+**2. Write a `Caddyfile`.** Pick the block that matches how you reach the
+machine. Replace `7000` if Odysseus listens elsewhere — the macOS start script
+uses `7860`.
+
+Public domain, Caddy obtains and renews the certificate itself:
+
+```
+odysseus.example.com {
+	reverse_proxy 127.0.0.1:7000
+}
+```
+
+Tailscale, no public DNS needed — `tailscale cert` issues a browser-trusted
+certificate for a tailnet name and writes `<domain>.crt` and `<domain>.key`:
+
+```bash
+tailscale cert myhost.tailnet-name.ts.net
+```
+
+```
+myhost.tailnet-name.ts.net {
+	tls /path/to/myhost.tailnet-name.ts.net.crt /path/to/myhost.tailnet-name.ts.net.key
+	reverse_proxy 127.0.0.1:7000
+}
+```
+
+LAN with your own certificate — same shape, your own files:
+
+```
+odysseus.lan {
+	tls /path/to/cert.pem /path/to/key.pem
+	reverse_proxy 127.0.0.1:7000
+}
+```
+
+Give `tls` absolute paths: a service starts in a working directory you did not
+choose. If port 443 is already taken, append a port to the site address
+(`odysseus.example.com:8443`) and use it in the URL. That alone does not free
+port 80 — Caddy still binds it for the HTTP-to-HTTPS redirect, and fails to
+start with `listen tcp :80: bind: address already in use` if something else
+holds it. Turn the redirect off with a global block at the top of the file:
+
+```
+{
+	auto_https disable_redirects
+}
+```
+
+**3. Run it in the foreground first:**
+
+```bash
+caddy run --config ./Caddyfile
+```
+
+Once that works, run it as a service:
+
+```bash
+brew services start caddy          # macOS — reads $(brew --prefix)/etc/Caddyfile, not ./Caddyfile
+sudo systemctl enable --now caddy  # Linux, if your package installed the unit
+```
+
+Odysseus's own service is unchanged; the proxy runs alongside it. Under Docker,
+run the proxy as another container, or on the host pointing at the published
+port.
+
+**4. Point Odysseus at the new origin** in `.env`, then restart it.
+
+A proxy that exposes the HTTPS request scheme to Odysseus needs no `SECURE_COOKIES` setting. Only force it on when the proxy cannot expose that scheme:
+
+```bash
+# only if the proxy cannot expose the external HTTPS scheme to Odysseus:
+SECURE_COOKIES=true
+# only if you use remote MCP servers with OAuth:
+OAUTH_REDIRECT_BASE_URL=https://odysseus.example.com
+```
+
+Gmail OAuth needs nothing here when the proxy runs on the same host: the
+redirect URI is built from the incoming request, and uvicorn rewrites the
+scheme from `X-Forwarded-Proto` for proxies it trusts — by default only
+`127.0.0.1`. A proxy in a separate container or on another machine is not
+trusted, so pin the URI there:
+
+```bash
+GOOGLE_OAUTH_REDIRECT_URI=https://odysseus.example.com/api/email/oauth/google/callback
+```
+
+(uvicorn's own `FORWARDED_ALLOW_IPS` widens that trust, but it has to be in the
+environment uvicorn starts with — `.env` is read by the app afterwards, too
+late for it to take effect.)
+
+**5. Confirm HTTP/2 is really on:**
+
+```bash
+curl -s -o /dev/null -w '%{http_version}\n' https://odysseus.example.com/
+# 2
+```
+
+The status code is not the thing to check here — a logged-out request redirects
+to the login page, so `curl -I` shows `HTTP/2 302`, and the `HTTP/2` prefix is
+the part that matters. The browser reports the same in the Network panel's
+Protocol column (`h2`); in Chrome and Firefox that column is hidden until you
+enable it by right-clicking the column headers.
+
+Three things bite when moving an existing install behind TLS:
+
+- Leave `SECURE_COOKIES` unset when Odysseus can see the external HTTPS scheme;
+  the cookie then follows the request automatically. If your proxy cannot expose
+  that scheme, set `SECURE_COOKIES=true` **at the same time** you stop serving
+  plain HTTP, not before. An explicit `true` applies to every login, so while an
+  HTTP entrypoint is still reachable the browser will reject the `Secure` cookie
+  there and login will appear to loop.
+- `OAUTH_REDIRECT_BASE_URL` defaults to `http://localhost:7000`. Unlike the
+  Gmail redirect URI it cannot be derived from a request — it is registered
+  with each MCP authorization server up front — so set it to the external
+  origin if you use remote MCP servers over OAuth.
+- Odysseus sends `Strict-Transport-Security` once it sees `X-Forwarded-Proto:
+  https`. HSTS applies to the whole hostname and ignores the port, so any other
+  plain-HTTP service on that same hostname becomes unreachable in browsers that
+  have visited Odysseus. Give Odysseus its own hostname, or strip the header at
+  the proxy (`header_down -Strict-Transport-Security` in Caddy).
+
+Server-sent events are not buffered by this configuration, so chat streaming
+arrives token by token; add `flush_interval -1` inside the `reverse_proxy`
+block if you want that pinned explicitly. nginx needs `proxy_buffering off;`
+for the same reason.
+
+Changing the external origin also affects state scoped to it. Service workers
+and their caches are origin-scoped, so moving to a different origin starts with
+a cold load. Cookies follow their own domain/path/security rules rather than
+being port-scoped: changing the hostname normally requires a new login, while
+changing only the scheme or port does not by itself guarantee that existing
+cookies disappear.
+
 Common internal-only ports from the default docs/compose setup:
 
 | Port | Service |
@@ -527,7 +697,7 @@ Key settings:
 | `AUTH_ENABLED` | `true` | Enable/disable login |
 | `LOCALHOST_BYPASS` | `false` | Development-only auth bypass for loopback requests. Keep false for shared/network deployments. |
 | `ALLOWED_ORIGINS` | `http://localhost,http://127.0.0.1` | Comma-separated exact permitted origins for cross-origin browser/API clients. |
-| `SECURE_COOKIES` | `false` | Set true when serving Odysseus through HTTPS at a trusted proxy or private access gateway. |
+| `SECURE_COOKIES` | derived from the request scheme | Marks session cookies `Secure` on HTTPS requests. Set true to force it on, false to force it off. |
 | `DATABASE_URL` | `sqlite:///./data/app.db` | Database connection string |
 | `CHROMADB_HOST` | `localhost` | ChromaDB host for vector memory. Docker overrides this to `chromadb`. |
 | `CHROMADB_PORT` | `8100` | ChromaDB port for manual host runs. Docker overrides this to `8000`. |

launch-windows.ps1

@@ -163,6 +163,10 @@ if (Test-Path $cudaBase) {
 }
 
 # 7. Start the server (use `python -m uvicorn` - bare `uvicorn` may not be on PATH)
+# -Port only reaches uvicorn as a flag. Everything that builds a URL for this
+# instance - internal_api_base(), companion pairing, the MCP OAuth callback -
+# reads APP_PORT, so set it too or they all assume 7000.
+$env:APP_PORT = $Port
 Write-Step ("Starting Odysseus at http://{0}:{1}" -f $BindHost, $Port)
 Write-Host "Press Ctrl+C to stop."
 Write-Host ""

licenses/KaTeX-MIT-LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2013-2020 Khan Academy and other contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

licenses/Mermaid-MIT-LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2014 - 2022 Knut Sveidqvist
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

mcp_servers/email_server.py

@@ -1802,7 +1802,6 @@ async def _ai_draft_reply_to_email(uid, folder="INBOX", reply_all=False, account
         from src.endpoint_resolver import (
             resolve_endpoint,
             resolve_utility_fallback_candidates,
-            resolve_chat_fallback_candidates,
         )
         from src.llm_core import llm_call_async_with_fallback
     except Exception as exc:
@@ -1843,13 +1842,6 @@ async def _ai_draft_reply_to_email(uid, folder="INBOX", reply_all=False, account
         utility_fallbacks = resolve_utility_fallback_candidates() or []
     for cand in utility_fallbacks:
         _add(*cand)
-    try:
-        chat_fallbacks = resolve_chat_fallback_candidates(owner=None) or []
-    except TypeError:
-        chat_fallbacks = resolve_chat_fallback_candidates() or []
-    for cand in chat_fallbacks:
-        _add(*cand)
-
     if not candidates:
         return {"error": "No LLM endpoint configured for AI reply"}
 

package-lock.json

@@ -5,13 +5,13 @@
   "packages": {
     "": {
       "devDependencies": {
-        "@antithesishq/bombadil": "^0.6.1"
+        "@antithesishq/bombadil": "^0.7.0"
       }
     },
     "node_modules/@antithesishq/bombadil": {
-      "version": "0.6.1",
+      "version": "0.7.0",
-      "resolved": "https://registry.npmjs.org/@antithesishq/bombadil/-/bombadil-0.6.1.tgz",
+      "resolved": "https://registry.npmjs.org/@antithesishq/bombadil/-/bombadil-0.7.0.tgz",
-      "integrity": "sha512-d1iufG3MI7gSMSiSmMeNdcMW+qR0yQXL2zdkVynC3n3DYgFJYlYXKUQzygmqU12m4RWlR5iOdQU1hsx5UT6+IA==",
+      "integrity": "sha512-alJmnphJ/iUoL5mCsnV3DwtajGy/sEQ3NJJCiMhgjqXshSq2BUtAs0vqdXEiiSkB8HbsOX5CLrAcaogYdwfAJg==",
       "dev": true,
       "license": "MIT",
       "bin": {

package.json

@@ -4,6 +4,6 @@
     "url": "https://github.com/odysseus-dev/odysseus.git"
   },
   "devDependencies": {
-    "@antithesishq/bombadil": "^0.6.1"
+    "@antithesishq/bombadil": "^0.7.0"
   }
 }

requirements-optional.txt

@@ -12,6 +12,16 @@
 # GPU-accelerated transcription — it's auto-detected, CPU is used otherwise.
 faster-whisper
 
+# Local text-to-speech via Kokoro-82M for the "local" TTS provider.
+# Kokoro 0.9.4 declares Python >=3.10,<3.13; Odysseus itself requires 3.11+,
+# so pip installs these extras on 3.11-3.12 and deliberately skips them on
+# Python 3.13+ (including the Python 3.14 container image). Kokoro declares
+# torch; the local provider still
+# requires a CUDA-enabled torch build and GPU at runtime. SoundFile is separate
+# in Kokoro's official install instructions and is not a transitive dependency.
+kokoro==0.9.4; python_version >= "3.11" and python_version < "3.13"
+soundfile; python_version >= "3.11" and python_version < "3.13"
+
 # DuckDuckGo as a search provider option.
 # Install if you want DDG in the search-provider dropdown.
 # Alternatives: SearXNG, Brave, Tavily, Serper, Google PSE.

routes/assistant_routes.py

@@ -16,7 +16,7 @@ from pydantic import BaseModel
 
 from core.database import SessionLocal, CrewMember, ScheduledTask
 from src.auth_helpers import get_current_user
-from core.auth import RESERVED_USERNAMES
+from src.owner_identity import REQUEST_SENTINEL_OWNERS
 from src.task_scheduler import compute_next_run
 
 
@@ -90,11 +90,12 @@ def setup_assistant_routes(task_scheduler) -> APIRouter:
     # check-in tasks seeded. Hitting any /assistant route under one of these
     # used to seed a full CrewMember + Morning/Midday/Evening tasks under that
     # owner, which then double-fired alongside the real user's check-ins.
-    # RESERVED_USERNAMES covers the same set; the `not owner` guard handles "".
+    # REQUEST_SENTINEL_OWNERS covers request-only identities; Default/Local is a
+    # reserved login name but remains a valid storage owner.
 
     async def _get_or_create(owner: str) -> CrewMember:
         """Return the per-owner assistant CrewMember, creating it on demand."""
-        if not owner or owner in RESERVED_USERNAMES:
+        if not owner or owner in REQUEST_SENTINEL_OWNERS:
             raise HTTPException(status_code=400, detail=f"Cannot seed assistant for {owner!r}")
         db = SessionLocal()
         try:

routes/auth_routes.py

@@ -22,6 +22,8 @@ from src.settings import (
     load_features as _load_features,
     save_features as _save_features,
     DEFAULT_SETTINGS,
+    RETIRED_SETTING_KEYS,
+    without_retired_settings,
 )
 from src.integrations import (
     load_integrations,
@@ -84,6 +86,33 @@ class SetOpenRegistrationRequest(BaseModel):
 SESSION_COOKIE = "odysseus_session"
 
 
+def _secure_cookie(request: Request) -> bool:
+    """Decide the ``Secure`` attribute of the session cookie.
+
+    ``SECURE_COOKIES`` stays authoritative when it holds an explicit value:
+    ``true`` always marks the cookie Secure (the documented knob for a TLS
+    proxy), ``false`` never does, which is the escape hatch for an install
+    that still answers on plain HTTP alongside HTTPS. Anything else —
+    unset, or the present-but-empty value docker-compose injects for a
+    variable the host has not defined — derives it from the request, so an
+    HTTPS login gets a Secure cookie without any configuration.
+
+    Either the connection scheme or ``X-Forwarded-Proto`` saying https is
+    enough, which is the same test ``core/middleware.py`` applies before it
+    sends HSTS. Uvicorn's proxy-headers middleware already folds that header
+    into the scheme for the proxies it trusts, so reading it here only adds
+    the case of a terminator that is not on a trusted address; the cost is
+    that a client talking to the app directly can set the header and lock
+    its own session out over plain HTTP.
+    """
+    configured = os.getenv("SECURE_COOKIES", "").strip().lower()
+    if configured in ("true", "false"):
+        return configured == "true"
+    # A chained proxy sends a list — the client-facing hop comes first.
+    forwarded_proto = request.headers.get("x-forwarded-proto", "").split(",")[0]
+    return request.url.scheme == "https" or forwarded_proto.strip().lower() == "https"
+
+
 def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
     router = APIRouter(prefix="/api/auth", tags=["auth"])
 
@@ -157,7 +186,7 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
             value=token,
             httponly=True,
             samesite="lax",
-            secure=os.getenv("SECURE_COOKIES", "false").lower() == "true",
+            secure=_secure_cookie(request),
             path="/",
         )
         if body.remember:
@@ -689,7 +718,7 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
         a scrubbed copy with secret keys blanked. The frontend uses this
         for keybinds + TTS prefs, so it stays callable without admin."""
         user = _get_current_user(request)
-        settings = _load_settings()
+        settings = without_retired_settings(_load_settings())
         if user and auth_manager.is_admin(user):
             return settings
         return scrub_settings(settings)
@@ -709,6 +738,8 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
             "agent_max_tool_calls": (0, 1000),  # 0 = unlimited
         }
         for key in DEFAULT_SETTINGS:
+            if key in RETIRED_SETTING_KEYS:
+                continue
             if key not in body:
                 continue
             val = body[key]
@@ -721,7 +752,7 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
                 val = max(lo, min(val, hi))
             current[key] = val
         _save_settings(current)
-        return current
+        return without_retired_settings(current)
 
     # ---- Integrations CRUD ----
 

routes/chat_helpers.py

@@ -15,7 +15,7 @@ from core.database import Session as DBSession, ModelEndpoint
 from src.llm_core import normalize_model_id
 from src.endpoint_resolver import normalize_base
 from src.context_compactor import maybe_compact, trim_for_context
-from src.model_context import estimate_tokens
+from src.model_context import estimate_tokens, get_context_length
 from src.auth_helpers import effective_user
 from src.prompt_security import untrusted_context_message
 from src.attachment_refs import attachment_ref
@@ -152,10 +152,38 @@ class ChatContext:
     # Uploads attached to this user turn, resolved and owner-checked for the
     # agent's private context. This is not emitted to the browser.
     uploaded_files: list = field(default_factory=list)
+    # Route-neutral prompt before any model-window compaction/trimming. This is
+    # retained only when explicit foreground fallbacks are enabled so each
+    # concrete candidate can apply its own context budget independently.
+    route_messages: list = field(default_factory=list)
 
 
 # ── Helpers ────────────────────────────────────────────────────────────── #
 
+def _allowed_models_from_privileges(privs: dict) -> Optional[frozenset[str]]:
+    if privs.get("block_all_models"):
+        return frozenset()
+    allowed_raw = privs.get("allowed_models")
+    allowed = allowed_raw if isinstance(allowed_raw, list) else []
+    restricted = bool(privs.get("allowed_models_restricted")) or bool(allowed)
+    return frozenset(model for model in allowed if isinstance(model, str)) if restricted else None
+
+
+def _allowed_models_for_request(request) -> Optional[frozenset[str]]:
+    """Return the caller's model allowlist, or ``None`` when unrestricted."""
+
+    try:
+        user = effective_user(request)
+    except Exception:
+        user = None
+    if not user:
+        return None
+    auth_manager = getattr(getattr(request.app, "state", None), "auth_manager", None)
+    if not auth_manager:
+        return None
+    privs = auth_manager.get_privileges(user) or {}
+    return _allowed_models_from_privileges(privs)
+
 def _enforce_chat_privileges(request, sess) -> None:
     """Apply the per-user privilege gates (allowed_models + max_messages_per_day)
     that both /api/chat and /api/chat_stream must enforce BEFORE any LLM work.
@@ -185,10 +213,8 @@ def _enforce_chat_privileges(request, sess) -> None:
     if privs.get("block_all_models"):
         raise HTTPException(403, f"Your account is not allowed to use model '{sess.model}'.")
 
-    allowed_raw = privs.get("allowed_models")
+    allowed_models = _allowed_models_from_privileges(privs)
-    allowed = allowed_raw if isinstance(allowed_raw, list) else []
+    if allowed_models is not None and sess.model and sess.model not in allowed_models:
-    restricted = bool(privs.get("allowed_models_restricted")) or bool(allowed)
-    if restricted and sess.model and sess.model not in allowed:
         raise HTTPException(403, f"Your account is not allowed to use model '{sess.model}'.")
 
     cap = int(privs.get("max_messages_per_day") or 0)
@@ -287,96 +313,6 @@ async def auto_name_session(session_manager, sess):
         logger.error(f"Auto-name failed for {sess.id}: {e}\n{traceback.format_exc()}")
 
 
-def try_fallback_endpoint(sess, session_id: str) -> dict | None:
-    """Find an alternative working endpoint when the current one fails.
-
-    Returns {"model": ..., "endpoint_url": ..., "endpoint_name": ...} or None.
-    """
-    import requests as _req
-    from src.endpoint_resolver import (
-        build_chat_url,
-        build_headers,
-        build_models_url,
-        normalize_base,
-        resolve_endpoint_runtime,
-    )
-    from src.chatgpt_subscription import is_chatgpt_subscription_base
-
-    current_url = sess.endpoint_url or ""
-    owner = getattr(sess, "owner", None)
-    db = SessionLocal()
-    try:
-        q = db.query(ModelEndpoint).filter(
-            ModelEndpoint.is_enabled == True
-        )
-        if owner:
-            from src.auth_helpers import owner_filter
-            q = owner_filter(q, ModelEndpoint, owner)
-        endpoints = q.all()
-    finally:
-        db.close()
-
-    for ep in endpoints:
-        base = normalize_base(ep.base_url)
-        # Skip current endpoint
-        if current_url and base in current_url:
-            continue
-        try:
-            base, api_key = resolve_endpoint_runtime(ep, owner=owner)
-        except Exception:
-            continue
-        ping_url = build_models_url(base)
-        headers = build_headers(api_key, base)
-        try:
-            if ping_url:
-                r = _req.get(ping_url, headers=headers, timeout=5)
-                r.raise_for_status()
-                data = r.json()
-                models = [m.get("id") for m in (data.get("data") or []) if m.get("id")]
-                if not models:
-                    models = [
-                        m.get("name") or m.get("model")
-                        for m in (data.get("models") or [])
-                        if m.get("name") or m.get("model")
-                    ]
-            else:
-                models = json.loads(ep.cached_models or "[]")
-            if not models:
-                continue
-            # Found a working endpoint — update session
-            new_model = models[0]
-            chat_url = build_chat_url(base)
-            new_headers = build_headers(api_key, base)
-            persisted_headers = {} if is_chatgpt_subscription_base(base) else new_headers
-
-            sess.model = new_model
-            sess.endpoint_url = chat_url
-            sess.headers = new_headers
-
-            # Persist
-            _db = SessionLocal()
-            try:
-                _db.query(DBSession).filter(DBSession.id == session_id).update({
-                    "model": new_model,
-                    "endpoint_url": chat_url,
-                    "headers": persisted_headers,
-                })
-                _db.commit()
-            finally:
-                _db.close()
-
-            logger.info(f"Fallback: switched session {session_id} from {current_url} to {ep.name} ({new_model})")
-            return {
-                "model": new_model,
-                "endpoint_url": chat_url,
-                "endpoint_name": ep.name,
-            }
-        except Exception:
-            continue
-
-    return None
-
-
 def extract_preset(chat_handler, preset_id) -> PresetInfo:
     """Extract preset parameters via chat_handler."""
     temperature, max_tokens, system_prompt, char_name = (
@@ -687,6 +623,9 @@ async def build_chat_context(
     use_enhanced_message: bool = False,
     agent_mode: bool = False,
     allow_tool_preprocessing: bool = True,
+    defer_context_shaping: bool = False,
+    continuation_context_message: str | None = None,
+    persist_user_message: bool = True,
 ) -> ChatContext:
     """Build the full context (preface + messages) for an LLM call.
 
@@ -710,14 +649,14 @@ async def build_chat_context(
     # Add user message to history. Nobody/incognito uses a request-local
     # transcript store instead of session history so stale saved chats cannot
     # bleed into context and the turn is not persisted.
-    if incognito:
+    if persist_user_message and incognito:
         user_meta = {"attachments": preprocessed.attachment_meta} if preprocessed.attachment_meta else None
         _append_incognito_message(session_id, "user", preprocessed.user_content, user_meta)
-    else:
+    elif persist_user_message:
         add_user_message(sess, chat_handler, preprocessed, incognito=False)
 
     # Fire events
-    if not incognito:
+    if persist_user_message and not incognito:
         fire_message_event(request, webhook_manager, session_id, sess, message, compare_mode)
 
     # Resolve owner-scoped prefs/context. Browser requests keep the cookie user;
@@ -729,7 +668,12 @@ async def build_chat_context(
         getattr(chat_handler, "upload_handler", None),
         getattr(sess, "owner", None),
     )
-    casual_low_signal = _is_casual_low_signal(message)
+    context_message = (
+        str(continuation_context_message).strip()
+        if continuation_context_message
+        else message
+    )
+    casual_low_signal = _is_casual_low_signal(context_message)
 
     # Memory enabled?
     mem_enabled = not incognito and not no_memory and uprefs.get("memory_enabled", True)
@@ -766,7 +710,15 @@ async def build_chat_context(
     # Build context preface
     # The stream path uses enhanced_message (with CoT/preprocessing applied),
     # the sync path uses text_for_context.
-    _ctx_msg = preprocessed.enhanced_message if use_enhanced_message else preprocessed.text_for_context
+    _ctx_msg = (
+        context_message
+        if continuation_context_message
+        else (
+            preprocessed.enhanced_message
+            if use_enhanced_message
+            else preprocessed.text_for_context
+        )
+    )
     _preface_kwargs = dict(
         message=_ctx_msg,
         session=sess,
@@ -830,13 +782,22 @@ async def build_chat_context(
         except Exception:
             logger.debug("Failed to add current date/time context", exc_info=True)
 
-    # Auto-compact
+    route_messages = list(messages)
-    messages, context_length, was_compacted = await maybe_compact(
+    # Explicit fallback routing must shape from the same route-neutral prompt
-        sess, sess.endpoint_url, sess.model, messages, sess.headers, owner=user,
+    # for every candidate. Running selected-model compaction here would mutate
-    )
+    # session history before we know which route can answer and would make a
+    # later larger-context candidate unable to recover discarded history.
+    if defer_context_shaping:
+        context_length = get_context_length(sess.endpoint_url, sess.model)
+        was_compacted = False
+    else:
+        messages, context_length, was_compacted = await maybe_compact(
+            sess, sess.endpoint_url, sess.model, messages, sess.headers, owner=user,
+        )
     _before_trim_messages = len(messages)
     _before_trim_tokens = estimate_tokens(messages)
-    messages = trim_for_context(messages, context_length)
+    if not defer_context_shaping:
+        messages = trim_for_context(messages, context_length)
     _after_trim_messages = len(messages)
     _after_trim_tokens = estimate_tokens(messages)
     _context_trimmed = _after_trim_messages < _before_trim_messages or _after_trim_tokens < _before_trim_tokens
@@ -860,6 +821,7 @@ async def build_chat_context(
         context_tokens_after_trim=_after_trim_tokens,
         auto_opened_docs=auto_opened_docs,
         uploaded_files=uploaded_files,
+        route_messages=route_messages,
     )
 
 

routes/cookbook_helpers.py

@@ -1204,6 +1204,41 @@ def _safe_env_prefix(ep: str | None) -> str | None:
     return f'[ -f "{path}" ] && source "{path}" || true'
 
 
+def _local_windows_bash_env_prefix(ep: str | None) -> str | None:
+    """Convert a frontend PowerShell venv prefix for the local Git Bash runner."""
+    if not ep:
+        return ep
+
+    prefix = ep.strip()
+    if not prefix.startswith("&"):
+        return ep
+
+    raw_path = prefix[1:].lstrip()
+    if not raw_path:
+        return ep
+    if raw_path.startswith("'"):
+        if len(raw_path) < 2 or not raw_path.endswith("'"):
+            return ep
+        quoted_path = raw_path[1:-1]
+        if "'" in quoted_path.replace("''", ""):
+            return ep
+        path = quoted_path.replace("''", "'")
+    else:
+        path = raw_path.rstrip()
+        if "'" in path or '"' in path:
+            return ep
+    if any(c in path for c in "\r\n;&|`$<>"):
+        return ep
+    if not path.replace("\\", "/").casefold().endswith("/scripts/activate.ps1"):
+        return ep
+
+    bash_path = _git_bash_path(path)
+    if "\\" in bash_path:
+        return ep
+    bash_path = bash_path[: -len("Activate.ps1")] + "activate"
+    return "source " + shlex.quote(bash_path)
+
+
 def _ssh_ps(host, script_path, port=None):
     """Build SSH command to run a PowerShell script on a Windows remote."""
     pf = f"-p {port} " if port and port != "22" else ""

routes/cookbook_routes.py

@@ -50,7 +50,7 @@ from routes.cookbook_helpers import (
     _SESSION_ID_RE, _validate_repo_id, _validate_serve_model_id, _validate_include, _validate_token,
     _validate_local_dir, _validate_gpus, _shell_path,
     _ps_squote, _bash_squote, _validate_serve_cmd, _parse_serve_phase, OLLAMA_MISSING_HINT,
-    _safe_env_prefix, _local_tooling_path_export, _append_serve_preflight_exit_lines,
+    _safe_env_prefix, _local_windows_bash_env_prefix, _local_tooling_path_export, _append_serve_preflight_exit_lines,
     _append_serve_exit_code_lines, _append_llama_cpp_linux_accel_build_lines, _cached_model_scan_script,
     load_stored_hf_token,
     _append_vllm_linux_preflight_lines, _ollama_bind_from_cmd, _pip_install_fallback_chain,
@@ -1336,7 +1336,7 @@ def setup_cookbook_routes() -> APIRouter:
             # Local: run hf download in the background (tmux on POSIX, a detached
             # process + logfile on Windows where tmux doesn't exist).
             if req.env_prefix:
-                lines.append(_safe_env_prefix(req.env_prefix))
+                lines.append(_safe_env_prefix(_local_windows_bash_env_prefix(req.env_prefix) if local_windows else req.env_prefix))
             else:
                 lines.append("deactivate 2>/dev/null; hash -r")
             # Show whether the HF token reached this run (masked) — tells a gated
@@ -2166,7 +2166,7 @@ def setup_cookbook_routes() -> APIRouter:
             if req.gpus:
                 runner_lines.append(f"export CUDA_VISIBLE_DEVICES='{req.gpus}'")
             if req.env_prefix:
-                runner_lines.append(_safe_env_prefix(req.env_prefix))
+                runner_lines.append(_safe_env_prefix(_local_windows_bash_env_prefix(req.env_prefix) if local_windows else req.env_prefix))
             else:
                 runner_lines.append("deactivate 2>/dev/null; hash -r")
             _append_venv_nvidia_library_path_lines(runner_lines, cmd=req.cmd)

routes/email_routes.py

@@ -5004,7 +5004,6 @@ def setup_email_routes():
             from src.endpoint_resolver import (
                 resolve_endpoint,
                 resolve_utility_fallback_candidates,
-                resolve_chat_fallback_candidates,
             )
             from src.llm_core import llm_call_async_with_fallback
 
@@ -5066,8 +5065,6 @@ def setup_email_routes():
                 pass
             for cand in resolve_utility_fallback_candidates(owner=owner) or []:
                 _add(*cand)
-            for cand in resolve_chat_fallback_candidates(owner=owner) or []:
-                _add(*cand)
             if not candidates:
                 return {"success": False, "error": "No LLM endpoint configured"}
 
@@ -5327,13 +5324,11 @@ def setup_email_routes():
             # Build a candidate chain so a stale session-stored API key
             # (the most common cause of "authentication failed" here)
             # doesn't kill AI Reply outright — fall through to the
-            # user's Utility / Default endpoints and the active Utility
+            # user's Utility / Default endpoints and active Utility fallback
-            # fallback chain. The retired default-fallback hook stays empty.
+            # chain. Dedupe by url+model so we don't retry the same endpoint.
-            # Dedupe by url+model so we don't retry the same broken endpoint.
             from src.llm_core import llm_call_async_with_fallback
             from src.endpoint_resolver import (
                 resolve_utility_fallback_candidates,
-                resolve_chat_fallback_candidates,
             )
             _seen = set()
             _candidates = []
@@ -5358,11 +5353,9 @@ def setup_email_routes():
                 _add(_d_url, _d_model, _d_headers)
             except Exception:
                 pass
-            # Active Utility fallbacks, then the retired default hook.
+            # Active Utility fallbacks last.
             for cand in resolve_utility_fallback_candidates(owner=owner) or []:
                 _add(*cand)
-            for cand in resolve_chat_fallback_candidates(owner=owner) or []:
-                _add(*cand)
             _messages = [
                 {"role": "system", "content": system_prompt},
                 {"role": "user", "content": user_msg},

routes/mcp/mcp_routes.py

@@ -475,7 +475,7 @@ def setup_mcp_routes(mcp_manager: McpManager):
                 return RedirectResponse(auth_url)
             else:
                 # Remote device — show paste-back page
-                return HTMLResponse(_oauth_authorize_page(auth_url, server_id, host, redirect_uri))
+                return HTMLResponse(_oauth_authorize_page(auth_url, server_id, redirect_uri))
         finally:
             db.close()
 
@@ -612,15 +612,13 @@ def setup_mcp_routes(mcp_manager: McpManager):
 def _oauth_authorize_page(
     auth_url: str,
     server_id: str,
-    host: str,
+    redirect_uri: str,
-    redirect_uri: str = "http://localhost:7000/api/mcp/oauth/callback",
 ) -> str:
     """Page with Google sign-in link and URL paste-back form for remote access."""
-    # Escape values interpolated into the page: `host` comes from the request
+    # Escape values interpolated into the page: `server_id` comes from the OAuth
-    # Host header and `server_id` from the OAuth state — neither is trusted.
+    # state and is not trusted.
     auth_url = html.escape(auth_url, quote=True)
     server_id = html.escape(server_id, quote=True)
-    host = html.escape(host, quote=True)
     redirect_uri = html.escape(redirect_uri, quote=True)
     return f"""<!DOCTYPE html>
 <html><head>
@@ -664,7 +662,15 @@ def _oauth_authorize_page(
   </div>
   <a class="auth-link" href="{auth_url}" target="_blank" rel="noopener">Sign in with Google</a>
   <div class="divider"></div>
-  <form method="POST" action="http://{host}/api/mcp/oauth/exchange/{server_id}">
+  <!-- Relative action: the browser resolves it against the origin this page was
+       served from, so the form follows the user through any proxy without the
+       app having to know the scheme or the host. An absolute http:// action is
+       blocked as mixed content on exactly the HTTPS deployments that need
+       paste-back, and request.url.scheme cannot be trusted to spot them —
+       uvicorn only honours X-Forwarded-Proto from a peer in
+       --forwarded-allow-ips, which defaults to 127.0.0.1 and excludes a proxy
+       arriving over the Docker bridge. -->
+  <form method="POST" action="/api/mcp/oauth/exchange/{server_id}">
     <p>Paste the URL from your browser after signing in:</p>
     <input type="text" name="callback_url" placeholder="{redirect_uri}?code=..." required>
     <br><button type="submit">Connect</button>

routes/model_routes.py

@@ -46,6 +46,7 @@ _ENDPOINT_SETTING_FIELDS = {
 }
 
 _ENDPOINT_FALLBACK_FIELDS = {
+    "foreground_model_fallbacks": "Foreground Model Fallbacks",
     "utility_model_fallbacks": "Utility Model Fallbacks",
     "vision_model_fallbacks":  "Vision Model Fallbacks",
 }
@@ -180,7 +181,12 @@ def _clear_user_pref_endpoint_refs(all_prefs: dict, ep_id: str) -> int:
     if not isinstance(all_prefs, dict):
         return 0
     users = all_prefs.get("_users")
-    pref_sets = users.values() if isinstance(users, dict) else [all_prefs]
+    # A mixed store can contain auth-disabled foreground policy at the root
+    # alongside named-owner preferences. Both are active namespaces; legacy
+    # `default_model_fallbacks` remains untouched by the field allowlist.
+    pref_sets = [all_prefs]
+    if isinstance(users, dict):
+        pref_sets.extend(users.values())
     cleared_users = 0
     for prefs in pref_sets:
         if isinstance(prefs, dict) and _clear_endpoint_settings_for_endpoint(prefs, ep_id):
@@ -1345,14 +1351,14 @@ def _legacy_visible_api_models(ep) -> List[str]:
 def _picker_models_for_endpoint(ep, base_url: str, kind: str):
     """Return model IDs that should appear in the picker for an endpoint.
 
-    API providers expose remote inventory from /v1/models. Treat that cache as
+    API providers expose remote inventory from /v1/models. Default to that
-    inventory, not approval: only manually pinned API models should appear in
+    visible inventory until an explicit pinned-model allow-list is saved.
-    the picker. Local/self-hosted endpoints keep the older hide-list behavior.
+    Local/self-hosted endpoints keep the older hide-list behavior.
     """
     pinned = _normalize_model_ids(getattr(ep, "pinned_models", None))
     if _picker_requires_pinning(base_url, kind):
         if not _has_explicit_pinned_models(ep):
-            pinned = _legacy_visible_api_models(ep) if _hidden_model_ids(ep) else []
+            pinned = _legacy_visible_api_models(ep)
         return pinned, pinned
     return _visible_models(
         _cached_model_ids(ep),
@@ -2336,9 +2342,7 @@ def setup_model_routes(model_discovery):
                 else:
                     response.headers["X-Model-Refresh-Status"] = "failed"
                     response.headers["X-Model-Refresh-Warning"] = "Model refresh failed or returned no models; kept cached models."
-            pinned = _normalize_model_ids(getattr(ep, "pinned_models", None))
+            _, pinned = _picker_models_for_endpoint(ep, base, kind)
-            if picker_requires_pinning and not _has_explicit_pinned_models(ep):
-                pinned = _legacy_visible_api_models(ep)
             pinned_set = set(pinned)
             return [
                 {

routes/personal_routes.py

@@ -1,11 +1,13 @@
 # routes/personal_routes.py
 """Routes for personal documents management."""
+import asyncio
 import os
 import logging
 import shutil
 import uuid
 from typing import Any, Dict, List, Tuple
 from fastapi import APIRouter, HTTPException, Query, Request, UploadFile, File, Depends
+from fastapi.concurrency import run_in_threadpool
 from src.request_models import DirectoryRequest
 from core.constants import BASE_DIR, PERSONAL_DIR, PERSONAL_UPLOADS_DIR
 from src.rag_singleton import get_rag_manager
@@ -18,7 +20,6 @@ UPLOADS_DIR = PERSONAL_UPLOADS_DIR
 
 logger = logging.getLogger(__name__)
 
-
 def _personal_upload_dir_for_owner(owner: str | None, *, create: bool = True) -> str:
     """Return the per-owner upload directory used for direct RAG uploads."""
     owner_segment = secure_filename((owner or "local").strip())[:80] or "local"
@@ -141,6 +142,22 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
     """
     router = APIRouter(prefix="/api/personal")
 
+    # Serializes directory index jobs across requests. Indexing runs in the
+    # threadpool (#5558), so concurrent requests would otherwise run in parallel
+    # and race PersonalDocsManager's unsynchronized list mutations and file
+    # writes; before the threadpool move they serialized on the blocked event
+    # loop, so one-at-a-time is behavior parity.
+    #
+    # An asyncio.Lock acquired in the async handler BEFORE offloading: a waiting
+    # request parks on the event loop instead of pinning a threadpool worker (an
+    # earlier threading.Lock taken INSIDE the worker meant queued jobs held pool
+    # tokens while blocked, starving every other run_in_threadpool caller).
+    # add/remove/reload all take this lock, so their mutations never interleave.
+    # Per-router (not module-global) so each app binds it to its own event loop.
+    # Scope is the single process: multi-worker deployments would need a shared
+    # lock (out of scope for #5558).
+    _index_job_lock = asyncio.Lock()
+
     def _rag():
         """Get the current RAG manager, retrying init if needed."""
         return get_rag_manager()
@@ -172,8 +189,12 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
         return {"files": files, "directories": directories}
     
     @router.post("/reload")
-    def api_personal_reload(owner: str = Depends(require_user), _admin: None = Depends(require_admin)):
+    async def api_personal_reload(owner: str = Depends(require_user), _admin: None = Depends(require_admin)):
-        personal_docs_manager.refresh_index()
+        # refresh_index() re-extracts text across every tracked directory —
+        # blocking work. Take the shared job lock (so it cannot race an add /
+        # remove) and run it off the event loop.
+        async with _index_job_lock:
+            await run_in_threadpool(personal_docs_manager.refresh_index)
         return {"ok": True, "count": len(personal_docs_manager.index)}
     
     @router.post("/add_directory")
@@ -207,12 +228,26 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
             # Use the RAGManager to index the directory
             rag = _rag()
             if rag:
-                result = rag.index_personal_documents(directory, owner=owner)
+                def _index_directory():
-                
+                    result = rag.index_personal_documents(directory, owner=owner)
+                    if result["success"]:
+                        # Also update the personal_docs_manager to track this
+                        # directory. Kept inside the offloaded call: it triggers
+                        # refresh_index(), which re-extracts text across tracked
+                        # directories.
+                        personal_docs_manager.add_directory(directory, index=False)
+                    return result
+
+                # Indexing walks, embeds, and stores the whole tree — minutes
+                # on a real directory. The handler is async, so calling it
+                # inline runs it on the event loop and every other request
+                # queues behind it until it finishes (#5558). Serialize on the
+                # async job lock BEFORE offloading so a queued request parks on
+                # the loop instead of pinning a threadpool worker.
+                async with _index_job_lock:
+                    result = await run_in_threadpool(_index_directory)
+
                 if result["success"]:
-                    # Also update the personal_docs_manager to track this directory
-                    personal_docs_manager.add_directory(directory, index=False)
-                    
                     return {
                         "success": True,
                         "message": f"Successfully indexed {result['indexed_count']} chunks from {directory}",
@@ -251,17 +286,25 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
 
             logger.info(f"Removing directory from RAG: {directory}")
 
-            # Always remove from personal_docs_manager tracking
-            if hasattr(personal_docs_manager, 'remove_directory'):
-                personal_docs_manager.remove_directory(directory)
-
-            # Remove from RAG vector store (best-effort)
             rag = _rag()
-            if rag:
+
-                try:
+            def _remove_directory():
-                    rag.remove_directory(directory)
+                # Always remove from personal_docs_manager tracking. This
-                except Exception as e:
+                # mutates the same unsynchronized list/index an add job touches
-                    logger.warning(f"RAG removal failed for directory {directory}: {e}")
+                # and re-extracts text (refresh_index), so it is blocking work.
+                if hasattr(personal_docs_manager, 'remove_directory'):
+                    personal_docs_manager.remove_directory(directory)
+                # Remove from RAG vector store (best-effort).
+                if rag:
+                    try:
+                        rag.remove_directory(directory)
+                    except Exception as e:
+                        logger.warning(f"RAG removal failed for directory {directory}: {e}")
+
+            # Same job lock as add/reload so remove cannot interleave with an
+            # in-flight add; offloaded off the event loop.
+            async with _index_job_lock:
+                await run_in_threadpool(_remove_directory)
 
             return {
                 "success": True,
@@ -289,54 +332,73 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
         total_failed = 0
         uploaded_files = []
 
-        for upload in files:
+        # Chunking, embedding and the tracking update are blocking work over the
-            try:
+        # same vector/tracking state add_directory mutates (#5634). Take the
-                file_path, stored_name, safe_name = _unique_personal_upload_path(upload_dir, upload.filename)
+        # shared job lock BEFORE offloading so a queued request parks on the loop
-                content_bytes = await upload.read(PERSONAL_UPLOAD_MAX_BYTES + 1)
+        # instead of pinning a threadpool worker, matching add_directory.
-                if len(content_bytes) > PERSONAL_UPLOAD_MAX_BYTES:
+        # Read and process one capped payload at a time so a multi-file request
-                    logger.warning(f"Rejected oversized personal upload: {upload.filename!r}")
+        # cannot retain len(files) * PERSONAL_UPLOAD_MAX_BYTES in memory.
-                    total_failed += 1
+        async with _index_job_lock:
-                    continue
+            for upload in files:
-                with open(file_path, "wb") as f:
+                try:
-                    f.write(content_bytes)
+                    file_path, stored_name, safe_name = _unique_personal_upload_path(
-
+                        upload_dir, upload.filename
-                ext = os.path.splitext(safe_name)[1].lower()
+                    )
-                if ext == ".pdf":
+                    content_bytes = await upload.read(PERSONAL_UPLOAD_MAX_BYTES + 1)
-                    from src.personal_docs import extract_pdf_text
+                    if len(content_bytes) > PERSONAL_UPLOAD_MAX_BYTES:
-                    text = extract_pdf_text(file_path)
+                        logger.warning(f"Rejected oversized personal upload: {upload.filename!r}")
-                else:
-                    text = content_bytes.decode("utf-8", errors="replace")
-
-                if not text or not text.strip():
-                    total_failed += 1
-                    continue
-
-                # Chunk and index
-                chunks = rag._split_into_chunks(text, chunk_size=500)
-                for i, chunk in enumerate(chunks):
-                    metadata = {
-                        "source": file_path,
-                        "filename": safe_name,
-                        "stored_filename": stored_name,
-                        "directory": upload_dir,
-                        "type": ext,
-                        "chunk_id": i,
-                    }
-                    if user:
-                        metadata["owner"] = user
-                    if rag.add_document(chunk, metadata):
-                        total_indexed += 1
-                    else:
                         total_failed += 1
+                        continue
 
-                uploaded_files.append(safe_name)
+                    def _index_upload():
-            except Exception as e:
+                        with open(file_path, "wb") as f:
-                logger.error(f"Failed to upload/index {upload.filename}: {e}")
+                            f.write(content_bytes)
-                total_failed += 1
 
-        # Track uploads directory
+                        ext = os.path.splitext(safe_name)[1].lower()
-        if uploaded_files and hasattr(personal_docs_manager, "add_directory"):
+                        if ext == ".pdf":
-            personal_docs_manager.add_directory(upload_dir, index=False)
+                            from src.personal_docs import extract_pdf_text
+                            text = extract_pdf_text(file_path)
+                        else:
+                            text = content_bytes.decode("utf-8", errors="replace")
+
+                        if not text or not text.strip():
+                            return 0, 1, None
+
+                        indexed = 0
+                        failed = 0
+                        chunks = rag._split_into_chunks(text, chunk_size=500)
+                        for i, chunk in enumerate(chunks):
+                            metadata = {
+                                "source": file_path,
+                                "filename": safe_name,
+                                "stored_filename": stored_name,
+                                "directory": upload_dir,
+                                "type": ext,
+                                "chunk_id": i,
+                            }
+                            if user:
+                                metadata["owner"] = user
+                            if rag.add_document(chunk, metadata):
+                                indexed += 1
+                            else:
+                                failed += 1
+                        return indexed, failed, safe_name
+
+                    indexed, failed, uploaded_name = await run_in_threadpool(_index_upload)
+                    total_indexed += indexed
+                    total_failed += failed
+                    if uploaded_name:
+                        uploaded_files.append(uploaded_name)
+                except Exception as e:
+                    logger.error(f"Failed to upload/index {upload.filename}: {e}")
+                    total_failed += 1
+
+            # Same transition, same lock: the tracking update must not land
+            # while another job is mid-write over the same state.
+            if uploaded_files and hasattr(personal_docs_manager, "add_directory"):
+                await run_in_threadpool(
+                    personal_docs_manager.add_directory, upload_dir, index=False
+                )
 
         return {
             "success": True,
@@ -349,38 +411,47 @@ def setup_personal_routes(personal_docs_manager, rag_manager, rag_available):
     async def delete_file_from_rag(filepath: str = Query(...), owner: str = Depends(require_user), _admin: None = Depends(require_admin)):
         """Delete a specific file from RAG index and optionally from disk."""
         try:
-            # Remove chunks from RAG vector store (best-effort)
+            def _delete_file():
-            removed = 0
+                # Remove chunks from RAG vector store (best-effort)
-            rag = _rag()
+                removed = 0
-            if rag:
+                rag = _rag()
-                try:
+                if rag:
-                    removed = rag.delete_by_source(filepath)
+                    try:
-                except Exception as e:
+                        removed = rag.delete_by_source(filepath)
-                    logger.warning(f"RAG removal failed for {filepath}: {e}")
+                    except Exception as e:
+                        logger.warning(f"RAG removal failed for {filepath}: {e}")
 
-            # Delete file from disk if it's in the caller's own uploads dir.
+                # Delete file from disk if it's in the caller's own uploads dir.
-            # Scope to the per-owner subdir, not the shared uploads root, so one
+                # Scope to the per-owner subdir, not the shared uploads root, so one
-            # admin can't delete another user's personal files by path.
+                # admin can't delete another user's personal files by path.
-            deleted_from_disk = False
+                deleted_from_disk = False
-            try:
-                abs_target = os.path.realpath(filepath)
-                base_abs = os.path.realpath(_personal_upload_dir_for_owner(owner, create=False))
-                in_uploads = (
-                    abs_target == base_abs
-                    or os.path.commonpath([abs_target, base_abs]) == base_abs
-                )
-            except ValueError:
-                # commonpath raises on mixed drives / non-comparable paths
-                in_uploads = False
-            if in_uploads and abs_target != base_abs:
                 try:
-                    os.remove(abs_target)
+                    abs_target = os.path.realpath(filepath)
-                    deleted_from_disk = True
+                    base_abs = os.path.realpath(_personal_upload_dir_for_owner(owner, create=False))
-                except FileNotFoundError:
+                    in_uploads = (
-                    pass  # already gone — race with another request or cleanup
+                        abs_target == base_abs
+                        or os.path.commonpath([abs_target, base_abs]) == base_abs
+                    )
+                except ValueError:
+                    # commonpath raises on mixed drives / non-comparable paths
+                    in_uploads = False
+                if in_uploads and abs_target != base_abs:
+                    try:
+                        os.remove(abs_target)
+                        deleted_from_disk = True
+                    except FileNotFoundError:
+                        pass  # already gone — race with another request or cleanup
 
-            # Exclude the file from the listing (persists across restarts)
+                # Exclude the file from the listing (persists across restarts)
-            personal_docs_manager.exclude_file(filepath)
+                personal_docs_manager.exclude_file(filepath)
+                return removed, deleted_from_disk
+
+            # Vector removal, the disk unlink and the exclusion write are one
+            # transition over the same state add_directory mutates (#5634), and
+            # all three block. Take the shared job lock BEFORE offloading, as
+            # add_directory does.
+            async with _index_job_lock:
+                removed, deleted_from_disk = await run_in_threadpool(_delete_file)
 
             return {
                 "success": True,

routes/prefs_routes.py

@@ -7,6 +7,10 @@ from src.auth_helpers import get_current_user
 from src.constants import USER_PREFS_FILE
 
 PREFS_FILE = USER_PREFS_FILE
+_FOREGROUND_POLICY_KEYS = (
+    "foreground_fallback_enabled",
+    "foreground_model_fallbacks",
+)
 
 
 def _load():
@@ -26,14 +30,27 @@ def _save(prefs):
 def _load_for_user(user: Optional[str] = None) -> dict:
     """Load preferences for a specific user."""
     all_prefs = _load()
-    if "_users" in all_prefs:
+    users = all_prefs.get("_users")
+    if isinstance(users, dict):
         if user is None:
             # Auth disabled — return first user's prefs for backward compat
-            users = all_prefs["_users"]
+            prefs = dict(next(iter(users.values()), {}))
-            return dict(next(iter(users.values()), {}))
+            # Foreground fallback consent is never borrowed from a named
-        return dict(all_prefs["_users"].get(user, {}))
+            # owner. Auth-disabled operation has a separate flat/root opt-in
-    # Legacy flat format — return as-is
+            # that remains inert when authentication is enabled again.
-    return dict(all_prefs)
+            for key in _FOREGROUND_POLICY_KEYS:
+                prefs.pop(key, None)
+                if key in all_prefs:
+                    prefs[key] = all_prefs[key]
+            return prefs
+        prefs = users.get(user, {})
+        return dict(prefs) if isinstance(prefs, dict) else {}
+    # A legacy flat store belongs only to auth-disabled single-user mode.
+    # Copying it into the first named user's new `_users` record during an
+    # auth transition would silently transfer another user's preferences and,
+    # critically, foreground fallback consent. Named owners therefore start
+    # with an empty record and must write their own preferences explicitly.
+    return dict(all_prefs) if user is None else {}
 
 
 def _save_for_user(user: Optional[str], prefs: dict):
@@ -45,17 +62,40 @@ def _save_for_user(user: Optional[str], prefs: dict):
         # `prefs` flat would overwrite the whole `_users` map and destroy every
         # other user's preferences. Instead write back into the same (first)
         # slot _load_for_user(None) reads from, preserving the others.
-        if "_users" in all_prefs:
+        users = all_prefs.get("_users")
-            users = all_prefs["_users"]
+        if isinstance(users, dict):
             first_key = next(iter(users), None)
             if first_key is not None:
-                users[first_key] = prefs
+                existing_named = users.get(first_key)
+                existing_named = (
+                    dict(existing_named)
+                    if isinstance(existing_named, dict)
+                    else {}
+                )
+                named_foreground = {
+                    key: existing_named[key]
+                    for key in _FOREGROUND_POLICY_KEYS
+                    if key in existing_named
+                }
+                users[first_key] = {
+                    key: value
+                    for key, value in prefs.items()
+                    if key not in _FOREGROUND_POLICY_KEYS
+                }
+                users[first_key].update(named_foreground)
+                for key in _FOREGROUND_POLICY_KEYS:
+                    if key in prefs:
+                        all_prefs[key] = prefs[key]
                 _save(all_prefs)
                 return
         _save(prefs)
         return
-    if "_users" not in all_prefs:
+    if not isinstance(all_prefs.get("_users"), dict):
-        all_prefs = {"_users": {}}
+        # Preserve the flat single-user object as inert legacy data while
+        # creating the first named-owner namespace. In particular, historical
+        # fallback values must not be deleted or copied into the new owner.
+        all_prefs = dict(all_prefs)
+        all_prefs["_users"] = {}
     all_prefs["_users"][user] = prefs
     _save(all_prefs)
 

routes/research/research_routes.py

@@ -15,7 +15,7 @@ from pydantic import BaseModel, Field
 from core.middleware import INTERNAL_TOOL_USER
 from src.endpoint_resolver import resolve_endpoint
 from src.auth_helpers import _auth_disabled, get_current_user
-from core.auth import RESERVED_USERNAMES
+from src.owner_identity import REQUEST_SENTINEL_OWNERS
 from src.constants import DEEP_RESEARCH_DIR
 
 _SESSION_ID_RE = re.compile(r"^[a-zA-Z0-9-]{1,128}$")
@@ -496,7 +496,7 @@ def setup_research_routes(research_handler, session_manager=None) -> APIRouter:
         user = require_privilege(request, "can_use_research")
         if user == INTERNAL_TOOL_USER:
             tool_owner = (request.headers.get("X-Odysseus-Owner") or "").strip()
-            if tool_owner and tool_owner not in RESERVED_USERNAMES:
+            if tool_owner and tool_owner not in REQUEST_SENTINEL_OWNERS:
                 auth_mgr = getattr(request.app.state, "auth_manager", None)
                 if auth_mgr is not None and getattr(auth_mgr, "is_configured", False):
                     try:

routes/skills_routes.py

@@ -18,6 +18,7 @@ from pydantic import BaseModel, Field
 
 from services.memory.skills import SkillsManager
 from src.auth_helpers import get_current_user
+from src.prompt_security import untrusted_context_message
 from core.middleware import require_admin
 
 logger = logging.getLogger(__name__)
@@ -107,6 +108,23 @@ def _skill_test_task(skill: dict) -> str:
     )
 
 
+def _skill_test_messages(md: str, task: str) -> list[dict]:
+    """Keep user-editable skill text out of the trusted system role."""
+    return [
+        {
+            "role": "system",
+            "content": (
+                "You are TESTING a skill. Follow the supplied reusable procedure "
+                "to complete the user's task for real, using available tools step "
+                "by step. If the skill is wrong, unclear, or references tools that "
+                "do not exist, do your best; the problems will be reviewed afterward."
+            ),
+        },
+        untrusted_context_message("skill under test", md),
+        {"role": "user", "content": task},
+    ]
+
+
 async def _eval_skill_run(skill_md: str, task: str, transcript: str,
                           url: str, model: str, headers: Optional[dict]) -> dict:
     """LLM-as-judge: grade a skill test run from its transcript. Advisory only.
@@ -411,7 +429,21 @@ async def _eval_skill_retrieval_precision(skill_md: str, others: list,
 _skill_test_jobs: dict = {}
 
 
-async def _run_skill_test_job(key, name, md, task, url, model, headers, owner, skills_manager=None):
+async def _run_skill_test_job(
+    key,
+    name,
+    md,
+    task,
+    url,
+    model,
+    headers,
+    owner,
+    skills_manager=None,
+    *,
+    messages=None,
+    transcript=None,
+    exact_approval=None,
+):
     """Background coroutine: run the skill in an agent loop, capture a condensed
     log + transcript, then have the judge grade it. Writes into _skill_test_jobs."""
     import json as _json
@@ -421,7 +453,7 @@ async def _run_skill_test_job(key, name, md, task, url, model, headers, owner, s
     if job is None:
         return
     log = job["log"]
-    transcript = []
+    transcript = transcript if isinstance(transcript, list) else []
     say_buf = []
 
     def _flush_say():
@@ -429,18 +461,12 @@ async def _run_skill_test_job(key, name, md, task, url, model, headers, owner, s
             log.append({"type": "say", "text": "".join(say_buf)})
             say_buf.clear()
 
-    messages = [
+    messages = list(messages) if isinstance(messages, list) else _skill_test_messages(md, task)
-        {"role": "system", "content":
-            "You are TESTING a skill. Below is a reusable skill (a procedure). Follow it "
-            "to complete the user's task for real, using your available tools, step by "
-            "step. If the skill is wrong, unclear, or references tools that don't exist, "
-            "do your best — the problems will be reviewed afterward.\n\n=== SKILL ===\n" + md},
-        {"role": "user", "content": task},
-    ]
     try:
         async for chunk in stream_agent_loop(
             url, model, messages, headers=headers,
             temperature=0.3, max_tokens=0, max_rounds=8, owner=owner,
+            exact_approval=exact_approval,
         ):
             if not chunk.startswith("data: ") or chunk.strip() == "data: [DONE]":
                 continue
@@ -458,8 +484,25 @@ async def _run_skill_test_job(key, name, md, task, url, model, headers, owner, s
             elif d.get("type") == "tool_output":
                 _flush_say()
                 out = str(d.get("output") or "")[:600]
-                log.append({"type": "tool_output", "output": out})
+                tool_log = {"type": "tool_output", "output": out}
+                approval = d.get("ask_user")
+                if isinstance(approval, dict):
+                    tool_log["ask_user"] = approval
+                log.append(tool_log)
                 transcript.append(f"[output] {out}\n")
+                if (
+                    isinstance(approval, dict)
+                    and approval.get("kind") == "tool_approval"
+                    and approval.get("approval_id")
+                ):
+                    # Manual skill tests have their own polling UI instead of a
+                    # chat session. Pause the run and retain only server-side
+                    # continuation state until the same owner approves/denies
+                    # this exact sealed action.
+                    job["status"] = "awaiting_approval"
+                    job["approval"] = approval
+                    job["_transcript"] = transcript
+                    return
             elif d.get("type") == "agent_step":
                 _flush_say()
                 log.append({"type": "agent_step", "round": d.get("round")})
@@ -471,6 +514,9 @@ async def _run_skill_test_job(key, name, md, task, url, model, headers, owner, s
         _flush_say()
         log.append({"type": "error", "error": str(e)})
 
+    job.pop("approval", None)
+    job.pop("_transcript", None)
+    job.pop("_run", None)
     log.append({"type": "evaluating"})
     try:
         job["verdict"] = await _eval_skill_run(md, task, "".join(transcript), url, model, headers)
@@ -694,12 +740,8 @@ async def _run_skill_test_once(md: str, task: str, url, model, headers, owner) -
     import json as _json
     from src.agent_loop import stream_agent_loop
     transcript = []
-    messages = [
+    approval_required = None
-        {"role": "system", "content":
+    messages = _skill_test_messages(md, task)
-            "You are TESTING a skill. Follow this skill's procedure to complete the task "
-            "for real, using your tools, step by step.\n\n=== SKILL ===\n" + md},
-        {"role": "user", "content": task},
-    ]
     try:
         # max_tokens explicitly set: passing 0 lets some upstreams (Ollama,
         # OpenAI-compat) generate an empty completion, which manifested as
@@ -719,11 +761,44 @@ async def _run_skill_test_once(md: str, task: str, url, model, headers, owner) -
                 transcript.append(f"\n[tool {d.get('tool')}] {str(d.get('command') or d.get('args') or '')[:300]}\n")
             elif d.get("type") == "tool_output":
                 transcript.append(f"[output] {str(d.get('output') or '')[:600]}\n")
+                approval = d.get("ask_user")
+                if (
+                    isinstance(approval, dict)
+                    and approval.get("kind") == "tool_approval"
+                ):
+                    approval_required = approval
+                    break
             elif d.get("type") == "agent_step":
                 transcript.append(f"\n--- round {d.get('round')} ---\n")
     except Exception as e:
         transcript.append(f"\n[run error] {e}\n")
     text = "".join(transcript)
+    if approval_required is not None:
+        # Unattended audits have no authority to approve and no UI that could
+        # resume this record. Destructively deny it now instead of leaving a
+        # reusable opaque grant pending until TTL/cap eviction.
+        try:
+            from src.tool_approvals import tool_approval_store
+            tool_approval_store.consume(
+                approval_required.get("approval_id"),
+                decision="deny",
+                owner=owner,
+                session_id=None,
+            )
+        except Exception:
+            logger.debug("Could not retire unattended skill approval", exc_info=True)
+        return text, {
+            "verdict": "inconclusive",
+            "confidence": 1.0,
+            "summary": (
+                "This automated audit reached an exact action that requires "
+                "a human approval; no action was executed."
+            ),
+            "issues": [
+                "Run this skill's manual test and review the sealed action."
+            ],
+            "approval_required": True,
+        }
     verdict = await _eval_skill_run(md, task, text, url, model, headers)
     return text, verdict
 
@@ -863,6 +938,26 @@ async def _audit_one_skill(skills_manager, skill, url, model, headers,
     transcript, verdict = await _run_skill_test_once(md, task, url, model, headers, owner)
     v = verdict.get("verdict")
     log(f"{name}: verdict = {v} ({verdict.get('summary', '')[:80]})")
+    if verdict.get("approval_required"):
+        # An unattended audit is not authority for an action influenced by the
+        # skill under test. Preserve the skill's current publication/confidence
+        # state and route the exact action to the manual test UI instead of
+        # letting a safety pause demote, rewrite, or auto-publish the skill.
+        skills_manager.set_audit(
+            name,
+            "inconclusive",
+            by_teacher=False,
+            worker_model=model,
+            owner=owner,
+        )
+        status = skill.get("status") or "draft"
+        log(f"{name}: {status} unchanged — exact action needs manual approval")
+        return {
+            "skill": name,
+            "result": "approval_required",
+            "verdict": verdict,
+            "status": status,
+        }
     if v == "pass":
         # Procedure works. If the reviewer still flagged metadata (tags/category/
         # when_to_use/description), do ONE fixer pass to correct the frontmatter
@@ -1431,6 +1526,19 @@ def setup_skills_routes(skills_manager: SkillsManager) -> APIRouter:
             logger.warning(f"Skill-test model resolve failed: {_e}")
 
         key = (user or "", name)
+        previous_job = _skill_test_jobs.get(key) or {}
+        previous_approval = previous_job.get("approval") or {}
+        if previous_approval.get("approval_id"):
+            try:
+                from src.tool_approvals import tool_approval_store
+                tool_approval_store.consume(
+                    previous_approval["approval_id"],
+                    decision="deny",
+                    owner=user,
+                    session_id=None,
+                )
+            except Exception:
+                logger.debug("Could not retire replaced skill approval", exc_info=True)
         _skill_test_jobs[key] = {
             "status": "running",
             "task": task,
@@ -1439,10 +1547,138 @@ def setup_skills_routes(skills_manager: SkillsManager) -> APIRouter:
             "started": _time.time(),
             "log": [{"type": "skill_test_start", "task": task, "skill": name, "model": model}],
             "verdict": None,
+            "_run": {
+                "md": md,
+                "url": url,
+                "model": model,
+                "headers": headers,
+                "owner": user,
+            },
         }
         _asyncio.create_task(_run_skill_test_job(key, name, md, task, url, model, headers, user, skills_manager))
         return {"ok": True, "status": "running", "skill": name, "model": model}
 
+    @router.post("/{skill_id}/test-approval")
+    async def approve_skill_test_action(request: Request, skill_id: str):
+        """Resume a manual skill test with one exact server-sealed action."""
+        import asyncio as _asyncio
+        from src.tool_approvals import tool_approval_store
+
+        user = _owner(request)
+        skills = skills_manager.load(owner=user)
+        match = next(
+            (s for s in skills if s.get("name") == skill_id or s.get("id") == skill_id),
+            None,
+        )
+        if not match:
+            raise HTTPException(404, "Skill not found")
+        _verify_owner(match, user)
+        name = match.get("name")
+        key = (user or "", name)
+        job = _skill_test_jobs.get(key)
+        if not job or job.get("status") != "awaiting_approval":
+            raise HTTPException(409, "This skill test is not awaiting an approval.")
+
+        body = await request.json()
+        if not isinstance(body, dict):
+            raise HTTPException(400, "Tool approval body must be a JSON object.")
+        approval_id = str(body.get("approval_id") or "")
+        decision = str(body.get("decision") or "").strip().lower()
+        expected = job.get("approval") or {}
+        if approval_id != str(expected.get("approval_id") or ""):
+            raise HTTPException(409, "This approval does not match the pending skill test action.")
+        if decision not in {"approve", "deny"}:
+            raise HTTPException(400, "Invalid tool approval decision.")
+
+        pending = tool_approval_store.peek(approval_id)
+        normalized_owner = str(user or "").strip().casefold()
+        if (
+            pending is None
+            or pending.owner != normalized_owner
+            or pending.session_id != ""
+        ):
+            raise HTTPException(409, "This tool approval is invalid or expired.")
+        exact_approval = tool_approval_store.consume(
+            approval_id,
+            decision=decision,
+            owner=user,
+            session_id=None,
+            # The button here says "Allow once" and there is no chat to carry a
+            # scope into, so the gate must re-arm behind the sealed action.
+            allow_continuation=False,
+        )
+
+        if decision == "approve" and exact_approval is None:
+            raise HTTPException(409, "This tool approval could not be consumed.")
+        job.pop("approval", None)
+        if decision == "deny":
+            job.pop("_transcript", None)
+            job.pop("_run", None)
+            job["log"].append({
+                "type": "approval_denied",
+                "text": "Exact action denied; the skill test stopped without executing it.",
+            })
+            job["verdict"] = {
+                "verdict": "inconclusive",
+                "confidence": 1.0,
+                "summary": "The test stopped because its exact action was denied.",
+                "issues": [],
+            }
+            job["status"] = "done"
+            return {"ok": True, "status": "done", "decision": "deny"}
+
+        run = job.get("_run") or {}
+        transcript = job.pop("_transcript", [])
+        # stream_agent_loop owns its per-round message list internally. Rebuild
+        # continuation context from the original untrusted skill plus the
+        # accumulated transcript so repeated approvals do not lose earlier
+        # approved results, while keeping every transcript byte tainted.
+        messages = _skill_test_messages(
+            run.get("md", ""),
+            job.get("task", ""),
+        )
+        if transcript:
+            messages.append(untrusted_context_message(
+                "skill test transcript",
+                "".join(str(item) for item in transcript),
+            ))
+        messages.extend([
+            {
+                "role": "assistant",
+                "content": str(expected.get("question") or "Allow this exact action once?"),
+            },
+            {
+                "role": "user",
+                "content": (
+                    f"Approved the exact {exact_approval.pending.tool_name} "
+                    "action shown above once."
+                ),
+            },
+        ])
+        job["status"] = "running"
+        job["log"].append({
+            "type": "approval_granted",
+            "text": (
+                f"Approved exact {exact_approval.pending.tool_name} action once; "
+                "resuming test."
+            ),
+        })
+        _asyncio.create_task(_run_skill_test_job(
+            key,
+            name,
+            run.get("md", ""),
+            job.get("task", ""),
+            run.get("url"),
+            run.get("model"),
+            run.get("headers"),
+            run.get("owner"),
+            skills_manager,
+            messages=messages,
+            transcript=transcript,
+            exact_approval=exact_approval,
+        ))
+        return {"ok": True, "status": "running", "decision": "approve"}
+
     @router.get("/{skill_id}/test-status")
     async def test_skill_status(request: Request, skill_id: str):
         """Current background-test state for a skill (status / log / verdict)."""
@@ -1459,6 +1695,7 @@ def setup_skills_routes(skills_manager: SkillsManager) -> APIRouter:
             "model": job.get("model"),
             "log": job.get("log", []),
             "verdict": job.get("verdict"),
+            "approval": job.get("approval"),
         }
 
     @router.post("/audit-all")

routes/task/__init__.py

@@ -0,0 +1,5 @@
+"""Task route domain package (slice 2p, #4082/#4071).
+
+Contains task_routes.py, migrated from the flat routes/ directory.
+Backward-compat shim at routes/task_routes.py re-exports from here.
+"""

scripts/migrate_searxng_settings.py

@@ -0,0 +1,166 @@
+#!/usr/bin/env python3
+"""Make retained SearXNG settings inherit defaults without replacing them."""
+
+from __future__ import annotations
+
+import os
+import stat
+import sys
+import tempfile
+from pathlib import Path
+
+import yaml
+from yaml.nodes import MappingNode
+from yaml.tokens import BlockMappingStartToken, FlowMappingStartToken
+
+
+_UTF8_BOM = b"\xef\xbb\xbf"
+
+
+def _parse_root_mapping(text: str) -> tuple[MappingNode | None, dict]:
+    """Parse settings with the same safe YAML semantics SearXNG uses."""
+    try:
+        loaded = yaml.safe_load(text)
+        node = yaml.compose(text, Loader=yaml.SafeLoader)
+    except yaml.YAMLError:
+        raise ValueError("settings file is not valid single-document YAML") from None
+
+    if loaded is None and node is None:
+        return None, {}
+    if not isinstance(loaded, dict) or not isinstance(node, MappingNode):
+        raise ValueError("settings root is not a mapping")
+    return node, loaded
+
+
+def _flow_mapping_start(text: str) -> int:
+    """Return the root flow mapping's opening-brace character offset."""
+    try:
+        for token in yaml.scan(text, Loader=yaml.SafeLoader):
+            if isinstance(token, FlowMappingStartToken):
+                return token.start_mark.index
+    except yaml.YAMLError:
+        pass
+    raise ValueError("flow-style settings mapping has no opening brace")
+
+
+def _newline_for(contents: bytes) -> bytes:
+    first_lf = contents.find(b"\n")
+    if first_lf > 0 and contents[first_lf - 1 : first_lf + 1] == b"\r\n":
+        return b"\r\n"
+    return b"\n"
+
+
+def _block_mapping_position(text: str, root: MappingNode | None) -> tuple[int, int]:
+    """Return a safe character offset and indent for a root block mapping key."""
+    if root is None:
+        return len(text), 0
+
+    try:
+        for token in yaml.scan(text, Loader=yaml.SafeLoader):
+            if not isinstance(token, BlockMappingStartToken):
+                continue
+            line_start = token.start_mark.index - token.start_mark.column
+            if not text[line_start : token.start_mark.index].strip():
+                return line_start, token.start_mark.column
+            return root.end_mark.index, token.start_mark.column
+    except yaml.YAMLError:
+        pass
+    return root.end_mark.index, root.start_mark.column
+
+
+def _add_block_default_inheritance(
+    contents: bytes, text: str, root: MappingNode | None
+) -> bytes:
+    newline = _newline_for(contents)
+    character_offset, indent_width = _block_mapping_position(text, root)
+    bom_length = len(_UTF8_BOM) if contents.startswith(_UTF8_BOM) else 0
+    offset = bom_length + len(text[:character_offset].encode("utf-8"))
+    separator = b""
+    if offset not in (0, bom_length) and not contents[:offset].endswith((b"\n", b"\r")):
+        separator = newline
+    addition = (
+        separator
+        + b" " * indent_width
+        + b"use_default_settings: true"
+        + newline
+    )
+    return contents[:offset] + addition + contents[offset:]
+
+
+def migrate_settings(path: Path) -> bool:
+    """Add the missing inheritance key atomically; return whether the file changed."""
+    source_stat = path.lstat()
+    if not stat.S_ISREG(source_stat.st_mode):
+        raise ValueError(f"settings path is not a regular file: {path}")
+
+    contents = path.read_bytes()
+    if not contents:
+        return False
+
+    text = contents.decode("utf-8-sig")
+    root, loaded = _parse_root_mapping(text)
+    if "use_default_settings" in loaded:
+        return False
+
+    if root is not None and root.flow_style:
+        start = _flow_mapping_start(text)
+        bom_length = len(_UTF8_BOM) if contents.startswith(_UTF8_BOM) else 0
+        offset = bom_length + len(text[: start + 1].encode("utf-8"))
+        separator = b", " if root.value else b""
+        updated = (
+            contents[:offset]
+            + b"use_default_settings: true"
+            + separator
+            + contents[offset:]
+        )
+    else:
+        updated = _add_block_default_inheritance(contents, text, root)
+    fd, temporary_name = tempfile.mkstemp(
+        prefix=f".{path.name}.odysseus-", dir=path.parent
+    )
+    temporary = Path(temporary_name)
+    try:
+        # chmod before chown: the Compose cap set is `cap_drop: ALL` plus
+        # CHOWN/SETGID/SETUID/DAC_OVERRIDE, with no FOWNER. Once the temporary
+        # file belongs to searxng:searxng — which every retained settings file
+        # does, because searxng's entrypoint chowns /etc/searxng — root can no
+        # longer chmod it and the migration dies with EPERM.
+        os.fchmod(fd, stat.S_IMODE(source_stat.st_mode))
+        os.fchown(fd, source_stat.st_uid, source_stat.st_gid)
+        with os.fdopen(fd, "wb") as handle:
+            fd = -1
+            handle.write(updated)
+            handle.flush()
+            os.fsync(handle.fileno())
+        os.replace(temporary, path)
+        directory_fd = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY)
+        try:
+            os.fsync(directory_fd)
+        finally:
+            os.close(directory_fd)
+    finally:
+        if fd >= 0:
+            os.close(fd)
+        temporary.unlink(missing_ok=True)
+    return True
+
+
+def main(argv: list[str]) -> int:
+    if len(argv) > 2:
+        print(f"usage: {Path(argv[0]).name} [settings.yml]", file=sys.stderr)
+        return 2
+
+    path = Path(argv[1]) if len(argv) == 2 else Path("/etc/searxng/settings.yml")
+    try:
+        changed = migrate_settings(path)
+    except (OSError, UnicodeError, ValueError) as exc:
+        print(f"SearXNG settings migration failed: {exc}", file=sys.stderr)
+        return 1
+
+    if changed:
+        print("Added use_default_settings inheritance to retained SearXNG settings")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv))

scripts/mlx_image_server.py

@@ -327,7 +327,12 @@ def list_models():
 
 @app.post("/v1/images/generations")
 def generate(req: ImageRequest):
-    model = req.model or _args.model
+    # The served model is the one this process was launched with. `req.model`
+    # is accepted for OpenAI wire compatibility and ignored, matching
+    # scripts/diffusion_server.py: honouring it would let a caller point the
+    # generator at any local directory or Hugging Face repo, and the HiDream
+    # branch runs a python script from inside that directory.
+    model = _args.model
     width, height = _size(req.size)
     out_images = []
     count = max(1, min(int(req.n or 1), 4))
@@ -393,7 +398,7 @@ async def edit_image(
     size: str = Form("1024x1024"),
     response_format: str = Form("b64_json"),
 ):
-    active_model = model or _args.model
+    active_model = _args.model  # pinned; see generate()
     if _is_lama_inpaint(active_model) or _is_ddcolor(active_model):
         image_raw = await image.read()
         mask_raw = await mask.read() if mask is not None else None

scripts/odysseus-webhook

@@ -2,7 +2,7 @@
 """odysseus-webhook — shell wrapper for scheduled-task webhook tokens.
 
 Tasks in the scheduled-task system can carry a `webhook_token`. Any
-HTTP POST to `/api/webhook/<token>` fires the task. This CLI lists,
+HTTP POST to `/api/tasks/<task-id>/webhook/<token>` fires the task. This CLI lists,
 rotates, and revokes those tokens.
 
     odysseus-webhook list                       # tasks that have a token
@@ -21,6 +21,7 @@ quiet_logs()
 
 import argparse, json, logging, os, secrets, sys
 from pathlib import Path
+from urllib.parse import quote
 
 try:
     from core.database import SessionLocal, ScheduledTask
@@ -53,6 +54,14 @@ def _summary(t: "ScheduledTask", reveal: bool = False) -> dict:
     }
 
 
+def _task_webhook_url(base: str, task_id: str, token: str) -> str:
+    """Build the live task-route URL without leaking ids into path syntax."""
+    root = (base or "http://localhost:7000").rstrip("/")
+    task_part = quote(str(task_id), safe="")
+    token_part = quote(str(token), safe="")
+    return f"{root}/api/tasks/{task_part}/webhook/{token_part}"
+
+
 def cmd_list(args):
     db = SessionLocal()
     try:
@@ -109,8 +118,7 @@ def cmd_url(args):
             fail(f"no task with id {args.id!r}")
         if not t.webhook_token:
             fail(f"task {args.id!r} has no webhook token (rotate one first)")
-        base = (args.base or "http://localhost:7000").rstrip("/")
+        url = _task_webhook_url(args.base, t.id, t.webhook_token)
-        url = f"{base}/api/webhook/{t.webhook_token}"
         emit({
             "task_id": t.id,
             "name": t.name,

services/docs/service.py

@@ -50,16 +50,46 @@ class DocsService:
             List of DocChunk objects
         """
         results = self.rag.search(query, k=top_k)
-        return [
+        chunks = []
-            DocChunk(
+
-                text=r.get("text", r.get("content", "")),
+        for result in results:
-                source=r.get("source", r.get("metadata", {}).get("source", "unknown")),
+            if not isinstance(result, dict):
-                score=r.get("score", 0.0),
+                continue
-                metadata=r.get("metadata"),
+
+            metadata = result.get("metadata")
+            if not isinstance(metadata, dict):
+                metadata = {}
+
+            text = result.get("document")
+            if text is None:
+                text = result.get("text")
+            if text is None:
+                text = result.get("content")
+            if text is None:
+                text = ""
+
+            source = result.get("source")
+            if source is None:
+                source = metadata.get("source")
+            if source is None:
+                source = "unknown"
+
+            score = result.get("similarity")
+            if score is None:
+                score = result.get("score")
+            if score is None:
+                score = 0.0
+
+            chunks.append(
+                DocChunk(
+                    text=text,
+                    source=source,
+                    score=score,
+                    metadata=metadata,
+                )
             )
-            for r in results
+
-            if isinstance(r, dict)
+        return chunks
-        ]
 
     async def index(self, directory: str) -> IndexResult:
         """
@@ -73,8 +103,8 @@ class DocsService:
         """
         result = self.rag.index_personal_documents(directory)
         return IndexResult(
-            indexed=result.get("indexed", 0),
+            indexed=result.get("indexed_count", result.get("indexed", 0)),
-            failed=result.get("failed", 0),
+            failed=result.get("failed_count", result.get("failed", 0)),
             errors=result.get("errors", []),
         )
 

services/memory/skill_importer.py

@@ -1,16 +1,18 @@
 """Import SKILL.md bundles from public GitHub (or skills.sh → GitHub) URLs."""
 from __future__ import annotations
 
+import ipaddress
 import logging
 import os
-import re
+import time
 from dataclasses import dataclass
-from typing import Dict, List, Optional, Tuple
+from typing import Dict, Iterable, List, Optional, Tuple, cast
 from urllib.parse import quote, urljoin, urlparse
 
+import httpcore
 import httpx
 
-from src.url_safety import check_outbound_url
+from src.url_safety import _default_resolver, check_outbound_url
 
 logger = logging.getLogger(__name__)
 
@@ -25,6 +27,7 @@ TEXT_NAMES = {"skill.md", "license", "license.md", "readme.md"}
 _GITHUB_HOSTS = frozenset({
     "github.com", "www.github.com", "api.github.com", "raw.githubusercontent.com",
 })
+_SKILLS_SH_HOSTS = frozenset({"skills.sh", "www.skills.sh"})
 
 
 def _github_host(url: str) -> str:
@@ -72,18 +75,158 @@ def _is_text_file(name: str) -> bool:
 _MAX_FETCH_REDIRECTS = 5
 
 
-def _check_fetch_url(url: str) -> None:
+def _validated_ips(raw_ips: List[str]) -> List[ipaddress._BaseAddress]:
-    """SSRF guard for skill-import fetches (defense-in-depth).
+    """Parse and de-duplicate one resolver snapshot in resolver order."""
+    ips: List[ipaddress._BaseAddress] = []
+    seen = set()
+    for raw in raw_ips:
+        if not isinstance(raw, str):
+            continue
+        try:
+            ip = ipaddress.ip_address(raw.split("%", 1)[0])
+        except ValueError:
+            continue
+        if ip in seen:
+            continue
+        seen.add(ip)
+        ips.append(ip)
+    return ips
 
-    Skill bundles only ever come from public GitHub, never an internal
+
-    address, so block private/loopback/link-local targets on every hop —
+def _resolve_and_check_url(url: str) -> List[ipaddress._BaseAddress]:
-    matching the hardened web-fetch path in
+    """Return the exact address snapshot approved for one fetch hop."""
-    ``services/search/content.py:_get_public_url`` rather than the lenient
+    resolved_ips: List[str] = []
-    default used for admin-configured model endpoints.
+
-    """
+    def _recording_resolver(host: str) -> List[str]:
-    ok, reason = check_outbound_url(url, block_private=True)
+        answers = list(_default_resolver(host))
+        resolved_ips[:] = answers
+        return answers
+
+    ok, reason = check_outbound_url(
+        url,
+        block_private=True,
+        resolver=_recording_resolver,
+    )
     if not ok:
-        raise SkillImportError(reason)
+        raise SkillImportError(f"outbound URL blocked: {reason}")
+
+    pinned_ips = _validated_ips(resolved_ips)
+    if not pinned_ips:
+        raise SkillImportError("outbound URL blocked: host did not resolve to a usable address")
+    return pinned_ips
+
+
+# Backward compatibility alias for tests importing _check_fetch_url directly
+_check_fetch_url = _resolve_and_check_url
+
+
+class _PinnedBackend(httpcore.NetworkBackend):
+    """Connect only to addresses from one validated DNS snapshot."""
+
+    def __init__(self, ips: List[ipaddress._BaseAddress]):
+        self._ips = [str(ip) for ip in ips]
+        self._real = httpcore.SyncBackend()
+
+    def connect_tcp(
+        self,
+        host: str,
+        port: int,
+        timeout: float | None = None,
+        local_address: str | None = None,
+        socket_options=None,
+    ):
+        deadline = None if timeout is None else time.monotonic() + timeout
+        last_exc: Optional[Exception] = None
+        for ip in self._ips:
+            remaining = None if deadline is None else max(0.0, deadline - time.monotonic())
+            try:
+                return self._real.connect_tcp(
+                    ip,
+                    port,
+                    remaining,
+                    local_address,
+                    socket_options,
+                )
+            except (httpcore.ConnectError, httpcore.ConnectTimeout) as exc:
+                last_exc = exc
+                if deadline is not None and time.monotonic() >= deadline:
+                    break
+        if last_exc is not None:
+            raise last_exc
+        raise httpcore.ConnectError("no validated address available")
+
+    def connect_unix_socket(self, path, timeout=None, socket_options=None):
+        return self._real.connect_unix_socket(path, timeout, socket_options)
+
+    def sleep(self, seconds: float) -> None:
+        return self._real.sleep(seconds)
+
+
+_HTTPCORE_TO_HTTPX_EXC = {
+    httpcore.ConnectError: httpx.ConnectError,
+    httpcore.ConnectTimeout: httpx.ConnectTimeout,
+    httpcore.LocalProtocolError: httpx.LocalProtocolError,
+    httpcore.NetworkError: httpx.NetworkError,
+    httpcore.PoolTimeout: httpx.PoolTimeout,
+    httpcore.ProtocolError: httpx.ProtocolError,
+    httpcore.ProxyError: httpx.ProxyError,
+    httpcore.ReadError: httpx.ReadError,
+    httpcore.ReadTimeout: httpx.ReadTimeout,
+    httpcore.RemoteProtocolError: httpx.RemoteProtocolError,
+    httpcore.TimeoutException: httpx.TimeoutException,
+    httpcore.UnsupportedProtocol: httpx.UnsupportedProtocol,
+    httpcore.WriteError: httpx.WriteError,
+    httpcore.WriteTimeout: httpx.WriteTimeout,
+}
+
+
+class _PinnedTransport(httpx.BaseTransport):
+    """Pin socket connects while preserving URL authority, Host, and TLS SNI."""
+
+    def __init__(self, ips: List[ipaddress._BaseAddress]):
+        self._pinned_ips = list(ips)
+        self._pool = httpcore.ConnectionPool(
+            ssl_context=httpx.create_ssl_context(),
+            http1=True,
+            http2=False,
+            network_backend=_PinnedBackend(ips),
+        )
+
+    def handle_request(self, request: httpx.Request) -> httpx.Response:
+        core_request = httpcore.Request(
+            method=request.method,
+            url=httpcore.URL(
+                scheme=request.url.raw_scheme,
+                host=request.url.raw_host,
+                port=request.url.port,
+                target=request.url.raw_path,
+            ),
+            headers=request.headers.raw,
+            content=request.stream,
+            extensions=request.extensions,
+        )
+        core_response = None
+        try:
+            core_response = self._pool.handle_request(core_request)
+            content = b"".join(cast(Iterable[bytes], core_response.stream))
+        except Exception as exc:
+            mapped = _HTTPCORE_TO_HTTPX_EXC.get(type(exc))
+            if mapped is not None:
+                raise mapped(str(exc)) from exc
+            raise
+        finally:
+            if core_response is not None:
+                core_response.close()
+
+        return httpx.Response(
+            status_code=core_response.status,
+            headers=core_response.headers,
+            content=content,
+            extensions=core_response.extensions,
+        )
+
+    def close(self) -> None:
+        self._pool.close()
 
 
 def _get_checked(
@@ -100,49 +243,76 @@ def _get_checked(
     hand lets us re-validate every hop, closing that blind-SSRF gap.
     """
     current = url
-    with httpx.Client(follow_redirects=False, timeout=timeout) as client:
+    for _ in range(_MAX_FETCH_REDIRECTS + 1):
-        for _ in range(_MAX_FETCH_REDIRECTS + 1):
+        pinned_ips = _resolve_and_check_url(current)
-            _check_fetch_url(current)
+        with httpx.Client(
+            transport=_PinnedTransport(pinned_ips),
+            follow_redirects=False,
+            timeout=timeout,
+        ) as client:
             r = client.get(current, headers=headers)
-            if r.status_code in (301, 302, 303, 307, 308):
+
-                location = r.headers.get("location")
+        if r.status_code in (301, 302, 303, 307, 308):
-                if not location:
+            location = r.headers.get("location")
-                    return r
+            if not location:
-                current = urljoin(str(r.url), location)
+                return r
-                continue
+            current = urljoin(str(r.url), location)
-            return r
+            continue
+        return r
     raise SkillImportError("too many redirects while fetching skill bundle")
 
 
 def parse_skill_source(url: str) -> ResolvedSource:
     """Normalize skills.sh / GitHub web URLs into owner/repo/ref/path."""
-    raw = (url or "").strip()
+    url = (url or "").strip()
-    if not raw:
+    if not url:
         raise SkillImportError("URL is required")
 
-    # skills.sh often links to GitHub; try to unwrap ?url= or redirect target later.
+    # ``urlparse`` only reports an unambiguous scheme when the URL carries the
-    if "skills.sh" in raw and "github.com" not in raw:
+    # ``scheme://`` form. Opaque schemes (``mailto:``, ``javascript:``) and a
-        r = _get_checked(raw, timeout=20.0)
+    # schemeless ``host:port`` both parse a "scheme" that is not one, so they
+    # fall through to the host check below and are rejected on the host instead.
+    scheme = urlparse(url).scheme.lower()
+    if scheme not in ("http", "https"):
+        if scheme and url.lower().startswith(f"{scheme}://"):
+            raise SkillImportError(f"unsupported URL scheme: {scheme}")
+        # Schemeless "github.com/owner/repo" — accept only a supported host.
+        rough_host = (urlparse("//" + url).hostname or "").lower()
+        if rough_host not in _GITHUB_HOSTS and rough_host not in _SKILLS_SH_HOSTS:
+            raise SkillImportError("Only GitHub or skills.sh URLs are supported")
+        url = "https://" + url
+
+    parsed = urlparse(url)
+    hostname = (parsed.hostname or "").lower()
+    if hostname not in _GITHUB_HOSTS and hostname not in _SKILLS_SH_HOSTS:
+        raise SkillImportError("Only GitHub or skills.sh URLs are supported")
+
+    # A skills.sh link is only usable if it redirects to an exact supported
+    # GitHub host. Scraping the page body for a github.com link cannot work:
+    # skill pages only ever link the repository root, never the skill's
+    # subdirectory, so the scrape resolves every skill in a repo to the same
+    # (wrong) bundle. Fail with an actionable message instead.
+    if hostname in _SKILLS_SH_HOSTS:
+        r = _get_checked(url, timeout=20.0)
         if r.status_code >= 400:
             raise _github_response_error(r)
         final = str(r.url)
-        _assert_github_url(final, context="redirect target")
+        if _github_host(final) not in _GITHUB_HOSTS:
-        # Page may embed a github link; prefer final URL if redirected.
+            raise SkillImportError(
-        if "github.com" in final:
+                "skills.sh did not redirect to GitHub — open the skill's "
-            raw = final
+                "repository on GitHub, navigate to the exact skill folder or "
-        else:
+                "SKILL.md file, and paste that URL; the repository-root link "
-            m = re.search(r"https?://github\.com/[^\s\"')]+", r.text or "")
+                "alone is not sufficient"
-            if m:
+            )
-                raw = m.group(0).rstrip(".,)")
+        url = final
 
-    parsed = urlparse(raw)
+    # Update parsed and hostname to reflect the new GitHub URL
-    host = _github_host(raw)
+    parsed = urlparse(url)
-    if host not in _GITHUB_HOSTS:
+    hostname = (parsed.hostname or "").lower()
-        raise SkillImportError(
-            "Only GitHub URLs are supported (https://github.com/... or raw.githubusercontent.com/...)"
-        )
 
-    if host == "raw.githubusercontent.com":
+    _assert_github_url(url)
+
+    if hostname == "raw.githubusercontent.com":
         # /owner/repo/ref/path/to/file
         bits = [p for p in parsed.path.split("/") if p]
         if len(bits) < 4:

services/search/content.py

@@ -2,22 +2,18 @@
 
 import copy
 import io
-import ipaddress
 import json
 import os
 import re
 import logging
-import socket
-import ssl
 from datetime import datetime, timedelta
-from typing import Iterable, List, cast
+from typing import List
-from urllib.parse import urljoin, urlparse
 
 import httpx
-import httpcore
 from bs4 import BeautifulSoup
 
 from src.constants import WEB_FETCH_SOFT_MAX_BYTES, WEB_FETCH_HARD_MAX_BYTES, WEB_FETCH_USER_AGENT
+from src import outbound_fetch as _outbound_fetch
 
 from .analytics import RateLimitError, error_logger
 from .cache import (
@@ -29,336 +25,40 @@ from .cache import (
 
 logger = logging.getLogger(__name__)
 
-_PRIVATE_NETWORKS = (
+def _is_private_address(addr):
-    ipaddress.ip_network("0.0.0.0/8"),
+    return _outbound_fetch._is_private_address(addr)
-    ipaddress.ip_network("10.0.0.0/8"),
-    ipaddress.ip_network("127.0.0.0/8"),
-    ipaddress.ip_network("169.254.0.0/16"),
-    ipaddress.ip_network("172.16.0.0/12"),
-    ipaddress.ip_network("192.168.0.0/16"),
-    ipaddress.ip_network("::1/128"),
-    ipaddress.ip_network("fc00::/7"),
-    ipaddress.ip_network("fe80::/10"),
-)
 
 
-def _is_private_address(addr: ipaddress._BaseAddress) -> bool:
+def _resolve_hostname_ips(hostname):
-    if isinstance(addr, ipaddress.IPv6Address) and addr.ipv4_mapped is not None:
+    return _outbound_fetch._resolve_hostname_ips(hostname)
-        addr = addr.ipv4_mapped
+
-    return (
+
-        addr.is_private
+def _public_http_url(url):
-        or addr.is_loopback
+    return _outbound_fetch._public_http_url(url, resolver=_resolve_hostname_ips)
-        or addr.is_link_local
+
-        or addr.is_reserved
+
-        or addr.is_multicast
+def _resolve_public_ips(url):
-        or addr.is_unspecified
+    return _outbound_fetch._resolve_public_ips(url, resolver=_resolve_hostname_ips)
-        or any(addr in net for net in _PRIVATE_NETWORKS)
+
+
+_PinnedBackend = _outbound_fetch._PinnedBackend
+_PinnedTransport = _outbound_fetch._PinnedTransport
+BodyTooLargeError = _outbound_fetch.BodyTooLargeError
+_CappedFetch = _outbound_fetch._CappedFetch
+
+
+def _get_public_url(url, headers, timeout, max_redirects=5, max_bytes=None):
+    return _outbound_fetch._get_public_url(
+        url,
+        headers=headers,
+        timeout=timeout,
+        max_redirects=max_redirects,
+        max_bytes=max_bytes,
+        resolve_public_ips=_resolve_public_ips,
+        transport_factory=_PinnedTransport,
     )
 
 
-def _resolve_hostname_ips(hostname: str) -> list[ipaddress._BaseAddress]:
-    try:
-        infos = socket.getaddrinfo(hostname, None)
-    except Exception:
-        return []
-    out = []
-    for info in infos:
-        try:
-            out.append(ipaddress.ip_address(info[4][0]))
-        except Exception:
-            continue
-    return out
-
-
-def _public_http_url(url: str) -> bool:
-    try:
-        parsed = urlparse(url)
-        if parsed.scheme not in ("http", "https"):
-            return False
-        host = (parsed.hostname or "").strip()
-        if not host:
-            return False
-        lower = host.lower()
-        if lower in ("localhost", "metadata", "metadata.google.internal"):
-            return False
-        if lower.endswith((".local", ".localhost", ".internal", ".lan", ".intranet")):
-            return False
-        try:
-            return not _is_private_address(ipaddress.ip_address(host))
-        except ValueError:
-            pass
-        addrs = _resolve_hostname_ips(host)
-        return bool(addrs) and not any(_is_private_address(a) for a in addrs)
-    except Exception:
-        return False
-
-
-def _resolve_public_ips(url: str) -> list[ipaddress._BaseAddress]:
-    parsed = urlparse(url)
-    if parsed.scheme not in ("http", "https") or not parsed.hostname:
-        raise httpx.RequestError(f"Blocked non-public URL: {url}")
-    host = (parsed.hostname or "").strip().lower()
-    if host in ("localhost", "metadata", "metadata.google.internal"):
-        raise httpx.RequestError(f"Blocked non-public hostname: {host}")
-    try:
-        ip = ipaddress.ip_address(host)
-        if _is_private_address(ip):
-            raise httpx.RequestError(f"Blocked non-public IP literal: {host}")
-        return [ip]
-    except httpx.RequestError:
-        raise
-    except ValueError:
-        pass
-    addrs = _resolve_hostname_ips(host)
-    if not addrs or any(_is_private_address(a) for a in addrs):
-        raise httpx.RequestError(f"Blocked non-public URL: {url}")
-    return addrs
-
-
-class _PinnedBackend(httpcore.NetworkBackend):
-    """Network backend that connects to a pre-resolved IP.
-
-    httpcore derives the TLS SNI and the ``Host`` header from the URL's
-    origin, not from the host argument passed to ``connect_tcp``. So
-    routing the TCP connect to a resolved IP while leaving the URL
-    untouched keeps SNI / vhost behaviour correct and closes the
-    DNS-rebinding TOCTOU between the SSRF check and the connect.
-    """
-
-    def __init__(self, ip: ipaddress._BaseAddress):
-        self._ip = str(ip)
-        self._real = httpcore.SyncBackend()
-
-    def connect_tcp(
-        self,
-        host: str,
-        port: int,
-        timeout: float | None = None,
-        local_address: str | None = None,
-        socket_options=None,
-    ):
-        return self._real.connect_tcp(
-            self._ip, port, timeout, local_address, socket_options
-        )
-
-    def connect_unix_socket(self, path, timeout=None, socket_options=None):
-        return self._real.connect_unix_socket(path, timeout, socket_options)
-
-    def sleep(self, seconds: float) -> None:
-        return self._real.sleep(seconds)
-
-
-# Map httpcore exception classes to their httpx equivalents. Built
-# once at import time from the public exception classes; avoids any
-# import of httpx's private transport machinery. httpcore's
-# ``ConnectionNotAvailable`` is a pool-internal signal (the pool will
-# close and retry on its own) — we never expect to see it surface to
-# a transport caller, so it has no httpx counterpart here.
-_HTTPCORE_TO_HTTPX_EXC = {
-    httpcore.ConnectError: httpx.ConnectError,
-    httpcore.ConnectTimeout: httpx.ConnectTimeout,
-    httpcore.LocalProtocolError: httpx.LocalProtocolError,
-    httpcore.NetworkError: httpx.NetworkError,
-    httpcore.PoolTimeout: httpx.PoolTimeout,
-    httpcore.ProtocolError: httpx.ProtocolError,
-    httpcore.ProxyError: httpx.ProxyError,
-    httpcore.ReadError: httpx.ReadError,
-    httpcore.ReadTimeout: httpx.ReadTimeout,
-    httpcore.RemoteProtocolError: httpx.RemoteProtocolError,
-    httpcore.TimeoutException: httpx.TimeoutException,
-    httpcore.UnsupportedProtocol: httpx.UnsupportedProtocol,
-    httpcore.WriteError: httpx.WriteError,
-    httpcore.WriteTimeout: httpx.WriteTimeout,
-}
-
-
-class _PinnedTransport(httpx.BaseTransport):
-    """Transport that pins every TCP connect to a pre-resolved IP.
-
-    Uses only the public ``httpcore`` and ``httpx`` APIs — no
-    subclassing of ``httpx.HTTPTransport``, no reads of private
-    ``httpcore.ConnectionPool`` attributes, no imports from
-    ``httpx private transport internals``. The URL is passed through unchanged so SNI
-    / vhost work as if httpx had been given the hostname directly;
-    only the TCP destination is pinned, closing the DNS-rebinding
-    TOCTOU between the SSRF check and the connect.
-    """
-
-    def __init__(self, ip: ipaddress._BaseAddress, *, http2: bool = False):
-        self._pool = httpcore.ConnectionPool(
-            ssl_context=ssl.create_default_context(),
-            http1=True,
-            http2=http2,
-            network_backend=_PinnedBackend(ip),
-        )
-
-    def __enter__(self):
-        self._pool.__enter__()
-        return self
-
-    def __exit__(self, exc_type=None, exc_value=None, traceback=None) -> None:
-        self._pool.__exit__(exc_type, exc_value, traceback)
-
-    def handle_request(self, request: httpx.Request) -> httpx.Response:
-        httpcore_req = httpcore.Request(
-            method=request.method,
-            url=httpcore.URL(
-                scheme=request.url.raw_scheme,
-                host=request.url.raw_host,
-                port=request.url.port,
-                target=request.url.raw_path,
-            ),
-            headers=request.headers.raw,
-            content=request.stream,
-            extensions=request.extensions,
-        )
-        try:
-            httpcore_resp = self._pool.handle_request(httpcore_req)
-            # Eager materialisation matches the original
-            # ``response.text`` usage in fetch_webpage_content. The
-            # sync pool's stream is a plain Iterable[bytes] despite
-            # the httpcore type hint unioning the async variant.
-            content = b"".join(cast(Iterable[bytes], httpcore_resp.stream))
-        except Exception as exc:
-            mapped = _HTTPCORE_TO_HTTPX_EXC.get(type(exc))
-            if mapped is not None:
-                raise mapped(str(exc)) from exc
-            raise
-
-        return httpx.Response(
-            status_code=httpcore_resp.status,
-            headers=httpcore_resp.headers,
-            content=content,
-            extensions=httpcore_resp.extensions,
-        )
-
-    def close(self) -> None:
-        self._pool.close()
-
-class BodyTooLargeError(Exception):
-    """The server declared a body larger than the hard fetch ceiling."""
-
-    def __init__(self, url: str, declared_bytes: int):
-        self.url = url
-        self.declared_bytes = declared_bytes
-        super().__init__(
-            f"response body is {declared_bytes:,} bytes, over the "
-            f"{WEB_FETCH_HARD_MAX_BYTES:,}-byte hard cap"
-        )
-
-
-class _CappedFetch:
-    """Result of a size-capped streaming GET.
-
-    Carries just what fetch_webpage_content needs from an httpx.Response,
-    plus the cap bookkeeping: the (possibly truncated) body, whether the
-    cap cut it short, and the size the server declared via Content-Length
-    (wire bytes; None when absent).
-    """
-
-    __slots__ = ("status_code", "headers", "content", "truncated",
-                 "declared_bytes", "encoding", "url")
-
-    def __init__(self, status_code, headers, content, truncated,
-                 declared_bytes, encoding, url):
-        self.status_code = status_code
-        self.headers = headers
-        self.content = content
-        self.truncated = truncated
-        self.declared_bytes = declared_bytes
-        self.encoding = encoding
-        self.url = url
-
-    @property
-    def text(self) -> str:
-        return self.content.decode(self.encoding or "utf-8", errors="replace")
-
-    def raise_for_status(self):
-        if self.status_code >= 400:
-            request = httpx.Request("GET", self.url)
-            raise httpx.HTTPStatusError(
-                f"HTTP {self.status_code} for {self.url}",
-                request=request,
-                response=httpx.Response(self.status_code, request=request),
-            )
-
-
-def _get_public_url(url: str, headers: dict, timeout: int, max_redirects: int = 5,
-                    max_bytes: int = None) -> "_CappedFetch":
-    """Capped streaming GET with SSRF-guarded, DNS-pinned manual redirects.
-
-    Each hop is resolved once, validated as public, and then the actual TCP
-    connection is pinned to that resolved IP. The request URL is left unchanged
-    so Host and TLS SNI keep the original hostname.
-    """
-    cap = min(max_bytes or WEB_FETCH_SOFT_MAX_BYTES, WEB_FETCH_HARD_MAX_BYTES)
-    current = url
-    for _ in range(max_redirects + 1):
-        ips = _resolve_public_ips(current)
-
-        # Force identity transfer-encoding. With gzip/deflate the wire bytes
-        # and Content-Length can be a small fraction of the decoded body, so a
-        # tiny compressed response could pass the hard-cap preflight and then
-        # expand past the ceiling in one decoded chunk before the streamed cap
-        # below can slice it.
-        req_headers = dict(headers or {})
-        req_headers["Accept-Encoding"] = "identity"
-
-        with httpx.Client(
-            headers=req_headers,
-            timeout=timeout,
-            follow_redirects=False,
-            transport=_PinnedTransport(ips[0]),
-        ) as client:
-            with client.stream("GET", current) as response:
-                if response.status_code in (301, 302, 303, 307, 308):
-                    location = response.headers.get("location")
-                    if not location:
-                        return _CappedFetch(response.status_code, response.headers, b"",
-                                            False, None, response.encoding, str(response.url))
-                    current = urljoin(str(response.url), location)
-                    continue
-
-                # A server can ignore the identity request and still return a
-                # compressed body; httpx.iter_bytes would then decode it, and a
-                # tiny gzip can balloon into one decoded chunk far past the cap.
-                # Refuse compressed Content-Encoding so the streamed cap stays
-                # a real memory bound.
-                enc = (response.headers.get("content-encoding") or "").strip().lower()
-                if enc and enc != "identity":
-                    raise httpx.RequestError(
-                        f"Refusing compressed response (Content-Encoding: {enc}) after "
-                        "requesting identity: cannot bound decoded body size",
-                        request=httpx.Request("GET", current),
-                    )
-
-                declared = None
-                raw_len = response.headers.get("content-length")
-                if raw_len and raw_len.isdigit():
-                    declared = int(raw_len)
-
-                if declared is not None and declared > WEB_FETCH_HARD_MAX_BYTES:
-                    raise BodyTooLargeError(current, declared)
-
-                chunks = []
-                read = 0
-                truncated = False
-                for chunk in response.iter_bytes():
-                    read += len(chunk)
-                    if read > cap:
-                        keep = cap - (read - len(chunk))
-                        if keep > 0:
-                            chunks.append(chunk[:keep])
-                        truncated = True
-                        break
-                    chunks.append(chunk)
-
-                return _CappedFetch(response.status_code, response.headers,
-                                    b"".join(chunks), truncated, declared,
-                                    response.encoding, str(response.url))
-
-    raise httpx.RequestError("Too many redirects", request=httpx.Request("GET", current))
-
 # PDF extraction (optional dependency)
 try:
     from pdfminer.high_level import extract_text as pdf_extract_text

specs/_readme.md

@@ -0,0 +1,88 @@
+# Specs DocumentMap
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+This folder is the compact implementation-truth map for humans and coding agents working on Odysseus. Read this file first, then open only the subsystem specs that match the work.
+
+Specs are living notes about current code shape and intended contracts. They are not product marketing, not PR planning, not templates, and not a replacement for source inspection or tests.
+
+This `_readme.md` is the DocumentMap and control document. It is intentionally exempt from subsystem `Scope` and `Current Gaps` sections; keep it limited to the quality contract, working rules, subsystem map, and cross-cutting update triggers.
+
+## Quality Contract
+
+Each subsystem spec should stay compact and useful under context pressure:
+
+- Start with `Last updated: dev@<short-sha> | YYYY-MM-DD`, using the
+  upstream `dev` commit the spec text was inspected against.
+- Use a concrete `Scope` section that names real files, route surfaces, frontend modules, data stores, and integration points.
+- Use domain-specific sections. Do not force every spec into the same headings when the subsystem needs `Streaming`, `Tool Results`, `Optional Dependencies`, `Current Gaps`, or another focused section.
+- State ownership clearly: which file owns a mapping, which layer only forwards state, and which caller requests behavior without owning implementation.
+- Include runtime behavior bullets for flows that matter.
+- Include "Current call sites include" when behavior is spread across many files.
+- Record transitional compatibility notes, especially `src/` versus `services/` duplication.
+- Record degraded, optional, or platform behavior where it changes runtime expectations.
+- Record policy/provenance where relevant: untrusted context, encrypted secrets, API token scopes, optional dependency/license implications, generated media, or user data.
+- End with `Current Gaps` only when there is a real known gap, not as filler.
+
+If code and specs disagree, treat code as ground truth. Update specs only when
+the current task explicitly includes spec maintenance or the PR intentionally
+includes specs; otherwise report the drift in the relevant issue, PR review, or
+project documentation.
+
+## Working Rules
+
+- Start here before substantial work.
+- Read the related subsystem spec before changing code in that area. For cross-cutting work, include the owning domain spec plus route/runtime, auth/security, persistence, frontend, tool/context, integration, and testing/devops specs as applicable.
+- Treat specs as read-only context during ordinary project work, PR review, and code review. Do not edit specs unless the user explicitly asks for spec work or the current PR intentionally includes spec changes.
+- During explicit spec-maintenance work, update the related spec when source inspection shows behavior, ownership, security boundaries, data shape, import paths, or implementation contracts have changed.
+- During ordinary work, record source/spec drift in the relevant issue, PR review, or project documentation instead of mutating specs.
+- Keep specs dense but readable. Prefer current facts and invariants over broad explanation.
+- Every non-index `specs/*.md` file should appear exactly once in the Subsystem Map with a one-line description and no dead link.
+- Specs contain implementation truth. Planning, research, branch notes, and decisions belong in tracked project docs. Drafts, audit reports, raw exports, and exploratory gap lists are not authoritative until promoted into tracked docs or specs.
+- Use repo source and these specs as the authority for Odysseus architecture. Do not treat global skill registries or external agent metadata as repo ground truth.
+
+## Subsystem Map
+
+- [runtime.md](runtime.md): FastAPI startup, router registration, static serving, lifespan, app-wide middleware.
+- [auth-security.md](auth-security.md): auth, privileges, API tokens, security headers, untrusted data, SSRF and admin boundaries.
+- [persistence.md](persistence.md): SQLite models, startup migrations, encrypted columns, ownership columns, data directory rules.
+- [chat.md](chat.md): chat routes, sessions, streaming, uploads-in-chat, compare handoff, research/chat mode dispatch.
+- [compare.md](compare.md): model A/B comparison runs, voting/history, compare frontend panes, compare ownership.
+- [llm-models.md](llm-models.md): LLM provider calls, endpoint discovery, model context length, fallbacks, model endpoints.
+- [model-capability-canonical.md](model-capability-canonical.md): canonical provider/model capability shapes, evidence, payload resolution, and safe fallback.
+- [model-quirks.md](model-quirks.md): model-specific behavior observations, evidence, and promotion gates.
+- [model-providers/_readme.md](model-providers/_readme.md): provider-by-provider API/catalog shape index and compatibility status.
+- [agent-tools.md](agent-tools.md): agent loop, tool schemas, tool execution, tool retrieval, tool security, MCP tool exposure.
+- [context-building.md](context-building.md): URL/search/RAG/memory/skills/YouTube/email/tool-output context, untrusted wrapping, unavailable context, intent boundaries.
+- [search.md](search.md): web search providers, ranking, cache/analytics, URL fetch/content extraction, `src.search`/`services.search` split.
+- [documents-rag-uploads.md](documents-rag-uploads.md): uploads, documents, PDF/form handling, personal docs, RAG/vector stores.
+- [memory-skills.md](memory-skills.md): memory storage, semantic memory, skill extraction/formatting, owner isolation.
+- [research.md](research.md): deep research jobs, synthesis, sources, research library, research UI panel.
+- [calendar-tasks-notes.md](calendar-tasks-notes.md): CalDAV calendars, scheduled tasks, reminders, assistant runs, notes/todos.
+- [email-contacts.md](email-contacts.md): IMAP/SMTP email, email library, scheduled mail, contacts/CardDAV.
+- [gallery-editor-media.md](gallery-editor-media.md): gallery, generated media, image editor drafts, signatures, emoji/font helpers.
+- [cookbook-hwfit.md](cookbook-hwfit.md): model downloads, local/remote model serving, hardware detection, fit ranking.
+- [speech.md](speech.md): STT and TTS services, routes, settings, optional dependencies.
+- [frontend.md](frontend.md): static SPA, module loading, UI conventions, major JS areas, no-build frontend shape.
+- [integrations.md](integrations.md): Codex/Claude scoped APIs, companion pairing, webhooks, external agent access.
+- [shell-mcp.md](shell-mcp.md): shell execution, background jobs, MCP manager, built-in MCP servers.
+- [settings-admin.md](settings-admin.md): settings, preferences, presets, backup/import/export, diagnostics, admin wipe.
+- [testing-devops.md](testing-devops.md): pytest, JS tests, Docker, scripts, requirements, local dev expectations.
+
+## Cross-Cutting Spec Update Triggers
+
+Use these triggers only during explicit spec-maintenance work or a PR that
+intentionally includes specs. For ordinary work and code review, use the same
+list to choose which specs to read and where to report drift.
+
+- New route file or route prefix: update [runtime.md](runtime.md) and the owning subsystem spec.
+- New SQLAlchemy model, column migration, durable JSON/local store, data directory, backup/import domain, or non-SQL persistence behavior: update [persistence.md](persistence.md) and the owning subsystem spec.
+- New tool, tool schema, agent prompt rule, or tool security behavior: update [agent-tools.md](agent-tools.md) and [context-building.md](context-building.md) if it adds model context.
+- New MCP runtime/config/built-in behavior: update [shell-mcp.md](shell-mcp.md), [agent-tools.md](agent-tools.md), and [context-building.md](context-building.md) when MCP tool results enter model context.
+- New external content source, tool result, MCP/app API result, or integration result shown to an LLM: update [context-building.md](context-building.md) and [auth-security.md](auth-security.md).
+- New API-token scope, scoped external API, webhook, companion/pairing route, generic integration provider, or external-agent helper bundle: update [integrations.md](integrations.md), [auth-security.md](auth-security.md), and the owning subsystem spec.
+- New secret store, decrypted-secret return path, settings backup/import/export behavior, diagnostics/log output, vault/tool secret flow, `.env*` policy change, or credential-bearing CLI output: update [auth-security.md](auth-security.md), [settings-admin.md](settings-admin.md), [testing-devops.md](testing-devops.md), and the owning subsystem spec.
+- New optional dependency, degraded fallback, platform/Docker/native/launcher difference, GPU overlay behavior, or retired compatibility shim: update [testing-devops.md](testing-devops.md) and the owning subsystem spec; also update [runtime.md](runtime.md), [llm-models.md](llm-models.md), [shell-mcp.md](shell-mcp.md), [cookbook-hwfit.md](cookbook-hwfit.md), or [persistence.md](persistence.md) when that layer owns the behavior.
+- New frontend module or modal/tool surface: update [frontend.md](frontend.md) and the owning subsystem spec.
+- New static/PWA/service-worker/cache/CSP behavior: update [frontend.md](frontend.md), [runtime.md](runtime.md), and [auth-security.md](auth-security.md) when headers or trust boundaries change.
+- New CLI script: update [testing-devops.md](testing-devops.md) and the owning subsystem spec.

specs/agent-tools.md

@@ -0,0 +1,157 @@
+# Agent Tools
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+This spec covers agent/tool behavior in:
+
+- `src/agent_loop.py`;
+- `src/llm_core.py`;
+- `src/tool_schemas.py`;
+- `src/tool_execution.py`;
+- `src/tool_policy.py`;
+- `src/tool_index.py`;
+- `src/tool_parsing.py`;
+- `src/tool_security.py`;
+- `src/tool_capabilities.py`;
+- `src/tool_approval_scopes.py`;
+- `src/tool_approvals.py`;
+- `src/attachment_refs.py` and shared upload lifecycle helpers in
+  `src/upload_handler.py` / `src/tool_utils.py`;
+- `src/tool_implementations.py`;
+- `src/tools/*.py`;
+- `src/builtin_actions.py`;
+- `src/ai_interaction.py`;
+- `src/action_intents.py`;
+- `src/goal_based_extractor.py`;
+- `src/teacher_escalation.py`;
+- `src/agent_tools/` modules and compatibility facade;
+- `src/mcp_manager.py`;
+- `src/builtin_mcp.py`;
+- `src/bg_jobs.py` and `src/bg_monitor.py`;
+- `routes/chat_routes.py`, `routes/chat_helpers.py`, `routes/model_routes.py`, `routes/skills_routes.py`, canonical `routes/mcp/mcp_routes.py` plus its shim, and `routes/workspace_routes.py`;
+- `mcp_servers/*.py`;
+- frontend stream/admin/settings files that display tool events, workspaces, and disabled tools;
+- `tests/test_agent_loop.py`, `tests/test_tool_*`, and focused MCP/public-policy/schema tests.
+
+## Agent Loop
+
+`src.agent_loop` owns agent prompt assembly, request-local current date/time insertion, tool retrieval, prompted tool-block handling, native tool-call consumption after `llm_core` normalizes provider events, multi-round execution, tool result insertion, final metrics, and fallback responses. It requests context from documents, skills, tool retrieval, and messages; it should not own domain-specific business logic for every tool. Its prompt rules now bias structured/long-form writing toward living documents, route active compose/email drafts back into existing email documents, and prefer first-class `web_search`/`web_fetch` tools over shell/Python/curl for current web lookups when web tools are enabled.
+
+`src.llm_core` owns provider payloads, native tool-schema emission, and provider stream parsing. `agent_loop` consumes normalized tool-call events and decides whether and how to execute them.
+
+Agent mode enters through chat routes, including auto-escalation from intent helpers, detached `agent_runs` streaming, resume/stop behavior, and frontend tool-event rendering.
+
+Guide-only/no-tools turns are runtime policy, not prompt advice. `src.tool_policy` detects strong latest-turn directives such as guide-only mode, no-tools mode, and explicit requests not to use tools; it builds a `ToolPolicy` that hides schemas, disables known native tools, disables MCP for that turn, skips tool retrieval, suppresses local/workspace context injection, blocks document streaming/teacher escalation, and gives `tool_execution` a final execution backstop.
+
+Plan mode is a read-only investigation path inside the same loop. It adds a denylist for known mutating tools, filters write/unknown MCP tools, prepends plan-mode instructions, and uses the `update_plan` tool only after a plan is approved for execution. The backend path still exists for compatibility, but current browser chat forces incoming `plan_mode` off and the old plan-window UI module is gone.
+
+Workspace mode is request-scoped. Admin chat can send a workspace directory selected through `static/js/workspace.js`; `agent_loop` injects that fact early in the prompt and `tool_execution` confines bash, python, read/write/edit-file, and code-navigation tools to that root. `routes.workspace_routes` owns admin-only browse/vet APIs, skips hidden/symlink directory traversal, caps listings, and rejects sensitive/root paths before a workspace reaches chat.
+
+## Tool Registry
+
+Tool registration is split:
+
+- `src.agent_tools` is now a package/facade. `TOOL_HANDLERS` maps native tool names to handler functions across filesystem, subprocess, web, document, interaction, model-interaction, background-job, session, and admin modules, while `TOOL_TAGS` keeps compatibility metadata and the global MCP manager handle;
+- `src.tools` owns domain do_* implementations for calendar, contacts, Cookbook, image, notes, research, search, system, and vault tools. `src.tool_implementations` is now a compatibility facade that re-exports those symbols and lazy-loads admin manage_* symbols to avoid circular imports;
+- `src.agent_tools.admin_tools` owns admin manage_* tools for endpoints, MCP, webhooks, tokens, and settings, including command validation for `manage_mcp`;
+- `src.tool_parsing._TOOL_NAME_MAP` owns aliases and prompted-block parsing;
+- `src.tool_schemas.FUNCTION_TOOL_SCHEMAS` and `function_call_to_tool_block()` own native schema and native-call conversion;
+- `src.tool_index.BUILTIN_TOOL_DESCRIPTIONS` owns retrieval text;
+- `src.tool_execution.execute_tool_block()` owns dispatch and hard execution gates;
+- `routes.model_routes.py` and frontend settings/admin surfaces expose global disabled-tool controls.
+
+When adding, removing, or renaming a tool, update the registry chain, execution dispatch, retrieval text, prompt wording, disabled-tool UI, and tests together.
+
+`src.tool_index.ALWAYS_AVAILABLE` is the retrieval catalog for high-frequency tools such as shell/python, web search/fetch, read/write/edit-file, code-nav, `manage_memory`, `ask_user`, `update_plan`, selected Cookbook serve controls, and `app_api`. Current prompt/schema assembly preserves only selected base tools unconditionally, then adds intent-, skill-, and retrieval-relevant tools so unrelated schemas do not flood small contexts.
+
+## Tool Retrieval And Execution
+
+`src.tool_index.ToolIndex` owns candidate retrieval using embeddings/keywords and cached index data. Security filtering is not its hard boundary: `agent_loop` hides unavailable schemas, and `tool_execution` blocks disabled, admin-only, and public-restricted calls before dispatch.
+
+`src.tool_execution` owns built-in tool execution, MCP dispatch, path confinement, background markers, output truncation, internal HTTP loopback, owner/admin checks, policy-blocked execution results, and formatting tool results for the model/UI. File tools support exact edit diffs, full-file writes, read line ranges, and workspace confinement. Code-navigation tools (`grep`, `glob`, `ls`) prefer `rg`/structured filesystem traversal over ad hoc shell commands. Uploaded-file context uses stable `attachment_ref` manifests and owner-checked URIs; a compatibility local path is exposed only after upload-root and tool-root confinement. Shared truncation, upload-handler registration, and MCP manager compatibility helpers live in `src.tool_utils`.
+
+Tool retrieval has domain-specific hooks beyond generic similarity: contact queries can surface `resolve_contact`/`manage_contact`; matched skills can add `manage_skills` and their required toolsets to the relevant tool set; explicit admin intents can include admin schemas so prompt text and native schema emission match.
+
+Interaction/session/model helper tools are native first-class tools, not prompt-only conventions. `ask_user` and `update_plan` live in `src.agent_tools.interaction_tools`, model delegation/listing helpers live in `model_interaction_tools`, session creation/list/send/manage helpers live in `session_tools`, and `manage_bg_jobs` lives in `bg_job_tools`.
+
+Prompted-tool parsing includes recovery paths for local/provider text leaks: bare JSON after a web-tool mention, OpenAI-style raw `{"function": ...}` payloads, StepFun/Gemma/DSML markup, Hermes/Qwen JSON bodies nested inside `tool_call` wrappers, and `<function_model><function_call>...</function_call><parameters>...</parameters></function_model>` wrappers from local MLX/Exo models. The Qwen bare end marker requires its pipe delimiter so ordinary text cannot terminate a tool block. Non-dict JSON arguments are rejected back to empty args instead of crashing the turn, common `tex` typos normalize to `text`, and delimiter scans are forward-only so unterminated tool markup cannot drive quadratic rescans. Executed raw tool JSON is stripped from assistant text afterward; this is still not a general-purpose JSON-command parser.
+
+Current call sites include:
+
+- agent mode tool calls from `src.agent_loop`;
+- MCP route configuration and built-in MCP registration;
+- background job monitoring and auto-continue;
+- skill tests, teacher escalation, scheduled tasks, and background follow-up loops;
+- UI-control and AI interaction helpers.
+
+## Streaming And Continuations
+
+Agent streaming emits normal content plus tool progress/output, document stream/update, ask-user choices, plan updates, budget, round exhaustion, loop-breaker, intent-nudge exhaustion, metrics, teacher escalation, research anchor, and finish/error events. Frontend chat stream code and detached replay depend on stable event names. If the stream generator closes while awaiting an in-flight tool, the loop cancels and awaits that tool task so subprocess-backed work is not left orphaned.
+
+Long-running bash jobs can be detached with background markers. `src.bg_jobs` owns persistent job state/result files; `src.bg_monitor` owns auto-continuation when jobs finish. Detached chat runs are in-memory and do not survive server restart, while background job state is disk-backed.
+
+Loop-breaker final-answer rounds, explicit repeated-tool/intent-nudge guard events, round-cap continuation signals, optional verifier retries, and teacher escalation are recovery behavior owned by `agent_loop` and `src.teacher_escalation`.
+
+Approval replay injects the sealed first tool result before the resumed model round. If that replay round has neither assistant prose nor reasoning, `_append_tool_results()` omits the empty assistant spacer so Anthropic-compatible payloads do not contain a rejected non-final empty assistant message; reasoning-only carriers remain a documented compatibility edge.
+
+## Security And Policy
+
+- `src.tool_security` owns non-admin blocked-tool decisions.
+- Non-admin users must not reach admin tools through agent mode, MCP, retrieval, or loopback calls.
+- Agent owner is passed from chat route `get_current_user(request)`. In `AUTH_ENABLED=false` mode this is `None`, not the `""` value returned by route dependencies. `blocked_tools_for_owner()`, schema hiding, and `execute_tool_block()` all use that owner.
+- Current dev tool security treats explicit `AUTH_ENABLED=false` as single-user even when an auth store exists, while auth-enabled pre-setup callers remain non-admin.
+- Path-based tools must remain confined to allowed roots and reject sensitive paths. Sensitive-path checks are case-insensitive and apply to direct file tools and code-navigation tools; `grep`/`glob`/`ls` must not become existence or content oracles for `.env`, SSH/GPG material, `id_rsa`, and similar denylisted paths.
+- Tool output is bounded/truncated where native execution owns the path, including displayed agent-tool output through the shared truncation helper. MCP output must be treated as untrusted; central MCP-output truncation before model re-entry remains a gap.
+- Provider-emitted native tool calls are requests, not authorization. `tool_execution` and route-level policy remain the authority.
+- `src.tool_capabilities` classifies each tool's effects and result integrity. Once external/workspace-untrusted content becomes model-visible, the request/session security context permits only explicitly low-impact tools without interruption and requires exact approval for high-impact, unknown, and arbitrary MCP calls.
+- `src.tool_approvals` seals an opaque, expiring exact first action plus server-only selected tools and continuation query to owner, session, origin run, tool content, workspace, capability snapshot, and—when relevant—document id/version/content digest. Chat choices grant the resumed task or the same chat session; both consume the exact first action, task scope bypasses the gate only during that resumed run, and chat scope is reconstructed only from a resolved card bound to the exact session id. The browser never receives selected tools/query and submits only task/chat/deny. Non-chat callers retain single-action behavior; new normal turns and superseding actions retire unresolved approvals without clearing taint.
+- Tool results that expose remote or stored untrusted content arm the gate even when their tool status is failed. Content-free failures and server-generated policy/approval placeholders do not. Native/provider tool messages and fenced results carry model-visible untrusted metadata/wrapping instead of relying on prompt wording alone.
+- Attachment-bearing document, note, and calendar tools owner-reserve internal
+  upload references before durable writes and fail without mutation when the
+  referenced upload is unavailable.
+- Guide-only/no-tools mode blocks tools before prompt assembly, before execution, and in chat preprocessing paths that would otherwise fetch context or start tool-backed research.
+- Plan mode is policy, not prompt advice: mutating native tools are disabled through schema-derived detection plus a static backstop, and write/unknown MCP tools are hidden and runtime-blocked for that turn.
+
+## Internal Loopback
+
+`do_app_api()` is implemented in `src.tools.system` and re-exported by `src.tool_implementations`. It owns generic app API loopback, OpenAPI discovery, method/path blocklists, and fixed local target behavior. `_internal_headers()` adds the process-secret internal-tool token and optional `X-Odysseus-Owner`; `core.middleware.require_admin()` and auth middleware own the corresponding bypass and owner-stamping rules. Route-specific owner handling must still be audited.
+
+## MCP
+
+`src.mcp_manager` owns configured MCP server lifecycle, discovered tool state, qualified MCP names, OpenAI schema conversion, call routing, generation invalidation, and connect/disconnect status. It supports stdio, SSE, and Streamable HTTP transports; Streamable HTTP can publish a `needs_auth` state and uses `src.mcp_oauth` for OAuth/OIDC-style authorization, token refresh, and encrypted token storage. Arbitrary MCP tools classify fail-high for approvals. `src.builtin_mcp` owns built-in server registration and the native-vs-MCP split. `mcp_servers/` owns server-specific tools for email, image generation, memory, RAG, and optional browser tooling.
+
+Native bash, python, file, web search, and web fetch tools continue through native fallback even when MCP is unavailable. Browser MCP is optional and can be skipped when cached Playwright/NPX packages are missing. Public users get no MCP schemas, and any `mcp__*` execution attempt must be blocked.
+
+MCP prompt/schema rendering includes server-provided input schemas, but names, types, and parameter hint text are sanitized and length-capped before entering the prompt. Per-server disabled tools filter listings, prompt descriptions, and function schemas; execution-time disabled-tool enforcement remains a separate hardening item.
+
+## Intent And Recovery Helpers
+
+`src.action_intents` owns deterministic chat-to-agent promotion hints and returns a category/reason so route logs can explain auto-escalation decisions. Explicit web-search language is category `web`; it can promote the turn into agent mode and narrow tools toward web search/fetch, but route policy requires explicit web-search enablement and honors explicit denial. It must avoid promoting explanatory questions into agent mode. `src.builtin_actions` owns scheduler/background actions outside the normal live agent loop. `src.teacher_escalation` owns recovery/escalation and skill-creation flows. `src.goal_based_extractor` is research-adjacent and should stay cross-referenced from research behavior rather than treated as ordinary tool execution.
+
+When an email reader is active, browser chat passes active email metadata and the agent loop injects it as protected, untrusted context so default reply/draft behavior targets the selected message. Active email compose documents are handled as existing email drafts rather than generic new-document requests.
+
+## Degraded Behavior
+
+- ToolIndex can degrade to keyword selection when embeddings, Chroma, index
+  warmup, or vector retrieval timeouts fail.
+- Agent mode can degrade from native function schemas to prompted fenced-block parsing based on provider/tool-support heuristics. Local Ollama `/v1` and native `/api` endpoints default to text tools unless the endpoint explicitly advertises `supports_tools`; `gpt-oss` remains text-tool by default unless the endpoint opts in.
+- MCP startup failure is non-critical; route/status surfaces expose per-server errors.
+- `ODYSSEUS_DISABLE_MCP`, missing `mcp`, uncached browser MCP packages, and per-server disabled tools can remove tools without blocking the app.
+- Global `builtin_browser` disable behavior may not currently match qualified `mcp__builtin_browser__*` tool names.
+
+## Current Gaps
+
+- Tool descriptions are duplicated across `FUNCTION_TOOL_SCHEMAS`, agent prompt sections, and `BUILTIN_TOOL_DESCRIPTIONS`.
+- Agent prompts remain heavy for small local context windows.
+- Some AI-control helpers are still globally wired from app startup rather than a narrower service layer.
+- Tool registry consistency is manual across handler maps, tags, aliases, schemas, retrieval descriptions, execution dispatch, settings/model routes, and frontend toggles.
+- MCP disabled-tool changes can stale-cache tool retrieval because disabled maps are not always an index generation input.
+- External MCP output still needs a single central size cap before model re-entry; untrusted-result metadata and the post-external-context action gate now cover the prompt-injection/authorization boundary.
+- Auth-disabled/no-login owner propagation is inconsistent between route dependencies and chat/agent execution, so tool-security and native tool storage behavior need dedicated regression coverage.
+- Agent tests mostly cover helpers and targeted regressions, including round-cap
+  and disconnect cancellation paths, but not an end-to-end fake-LLM
+  `stream_agent_loop` path with retrieval, native schemas, prompted blocks,
+  disabled/admin hiding, MCP tools, plan/workspace state, user-time context, and
+  tool-result SSE.

specs/architecture-runtime-inventory.md

@@ -1,412 +0,0 @@
-# Architecture Runtime Inventory
-
-> **Purpose**: Phase 0 planning baseline for codebase readability improvements (#4071).
-> **Parent issue**: [#4082](https://github.com/odysseus-dev/odysseus/issues/4082)
-> **Last updated**: dev@b58af42 | 2026-06-16
-> **Status**: Draft — to be reviewed before follow-up slices open.
-> **Snapshot basis**: Importer / file / import-line counts are refreshed to `dev@b58af42` (2026-06-16) and are recomputable via the commands in §3.4. **Line counts** in §2.1 / §2.2 are a snapshot from an earlier baseline and drift as `dev` moves — recompute any of them with `wc -l <file>`. This inventory tracks structure and risk, not live metrics.
-
-This document maps the current runtime module structure, identifies high-risk boundaries, and recommends safe first refactor slices. It does **not** move files, change imports, or alter runtime behavior.
-
----
-
-## 1. Current Structure Overview
-
-### 1.1 Top-Level Layout
-
-```
-odysseus/
-├── app.py                    # FastAPI app entrypoint (1,145 lines)
-├── conf/                     # Configuration (config.py, settings.py, settings_scrub.py)
-├── src/                      # 95 flat .py files + 2 subdirectories
-│   ├── agent_tools/          # Tool helpers: document, filesystem, subprocess, web
-│   └── search/               # Search subsystem
-├── routes/                   # 54 flat .py files — HTTP route handlers
-├── core/                     # 10 files — database models, auth, middleware, session
-├── mcp_servers/              # 5 files — MCP server implementations
-├── scripts/                  # CLI tools and one-shot scripts
-├── static/                   # Frontend HTML/CSS/JS
-├── tests/                    # 583 test files (~54,800 lines)
-└── services/                 # (exists as needed)
-```
-
-### 1.2 Directory Flatness Metric
-
-| Directory | Flat `.py` Files | Subdirectories | Concern |
-|-----------|-----------------|----------------|---------|
-| `src/` | **95** | 2 (`agent_tools/`, `search/`) | No domain grouping; 95 files in one directory |
-| `routes/` | **54** | 0 | All route handlers in one flat directory |
-| `core/` | 10 | 0 | Manageable, but `database.py` is oversized |
-
----
-
-## 2. Largest Runtime Modules
-
-### 2.1 Python Backend
-
-| Rank | File | Lines | Classes | Functions | Risk |
-|------|------|-------|---------|-----------|------|
-| 1 | `src/tool_implementations.py` | **4,032** | 0 | ~48 | **HIGH** |
-| 2 | `routes/email_routes.py` | **3,245** | — | — | **MEDIUM** |
-| 3 | `routes/cookbook_routes.py` | **2,969** | — | — | **MEDIUM** |
-| 4 | `src/agent_loop.py` | **2,961** | 0 | ~24 | **HIGH** |
-| 5 | `src/task_scheduler.py` | **2,330** | — | 5 | MEDIUM |
-| 6 | `routes/model_routes.py` | **2,266** | — | — | MEDIUM |
-| 7 | `core/database.py` | **2,265** | 28 | ~59 helpers | **HIGH** |
-| 8 | `src/builtin_actions.py` | **2,262** | 2 | ~24 | MEDIUM |
-| 9 | `src/llm_core.py` | **2,164** | — | — | MEDIUM |
-| 10 | `mcp_servers/email_server.py` | 2,197 | — | — | LOW (separate process) |
-| 11 | `src/visual_report.py` | 1,918 | — | — | LOW |
-| 12 | `routes/gallery_routes.py` | 1,896 | — | — | LOW |
-| 13 | `src/ai_interaction.py` | 1,846 | — | — | MEDIUM |
-| 14 | `routes/document_routes.py` | 1,717 | — | — | LOW |
-| 15 | `routes/skills_routes.py` | 1,648 | — | — | LOW |
-
-**Heuristic**: Files > 2,000 lines with 20+ public symbols and many importers are the highest-risk splits. Files 1,000–2,000 lines are medium-risk if tightly coupled.
-
-### 2.2 Frontend
-
-| File | Lines | Concern |
-|------|-------|---------|
-| `static/style.css` | **36,653** | Entire app CSS in one file (tracked separately in #2617) |
-| `static/js/document.js` | **9,776** | Single JS file for document functionality |
-| `static/js/slashCommands.js` | 6,498 | |
-| `static/js/settings.js` | 5,266 | |
-| `static/js/emailLibrary.js` | 5,217 | |
-| `static/js/notes.js` | 5,124 | |
-| `static/js/chat.js` | 4,985 | |
-| `static/app.js` | 4,090 | |
-
-**Note**: Frontend modularization is tracked separately in #2617 (CSS) and is not the focus of this Phase 0 inventory. Frontend is listed here for completeness but follow-up slices should target Python backend boundaries first.
-
----
-
-## 3. Import Dependency Graph
-
-### 3.1 Who Depends on `core/database.py`
-
-**102 files** import from `core.database` — this is the most depended-upon module:
-
-- All route handlers (`routes/*.py`)
-- Most `src/*.py` files
-- `core/session_manager.py`, `core/auth.py`
-- Multiple test files
-
-**Implication**: Any split of `core/database.py` is the highest-risk refactor. It should be tackled **last**, never first.
-
-### 3.2 Who Depends on `src/tool_implementations.py`
-
-**17 files** import from `src.tool_implementations`:
-- `src/agent_loop.py`, `src/builtin_actions.py`, `src/tool_index.py`
-- `src/task_scheduler.py`, `src/tool_policy.py`
-- Various tests
-
-### 3.3 Who Depends on `src/agent_loop.py`
-
-**22 files** import from `src.agent_loop`:
-
-- `src/tool_policy.py`, `src/teacher_escalation.py`, `src/bg_monitor.py`
-- `src/task_scheduler.py`
-- Multiple test files
-
-### 3.4 Cross-Layer Import Violations
-
-**`src/` importing from `routes/`** (backwards dependency — domain logic depending on HTTP layer):
-
-```
-src/tool_implementations.py ──→ routes/calendar_routes.py
-src/tool_implementations.py ──→ routes/cookbook_helpers.py
-src/tool_implementations.py ──→ routes/email_helpers.py
-src/tool_implementations.py ──→ routes/email_pollers.py
-src/tool_implementations.py ──→ routes/email_routes.py
-src/tool_implementations.py ──→ routes/model_routes.py
-src/tool_implementations.py ──→ routes/note_routes.py
-src/tool_implementations.py ──→ routes/prefs_routes.py
-```
-
-> These are **runtime imports** (inside function bodies, not at module top), which mitigates circular import risk but indicates fuzzy layer boundaries. Function-level inline imports from the HTTP layer into business logic are a code smell.
-
-**Import counts (top-level)**:
-| Direction | Count | Notes |
-|-----------|-------|-------|
-| `routes/` → `src/` | **374** | Expected: HTTP handlers call domain logic |
-| `routes/` → `core/` | **126** | Expected: handlers access DB models |
-| `src/` → `routes/` | **31** | **Unexpected**: domain logic reaching into HTTP layer (direct grep of import lines referencing `routes/`) |
-| `src/` → `core/` | **106** | Acceptable but could be reduced with a data-access layer |
-
-> **How the metrics in this document are computed** — recompute against current `dev` before treating any count as authoritative (the tree drifts; these numbers are a snapshot, not a live value):
-> - `src/` flat `.py` files: `find src -maxdepth 1 -name '*.py' | wc -l`
-> - `tests/` test files: `find tests -name 'test_*.py' | wc -l`
-> - `core.database` importers: `grep -rlE '(from|import) +core\.database' --include='*.py' . | grep -v core/database.py | wc -l`
-> - `src.agent_loop` importers: `grep -rlE '(from|import) +src\.agent_loop' --include='*.py' . | grep -v src/agent_loop.py | wc -l`
-> - Cross-layer import lines: `grep -rhE '(from|import) +<pkg>' --include='*.py' <dir>/ | wc -l` (e.g. `(from|import) +routes` over `src/`)
-
----
-
-## 4. Route Ownership Map
-
-Routes can be grouped into logical feature domains. Current flat structure obscures these boundaries:
-
-| Domain | Route Files | Total Lines | Review Complexity |
-|--------|-------------|-------------|-------------------|
-| **Email** | `email_routes.py`, `email_helpers.py`, `email_pollers.py` | 5,936 | HIGH — most complex domain |
-| **Chat / Agent** | `chat_routes.py`, `chat_helpers.py`, `shell_routes.py`, `codex_routes.py`, `skills_routes.py` | 6,365 | HIGH — core interaction surface |
-| **Cookbook** | `cookbook_routes.py`, `cookbook_helpers.py`, `cookbook_output.py` | 4,110 | MEDIUM |
-| **Model / LLM** | `model_routes.py`, `assistant_routes.py`, `copilot_routes.py` | 2,764 | MEDIUM |
-| **Calendar / Contacts** | `calendar_routes.py`, `contacts_routes.py` | 2,336 | MEDIUM |
-| **Documents** | `document_routes.py`, `document_helpers.py` | 1,954 | LOW |
-| **Auth** | `auth_routes.py`, `api_token_routes.py`, `device_flow.py` | 1,171 | LOW |
-| **Tasks** | `task_routes.py` (standalone) | 1,157 | LOW |
-| **Session** | `session_routes.py` (standalone) | 1,287 | LOW |
-| **Gallery** | `gallery_routes.py`, `gallery_helpers.py` | 1,896 | LOW |
-| **Memory** | `memory_routes.py` | — | LOW |
-| **Research** | `research_routes.py` | — | LOW |
-| **MCP** | `mcp_routes.py` | — | LOW |
-| **Notes** | `note_routes.py` | — | LOW |
-| **Other** | `prefs_routes.py`, `upload_routes.py`, `vault_routes.py`, `webhook_routes.py`, `workspace_routes.py`, `search_routes.py`, `history_routes.py`, `hwfit_routes.py`, `preset_routes.py`, `signature_routes.py`, `backup_routes.py`, `cleanup_routes.py`, `diagnostics_routes.py`, `embedding_routes.py`, `emoji_routes.py`, `font_routes.py`, `stt_routes.py`, `tts_routes.py`, `compare_routes.py`, `personal_routes.py`, `editor_draft_routes.py`, `admin_wipe_routes.py`, `chatgpt_subscription_routes.py` | 2,000+ | LOW individual, HIGH cumulative |
-
----
-
-## 5. Tool Registry & Implementation Boundaries
-
-### 5.1 Current Tool Architecture
-
-| Component | File | Lines | Role |
-|-----------|------|-------|------|
-| Tool schemas | `src/tool_schemas.py` | 1,392 | JSON Schema tool definitions (Duck-TypedDict) |
-| Tool index | `src/tool_index.py` | 542 | RAG-based tool retrieval from ChromaDB |
-| Tool implementations | `src/tool_implementations.py` | 4,032 | 33 `do_*` functions — all tool execution logic |
-| Tool security | `src/tool_security.py` | — | Owner-scoped tool blocking |
-| Tool policy | `src/tool_policy.py` | — | Guide-only directive, plan-mode disabled tools |
-| Tool utils | `src/tool_utils.py` | — | Shared tool helpers |
-
-### 5.2 Tool Implementation Categories
-
-The 33 `do_*` functions in `tool_implementations.py` fall into natural domain groups — the basis for slice 1's split in §6.2:
-
-| Category | `do_*` functions | Count |
-|----------|------------------|-------|
-| **System / config** | `do_manage_skills`, `do_manage_tasks`, `do_manage_endpoints`, `do_manage_mcp`, `do_manage_webhooks`, `do_manage_tokens`, `do_manage_settings`, `do_api_call`, `do_app_api` | 9 |
-| **Cookbook / model serving** | `do_download_model`, `do_serve_model`, `do_list_served_models`, `do_stop_served_model`, `do_tail_serve_output`, `do_list_downloads`, `do_cancel_download`, `do_search_hf_models`, `do_adopt_served_model`, `do_list_cookbook_servers`, `do_list_serve_presets`, `do_serve_preset`, `do_list_cached_models` | 13 |
-| **Notes** | `do_manage_notes` | 1 |
-| **Calendar** | `do_manage_calendar` | 1 |
-| **Search** | `do_search_chats` | 1 |
-| **Research** | `do_manage_research`, `do_trigger_research` | 2 |
-| **Contacts** | `do_resolve_contact`, `do_manage_contact` | 2 |
-| **Vault** | `do_vault_search`, `do_vault_get`, `do_vault_unlock` | 3 |
-| **Image** | `do_edit_image` | 1 |
-| | **Total** | **33** |
-
-> Low-level tools (filesystem, subprocess, web fetch, document parsing) live in `src/agent_tools/`, **not** in `tool_implementations.py` — out of scope for this split.
-
----
-
-## 6. Risk Assessment & Candidate Slice Ranking
-
-> **Candidate proposals, not a committed plan.** The rankings, package shapes (e.g. `src/pkg/`, `src/domain/`, `src/infra/`, `src/api/`), split ordering, and route-grouping strategy below are **options for maintainer discussion**. Per #4082/#4071, slice ownership and order are settled by maintainers before any follow-up PR. §1–§3 above are the factual current-state inventory.
-
-### 6.1 Risk Scale
-
-| Level | Criteria |
-|-------|----------|
-| **LOW** | File has ≤3 importers AND ≤500 lines, OR is a pure refactor with clear boundaries |
-| **MEDIUM** | File has 4–15 importers OR 500–1,500 lines |
-| **HIGH** | File has 16+ importers OR >2,000 lines, OR has cross-layer import violations |
-
-### 6.2 Ranked Split Candidates
-
-| Priority | Target | Risk | Rationale |
-|----------|--------|------|-----------|
-| **1** | `src/tool_implementations.py` → `src/tools/*.py` | **MEDIUM** | 4,032 lines → ~10 files by tool category. Already has natural boundaries. 17 importers, tracked in #3629. Use `__init__.py` shim to keep existing imports working. |
-| **2** | `routes/` → domain subdirectories (one domain per PR) | **MEDIUM** | 54 flat files. Done **one domain at a time** (e.g. a standalone PR for the email domain, then chat, …), not a broad reorganization — route modules carry helper imports, registration assumptions, and test import paths. |
-| **3** | `src/agent_loop.py` → `src/agent/loop.py` + submodules | **MEDIUM-HIGH** | 2,961 lines, 24 functions. Can extract prompt building, classification, verification, and runaway detection. Tracked in #3266. |
-| **4** | `src/` → `src/pkg/`, `src/domain/`, `src/infra/`, `src/api/` | **MEDIUM** | Structural reorganization. Split flat `src/` into layered packages. Must come after routes and tools are stable. |
-| **5** | `routes/email_*.py` consolidation | **LOW** | Already grouped by filename prefix. Low-risk cleanup within the email domain. |
-| **6** | `core/database.py` → `src/infra/database/models/*.py` | **HIGH** | 28 classes, 102 importers. Highest-risk split. Must be **last** in any sequence. Requires careful import shim strategy. |
-| **7** | Frontend CSS modularization | **MEDIUM** | 36,653 lines. Tracked in #2617. Separate timeline from backend work. |
-| **8** | Frontend JS modularization | **MEDIUM** | 9,776 lines in `document.js`. Introduce ES modules at minimum. |
-
-### 6.3 Candidate First 3 Behavior-Preserving Slices
-
-**Slice 1: Split `tool_implementations.py`** (Lowest-risk high-impact)
-
-- Create `src/tools/` package with one file per tool category
-- Add `src/tools/__init__.py` re-exporting all symbols with current names
-- Update 17 importers to use new paths (can be deferred via shim)
-- Validation: `python -m pytest tests/ -x -q` + manual smoke test of tool execution
-- Reference: #3629
-
-**Slice 2: Group `routes/` by domain** (one domain per PR, not a broad sweep)
-
-Route modules carry helper imports, router registration assumptions, and test import paths, so this must be done **one domain at a time** rather than as a single reorganization PR. Example sequence (each its own PR):
-
-- PR 2a: move the **email** domain (`email_routes.py`, `email_helpers.py`, `email_pollers.py`) → `routes/email/` + shim
-- PR 2b: move the **chat/agent** domain → `routes/chat/` + shim
-- PR 2c: move the **cookbook** domain → `routes/cookbook/` + shim
-- …and so on per domain from §4
-
-Each PR: add `__init__.py` re-exporting old names, update `app.py` router imports, validation `python app.py` starts clean. **No behavior change** — pure file reorganization.
-
-**Slice 3: Extract `agent_loop.py` submodules** (Improve reviewability)
-
-- Move prompt assembly → `src/agent/prompt.py`
-- Move request classification → `src/agent/classifier.py`
-- Move sub-agent verification → `src/agent/verifier.py`
-- Move runaway detection → `src/agent/runaway.py`
-- Move context management → `src/agent/context.py`
-- Keep `src/agent/loop.py` as the main orchestration module
-- Validation: `python -m pytest tests/test_agent_loop.py tests/test_loop_breaker_runaway.py -v`
-
----
-
-## 7. Safety Guardrails for Follow-Up Work
-
-Per maintainer guidance in #4082 and #4071:
-
-- [ ] **One domain/slice per PR** — never mix multiple reorganizations
-- [ ] **No behavior changes** mixed with file moves — pure reorganization only
-- [ ] **Keep compatibility shims** — `__init__.py` re-exports for all existing import paths
-- [ ] **Add or identify focused tests** before risky splits
-- [ ] **Do not start with `core/database.py`** or broad route movement unless this inventory shows a safe boundary
-- [ ] **Prefer small, reviewable slices** over large restructures
-- [ ] **No packaging/runtime/tooling migration** mixed into file moves
-- [ ] **No frontend framework migration** inside this stabilization lane
-- [ ] **Validate with `python -m compileall`** — every PR must pass CI checks
-- [ ] **Validate with `pytest`** — run the full test suite before opening each PR
-
----
-
-## 8. Validation Commands
-
-Each follow-up PR should be verifiable with these commands before submission:
-
-```bash
-# Syntax check — must pass with zero errors
-python -m compileall src/ routes/ core/ conf/
-
-# Full test suite — must match baseline pass rate
-python -m pytest tests/ -x -q
-
-# Import shim verification — existing import paths must still work
-python -c "from src.tool_implementations import do_search_chats; print('OK')"
-
-# App startup smoke test (if backend touched)
-timeout 5 python app.py 2>&1 | head -5 || true
-```
-
----
-
-## 9. Open Questions
-
-1. Is `#2538` (specs ground truth) the canonical behavior map baseline, and should this inventory be kept in sync with those specs once merged?
-2. Should route grouping follow the domain map proposed here, or is there a different taxonomy preferred by maintainers?
-3. For the `tool_implementations.py` split (#3629), is the tool categorization in §5.2 acceptable, or should it follow a different grouping?
-4. Should compatibility shims (`__init__.py`) be temporary (removed in a follow-up wave) or permanent?
-5. Should an ADR (Architecture Decision Record) document be started to track decisions made during this process?
-
----
-
-## 10. Future Direction (NOT current state)
-
-The following are **future refactor targets** (candidate directions **pending maintainer agreement**, not committed), recorded here so this inventory does not imply they exist today. None of them are present in the current `dev` tree:
-
-- `main.py` — proposed rename of the `app.py` entrypoint. Today the app boots via `app.py`.
-- `src/agent/` — proposed package to hold `agent_loop.py` submodules (prompt/classifier/verifier/runaway/context). Today `agent_loop.py` is a single flat file in `src/`.
-- `src/infra/`, `src/domain/`, `src/pkg/`, `src/api/` — proposed layered reorganization of the flat `src/` directory (slice 4 in §6).
-
-These become real only when the corresponding slices land.
-
----
-
-## Appendix A: File Listing
-
-### `src/` (95 files — 61 shown; run `ls src/*.py` for the full list)
-
-```
-agent_loop.py          tool_implementations.py   tool_schemas.py
-tool_index.py          tool_security.py          tool_policy.py
-tool_utils.py          builtin_actions.py        task_scheduler.py
-llm_core.py            model_context.py          model_discovery.py
-session_search.py      context_budget.py         context_compactor.py
-ai_interaction.py      action_intents.py         agent_runs.py
-app_helpers.py         app_initializer.py        config.py
-database.py            memory.py                 memory_provider.py
-secret_storage.py      prompt_security.py        url_security.py
-url_safety.py          rate_limiter.py           cleanup_service.py
-readiness.py           service_health.py         exceptions.py
-request_models.py      assistant_log.py          bg_monitor.py
-builtin_mcp.py         chat_helpers.py           chroma_client.py
-document_processor.py  embedding_lanes.py        deep_research.py
-research_handler.py    research_utils.py         personal_docs.py
-rag_manager.py         rag_singleton.py          topic_analyzer.py
-visual_report.py       youtube_handler.py        pdf_forms.py
-pdf_form_doc.py        pdf_runtime.py            caldav_writeback.py
-email_thread_parser.py text_helpers.py           user_time.py
-teacher_escalation.py  cookbook_serve_lifecycle.py
-chatgpt_subscription.py  mcp_manager.py
-```
-
-### `routes/` (54 files)
-
-```
-__init__.py    _validators.py
-auth_routes.py              api_token_routes.py       device_flow.py
-chat_routes.py              chat_helpers.py           shell_routes.py
-codex_routes.py             skills_routes.py
-email_routes.py             email_helpers.py          email_pollers.py
-cookbook_routes.py          cookbook_helpers.py       cookbook_output.py
-model_routes.py             assistant_routes.py       copilot_routes.py
-calendar_routes.py          contacts_routes.py
-document_routes.py          document_helpers.py
-gallery_routes.py           gallery_helpers.py
-task_routes.py              session_routes.py
-note_routes.py              memory_routes.py          research_routes.py
-mcp_routes.py               search_routes.py          history_routes.py
-webhook_routes.py           workspace_routes.py       upload_routes.py
-vault_routes.py             prefs_routes.py           preset_routes.py
-signature_routes.py         personal_routes.py        hwfit_routes.py
-backup_routes.py            cleanup_routes.py         diagnostics_routes.py
-embedding_routes.py         emoji_routes.py           font_routes.py
-stt_routes.py               tts_routes.py             compare_routes.py
-editor_draft_routes.py      chatgpt_subscription_routes.py    admin_wipe_routes.py
-```
-
-### `core/` (10 files)
-
-```
-__init__.py    constants.py    database.py    models.py
-auth.py        middleware.py   session_manager.py   exceptions.py
-atomic_io.py   platform_compat.py
-```
-
----
-
-## Appendix B: Key Import Relationships
-
-```
-core/database.py  ←── 102 importers (routes/*, src/*, core/*, tests/*)
-    ↑
-    ├── routes/auth_routes.py
-    ├── routes/email_routes.py
-    ├── src/builtin_actions.py
-    ├── src/task_scheduler.py
-    ├── src/tool_implementations.py (inline)
-    └── ...97 more
-
-src/tool_implementations.py  ←── 17 importers
-    ↑
-    ├── src/agent_loop.py
-    ├── src/builtin_actions.py
-    ├── src/tool_index.py
-    ├── src/task_scheduler.py
-    ├── src/tool_policy.py
-    └── ...12 more (mostly tests)
-
-src/agent_loop.py  ←── 22 importers
-    ↑
-    ├── src/tool_policy.py
-    ├── src/teacher_escalation.py
-    ├── src/bg_monitor.py
-    ├── src/task_scheduler.py
-    └── 18 more (incl. tests)
-```

specs/auth-security.md

@@ -0,0 +1,169 @@
+# Auth And Security
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+This spec covers current security and trust-boundary behavior in:
+
+- `core/auth.py`;
+- `core/middleware.py`;
+- `core/log_safety.py`;
+- `core/database.py`;
+- `app.py` auth middleware and token cache;
+- `src/auth_helpers.py`;
+- `src/owner_identity.py`;
+- `src/tool_approval_scopes.py`, `src/tool_approvals.py`, and `src/tool_capabilities.py`;
+- `src/tool_security.py`;
+- `src/tool_execution.py`;
+- `src/task_action_policy.py`;
+- `src/prompt_security.py`;
+- `src/url_safety.py` and `src/url_security.py`;
+- `src/host_docker_access.py`;
+- `src/attachment_refs.py` and upload lifecycle enforcement in
+  `src/upload_handler.py` / `routes/upload_routes.py`;
+- `src/secret_storage.py`;
+- `src/api_key_manager.py`;
+- `src/integrations.py`;
+- `src/webhook_manager.py`;
+- `src/generated_images.py`;
+- `scripts/diffusion_server.py`;
+- `scripts/mlx_image_server.py`;
+- `companion/routes.py` and `companion/pairing.py`;
+- `routes/auth_routes.py`, `routes/api_token_routes.py`, and canonical `routes/vault/vault_routes.py` plus its top-level compatibility shim;
+- admin-gated call sites in route files;
+- `THREAT_MODEL.md` and `SECURITY.md`.
+
+## Trust Boundary
+
+Odysseus is a trusted-user private-network app. Admins intentionally have powerful local capabilities: shell, files, email, calendar, MCP, model serving, vault, settings, and API token management. The security model prevents unauthenticated access, non-admin escalation, prompt-injection through untrusted content, and accidental exposure of internal services.
+
+`THREAT_MODEL.md` owns high-level security framing, but implementation claims here should be verified against current code when the threat model is stale. This spec records the implementation map that contributors should check before changing auth or untrusted-context flows. Security-header runtime details live in `runtime.md`.
+
+## Auth Ownership
+
+- `core.auth.AuthManager` owns users, password hashing, TOTP/backup codes, reserved usernames, privilege defaults, admin promote/demote state, and auth settings stored in `data/auth.json`. Auth config/setup mutations are lock-guarded, and session tokens are persisted separately in `data/sessions.json` behind their own lock.
+- `app.py` owns request-time auth middleware, token-cache rebuild/invalidation, auth exemptions, API-token verification, and internal-tool identity stamping.
+- `routes/auth_routes.py` owns HTTP endpoints for setup, signup/login/logout, 2FA, users, privileges, auth features, and integration settings.
+- `core.middleware.require_admin()` owns the normal admin gate. Local wrappers must document and test any intentional divergence from that boundary.
+- `src.auth_helpers.effective_user()` owns cookie/API-token owner attribution for selected route code. `require_user()` owns route-level degraded user resolution, `require_privilege()` owns privilege checks, and `owner_filter()` owns shared/null-owner query compatibility.
+
+Reserved usernames include request-only sentinels `internal-tool`, `api`, `demo`, and `system`, plus the storage-only Default/Local owner `__odysseus_local__`. Loaded auth data drops reserved user records, and create/rename flows must reject real users with those names. `src.owner_identity` is the canonical owner vocabulary and `auth_disabled()` parser.
+
+## Auth Runtime Flow
+
+`AuthMiddleware` is the outer request gate because FastAPI middleware executes in reverse add order. It can return API `401` JSON or browser `/login` redirects before timeout/security-header middleware reaches the route.
+
+Public/auth-exempt surfaces are limited to setup, signup/login/logout/status, feature/settings/integration preset reads, health/version/login, `/static/*`, and task webhook trigger paths. `routes/task/task_routes.py` owns validation of `POST /api/tasks/{task_id}/webhook/{token}` path credentials.
+
+Login issues an `HttpOnly`, `SameSite=Lax` cookie with a seven-day max age when "remember" is enabled. `_secure_cookie()` (`routes/auth_routes.py:89`) decides the `Secure` attribute: an explicit `SECURE_COOKIES` of `true` or `false` is authoritative, and any other value, including unset and the present-but-empty value docker-compose injects, derives it from the request, marking the cookie `Secure` when the connection scheme or the first `X-Forwarded-Proto` hop is https. TOTP is checked before session issuance. Logout, password changes, user deletion, rename flows, expired sessions, and deleted-user sessions must keep revocation/migration behavior intact.
+
+Deleting a user revokes that user's browser sessions and API-token rows, then the admin delete route invalidates the in-memory bearer-token cache so already-cached tokens stop authenticating.
+
+Rename first changes the auth username, then migrates owner-bearing DB rows and disk-backed stores. Current rename coverage includes user preferences, active/disk research state, `memory.json`, upload metadata and owner-qualified upload index keys, skills frontmatter/usage state, cached browser sessions, and API-token cache invalidation. If owner migration fails after the auth rename, the route attempts to roll auth back to the old username instead of leaving a split identity.
+
+Admin promotion/demotion is a live auth flag change through `AuthManager.set_admin()` and `PUT /api/auth/users/{username}/admin`. Demotion refuses to remove the last admin, permits self-demotion when another admin remains, restores the pre-admin privilege map when available, and does not revoke sessions or API tokens because later admin checks read the current `is_admin` flag.
+
+## Owner Attribution
+
+Cookie requests use the real username. Bearer-token requests are stamped as `request.state.current_user = "api"` plus `api_token_owner`, `api_token_scopes`, and token id. Routes that support API-token access must explicitly use `effective_user()` or route-local scope helpers instead of treating `"api"` as an owner.
+
+Internal loopback calls may stamp `current_user = "internal-tool"` or a validated `X-Odysseus-Owner` username. Network/proxy validation for that bypass lives in `app.py`; `require_admin()` trusts the stamped sentinel or raw internal header and should be used behind equivalent middleware control.
+
+Missing-owner values remain state-dependent at legacy call sites, but new storage-facing code has one normalization contract:
+
+- Auth-enabled, configured auth with no `current_user` is unauthenticated and should fail closed at route dependencies.
+- `AUTH_ENABLED=false` is an explicit local single-user/no-login mode. Existing route dependencies can still return `""`, and admin gates allow the local operator. `effective_storage_owner()` and `storage_owner_for_request()` normalize an absent owner to `__odysseus_local__` only in this mode.
+- Chat/agent code that reads `get_current_user(request)` directly gets `None` when auth middleware is disabled, because no middleware stamps request state.
+- SQL `NULL`/JSON missing owners remain legacy/shared compatibility data, not the same thing as a logged-out authenticated caller.
+- `"api"` and `"internal-tool"` are request sentinels. They must not be persisted as normal storage owners unless a route explicitly defines that behavior.
+- `__odysseus_local__` is a valid storage owner but never a login or request sentinel. Adoption is incremental: callers that do not use the storage-owner helper can still expose older `None`/empty/null compatibility behavior.
+
+Authenticated `manage_tasks` mutations require an exact stored task-owner
+match and reject both cross-owner and legacy null-owner rows. The `owner=None`
+agent path keeps deliberate auth-disabled single-user compatibility, including
+unscoped list/create/mutation behavior.
+
+Owner-scoped route code should use `require_user()` or equivalent policy before querying per-owner data. Current note CRUD/reorder/reminder routes do this so an auth-enabled request that reaches the route without identity returns `401` instead of falling into single-user/null-owner compatibility behavior.
+
+Scheduled task actions attribute differently again. `_execute_action` (`src/task_scheduler.py:1231`) invokes the action with `owner=task.owner` read from the stored `ScheduledTask` row, so no request and no resolved principal are in flight. These trigger paths converge there: schedule, event bus, manual run (`routes/task/task_routes.py:865`), the `manage_tasks` agent tool (`src/tools/system.py:469`), webhook triggers (`routes/task/task_routes.py:1045`), which are unauthenticated by design with the token as the only credential and execute under the stored `task.owner`, and success-chained tasks (`src/task_scheduler.py:1063-1074`), which additionally require the chained target to share `task.owner` and reject cycles. Trigger-side ownership checks use the `if user and task.owner != user` shape, so a falsy caller skips them. Action bodies that reach owner-scoped storage must treat `task.owner` as the authority; route-level `require_user()` never runs on this path.
+
+## API Tokens And Scoped Integrations
+
+`routes/api_token_routes.py` owns token CRUD and scope normalization. Partial updates preserve existing scopes unless new scopes are supplied, write scopes imply the matching read scopes where applicable, and Cookbook scopes are part of the normalized scope set. `app.py` caches active token prefix rows and verifies bearer tokens with bcrypt. API-token requests set `request.state.current_user = "api"` plus token owner/scopes.
+
+Current call sites include Codex/Claude scoped APIs, `/api/v1/chat`, webhooks, selected session routes, companion pairing, and external integrations. `/api/codex/*` and `/api/v1/chat` enforce route-local scopes; companion and selected session routes use owner attribution. `companion/pairing.py` can mint chat-scoped tokens outside normal token CRUD.
+
+Admin token CRUD is cookie/admin gated. Update/delete operations check token ownership, and cache rebuild ignores active tokens whose owner no longer maps to a known auth user. Scoped route code must use the token owner and declared scopes instead of falling back to cookie-user assumptions.
+
+## Internal Tool Loopback
+
+Agent tools call admin-gated HTTP routes through an in-process loopback. `core.middleware.INTERNAL_TOOL_TOKEN` owns the random per-process secret. `app.py` only accepts this bypass from direct loopback clients without proxy-forwarding headers.
+
+`src.tool_security` owns non-admin tool blocking. Non-admin users must not reach admin tools through agent mode, MCP tools, or loopback calls.
+
+`src.tool_security.owner_is_admin_or_single_user()` treats explicit `AUTH_ENABLED=false` as intentional single-user mode even when an auth store already exists, while keeping pre-setup auth-enabled callers non-admin.
+
+Current admin gates include `require_admin()` call sites across admin wipe, backup, contacts, Cookbook, diagnostics, embeddings, MCP, model, personal docs, presets, skills, uploads, vault, webhook, and companion routes. Local wrappers also exist in auth routes, shell routes, and task action policy; changes to those wrappers need the same trust-boundary review as `require_admin()`. Scheduled task action policy treats `run_local`, `run_script`, `ssh_command`, and `cookbook_serve` as admin-only action tasks across create/update/manual-run/webhook/scheduler execution.
+
+`tidy_research` can remove only empty or unparseable research JSON. Because a broken file has no trustworthy owner stamp, the action checks `owner_is_admin_or_single_user()` before enumerating files; regular users and the pre-setup window cannot run that global unattributable-file sweep.
+
+## Untrusted Context Policy
+
+`src.prompt_security` owns the model-facing untrusted data contract:
+
+- `UNTRUSTED_CONTEXT_POLICY` states the policy in system prompt text.
+- `untrusted_context_message(label, content)` wraps external content as user-role data with `metadata.trusted = False`, provenance metadata, and a default `tool_gate_untrusted` marker. Guard-like labels/content are escaped so source text cannot counterfeit the wrapper boundary.
+
+Current untrusted surfaces include fetched URLs, web results, emails, memories, skills, notes, documents, active editor content, and tool output sourced from outside the server. Injecting those as trusted system instructions is a security bug.
+
+`src.tool_capabilities` classifies native and MCP tools by effects and result integrity. After external/workspace-untrusted context becomes model-visible, `ToolRunSecurityContext` keeps a server-owned taint for the session turn: only explicitly low-impact tools can run immediately, while write, execute, network-egress, UI/external-side-effect, admin, destructive, unknown, and arbitrary MCP actions require exact approval. Failed tools can still arm the gate when their result carries remote or stored payload; content-free failures and server-generated blocked/approval placeholders do not.
+
+`src.tool_approvals` owns opaque approvals sealed to the owner, session, origin run, exact first tool name/content, workspace, capability effects/result integrity, selected continuation tool set/query, and expiry. Document actions additionally seal document id, version, content digest, and workspace. Chat cards offer task scope, chat-session scope, or deny: both allow choices consume and execute the exact sealed first action after current-policy/freshness checks, task scope bypasses the gate only for the resumed task, and chat-session scope persists a resolved session-bound grant for later turns in that same chat. The browser submits only the opaque decision and cannot replace the sealed action, selected tools, query, composer text, or attachments. Non-chat callers retain single-action scope. A new ordinary turn or superseding action retires an unresolved approval without clearing taint.
+
+## URL, Path, And Secret Policy
+
+- `src/url_security.py` owns public HTTP(S) validation for integration/API-token supplied URLs. It should fail closed for private IP, loopback, invalid scheme, and unsafe redirect targets.
+- `src/url_safety.py` owns local-first outbound URL safety for model endpoints and similar local services. Loopback/LAN can be allowed by default, and private-IP blocking is an explicit caller policy. Strict `block_private=True` also rejects RFC 6598 shared/CGNAT space (`100.64.0.0/10`) explicitly because Python does not classify that range as private.
+- `core.log_safety.redact_url()` strips URL userinfo, query strings, and fragments before endpoint URLs enter logs. Model, chat/research endpoint, contact/CardDAV, and similar diagnostics should use this helper instead of logging raw admin-configured URLs.
+- `src.webhook_manager` validates webhook URLs at create and delivery time,
+  rejects private/internal targets, disables redirects, and pins delivery to
+  the public IP set that passed validation immediately before the request.
+- `src.integrations` owns admin-configured integration base URLs and secret
+  masking. `api_call` accepts only relative paths, rejects link-local/metadata destinations through `src.url_safety`, can additionally block RFC1918/loopback/private targets with `INTEGRATION_API_BLOCK_PRIVATE_IPS=true`, and pins requests to the IP set that passed SSRF validation while preserving the intended Host/TLS identity.
+- `src.outbound_fetch` owns reusable public-URL classification, validates every redirect hop, rejects private/local resolved addresses, and pins the HTTP connection to the validated public IP while preserving original URL/SNI/Host semantics. `services.search.content` adapts that transport for extraction and caching.
+- Path-based tools, upload/document/gallery/signature/generated-image routes, embedding cache paths, and research JSON helpers must stay confined to allowed roots and owner-scoped files. Native file/code-navigation tools also apply a case-insensitive sensitive-path denylist so `grep`, `glob`, `ls`, direct reads, and writes cannot reveal `.env`, SSH/GPG material, private-key filenames, or similar secret paths.
+- Durable upload references are owner-reserved before chat/session, document,
+  note, or calendar writes. Cleanup scans every current durable reference
+  surface and fails closed on incomplete discovery or inconsistent upload-index
+  state rather than deleting a possibly live upload.
+- File-backed SQLite startup restricts `app.db` and existing rollback/WAL/SHM
+  sidecars to `0600` on POSIX after resolving the real path from the parsed
+  engine URL. Windows, in-memory, and non-SQLite databases are excluded, and
+  failed POSIX restriction is logged as a secret-file warning.
+- Secret-like DB columns use `EncryptedText` or `src.secret_storage`. Email passwords and Google OAuth mail tokens are encrypted manually in `EmailAccount` string columns; Google OAuth state is HMAC-signed and callback writes are owner-checked before token storage. `src.api_key_manager` keeps provider API keys encrypted in `data/api_keys.json`, writes by loading the raw encrypted dict so saving one provider does not rewrite other providers' keys as plaintext, and restricts local key-file permissions where the platform supports chmod. Vault state in `data/vault.json` is a chmod-restricted JSON secret store, not Fernet-encrypted DB storage. Do not log or return decrypted secrets except for intentional admin vault retrieval flows with audit/reason checks.
+- `.env` files are secrets-only inputs and should not be read or printed during agent work.
+
+`scripts/diffusion_server.py` is a local model-serving helper with its own web surface. It defaults CORS to deny, installs a trusted-host allowlist for loopback/bind addresses, and only extends Host/CORS through explicit CLI flags.
+
+`scripts/mlx_image_server.py` serves exactly the model selected when the process starts. OpenAI-compatible request `model` fields are accepted but ignored for generation and edits, so an unauthenticated caller cannot select another local directory or Hugging Face repository and drive model-specific script/bridge execution.
+
+Host Docker socket access is a high-trust admin/deployment choice, not a normal container capability. Default Docker Compose does not mount `/var/run/docker.sock`; `src.host_docker_access` only reports local Docker available inside a container when `ODYSSEUS_ENABLE_HOST_DOCKER=true` and the socket exists. Remote SSH Docker/Cookbook workflows remain the safer default.
+
+## Degraded And Compatibility Behavior
+
+- `AUTH_ENABLED=false` skips `AuthMiddleware` and `src.auth_helpers.require_user()` returns `""` from any host. This preserves local single-user/no-login operation; it is not permission for auth-enabled logged-out callers. Storage code that adopts `storage_owner_for_request()` receives the reserved Default/Local owner; direct `get_current_user()` readers still receive `None`. Owner-scoped routes that tolerate no-login mode should call the appropriate route or storage helper so auth-enabled anonymous requests fail closed.
+- First-run setup mode redirects browser requests to `/login`, returns API `401 Setup required`, and keeps setup/status/login surfaces auth-exempt. Setup/signup/login are rate-limited; status is exempt but not rate-limited. Route helper fallbacks only tolerate unconfigured anonymous access from loopback.
+- User privilege checks distinguish legacy empty `allowed_models=[]` from explicit no-model access through `allowed_models_restricted=True`.
+- `LOCALHOST_BYPASS` in `app.py` only applies to direct loopback clients and excludes proxy/tunnel headers. Helper fallback code is weaker and should not be treated as the primary bypass boundary.
+- Legacy migrations claim null-owner SQL/JSON data for the primary admin when possible, and startup repeats a null-owner sweep hourly. Remaining null-owner rows are surface-specific compatibility data that must be deliberately included, no-oped for single-user mode, or rejected for strict ownership gates.
+- `.env` is loaded with `utf-8-sig`, so Windows BOM auth flags still parse.
+
+## Current Gaps
+
+- There is no shell/filesystem sandbox for admin tools.
+- Token scopes remain coarse for some surfaces.
+- `app.py` AuthMiddleware lacks direct regression coverage for bearer-token state/cache behavior, trusted-loopback proxy-header rejection, and internal-tool owner stamping.
+- Codex/Claude scoped route enforcement still needs stronger regression coverage.
+- `THREAT_MODEL.md` still has stale token-scope and `/api/v1/chat` SSRF gap text that should be reconciled with current route validation.
+- The Default/Local owner contract is canonical but only incrementally adopted; route helper `""`, chat/agent `None`, SQL/JSON null-owner compatibility, and calendar fallback owner behavior still need domain-by-domain migration decisions.

specs/calendar-tasks-notes.md

@@ -0,0 +1,186 @@
+# Calendar, Tasks, And Notes
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+This spec covers calendar, reminders, tasks, assistant runs, and notes in:
+
+- app route wiring, auth exemptions, and scheduler startup in `app.py`;
+- canonical database models in `core/database.py`, with `src/database.py` as a compatibility re-export;
+- `routes/calendar_routes.py`, `src/caldav_sync.py`, and `src/caldav_writeback.py`;
+- canonical `routes/task/task_routes.py`, compatibility shim `routes/task_routes.py`, `src/task_scheduler.py`, `src/task_endpoint.py`, `src/event_bus.py`, and `src/interactive_gate.py`;
+- shared privileged task-action policy in `src/task_action_policy.py`;
+- `routes/assistant_routes.py`;
+- canonical `routes/note/note_routes.py`, compatibility shim
+  `routes/note_routes.py`, `src/builtin_actions.py`, and `src/action_intents.py`;
+- agent/tool call sites in `src/tool_index.py` and `src/tool_implementations.py`;
+- scoped Codex wrappers in `routes/codex_routes.py`;
+- database models `CalendarCal`, `CalendarEvent`, `ScheduledTask`, `TaskRun`, `Note`, and `CrewMember`;
+- direct DB CLIs `scripts/odysseus-calendar`, `scripts/odysseus-notes`, and `scripts/odysseus-tasks`;
+- frontend modules `static/js/calendar.js`, `static/js/calendar/*`, `static/js/tasks.js`, `static/js/notes.js`, and `static/js/assistant.js`;
+- tests covering calendar routes/utilities, CalDAV, recurrence, timezone handling, scheduler behavior, task webhooks, notes CLI/tool behavior, and task CLI behavior.
+
+## Calendar
+
+`routes/calendar_routes.py` owns `/api/calendar` behavior: config, multi-account CalDAV CRUD, connection test, sync, local calendar CRUD, event CRUD, recurrence expansion, ICS import/export, quick parse, and user timezone offset handling.
+
+`src.caldav_sync` owns CalDAV fetch/sync. `src.caldav_writeback` owns pushing local changes back to remote calendars. Calendar routes request those behaviors; they do not own CalDAV protocol details.
+
+Runtime behavior:
+
+- local default calendars are created lazily per owner with stable UUID5 candidates. Default creation remains inside the caller's transaction so a failed event write cannot leave an orphaned calendar; SQLite serializes the absent-row check with `BEGIN IMMEDIATE`, other backends recover insert races inside a savepoint, and renamed-owner ID collisions advance through deterministic slots. List-only callers explicitly commit the lazy default.
+- route-level no-login calendar access normalizes empty owner values to `ODYSSEUS_FALLBACK_OWNER` or `owner@localhost`, so route-created calendar rows do not use the empty string as their storage owner;
+- CalDAV account config lives in per-user prefs as `caldav_accounts`, with the legacy `/api/calendar/config` route reading/upserting the first account;
+- recurring rules are expanded server-side, including compound recurrence IDs;
+- RRULE expansion is capped and marks truncated responses;
+- event datetimes preserve UTC/local metadata through `CalendarEvent.is_utc` where supported;
+- CalDAV pull uses a bounded sync window, scopes existing UID lookups to the synced calendar, stamps account ids and remote metadata on local calendars, maps Google principal URLs to event collections, preserves locally-created or writeback-pending events that are not yet remote-owned, and deletes stale in-window remote events only when remote object parsing did not fail;
+- CalDAV writeback stores `remote_href`/`remote_etag`, clears `caldav_sync_pending` only after successful remote writes, and leaves create/update/delete pending markers for retry on failure;
+- pull and writeback paths always close their `DAVClient`, including discovery,
+  database, and remote-write failure paths;
+- sync direction can be pull, push, or both, and pending local writeback rows are included even before remote href metadata exists;
+- ICS import is per-owner, capped, creates fresh local IDs in the target import calendar, and preserves zero-duration events as visible imported rows rather than dropping them as empty ranges;
+- writeback is best-effort and local SQLite remains source of truth when remote writes fail.
+
+Calendar credentials are encrypted at rest and are not returned to clients. CalDAV URL validation rejects unsafe schemes, credentials, fragments, localhost names, bad ports, unsafe IP literals, and hostnames resolving to disallowed addresses, with `ODYSSEUS_ALLOW_PRIVATE_CALDAV=1` as the explicit private-IP escape hatch. CalDAV sync/writeback clients disable redirects so credentials are not followed to another origin. The connection-test client keeps proxy/environment trust disabled but explicitly loads an operator `SSL_CERT_FILE` or `REQUESTS_CA_BUNDLE` when the file exists so private/self-signed deployments use the same CA trust intent as real sync.
+
+## Tasks And Assistant Runs
+
+`src.task_scheduler.TaskScheduler` owns scheduled task execution, next-run computation, strict single-slot execution, queued/running cleanup at startup, overdue next-run advancement, webhook-triggered tasks, notifications, run records, chained tasks, and event-triggered actions.
+
+Cookbook serve scheduling crosses this domain. The Cookbook UI creates `cookbook_serve` scheduled tasks, can mirror them as Cookbook calendar events with `cookbook_event_uid`, and task deletion cleans up the linked event when present, falling back to exact-summary matching for legacy events without a stored UID. Cookbook command execution/lifecycle details stay in `cookbook-hwfit.md`.
+
+`routes.task.task_routes` owns task CRUD, status, manual run/stop/cancel, pause/resume, owner-scoped run/activity history, metadata, onboarding defaults, cache clearing, parse endpoints, and webhook-token regeneration. `app.py` imports the canonical package path; `routes/task_routes.py` replaces its module entry with the canonical module for legacy import and monkeypatch compatibility. Chained-task `then_task_id` values are validated as same-owner relationships on create/update, and scheduler execution also rejects cross-owner or cyclic chains.
+
+Task webhook paths are auth-exempt at the app middleware layer only for `/api/tasks/{task_id}/webhook/{token}`. The route still validates active task state plus task-specific webhook token before dispatch.
+
+Task runtime behavior:
+
+- task runs move through queued/running/success/error/skipped/aborted states;
+- scheduler/background execution can wait for `src.interactive_gate` to report a quiet foreground window, and running background work can use browser heartbeat/chat-stream activity as a cancellation/defer signal where implemented;
+- output targets include chat sessions, notifications, email, and MCP delivery paths;
+- LLM and research tasks can carry a built-in `character_id` persona prompt that the scheduler prepends at execution time;
+- task-created chat sessions can be foldered under `Tasks`, and startup migration backfills task/research folders for legacy sessions;
+- event-bus triggers persist counters and `next_run` before scheduler handoff;
+- the in-process scheduler is gated by `ODYSSEUS_INPROCESS_TASKS`, and multiple enabled app processes can double-run work.
+- action tasks with `run_local`, `run_script`, `ssh_command`, or
+  `cookbook_serve` are admin-only. `routes.task_routes` enforces this on
+  create/update/manual run and hides those actions from `/meta/actions` for
+  non-admin owners; webhook and scheduler execution pause the task and clear
+  `next_run` if an admin-only action belongs to a non-admin owner.
+- background LLM task execution uses the background workload path, and the
+  scheduler can abort/cancel active in-process task runs when foreground browser
+  activity appears.
+- `tidy_research` scans all persisted research files because broken JSON has no trustworthy owner stamp, so it runs only for admins or the explicit auth-disabled single-user operator and refuses regular/pre-setup callers before enumeration.
+
+`routes.assistant_routes.py` owns crew/assistant settings and run-status surfaces that use the scheduler. `TaskScheduler.ensure_assistant_defaults()` currently seeds the personal assistant crew member and pinned assistant session, but no longer auto-creates Morning/Midday/Evening check-in tasks. Existing crew-linked check-in tasks are still rendered and managed when present.
+
+## Notes And Reminders
+
+`routes.note.note_routes` owns notes/todos/reminders, and `app.py` imports that
+canonical path. `routes.note_routes` replaces its module entry with the
+canonical module for legacy import and monkeypatch compatibility. Notes are
+SQLAlchemy `Note` rows and can include due dates, ordering, images, repeat
+state, AI classification, source/session provenance, and agent session
+linkage.
+
+Notes CRUD/reorder/reminder routes resolve the acting owner through `require_user()`: auth-enabled anonymous requests fail closed before hitting owner-scoped queries, while documented no-login/single-user modes still resolve to the compatibility owner path.
+
+Reminder policy:
+
+- "remind me at 5pm" should become a todo/note with a due date;
+- calendar event alarm/reminder UI writes reminder Notes;
+- calendar events are for scheduled time blocks, meetings, appointments, or explicit calendar requests;
+- creating a calendar event named "Reminder" does not create notification behavior.
+
+Built-in reminder/persona prompt text is mirrored server-side for reminder synthesis and scheduled task execution; frontend persona selectors are UI over that server-owned id map, not the authority.
+
+Reminder dispatch is Note-owned:
+
+- `dispatch_reminder()` owns browser, email, ntfy, generic webhook, in-app notification, optional LLM reminder text, and dedupe behavior;
+- the scheduler note scanner calls note-ping actions for backend due-note delivery with per-owner notification state, and calendar-event reminders are treated as Note-owned reminders rather than separate scheduler event pings;
+- the notes frontend has a browser-tab fallback for visible sessions;
+- calendar frontend reminder UI stores reminder records as Notes, not calendar-event notification jobs.
+
+Email/ntfy failures degrade into channel result fields rather than blocking every reminder path. ntfy and generic webhook reminder URLs run through outbound URL safety checks, with `REMINDER_WEBHOOK_BLOCK_PRIVATE_IPS` controlling whether private/LAN targets are allowed. ntfy notification titles are converted to ASCII with replacement and capped at 200 characters before entering HTTP headers. Reminder dedupe uses owner-scoped cache files under `data/`.
+
+## Agent, Codex, And CLI Surfaces
+
+`do_manage_tasks`, `do_manage_notes`, and `do_manage_calendar` own agent-side writes. `do_manage_calendar` supports batch event creation plus list range aliases (`start`, `start_time`, `start_date`, `range_start`, `from`, `dtstart`, `since`, and matching end aliases), calendar name/short-id lookup, importance/tag aliases, and reminder offsets expressed as numbers, minute/hour words, or common abbreviations such as `min`/`mins`/`hr`/`hrs`. If a model supplies a loose `query`, `date_range`, or `range` without explicit start/end datetimes, `list_events` returns an error asking the caller to resolve the range and call again instead of guessing. Event classification reads `Memory.text` for personal context before LLM classification. `src.tool_index` encodes the reminder policy that notes/todos own reminders while calendar events own time blocks.
+
+Agent native tool owner handling is not uniform today. `do_manage_tasks()` filters lists only when `owner` is truthy and creates tasks with the passed owner, so `owner=None` can create legacy/null-owner tasks. For authenticated/non-empty owners, edit/delete/pause/resume/run require an exact stored owner match and reject both cross-owner and null-owner rows; `owner=None` retains single-user compatibility. `do_manage_notes()` list/query behavior distinguishes `None` from `""`, with `None` acting as broader single-user compatibility while `""` filters to empty-owner rows in some paths. `do_manage_calendar()` query helpers filter only when owner is not `None`, while calendar creation routes through the calendar fallback owner for default calendars. These are compatibility behaviors, not a cross-user sharing model.
+
+Note and calendar route/tool writers owner-reserve any canonical internal upload
+references in content, checklist/color/image fields, descriptions, and
+locations before their database writes. Missing or wrong-owner uploads fail the
+write instead of creating a dangling durable reference; reservations serialize
+with upload cleanup.
+
+Chat forwards browser timezone offset and IANA timezone name so natural-language note/calendar tools can anchor dates to the user clock. A valid IANA zone wins over the fixed offset for current-time/DST reasoning; invalid or absent names fall back to the offset and then server-local/UTC compatibility behavior. Chat can auto-promote note/calendar/reminder intents to agent mode.
+
+Codex todo/calendar wrappers enforce bearer-token owner and `todos:*` or `calendar:*` scopes, then delegate to note/calendar behavior as the token owner. Normal calendar/task/note routes are current-user/cookie routes and should not be treated as scoped bearer-token APIs unless they explicitly use token owner/scope policy.
+
+Direct DB CLIs are local compatibility tools. They bypass HTTP route behavior, CalDAV writeback, and some owner/timezone parsing policy.
+
+## Event Bus
+
+`src.event_bus` owns event-triggered task counters and scheduler handoff. Current emitters include chat/session/document/memory/research/email/skill paths. Ownerless events resolve to a primary configured user instead of broadcasting to every owner.
+
+The current event bus is not a calendar-event emitter despite the adjacent calendar/task/reminder domain.
+
+## Timezone And Date Semantics
+
+- calendar events store offset-aware input as UTC/naive fields plus `is_utc`;
+- note `due_date` uses ISO-like strings interpreted through note/tool parsers;
+- chat forwards browser UTC offset into `routes.calendar_routes` request-local state for natural-language date anchoring in calendar/note tool parsing;
+- generic scheduled task clock times are stored as UTC values after local conversion;
+- assistant check-ins can use an IANA timezone on `CrewMember`, with UTC fallback.
+
+Dateutil fallbacks strip timezone-aware parser results back to the naive-UTC contract before recurrence/window comparisons. Calendar agent list tools accept current range aliases implemented by `src.tool_implementations`, and equal/same-day start/end ranges are normalized to a one-day window instead of silently returning no rows.
+
+Natural-language parsers prefer time-first interpretations for short reminder/event phrases where the user supplies a clock time before a date phrase.
+
+Calendar frontend week-start preference is browser-local (`cal-week-start`) with Monday/Sunday controls; it is not persisted as a server preference.
+
+Natural-language date parsing and timezone behavior are compatibility-sensitive and need route/tool/frontend regression coverage when changed. Request-local timezone context is ephemeral and must not be persisted as user state. A valid browser IANA timezone is authoritative over a possibly stale or wrong-sign fixed offset because it carries daylight-saving rules.
+
+## Degraded And Optional Behavior
+
+- CalDAV sync no-ops with shaped errors when unconfigured, invalid, offline, or missing the optional `caldav` dependency.
+- CalDAV writeback failures are non-fatal to local calendar writes and are mostly visible through logs.
+- Missing or invalid `croniter` rejects cron schedules or yields no next run.
+- Missing timezone support falls back to UTC or legacy behavior.
+- ICS import depends on `icalendar`; missing dependency can fail before route-shaped error handling today.
+- Notes reminders can still use local browser fallback when backend email/ntfy channels fail.
+- App backup import/export does not currently include calendar events, scheduled tasks, task runs, or notes; calendar ICS import/export is separate and calendar-only.
+
+## Security And Provenance
+
+Calendar, task, note, and assistant routes are owner-scoped for normal users. Legacy null-owner behavior is compatibility-sensitive and should not silently grant authenticated owners broad mutation rights.
+
+Because auth-disabled chat owners can arrive as `None`, tool-created rows may not use the same owner value as route-created rows. Multi-user or owner-model changes must audit both route and agent paths.
+
+Task creation/update/manual run/webhook/scheduler execution blocks shell-like and Cookbook serve action types for non-admin users through `src.task_action_policy`, and tool security blocks privileged task/calendar tools for non-admin use. Assistant defaults reject synthetic owners such as `api` and `internal-tool`.
+
+Note routes store caller-provided `source`, `session_id`, `image_url`, and agent-session provenance. Canonical internal upload references in persisted note/calendar fields are owner-reserved before writes, and upload-backed bytes remain protected when fetched through upload routes. Arbitrary non-upload image/provenance URLs are not otherwise normalized or validated by note storage.
+
+## Testing Coverage
+
+Existing coverage is strongest around CalDAV URL hardening/writeback, client cleanup and operator CA handling, bidirectional/pending CalDAV sync markers, CalDAV UID calendar scoping, calendar recurrence/timezone helpers, owner-scoped calendar basics, exact-owner task-tool mutations, scheduler restart/cancel/next-run behavior, webhook auth-exemption source shape, canonical/legacy note-module identity, note-route unauthenticated fail-closed behavior, note/calendar attachment reservations, notes CLI/tool due-date behavior, calendar reminder abbreviation parsing, task CLI preview, task persona fields, and same-owner chained task validation.
+
+Route-level coverage is thinner for full calendar route behavior, task CRUD/security/run controls, live webhook token dispatch, notes owner CRUD/reminder delivery, assistant defaults/run status, event-bus triggers, Codex todo/calendar scopes, and frontend panel wiring.
+
+## Current Gaps
+
+- CardDAV still needs URL hardening parity with CalDAV; CalDAV now resolves hostnames during validation and revalidates writeback URLs.
+- `do_manage_notes()` should match HTTP note-route owner behavior for legacy null-owner notes.
+- Auth-disabled agent tools can produce or read broader owner scopes than route handlers because they receive `owner=None`; tasks, notes, and calendar need aligned policy/tests.
+- Task webhook tests should keep exercising live route token behavior and
+  admin-only action blocking, not only middleware/source strings.
+- Reminder delivery needs tests across frontend `/fire-reminder`, backend `dispatch_reminder()`, scheduler note pings, channel degradation, and dedupe.
+- Codex todo/calendar scope and owner mapping needs dedicated regression coverage.
+- Direct DB CLIs need either documented route-bypassing support status or shared helpers to avoid owner/timezone/writeback drift.
+- `scripts/odysseus-webhook` builds the live `/api/tasks/{task_id}/webhook/{token}` path with percent-encoded path segments; its direct DB token rotation/revocation behavior remains a local compatibility surface.
+- Assistant default documentation/code comments still mention check-ins that are no longer auto-seeded.
+- App backup import/export does not cover the calendar/task/note rows described by this spec.

specs/chat.md

@@ -0,0 +1,154 @@
+# Chat
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+This spec covers current chat behavior in:
+
+- `routes/chat_routes.py` and `routes/chat_helpers.py`;
+- `routes/session_routes.py` and canonical `routes/history/history_routes.py`,
+  with `routes/history_routes.py` as a compatibility shim;
+- `src/chat_helpers.py`;
+- `src/agent_runs.py`;
+- `src/chat_handler.py` and `src/chat_processor.py`;
+- `core/session_manager.py` and `core/models.py`;
+- `src/attachment_refs.py` and `src/upload_handler.py` for durable attachment
+  references and write reservations;
+- `src/context_budget.py`, `src/context_compactor.py`, and `src/topic_analyzer.py`;
+- `src/foreground_model_routing.py`, `src/tool_approval_scopes.py`, `src/tool_approvals.py`, and `src/tool_capabilities.py`;
+- `routes/workspace_routes.py` for workspace selection support;
+- frontend modules `static/js/chat.js`, `static/js/chatStream.js`, `static/js/chatRenderer.js`, `static/js/sessions.js`, `static/js/search-chat.js`, `static/js/compare/stream.js`, `static/js/workspace.js`, `static/js/composerArrowUpRecall.js`, `static/js/streamingSegmenter.js`, `static/js/group.js`, and `static/js/notes.js`;
+- integration points with uploads, documents, compare, research, agent tools, memory, RAG, search, and model endpoints.
+
+## Session Ownership
+
+`core.session_manager.SessionManager` owns session persistence and message writes. `routes/session_routes.py` owns session list/create/update/archive/delete/folder/importance behavior for the sidebar. `routes.history.history_routes` owns history/topic surfaces, with `routes/history_routes.py` kept as a compatibility shim.
+
+`core.models.Session` and `ChatMessage` are pure data containers. They do not own persistence; `Session.add_message()` delegates to the configured session manager when present.
+
+Startup session discovery selects non-archived sessions by the existence of persisted `ChatMessage` rows rather than trusting the denormalized `Session.message_count`. It computes authoritative counts only for the bounded discovery set, then keeps full message hydration lazy.
+
+## Streaming
+
+`routes/chat_routes.py` owns `/api/chat`, `/api/chat_stream`, detached stream resume/stop/status, injected context, chat-message search, and rewrite routes. Streaming is the main UI path.
+
+`static/js/chat.js` owns send/abort/continue UI state, the main fetch/read loop, SSE parsing, rendering dispatch, workspace form wiring, and background/resumable stream tracking. `static/js/chatStream.js` owns UI-control event handling and stream/research notification helpers. `static/js/sessions.js` polls server stream status after refresh or session switch. `static/js/composerArrowUpRecall.js` owns prompt recall from the composer when the caret is at the top of an empty input.
+
+Runtime behavior:
+
+- the `/api/chat*` prefix is exempt from the global request hard timeout;
+- browser chat sends `X-Tz-Offset` and an IANA timezone name; request-local helpers prefer a valid IANA zone for DST-aware current-time reasoning, then fall back to the fixed offset;
+- browser chat can send a selected workspace path; route code only resolves it for admin/single-user flows, validates it as an existing directory, and forwards it so agent file/shell tools are confined by `src.tool_execution`;
+- stream callbacks can outlive a deleted session, so persistence must fail closed instead of recreating orphan messages;
+- message metadata carries timestamps, metrics, tool events, sources, hidden
+  thinking/reasoning text when providers expose it separately, context-trim
+  metrics, structured attachment references, and related UI state;
+- metadata preserves requested and actual reply models and endpoints, per-round route transitions, and answering-route cost attribution; stable session ids remain available so prompt/sequence-memory and KV-cache paths can address the same conversation consistently;
+- multimodal content can be a list of content blocks for the live provider call,
+  while persistence collapses raw media into readable text and stable
+  attachment-reference lines;
+- agent streams forward explicit round-cap, tool-budget, repeated-tool-loop,
+  and intent-without-action guard events so the frontend can distinguish a
+  controlled stop from a stalled response.
+
+`src.agent_runs` owns detached in-memory stream runs, replay buffers, replacement cancellation, resume subscribers, explicit stop, and terminal-buffer eviction. Closing the SSE connection does not necessarily stop generation. `static/js/chat.js` can live-resume a still-running detached stream through `/api/chat/resume/{session_id}`; rich responses reload from DB for canonical rendering. Detached runs are process-local and do not survive server restart.
+
+Provider adapters live below chat in `src.llm_core`. Chat consumes normalized SSE output, fallback/error events, reasoning/tool deltas, and metrics. Foreground chat is strict to the selected route by default. Only the selected owner can opt in through `foreground_fallback_enabled` plus ordered `foreground_model_fallbacks`; the retired `default_model_fallbacks` key is ignored. Eligible pre-content availability failures can advance through at most ten owner-visible exact model candidates, while missing configuration/endpoints, provider/schema errors, clean empty completions, and post-content failures remain on the selected route and surface an error. Once a route produces substantive text/reasoning or a tool call it is pinned as the answering route.
+
+Fallback candidates receive route-neutral context shaping. Only compaction performed for the answering route is persisted. Chat and agent metadata record requested/actual model and endpoint identity, round-by-round route transitions, and costs against the route that actually answered; the browser renders same-model endpoint changes as well as model changes.
+
+## Context Preface
+
+`routes.chat_helpers.build_chat_context()` owns the shared route pipeline: preset extraction, preprocessing, user-message persistence, incognito/no-memory/RAG/skills flags, prefetched compare search, YouTube transcript context, research-spinoff grounding, model normalization, and compaction.
+
+`src.chat_processor.ChatProcessor.build_context_preface()` owns source preface construction. It can add memory, RAG, web search, URL page content, and skills index context before the model call.
+
+Chat preface enhances the model's context. It must not rewrite the user message or force literal-vs-fetch interpretation before the model sees the request. See [context-building.md](context-building.md).
+
+Chat-owned external context must enter the model through `untrusted_context_message()` unless a different treatment is explicitly documented. This includes memory, RAG, web search, URL fetches, prefetched search context, YouTube transcripts, research injection, and manual context injection.
+
+## Modes And Handoffs
+
+Chat can dispatch to normal LLM calls, agent mode, research mode, or compare-related flows. Session mode is stored on `sessions.mode`.
+
+Legacy plan-mode backend plumbing still exists below chat, but `routes/chat_routes.py` currently forces browser/form `plan_mode` input off and the old visible plan window frontend module is not part of the current SPA. Treat plan-mode changes as compatibility work unless the UI contract is intentionally reintroduced.
+
+Current call sites include:
+
+- chat/research dispatch in `routes/chat_routes.py`;
+- agent execution in `src/agent_loop.py`;
+- deep research orchestration in `src/research_handler.py`;
+- compare entry points in canonical `routes/compare/compare_routes.py` and frontend compare modules.
+
+Agent-mode tool access is gated in layers. Chat route toggles and privileges
+build a disabled-tool set; incognito and compare mode remove persistence-heavy
+or UI-breaking tools; `src.action_intents.message_needs_tools()` provides
+conservative regex auto-escalation hints; `src.agent_loop`,
+`src.tool_security`, `src.tool_execution`, and internal loopback validation
+remain server-side enforcement owners.
+
+`allow_bash` and `allow_web_search` can be read from the JSON request body for browser chat posts that do not submit traditional form fields.
+
+Web search tools are per-turn explicit opt-in. Either `allow_web_search=true`
+or `use_web=true` can enable `web_search`/`web_fetch`, but an explicit
+`allow_web_search=false` wins over `use_web=true` and keeps those tools
+disabled. Explicit latest-turn web-search intent can still auto-escalate into
+agent mode and narrows the available tool set toward `web_search`/`web_fetch`,
+but it no longer re-enables web tools after an explicit denial or global
+disable.
+
+Guide-only/no-tools requests build an effective tool policy before preprocessing and agent dispatch. That policy suppresses tool-backed preprocessing/background extraction/research, disables schemas and MCP for the turn, and is still enforced by `src.tool_execution` if a model emits a tool call anyway.
+
+When route context is trimmed without full compaction, chat emits a
+`context_trimmed` SSE event and carries before/after message/token counts into
+metrics. Provider reasoning/thinking deltas are streamed for live UI handling
+but kept out of the visible saved assistant content and stored in metadata when
+available.
+
+## Attachments
+
+`src.chat_handler.ChatHandler.preprocess_message()` owns owner-scoped upload-id resolution, attachment metadata, YouTube transcript/comment preprocessing, image/VL behavior, and enhanced text used by chat. `src.document_processor.build_user_content()` owns conversion of uploaded/chat-attached files into model-ready text or multimodal blocks. `src.attachment_refs` owns persisted text/reference normalization, and `SessionManager` owner-reserves attachment ids before appending or replacing durable message rows. `static/js/fileHandler.js` owns frontend pending-file state.
+
+Attachment-only sends are valid. Missing or unauthorized ids are skipped during preprocessing, while a missing/wrong-owner durable reference aborts a message/history replacement before existing transcript rows are removed. Upload failures keep pending files for retry, unsupported media can degrade to text markers, optional Office/PDF/VL dependencies can emit extraction banners, Office attachments can create markdown documents when extracted server-side, and fillable-PDF auto-document failures fall back to normal PDF extraction. `chat_messages.content` and FTS do not retain provider data URLs; structured references stay in metadata for reloads. Chat does not own upload bytes or durable document storage; it requests document/upload behavior from those subsystems.
+
+Frontend chat distinguishes normal resend from regenerate-from-here: normal resend appends a fresh user copy and carries upload IDs where available, while regeneration truncates from the selected point. AI-message delete prompts before removing the AI response plus preceding user turn. Desktop Enter submits; mobile Enter inserts a newline unless another platform-specific send control is used.
+
+Native document tool outputs can open or refresh the document editor from
+tool-result metadata, so the UI can recover if a later `doc_update` stream event
+is missed. The chat renderer also hides raw/incomplete leaked tool JSON and
+document fences from normal transcript text.
+
+When untrusted external/workspace content has entered the agent context, high-impact tool calls pause as exact approval cards instead of executing. The browser can allow the rest of the interrupted task, allow this chat session, or deny; it submits only the opaque id/decision with an empty control-plane message and does not mutate the composer. The server restores the sealed first action plus private selected tools/query, revalidates policy and document freshness, consumes the first action, and resumes without persisting a synthetic user message. Task scope ends with that resumed run. Chat scope persists the resolved card and marks later context only for that exact session; forks do not inherit it. A normal message retires an unresolved card while preserving taint.
+
+## Security And Provenance
+
+`/api/chat` and `/api/chat_stream` verify session ownership before loading the session. Chat privilege gates enforce allowed models and daily message caps before LLM work. Active document injection, session auth/header recovery, endpoint repair, upload-id resolution and reservation, memory/RAG retrieval, and post-response work must stay owner-scoped.
+
+The scoped API-token chat surface is `/api/v1/chat`. Browser chat routes can receive bearer-auth state from middleware, but route code must not assume `"api"` is a durable owner; API-token support requires explicit scope checks and token-owner attribution.
+
+Incognito disables memory, skill, and chat-history tools and skips assistant DB persistence, but current user-message persistence and later cleanup are not a strict no-write guarantee. Treat incognito changes as security-sensitive until that contract is clarified.
+
+## Search Boundary
+
+`GET /api/search` in `routes/chat_routes.py` is chat-message search for the UI and slash commands. Web search routes are owned by canonical `routes/search/search_routes.py`; chat and agent web context call through `src.search`, compatibility shims, and search content fetchers. Do not confuse chat-history search with external web retrieval.
+
+## Degraded And Compatibility Behavior
+
+- Missing ChromaDB, embeddings, memory vectors, RAG managers, or skills indexes should remove injected context or fall back to keyword/text behavior without failing chat.
+- Direct URL prefetch failures become compact untrusted context stating that the page was not read, with only transport-owned HTTP/size/rate-limit status where recognized; raw URLs, exception text, and response-controlled diagnostics are not echoed into logs or model context.
+- Sessions hydrate legacy string headers and multimodal JSON-array content, export text/HTML/Markdown after flattening non-string blocks, can lazy-load from DB when cached state is empty, and preserve old history/index delete behavior where needed.
+- Initial shell/session loading is non-blocking: the sidebar can render before a selected transcript is hydrated, and full transcript hydration is deferred until display or a model send requires it.
+- Chat repairs empty selected models and orphaned endpoint references before provider calls when possible.
+- Deleted-session stream writes fail closed.
+- Docker/native endpoint differences are owned by runtime/model setup, but chat sessions depend on the saved endpoint URLs and headers.
+- Copying a response from the UI copies the displayed answer text and omits hidden reasoning/thinking segments.
+
+## Current Gaps
+
+- Chat, agent, research, and compare orchestration still meet in a large route file.
+- Context preface behavior is spread across `routes/chat_helpers.py`, `src/chat_processor.py`, route injections, and agent/tool paths.
+- Detached stream lifecycle spans `routes/chat_routes.py`, `src/agent_runs.py`, `static/js/chat.js`, `static/js/sessions.js`, and non-chat callers.
+- Some frontend stream state is still global/module-level in `static/js/chat.js` and needs careful session isolation when adding background or resumable flows.
+- Chat lacks route-level SSE regression tests for `/api/chat_stream`, live resume/stop/status, mode handoff, persistence metadata, partial-save behavior, attachment/doc-update events, browser timezone offset/workspace handling, and literal URL context intent.
+- Bearer-token behavior on browser chat routes and incognito persistence need explicit contract decisions and regression coverage.

specs/compare.md

@@ -0,0 +1,79 @@
+# Compare
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+This spec covers model A/B comparison behavior in:
+
+- canonical `routes/compare/compare_routes.py`, with `routes/compare_routes.py` as a compatibility shim;
+- `routes/session_routes.py`;
+- `routes/chat_routes.py` and `routes/chat_helpers.py`;
+- `routes/model_routes.py`;
+- canonical `routes/search/search_routes.py`, with `routes/search_routes.py` as a compatibility shim;
+- `core/database.py` model `Comparison`;
+- `src/llm_core.py` and `src/endpoint_resolver.py`;
+- frontend modules under `static/js/compare/`;
+- `static/js/chat.js`, `static/js/sessions.js`, `static/js/models.js`, and `static/js/slashCommands.js`;
+- `tests/test_compare_*` and focused blind-compare redaction tests.
+
+## Runtime Behavior
+
+The active text compare UI creates ordinary `[CMP]` sessions through `/api/session`, then streams each pane through `/api/chat_stream` with `compare_mode=true`. Search compare is a separate branch: it can query `/api/search/query` directly and its synthesis sessions use ordinary chat streaming without `compare_mode=true`. `static/js/compare/index.js` owns compare orchestration, session creation, execution order, search-mode branching, and export actions. `static/js/compare/panes.js` owns pane add/remove/swap/reroll lifecycle. `static/js/compare/stream.js` owns pane streaming and event rendering.
+
+`routes/compare/compare_routes.py` owns the `/api/compare` HTTP surface for alternate/legacy start/vote/history/delete behavior and the active `/api/compare/record` vote-summary endpoint. The top-level module is a compatibility alias. Legacy `/api/compare/start` uses neutral helper-session names and withholds model identities/mapping from the start response while blind mode is active. It does not own provider-specific payload behavior.
+
+Current call sites include:
+
+- `/api/session` compare session creation and cleanup in compare frontend modules;
+- `/api/chat_stream` pane execution through chat routes and detached stream infrastructure, streamed directly into panes so upstream generation stops promptly when panes are stopped;
+- `/api/models` and probe routes for model/endpoint selection;
+- search-provider compare mode through `routes/search/search_routes.py`;
+- `/api/compare/record` as a fire-and-forget backend vote summary, while active scoreboard state is localStorage-backed.
+
+`Comparison` rows currently persist vote/history metadata: prompt, first model identifiers, winner, blind flag, optional N-model JSON in `blind_mapping`, vote timestamp, and owner. Response and metric columns exist in the schema but are not populated by the active compare UI flow. Compare history must be owner-scoped.
+
+Frontend compare behavior is split by responsibility:
+
+- `state.js` owns local compare state;
+- `selector.js`, `models.js`, and `probe.js` own endpoint/model selection and probe UI;
+- `panes.js` and `stream.js` own paired response rendering;
+- `vote.js` and `scoreboard.js` own voting and history display.
+
+Compare panes can receive `ask_user` or tool-approval controls from the shared chat stream. `static/js/compare/stream.js` routes those controls into the main chat renderer/control plane, pauses pane completion/autograding while a choice is pending, and can resume the pane after the user decision; compare orchestration keeps its busy state until those continuations settle.
+
+Mobile compare layout collapses multi-pane grids to a single column so panes
+remain readable on narrow screens while the desktop grid still uses the
+selected column count.
+
+## Ownership Boundaries
+
+Compare owns paired evaluation flow and pane state. Chat routes own the actual stream execution path for compare panes. LLM provider code owns model-call mechanics. Session/model routes own endpoint-id resolution, owner-filtered endpoint/model visibility, header copying, and deleted-endpoint failures.
+
+`compare_mode` in chat strips compare-breaking tools, disables document tools for `[CMP]` sessions, skips some research clarification, and suppresses memory, skill, and webhook side effects after pane responses.
+
+Compare frontend code is part of the app DOM security surface. Current stream/search rendering sanitizes probe labels and tool labels, constrains search-result links to HTTP(S), uses safe generated-image display sources, and opens compare export/image popups with opener isolation.
+
+## Policy Notes
+
+- Current blind compare is UI/API masking until vote/reveal, not a full confidentiality boundary. `[CMP]` session names and session-list model fields are redacted for helper sessions, and legacy `/api/compare/start` withholds model identity/mapping while blind. Client-side selected model state and privileged/local inspection can still expose identity.
+- Compare endpoint lists and secondary endpoint lookups use owner filtering so users see and resolve only shared or owned endpoints.
+- Non-admin compare session creation must use registered owner-visible endpoints; compare must not allow arbitrary raw endpoint URLs to bypass session-route endpoint policy.
+- Prefetched search, URL, RAG, and research context entering compare panes must use the untrusted-context wrapper.
+- Compare panes use chat's foreground routing contract: selected routes are strict unless that owner explicitly enabled ordered foreground fallbacks. Verify each pane still reaches its intended route and that any opt-in route transition or error is visible.
+
+## Degraded And Compatibility Behavior
+
+- Missing/offline endpoints are surfaced by model/session routes; chat can clear orphaned endpoint references and recover empty models when possible.
+- Compare streams inherit chat's opt-in, eligible-pre-output-only foreground fallback and provider-normalized SSE events, but compare frontend handling for errors and model/endpoint route transitions is thinner than chat's stream path.
+- Shared legacy `ModelEndpoint.owner == NULL` rows remain visible through owner filters. Legacy `Comparison.owner == NULL` rows are not treated as shared for authenticated vote/delete/history flows.
+- `/api/compare/start` and `/{comp_id}/vote` remain implemented but are not the active frontend path.
+
+## Current Gaps
+
+- Blind mode is not a confidentiality boundary; client/local state can still expose model identity before vote.
+- `/api/compare/start` accepts raw endpoint URLs and can diverge from `/api/session` endpoint-owner/raw-endpoint policy.
+- `src/agent_loop.py` advertises stale compare app API endpoints.
+- Compare streaming and chat streaming are separate frontend paths but share model/provider infrastructure; regressions can happen when provider event shape changes.
+- Compare frontend needs explicit fallback/error event handling parity with chat streaming.
+- Compare tests cover endpoint owner helper behavior, blind compare redaction, ask-user/tool-approval routing, and portable JS helpers, but not full active `/api/session` pane creation, frontend pane lifecycle, or complete SSE fallback/error handling.

specs/context-building.md

@@ -0,0 +1,113 @@
+# Context Building
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+This spec covers model-context construction in:
+
+- `src/chat_processor.py`;
+- `src/chat_handler.py` and `src/youtube_handler.py`;
+- `routes/chat_helpers.py` and context injection in `routes/chat_routes.py`;
+- `src/agent_loop.py`;
+- `src/tool_execution.py`;
+- `src/attachment_refs.py` and uploaded-file manifest construction in
+  `routes/chat_helpers.py`;
+- `src/tool_policy.py`;
+- `src/prompt_security.py`;
+- `src/tool_capabilities.py`, `src/tool_approval_scopes.py`, and `src/tool_approvals.py`;
+- transport primitives in `src/outbound_fetch.py` plus fetch/extraction adapters in `src/search/content.py` and `services/search/content.py`;
+- search orchestration in `services/search/core.py` and the compatibility wrapper in `src/search/core.py`;
+- RAG and personal docs in `src/rag_singleton.py`, `src/rag_vector.py`, `src/rag_manager.py`, and `src/personal_docs.py`;
+- research flows in `src/deep_research.py`, `src/research_handler.py`, and `services/research/research_handler.py`;
+- memory and skills in `src/memory.py` and `services/memory/*`;
+- related policy in `THREAT_MODEL.md`.
+
+## Contract
+
+Context-building tools gather evidence. They do not own user-intent routing.
+
+Runtime rules:
+
+- if external context is available, add it as compact untrusted source data;
+- if an attempted source is unavailable and relevant, represent the unavailable state explicitly with source and reason when known;
+- preserve the user's original message for the model;
+- do not use regex preprocessing to force literal-vs-fetch intent;
+- do not disable tools or force a reply style solely because preprocessing found a URL.
+
+## Untrusted Data
+
+`src.prompt_security` owns the untrusted wrapper:
+
+- `UNTRUSTED_CONTEXT_POLICY` states global model policy;
+- `untrusted_context_message(label, content)` wraps source content as user-role data with `metadata.trusted = False`, provenance origin, and an `arm_tool_gate`/`tool_gate_untrusted` signal that defaults to arming the server-owned tool gate.
+
+Current untrusted context sources include:
+
+- fetched URLs and web search results;
+- webpage content passed into deep-research extraction;
+- YouTube transcripts/comments;
+- RAG/personal document chunks;
+- memories and skills;
+- notes and active editor documents;
+- emails and attachments;
+- tool output from external/user-controlled data.
+
+Live multimodal provider blocks can contain data URLs, but persisted and
+tool-facing context uses stable attachment references. Tool manifests carry an
+`odysseus://attachment/<id>` URI and owner-checked read policy; local paths are
+compatibility data added only after owner and root-confinement checks. Persisted
+chat context keeps readable text/reference lines rather than reinserting raw
+media bytes into later turns or search state.
+
+## URL, Search, And Tool-Derived Context
+
+Chat URL prefetch and agent `web_fetch` are different paths. Chat prefetch happens before the model call; `web_fetch` is a tool the model may choose later. Both should converge on the same intent: enrich context when content is available, represent unavailable content when it is not, and let the model interpret the user request.
+
+Search results and fetched pages are evidence. `web_search` should not force a page fetch unless its explicit contract says it does. Failed fetches should not crash chat or silently imply content was read. Canonical search content fetchers can extract readable text from HTML, `text/*`, Markdown, `.txt`, `.json`, and `.jsonl` responses and should return shaped error results for HTTP status failures. URL fetches validate every redirect hop and pin the outbound connection to a public IP resolved during validation, so context-building callers do not need a second DNS-rebinding guard.
+
+Current behavior is not yet unified:
+
+- successful chat URL prefetch is wrapped as untrusted context; failed prefetch now adds a compact untrusted statement that the page was not read, recognizes only transport-owned HTTP/size/rate-limit categories, and suppresses raw exception/response text;
+- agent `web_fetch` returns explicit URL-specific tool errors for timeout, unsupported scheme, fetch failure, or no readable text;
+- comprehensive search reports provider-chain failures, but individual page-fetch failures can be logged and omitted;
+- YouTube fetching is owned by `ChatHandler`/`youtube_handler`, while `routes.chat_helpers` only wraps the resulting transcript/comment strings.
+
+`src.outbound_fetch` owns reusable synchronous public-URL classification, per-hop DNS resolution/pinning, redirect handling, and body budgets. `services/search/core.py` owns `comprehensive_web_search()` orchestration. `services.search.content` owns content extraction and adapts the shared transport; `src/search/core.py` and `src/search/content.py` preserve compatibility imports without a second implementation.
+
+## Tool Result Envelope
+
+`src.tool_execution` executes and formats tools. Tool output caps live in `src.constants` and are re-exported through older facades; shared native-tool truncation lives in `src.tool_utils`. `src.agent_loop._append_tool_results()` owns model re-entry: native tool calls return as provider-style `role: "tool"` messages with untrusted metadata, while fenced-tool results use the untrusted wrapper. Classification considers both the requested tool and the result payload, so remote or stored model-visible content can arm the session gate even on a failed tool status.
+
+Taint is server-owned continuation state, not a model instruction. After untrusted external/workspace context, low-impact reads can continue, but high-impact, unknown, and arbitrary MCP actions become proposals that produce an exact approval card. The server seals the exact first action plus private continuation tool/query state; document actions also bind the current document version and digest. A chat decision can allow the resumed task or persist a grant for later turns in that exact chat, while non-chat callers remain single-action. Blocked/approval placeholders and content-free failures do not recursively arm the gate.
+
+Context budgeting uses known model context windows when available. `src.context_budget` treats the default 6000-token value as an automatic sentinel, scales to a capped fraction of known context length for non-explicit budgets, and leaves unknown windows on conservative defaults.
+
+Side-effect enforcement lives outside context building. Chat route disabled-tool policy, `src.tool_security`, `src.tool_execution`, and `do_app_api()` block unsafe tool execution; prompt wording alone is not the authority.
+
+Guide-only/no-tools policy can suppress context acquisition before the model call. `src.tool_policy` feeds chat route preprocessing and agent-loop assembly so tool-backed search/research/memory/RAG/skills/local-context paths are skipped when the latest user turn explicitly forbids tools.
+
+## Degraded And Optional Dependencies
+
+- ChromaDB, HTTP embeddings, and FastEmbed are installed/expected in normal setups but must degrade cleanly when a service, package, or embedding backend is unavailable.
+- `src.rag_singleton.get_rag_manager()` owns RAG startup retry throttling; `src.rag_vector.VectorRAG` is the live owner-filtered path; `src.rag_manager.RAGManager` is compatibility/backward-compat behavior.
+- Memory-vector and tool-index retrieval can fall back to keyword/text behavior when vector stores or embeddings fail.
+- Docker compose and native installs use different Chroma host defaults; model endpoint loopback rewriting is owned by model/runtime specs.
+
+## Current Call Sites Include
+
+- `ChatProcessor.build_context_preface()` for memory, RAG, web search, URL content, and skills index;
+- `ChatHandler.preprocess_message()` and the canonical `services.youtube.youtube_handler` import path for YouTube fetch/format, then `routes/chat_helpers.py` for wrapping prefetched search/Youtube context;
+- `routes/chat_routes.py` research context injection;
+- `src.agent_loop` for active editor document, skill context, and tool-result reinsertion;
+- uploaded-file manifest/reference context for agent tools and later chat turns;
+- `src.tool_execution` for `web_search`, `web_fetch`, file, shell, MCP, and other tool outputs;
+- `src.deep_research` and research handlers for search/fetch/extract flows used by research jobs, with fetched webpage text wrapped before extraction and analyzed URLs tracked separately from source snippets.
+
+## Current Gaps
+
+- URL/search context result shape is not unified across chat prefetch, agent tools, and research.
+- Failed fetch representation remains inconsistent outside direct chat URL prefetch, especially in comprehensive search and research aggregation.
+- Tool/context wording is spread across schema, prompt, and retrieval surfaces.
+- Source-specific wrapping and unavailable-state behavior still needs broader focused coverage for literal URL intent, research, RAG/memory/skills, and YouTube; external tool results and approval continuation now have dedicated gate/taint regressions.
+- Compare pre-search context is computed but may not be submitted through the current compare stream form.

specs/cookbook-hwfit.md

@@ -0,0 +1,195 @@
+# Cookbook And Hardware Fit
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+This spec covers model setup/serving and hardware fit in:
+
+- app route registration in `app.py`;
+- `routes/cookbook_routes.py`;
+- `src/cookbook_serve_lifecycle.py`;
+- `src/host_docker_access.py`;
+- Cookbook package/rebuild/shell integration in `routes/shell_routes.py`;
+- `routes/cookbook_helpers.py`;
+- `routes/hwfit_routes.py`;
+- `services/hwfit/*` and `services/hwfit/data/hf_models.json`;
+- durable Cookbook state through `routes.cookbook_helpers.COOKBOOK_STATE_FILE`;
+- helper/CLI scripts `scripts/odysseus-cookbook`, `scripts/add_hwfit_models.py`, `scripts/hf_download.py`, and `scripts/diffusion_server.py`;
+- Docker overlays `docker-compose.gpu-*.yml`, `docker/gpu.*.yml`, `docker/host-docker.yml`, `scripts/check-docker-gpu.sh`, and `scripts/check-docker-amd-gpu.sh`;
+- frontend modules `static/js/cookbook*.js`, including Cookbook running, serve, download, diagnosis, progress, and HW Fit modules;
+- tests covering Cookbook helpers, routes, CLI state, package detection, frontend progress, HW Fit services, serve profiles, Docker GPU overlays, and GPU diagnostic scripts.
+
+## Current Call Sites Include
+
+- Cookbook modal and state modules in `static/js/cookbook*.js`;
+- package readiness/install and rebuild flows through `routes/shell_routes.py`;
+- direct shell exec/stream integration used by Cookbook task controls;
+- model endpoint setup and serve flows;
+- hardware-fit recommendations for model choices;
+- image-model recommendations for diffusion serving;
+- APFEL/local platform dependency paths where supported;
+- Docker GPU helper scripts and compose overlays;
+- the `odysseus-cookbook` CLI using the same Cookbook state file.
+
+## Cookbook Runtime
+
+`routes.cookbook_routes` owns model download, setup, SSH key, cached model scan, serve, GPU state, kill-pid, state sync, Hugging Face latest lookup, vLLM recipe lookup, serve diagnosis, and task-status endpoints. `src.cookbook_serve_lifecycle` bridges scheduled `cookbook_serve` tasks into serve/stop behavior; task/calendar scheduling ownership stays in `calendar-tasks-notes.md`.
+
+Access policy is split by surface:
+
+- download/setup/SSH key/cache scan/serve/GPU/kill/state/task-status are admin/internal-tool surfaces;
+- `/api/cookbook/hf-latest` is authenticated-user gated;
+- HW Fit routes are authenticated read/probe routes through normal middleware, not admin-only operations;
+- bearer API tokens do not satisfy Cookbook admin gates.
+
+Runtime behavior:
+
+- POSIX and most remote flows run detached through tmux;
+- local Windows uses detached process/log/pid behavior under `%TEMP%\\odysseus-tmux`; Python first publishes a valid Win32 fallback PID, then Git Bash may replace it with `/proc/$$/winpid` after a ready-file handoff, so PowerShell `Stop-Tree` can terminate the actual serving shell and children instead of receiving an MSYS PID. Frontend PowerShell venv activation is quoted safely and the local Git Bash runner converts a valid `Scripts\\Activate.ps1` prefix into `source <git-bash-path>/Scripts/activate` so the selected environment actually supplies the serve binary;
+- remote Windows uses PowerShell runner scripts;
+- missing `tmux`, `docker`, or serve-engine binaries return shaped errors where possible;
+- local Docker inside the Odysseus container is available only when the Docker CLI exists, `ODYSSEUS_ENABLE_HOST_DOCKER=true`, and `/var/run/docker.sock` is actually mounted as a socket; otherwise Cookbook should show the host-Docker access hint and prefer remote SSH Docker workflows;
+- model serve auto-registers LLM or image `ModelEndpoint` rows immediately, then frontend readiness probing can repair/create fallback endpoints;
+- diffusion-server serves are registered as image endpoints;
+- MLX image serves use `scripts/mlx_image_server.py`, which pins generation/edit dispatch to the model chosen at process start and ignores OpenAI-compatible per-request model selectors;
+- vLLM recipe routes fetch and cache model recipe manifests/YAML from `vllm-project/recipes`, normalize base args/env/dependencies/tool-calling/reasoning variants, and expose compatible strategy metadata for serve setup;
+- Hugging Face download/setup paths can detect and persist encrypted HF tokens for later Cookbook/agent use;
+- local and remote model paths can contain spaces or non-ASCII characters when helper validation/quoting accepts them;
+- task status handles tmux, remote Windows logs, local Windows PID/log files, HF cache completion checks, stale browser-state download guards, pip dependency-install success sentinels, exit-code wrappers, serve diagnosis snapshots, and scheduled serve lifecycle hooks;
+- scheduled serve lifecycle stop attempts only persist `status=stopped`, clear `_scheduledStopAtMs`, and delete auto-registered endpoints for sessions whose tmux/remote stop command succeeded or were already gone; failed stop attempts are logged without marking unrelated expired serves as stopped.
+
+`routes.cookbook_helpers` owns validation and command construction:
+
+- repository and model IDs;
+- local directories, SSH hosts/ports, GPU selectors, and tokens;
+- shell quoting for Bash and PowerShell;
+- pip/install fallback chains;
+- safe environment prefixes;
+- serve command validation;
+- user-shell PATH bootstrap, Git-Bash drive-path conversion, preflight, and exit-code helpers.
+
+Cookbook routes request shell/SSH behavior; they do not relax shell security.
+
+## Shell Dependencies
+
+`routes.shell_routes.py` owns Cookbook-adjacent package readiness/install, shell execution/streaming, and llama.cpp rebuild endpoints. The Cookbook UI calls these routes for dependency diagnosis, install/update actions, engine rebuilds, and tmux/reconnect/stop/kill flows. Windows uses detached log/PID wrappers where POSIX tmux is unavailable.
+
+These are admin-only code-execution surfaces and should be reviewed with Cookbook changes even though they are implemented outside `routes.cookbook_routes.py`.
+
+## State, Secrets, And Provenance
+
+Cookbook state lives under the shared data dir through the `COOKBOOK_STATE_FILE` constant, normally `data/cookbook_state.json`. Routes and the `odysseus-cookbook` CLI use the same state path.
+
+State behavior:
+
+- browser-facing state masks secrets;
+- server-side `env.hfToken` is encrypted before storage;
+- task payloads strip raw HF tokens;
+- browser local storage strips HF token values;
+- state POST has anti-wipe guards for server lists;
+- state POST rejects stale `done` download state when the latest shard/cache markers still show an incomplete download;
+- recent server-side tasks are preserved against stale browser overwrites;
+- task-status validates saved shell-bound fields before SSH/tmux commands.
+
+Cookbook auto-registered endpoints are currently shared/null-owner rows with no API key when created by backend serve registration. Browser fallback registration goes through the normal model-endpoint route. The desired ownership policy for Cookbook-created endpoints should remain explicit.
+
+HW Fit is an MIT-licensed llmfit adaptation; attribution lives in project acknowledgments/licenses.
+
+## Hardware Fit
+
+`services/hwfit/hardware.py` owns hardware detection across NVIDIA, AMD, Apple Silicon, Windows, CPU, RAM, available RAM, remote SSH, container/native probe context, and cached host detections.
+
+`services/hwfit/models.py`, `fit.py`, `profiles.py`, `image_models.py`, and
+`hf_discovery.py` own model catalog loading, normalization, API-backed dynamic
+catalog refresh, memory estimates, quantization labels, fit scoring, serve
+profile computation, image model ranking, and backend/format servability
+filtering.
+
+`routes/hwfit_routes.py` owns the HTTP surface and manual hardware override application.
+
+Runtime behavior:
+
+- hardware detection uses a cache with `fresh=true` bypass;
+- probe results include scope/container visibility metadata, and containerized no-GPU/low-RAM states can return user-facing visibility warnings with rescan/manual/copy-diagnostics actions;
+- manual hardware replacement is a what-if simulator, not additive hardware;
+- manual hardware accepts `cuda`, `rocm`, `metal`, `cpu_x86`, and `cpu_arm`
+  backends and must stay in lock-step with backend support in `fit.py`. Metal
+  simulation marks unified memory and filters toward locally servable GGUF/MLX
+  choices instead of CUDA/vLLM-only formats.
+- ignore switches can drop detected GPU/RAM before ranking;
+- homogeneous GPU grouping targets realistic multi-GPU pools;
+- image model ranking normalizes to a single-GPU fit view;
+- Metal/RDNA/backend restrictions can filter otherwise fit models.
+- Apple Silicon bandwidth estimates use chip/core-specific tables for M-series Max/Pro/Ultra variants and avoid matching non-Apple GPU names.
+- Windows and Apple/consumer-AMD paths filter toward GGUF/llama.cpp-compatible
+  choices. On multi-GPU systems, fixed GGUF target quantization that cannot be
+  served by the selected backend returns `no_fit` rather than `None`.
+
+## Platform And Degraded Behavior
+
+- Linux, Windows/PowerShell, macOS, Docker, NVIDIA, AMD, Apple Silicon, and CPU-only systems have different command paths.
+- Remote hosts are accessed through SSH helpers; Cookbook host/port/path inputs must be validated before command construction.
+- HW Fit remote host/port query values currently do not share all Cookbook route-level validation before SSH probing.
+- Missing local tools or failed installs should surface command/output/error detail where possible.
+- GPU overlays remain optional and do not break CPU-only deployments.
+- Docker GPU overlays pass host devices/env; they do not install CUDA/ROCm engines by themselves.
+- Default Docker Compose intentionally does not mount the host Docker socket. `docker/host-docker.yml` is an explicit high-trust overlay for operators who accept broad host-Docker control from inside the container.
+- NVIDIA Docker diagnostics are read-only by default, and `.env` edits/install actions require explicit flags.
+- AMD Docker diagnostics are read-only and do not mutate `.env`.
+- vLLM is rejected on unsupported Windows/macOS paths.
+- llama.cpp CPU-only and GPU fallback scripts should preserve usable CPU paths.
+- SSH probe failures, GPU driver errors, and no-GPU states should be distinguishable.
+- Remote SSH host/port validation is shared through route validators for Cookbook/HWFit paths.
+- Windows launcher/runtime Git Bash discovery includes per-user installs under `%LocalAppData%\\Programs\\Git`, and WSL/Git Bash detection shapes PATH handling for NVIDIA/remote flows.
+- macOS startup helpers start ChromaDB alongside the app path.
+- Ollama serve can auto-pick an available port, and scheduled task stop paths
+  verify stop success before persisting a stopped state.
+
+## Model Catalog And Latest Lookup
+
+HW Fit model scoring depends on bundled `services/hwfit/data/hf_models.json`,
+bundled `services/hwfit/data/mlx_community_models.json`, runtime dynamic caches
+under `DATA_DIR/hwfit/`, catalog normalization, and assumptions about model
+formats and quantization. `scripts/add_hwfit_models.py` updates the static HF
+catalog.
+
+Hugging Face latest lookup and HW Fit dynamic refresh use external Hub metadata
+and can degrade to empty, unknown-size, partial, or malformed-result behavior.
+`refresh_catalog=1` refreshes API-backed collection caches for MLX community
+and selected HF organization collections, with a 24-hour freshness guard and
+bundled JSON fallbacks when the network/cache is unavailable. HW Fit tolerates
+non-numeric `gpu_count` values from callers. Model normalization also treats
+non-string `parameter_count` and quantization fields as unknown rather than
+calling string methods and aborting the ranking pass. Catalog drift and dynamic
+latest-model metadata are separate sources of recommendation drift.
+
+## Security Policy
+
+Admin gates must stay in place for install, serve, kill, setup, state mutation, and shell-like actions. `/api/shell/exec` is an admin primitive used by Cookbook task control and must stay in this review boundary. Scheduled `cookbook_serve` tasks are admin-only action tasks; task create/update/manual run/webhook/scheduler execution must all reject or pause them for non-admin owners.
+
+Kill-pid guardrails:
+
+- admin-only;
+- PID floor;
+- signal allowlist;
+- validated remote host/port;
+- frontend confirmation for TERM/KILL cleanup.
+
+Shell-bound Cookbook inputs must pass helper validation before command construction. HF tokens, Cookbook state secrets, and endpoint API keys must remain encrypted or masked and must not be written back to clients in raw form. Host Docker socket access must stay opt-in and clearly distinguished from merely having a Docker CLI in the container.
+
+## Testing Coverage
+
+Existing coverage is strongest for helper validation/quoting, SSH host validation, pip fallback and dependency-completion regressions, cached scan scripts, serve profile computation, scheduled serve lifecycle state persistence, hardware detection/ranking across AMD/NVIDIA/macOS/manual/container modes, MLX/Metal ranking and request-model pinning, manual backend simulation, Docker GPU compose overlays, Cookbook CLI state, package detection, Windows venv/path/task helpers, non-numeric GPU counts, non-string model catalog fields, and selected frontend progress regressions.
+
+Route-level auth/security and degraded-return coverage is thinner for Cookbook admin routes, shell dependency routes, `/api/cookbook/hf-latest`, state/status edge cases, HW Fit routes, frontend JS behavior, and helper scripts such as `hf_download.py`, `add_hwfit_models.py`, and `diffusion_server.py`.
+
+## Current Gaps
+
+- Cookbook-created model endpoint ownership/shared/null-owner policy needs a deliberate decision.
+- `/api/shell/exec` and Cookbook package/rebuild routes need to remain cross-referenced with shell/admin specs because they are Cookbook-critical code-execution surfaces.
+- Cookbook route auth/security and degraded-return behavior need route-level tests.
+- `/api/cookbook/hf-latest` needs tests locking its user-authenticated access policy and failure behavior.
+- HW Fit routes need route-level tests around missing catalogs, manual overrides, `fit_only`, profiles, and image-model cases.
+- Dependency install/serve diagnosis remains split across Cookbook routes, shell routes, frontend diagnosis, optional binaries, and platform-specific scripts, even though longer serve-output tails are centralized through `routes/cookbook_output.py`.
+- Model catalog, quantization, backend, and Hugging Face metadata drift need ongoing maintenance.

specs/documents-rag-uploads.md

@@ -0,0 +1,205 @@
+# Documents, RAG, And Uploads
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+This spec covers file/document context, document storage, and vector retrieval in:
+
+- `app.py` and `src/app_initializer.py` route/manager wiring;
+- `routes/upload_routes.py`, `routes/personal_routes.py`, `routes/embedding_routes.py`, canonical `routes/document/document_routes.py` and `routes/document/document_helpers.py`, plus their top-level compatibility shims;
+- chat attachment paths in `routes/chat_routes.py`, `routes/chat_helpers.py`, `src/chat_handler.py`, and `src/chat_processor.py`;
+- `core/session_manager.py`, `src/attachment_refs.py`, `src/upload_handler.py`,
+  `src/upload_limits.py`, and the public reference contract in
+  `docs/attachments.md`;
+- `src/document_processor.py`, `src/document_actions.py`, `src/personal_docs.py`, and `src/markitdown_runtime.py`;
+- `src/rag_singleton.py`, `src/rag_vector.py`, `src/rag_manager.py`, `src/chroma_client.py`, `src/embeddings.py`, and `src/embedding_lanes.py`;
+- PDF/form helpers in `src/pdf_runtime.py`, `src/pdf_forms.py`, and `src/pdf_form_doc.py`;
+- `services/docs/service.py`;
+- document, upload, RAG, chat, email, and admin frontend callers in `static/app.js`, `static/js/chat.js`, `static/js/chatRenderer.js`, `static/js/fileHandler.js`, `static/js/document.js`, `static/js/documentLibrary.js`, `static/js/rag.js`, `static/js/admin.js`, `static/js/emailInbox.js`, and `static/js/slashCommands.js`;
+- tests covering upload, document, attachment, PDF, RAG, Chroma, MarkItDown, and embedding behavior.
+
+## Runtime Integration
+
+`app.py` registers upload, personal-doc/RAG, embedding, document, diagnostics, and Codex document routes. `src.app_initializer.initialize_managers()` creates `UploadHandler` and `PersonalDocsManager`, installs the upload handler on `SessionManager` and the shared tool helper, and startup attempts to initialize the RAG singleton. App route wiring passes that same handler to session/history, document, note, and calendar writers that can persist upload references.
+
+`src.rag_singleton.get_rag_manager()` returns the live `VectorRAG` instance when Chroma/embedding dependencies are reachable. Personal routes can retry the singleton and return explicit 503s when unavailable. Chat RAG uses the `PersonalDocsManager.rag_manager` captured during app initialization and can silently skip RAG if that manager is absent.
+
+## Uploads And Attachments
+
+`src.upload_handler.UploadHandler` owns upload IDs, safe filenames, upload metadata, owner rename rewrites, atomic `uploads.json` writes, content-type detection, and file storage under `data/uploads`. Upload IDs accept extensionless values or one sanitized alphanumeric extension.
+
+Upload-index reads track the live and `.bak` files by device, inode, size, nanosecond mtime, and ctime, then verify the combined signature after parsing. This catches same-timestamp corruption/replacement and prevents stale parsed data from being cached under a newer file identity. Non-destructive reads can recover from the backup; destructive cleanup requires a valid live index and never treats an older backup as deletion authority. Lifecycle writes can synchronize the backup so intentionally removed metadata is not resurrected.
+
+`src.upload_limits` owns central upload-size caps and environment overrides for chat attachments, gallery, transforms, memory import, personal uploads, email compose, STT audio, and ICS imports. Invalid configured limits fail fast at import so routes do not silently accept unsafe sizes. Docker installs `libmagic1` plus `python-magic` so `UploadHandler.detect_content_type()` can sniff bytes in the official image; native installs can fall back to extension/MIME guesses when `python-magic` is unavailable.
+
+`routes/upload_routes.py` owns:
+
+- `POST /api/upload`, returning uploaded file metadata;
+- reference-aware admin upload cleanup and stats;
+- `GET /api/upload/{file_id}`;
+- `GET/PUT /api/upload/{file_id}/vision` for editable OCR/vision cache;
+- thumbnail and masked owner/admin access behavior.
+
+It does not currently expose a general upload list/delete route. Download/preview responses that serve uploaded content should include `X-Content-Type-Options: nosniff` where route code owns the response so browser MIME sniffing does not widen accepted upload types.
+
+Readable/code-like upload handling includes common text/code extensions plus `.nix`; document processing renders recognized code-like text into fenced blocks with language metadata.
+
+Chat does not own attachment extraction. Runtime flow:
+
+- the frontend uploads files and submits attachment IDs;
+- `ChatHandler.preprocess_message()` resolves IDs with the session owner through `UploadHandler.resolve_upload()`, which enforces owner/admin access and no longer treats missing owner context as permission to read owned uploads;
+- vision/OCR cache and attachment metadata are prepared before model calls;
+- text-only models receive stripped multimodal blocks;
+- `src.document_processor.build_user_content()` produces model-ready text, PDF text, Office/EPUB text when MarkItDown or the DOCX fallback is available, image/multimodal blocks, truncation, and PDF/Office auto-document updates;
+- chat streams attachment, PDF-created `doc_update`, and `rag_sources` events where applicable.
+
+Extensionless image and audio attachments derive their data-URI subtype from
+the detected MIME type, so `image/png` and `audio/mpeg` uploads do not become
+invalid `data:image/;base64` or `data:audio/;base64` blocks when the filename
+has no extension.
+
+## Durable References And Cleanup
+
+`src.attachment_refs` owns the stable `attachment_ref` shape used outside raw
+upload storage: attachment id, name, MIME type, size, and optional checksum,
+creation time, dimensions, vision text/model, and gallery id. Live provider
+calls may still receive multimodal data URLs for the current turn, but durable
+chat content is normalized to readable text plus compact reference lines.
+Structured references remain in message attachment metadata, and chat FTS
+triggers omit inline media while startup migration scrubs legacy indexed data
+URLs.
+
+Agent/tool manifests expose `odysseus://attachment/<id>` with
+`read_policy: "owner_checked_upload"`. A compatibility filesystem path is
+included only after owner-aware upload resolution, upload-root confinement, and
+tool-readable-root checks; the stable contract for external tools is the URI
+and attachment id, not host layout.
+
+Writers reserve referenced uploads before committing durable state. This
+includes session message append/replace and history rewrites, document
+create/update and native document edits, note route/tool create/update,
+calendar/event route/tool create/update, and attachment-bearing session
+updates. A missing or wrong-owner reference aborts before destructive
+replacement and surfaces a route conflict or tool error. Reservations serialize
+with cleanup through the upload-index lock and refresh access time.
+
+Admin cleanup first scans chat content and attachment metadata, current and
+versioned documents including PDF markers, gallery filenames/hashes, note
+image/color/content/checklist fields, and calendar color/description/location
+fields. Reference discovery or index-integrity failure aborts cleanup; the
+lower-level API removes nothing without both completed id and hash snapshots.
+Only expired, unreferenced files with coherent id/path/owner/checksum/timestamp
+metadata are candidates. Matching index rows are persisted away before byte
+deletion and restored if deletion fails. This lock is process-local, so the
+documented race protection assumes the current single-worker deployment.
+
+## Living Documents And PDF
+
+`routes/document/document_routes.py` owns the HTTP document API: create/read/update/archive/delete, library listing, import/export, version history, tidy/AI tidy, PDF rendering/export, PDF form helpers, and email-attachment reply preparation. The top-level document route/helper modules remain compatibility aliases.
+
+`static/js/documentLibrary.js` owns local library state after archive/delete actions, including total counts and language chips. Server route truth still owns durable document state.
+
+`static/js/document.js` owns the browser document editor and markdown preview. Preview rendering applies code highlighting when highlight.js is present, renders Mermaid diagrams when the Mermaid runtime is available, refreshes after AI edits, and discards pending AI diffs before switching the active document.
+
+Document mutations also happen through agent tools, Codex document routes, email attachment import, and scripts. HTTP and native-agent document writers owner-reserve any internal upload/PDF references before persisting new current content or versions. Native document tool outputs include metadata that the browser can use to open/update the editor if a later stream update is missed. Those callers must preserve document owner, attachment, and version semantics.
+
+After external/workspace-untrusted context, a proposed document mutation is sealed into an exact approval with document id, current version, content digest, tool content, owner/session, and workspace. Approval continuation re-reads and verifies those fields before consuming the one-use authorization, so an intervening edit cannot apply a stale approved patch to new content.
+
+Email draft documents are a first-class document language. Create/update paths
+detect the `To`/`Subject`/header shape, coerce language to `email`, and preserve
+protected reply/forward headers such as `In-Reply-To`, `References`,
+`X-Source-UID`, `X-Source-Folder`, attachment headers, and quoted/original
+history when model or UI edits replace the draft body. Creating a draft for the
+same source UID/folder in the same session updates the active draft instead of
+creating a duplicate.
+
+`Document` rows own current content and owner. `DocumentVersion` rows own immutable snapshots. Document access should be owner-filtered, not session-id-only; the session document listing path still needs regression coverage for per-document owner filtering after the session owner check.
+
+PDF runtime behavior:
+
+- direct PDF import stores the upload through `UploadHandler`;
+- PDF library entries preserve metadata/preview behavior for source PDFs;
+- pypdf text extraction remains core;
+- PyMuPDF enables form detection, page rendering, page PNGs, annotation fill, render/export PDF, and form filling;
+- PDF render routes should return a shaped 503 when PyMuPDF is absent and use same-origin framing/download behavior for rendered pages;
+- imported PDFs become either plain `pdf_source` markdown or `pdf_form_source` markdown with sidecar field data;
+- PDF markers must resolve back through an upload owned by the caller;
+- signed-reply preparation uses document `source_email_*` provenance and verifies the document owner and signature owner. Source email account resolution still needs explicit owner-scoped coverage.
+
+Office/EPUB attachment extraction is optional and MarkItDown-backed for `.docx`, `.pptx`, `.xlsx`, `.xls`, and `.epub`; a pure-Python DOCX fallback can extract `word/document.xml`. When a session id is present, full extraction can be saved as a markdown `Document` while the chat-inline copy remains capped.
+
+## Personal Docs And RAG
+
+`src.personal_docs.PersonalDocsManager` owns personal-directory indexing and keyword retrieval.
+
+`src.rag_vector.VectorRAG` owns Chroma/embedding-backed indexing and owner-filtered retrieval. Chunk ids are owner-scoped so byte-identical chunks from different owners do not suppress each other. `src.rag_singleton` owns lazy initialization, retry throttling, and reset behavior.
+
+`routes/personal_routes.py` owns personal-doc and direct RAG-upload routes. Directory list/index/delete routes are admin-gated, and directory indexing runs in a worker thread so traversal/extraction does not block the async event loop. Direct RAG upload is user-authenticated, requires document privilege, forwards owner into the manager wrapper, writes unique files under per-owner subdirectories of `data/personal_uploads`, and has looser file-type validation than normal uploads.
+
+Current call sites include:
+
+- admin RAG pages and slash commands;
+- chat RAG preface building;
+- AI interaction and MCP RAG management tools;
+- CLI scripts for document/personal indexing.
+
+Some non-route tool/script paths can index ownerless or arbitrary directories and should be treated as compatibility-sensitive management surfaces.
+
+## Embedding Models
+
+`routes/embedding_routes.py` owns admin-gated embedding model and custom endpoint management. It validates custom endpoints with outbound URL checks, can persist and process-expose `EMBEDDING_API_KEY`, resets embedding/RAG/tool-index/Chroma state, and does not own document extraction.
+
+`src.embeddings` owns HTTP embedding fallback to FastEmbed and process-level endpoint state. `src.embedding_lanes` keeps custom HTTP embedding vectors separate from FastEmbed fallback vectors with lane-specific Chroma collections, migrates legacy unsuffixed collections into empty lanes, and dedupes query results across lanes. `src.chroma_client` owns native Chroma defaults and fast reachability checks.
+
+## Compatibility State
+
+`src.rag_manager.RAGManager` is a backward-compat wrapper. The live owner-aware vector path is `VectorRAG`.
+
+`services/docs/service.py` is a separate facade. It accepts live `VectorRAG` query rows (`document`, `similarity`, nested metadata source), retains legacy `text`/`content` and `score` fallbacks, skips non-object rows, and maps live `indexed_count`/`failed_count` plus legacy `indexed`/`failed` index summaries into its dataclasses.
+
+`src.database` re-exports `core.database`; document models and migrations live in `core.database`.
+
+## Optional And Degraded Behavior
+
+- ChromaDB/FastEmbed are default installed dependencies, but Chroma can be offline or unreachable.
+- Native Chroma defaults to `localhost:8100`; Docker uses the `chromadb:8000` compose service and persistent Chroma storage.
+- HTTP embeddings can fall back to FastEmbed; when both lanes exist, lane separation avoids Chroma dimension conflicts.
+- MarkItDown is optional for Office/EPUB extraction; chat attachments and personal directory indexing have clear degraded behavior, while direct RAG upload does not share the same extraction path.
+- PyMuPDF is optional, unlocks PDF form/render/fill paths, and carries AGPL implications when installed.
+- PyMuPDF-dependent document routes should use the shared runtime helper/error text so missing-dependency and license policy stay visible.
+- pypdf text extraction is core and should remain available without PyMuPDF.
+
+## Security And Provenance
+
+Uploaded files, documents, RAG chunks, extracted attachment text, OCR/vision text, PDF marker content, and source-email metadata are untrusted external or user-provided context when sent to an LLM.
+
+Concrete enforcement points include:
+
+- `UploadHandler.resolve_upload()` for upload ID validation, owner/admin access, and upload-dir confinement;
+- owner-checked write reservations before durable attachment references are
+  stored, sharing the upload-index lock with reference-aware cleanup;
+- PDF marker ownership checks before resolving source uploads;
+- personal-directory and personal-upload confinement helpers, including symlink/realpath checks before deleting uploaded files or removing indexed directories;
+- owner-filtered `VectorRAG.search(owner=...)`;
+- shared untrusted-context wrappers for RAG preface insertion.
+
+Extracted attachment text is currently appended into the user message rather than wrapped as a separate untrusted-context message. That is current behavior and a prompt-injection hardening gap.
+
+Bearer-token callers are not a scoped document/upload API surface today. Routes that treat token-authenticated users as owners need explicit scope/effective-user policy before they are considered safe token APIs.
+
+## Testing Coverage
+
+Existing useful coverage includes upload owner scope, upload IDs, upload atomicity, durable attachment reference normalization, message/document/note/calendar write reservations, fail-closed reference-aware cleanup, attachment budgets, `.nix` text upload handling, upload/PDF security regressions, Docker `libmagic`/`python-magic` upload detection, RAG owner fallback, Chroma fast-fail, MarkItDown runtime, PDF runtime, document-library counter updates, and selected document helper behavior.
+
+Route-level coverage is thinner for document CRUD, PDF import/render/export/fill, direct RAG upload, embedding admin/security behavior, and RAG unavailable states.
+
+## Current Gaps
+
+- Direct RAG upload still needs clearer file-type validation and MarkItDown/PDF extraction parity decisions.
+- Document `session_id` relinking and session document listing need owner-scope regressions.
+- Chat RAG can remain degraded after startup even if personal routes later initialize the RAG singleton.
+- PyMuPDF-dependent routes do not all share the same optional-runtime helper/error behavior.
+- Signed-reply preparation needs owner-scoped source email account/signature regression coverage.
+- Document/upload routes need explicit bearer-token scope/effective-user policy.
+- User-facing document/PDF/RAG route matrices need more regression coverage for owner denial, admin gates, unavailable services, and degraded optional dependencies.

specs/email-contacts.md

@@ -0,0 +1,209 @@
+# Email And Contacts
+
+Last updated: dev@2e2bb52 | 2026-08-16
+
+## Scope
+
+This spec covers mail and contacts in:
+
+- app wiring in `app.py`;
+- `core.database.EmailAccount`;
+- `routes/email_routes.py`, `routes/email_helpers.py`, and `routes/email_pollers.py`;
+- email threading in `src/email_thread_parser.py`;
+- email MCP tools in `mcp_servers/email_server.py`;
+- canonical contact/CardDAV routes in `routes/contacts/contacts_routes.py`,
+  with `routes/contacts_routes.py` as a compatibility shim;
+- Codex email bridge in `routes/codex_routes.py`;
+- document signed-reply flows in canonical `routes/document/document_routes.py` and document `source_email_*` fields;
+- reminder/task email senders in `routes/note_routes.py` and `src/task_scheduler.py`;
+- email/contact agent surfaces in `src/tool_implementations.py`, `src/tool_schemas.py`, `src/tool_index.py`, and `src/agent_loop.py`;
+- CLI wrappers `scripts/odysseus-mail` and `scripts/odysseus-contacts`;
+- frontend modules `static/js/emailInbox.js`, `static/js/emailLibrary.js`, `static/js/emailLibrary/*`, `static/js/emailShared.js`, `static/js/chatStream.js`, `static/js/document.js`, and `static/js/settings.js`;
+- tests under `tests/test_email_*`, `tests/test_contacts_*`, `tests/test_mail_cli_*`, `tests/test_mcp_email_*`, `tests/test_schedule_email_*`, email/contact JS tests, and email security regressions.
+
+## Current Call Sites Include
+
+- browser email inbox/library, compose, schedule, account, and attachment actions;
+- document-editor compose, recipient autocomplete, compose uploads, and signed-reply handoff;
+- Codex email read/draft/send routes using API-token scopes;
+- note reminder and task-output email delivery;
+- built-in email summary/reply/calendar/urgency actions;
+- scheduled email pollers and CLI one-shot pollers;
+- MCP email tools;
+- contact manager settings, compose contact autocomplete, agent contact tools, and contacts CLI.
+
+## Email Accounts And Transport
+
+`EmailAccount` rows own IMAP/SMTP configuration. Password fields are string columns containing encrypted ciphertext written with `src.secret_storage`; startup migrations handle legacy plaintext rows. Google OAuth account rows also carry `oauth_provider`, encrypted access/refresh tokens, token expiry, and an optional outbound `display_name`. Do not return decrypted credentials or OAuth tokens, or write them to logs.
+
+Exactly one default account per owner is enforced as a serialized database transition. Startup normalizes legacy duplicate defaults and installs a unique per-owner default constraint/index; first create, delete/promotion, set-default, demo teardown, and owner rename lock the relevant owner rows and commit atomically. Multi-owner rename acquires locks in canonical order so stale concurrent writers fail closed.
+
+`routes.email_helpers` owns:
+
+- account owner assertions and config fallback order;
+- IMAP/SMTP connection helpers and related transport utilities;
+- Google OAuth2 state signing/verification, token refresh, and XOAUTH2 framing;
+- SMTP security modes (`ssl`, `starttls`, `none`);
+- envelope recipients and Odysseus headers;
+- attachment extraction helpers;
+- email pre-retrieval context for AI reply drafting;
+- scheduled email, summary, reply, tag, calendar extraction, urgency, and signature-boundary side databases.
+
+Email config can fall back to legacy `data/settings.json` or environment variables when no scoped account is configured. Account discovery now owner-scopes the default/first-enabled fallback and can still match legacy account rows by IMAP username or from-address. That fallback remains compatibility-sensitive in multi-user contexts.
+
+Email owner semantics are route-local and compatibility-sensitive:
+
+- `routes.email_helpers._require_auth()` returns `""` in `AUTH_ENABLED=false` mode, rejects configured auth with no user, and only tolerates first-run anonymous loopback fallback.
+- Empty owner is treated as single-user compatibility: account-ownership assertions no-op, default/first-enabled account fallback can be global, and email cache clauses include `owner = '' OR owner IS NULL`.
+- Non-empty owners scope account/config/cache queries. Legacy ownerless account
+  rows are visible to an authenticated owner only when the row's IMAP username
+  or from-address matches that owner, so old unowned rows do not become global
+  cross-user accounts in configured multi-user deployments.
+
+`routes.email_routes` owns the HTTP mail surface:
+
+- account CRUD, test, default, and masked config reads;
+- Google OAuth authorize/callback for Workspace and .edu Gmail-style accounts;
+- list, search, read, folders, and contacts;
+- folder role resolution and UID fetch/search helpers used by the route surface;
+- owner-scoped route caches and IMAP pool behavior;
+- attachments, bulk attachment ZIP downloads, and attachment-to-document flows;
+- compose upload, draft/send, `wait_for_delivery`, Sent append, and source `\Answered` marking;
+- schedule/list/delete scheduled emails;
+- pending agent-draft approval/cancel flows;
+- mark read/unread/answered, spam flags, move, archive, and delete. IMAP move/delete/archive operations use UID commands for message identity and fail safe when the requested UID no longer exists; they never reinterpret a missing UID as a sequence number, which could mutate or expunge an unrelated message.
+
+Google OAuth behavior is account-owned:
+
+- `/api/email/oauth/google/authorize` requires an authenticated owner, checks account ownership, HMAC-signs state with account id, owner, and nonce, and redirects to Google with mail/userinfo scopes;
+- `/api/email/oauth/google/callback` verifies signed state before token exchange, re-checks the target account owner before writing tokens, stores access/refresh tokens encrypted, stores token expiry as a timestamp, and redirects with generic success/error codes rather than raw provider errors;
+- token refresh uses `GOOGLE_OAUTH_CLIENT_ID` and `GOOGLE_OAUTH_CLIENT_SECRET`, stores refreshed access tokens encrypted, and logs only generic/account-id context on failures;
+- SMTP and IMAP use XOAUTH2 when `oauth_provider == "google"`; OAuth accounts are send-capable without an SMTP password when host and user are configured;
+- outbound mail formats the `From` header with `display_name` when present.
+- authorize/callback redirect URIs derive their scheme and host from the mounted request unless `GOOGLE_OAUTH_REDIRECT_URI` explicitly pins a value; the browser preserves the selected SMTP security mode during connect and reopens Settings after the callback.
+
+MCP full-message read/reply/attachment fetches use IMAP `BODY.PEEK[]` rather than bare `RFC822`, so iCloud-style servers return the full body without marking messages seen. Poller UID handling must tolerate both bytes and string UIDs. Built-in signature-learning and daily-brief actions also use UID SEARCH/FETCH rather than sequence-number commands.
+
+IMAP helpers quote mailbox names, raise the Python IMAP line cap for large messages, close sockets after connect/login failures, and preserve Gmail FETCH attributes that follow header literals so unread flag state is not lost. Browser list routes offload blocking IMAP work from async handlers; browser search runs in FastAPI's threadpool, rejects CRLF query input, tokenizes quoted phrases/terms, searches FROM/TO/CC/SUBJECT/TEXT, can search Gmail All Mail when an INBOX query should include archived or labelled messages, and supports `scope=folder` when callers intentionally want the selected folder only. The local index fallback can return indexed results when IMAP returns empty or fails.
+
+## Runtime And Pollers
+
+Scheduled email rows live in `data/scheduled_emails.db` and are owner-scoped. Scheduled send times are normalized before storage.
+
+`routes.email_pollers` owns the scheduled-send poller and single-shot/task/CLI automation passes. Before SMTP work, each poller atomically claims a due row with a conditional `pending` to `sending` update; concurrent in-process/CLI pollers that lose the claim skip the row instead of sending a duplicate. Only the scheduled-send poller starts in-process by default when `ODYSSEUS_INPROCESS_POLLERS` allows it; Docker forwards that gate. Background email automation can also consult the foreground activity gate so auto actions do not compete with active browser/model work. Native cron/systemd can drive one-shot pollers through `scripts/odysseus-mail`.
+
+Manual and scheduled summaries use the shared LLM adapter and owner-scoped cache instead of constructing provider calls locally. Scheduled summaries use background fallback policy and yield to foreground work; provider exception text is shaped before it can reach the browser.
+
+Urgency delivery publishes through a serialized atomic checkpoint transaction. Generation and membership fences prevent stale scans from overwriting newer state; authoritative scans retire deleted/disabled accounts, partial failures preserve the prior checkpoint, concurrent account-scoped actions merge disjoint facts, and cancellation rolls back without publishing.
+
+Transport degraded behavior:
+
+- IMAP timeouts are clamped by configuration;
+- providers can use implicit SSL, STARTTLS, or plain connections;
+- poisoned IMAP sockets are reconnected around known provider failures;
+- SMTP-capable account fallback is used where supported;
+- route helpers, MCP, and CLI do not all share identical SMTP/IMAP parsing and security behavior today.
+
+## Caching And Staleness
+
+Email list/read behavior uses short route caches, longer read caches, capped warm prefetch, and owner/account-aware pool/cache keys. The frontend email library has its own session SWR cache, cache-buster refreshes, scheduled/search cache exclusions, and stale-row behavior when refresh fails.
+
+Opening an unread message is one authoritative backend IMAP operation. The read route fetches/parses the message and applies `\Seen` over the same connection; cached bodies still await one UID STORE, read-only mailboxes serve content without claiming a mark, and STORE failure returns the body with explicit failure state rather than caching a false read. Inbox/library clients deduplicate opens, carry immutable mailbox context, and ignore late responses after account, folder, or message changes.
+
+Library prewarm runs only while genuinely idle, as one bounded single-flight request for the default or last-used enabled account and initial page. Visible foreground work, panel lifecycle, account changes, or explicit reads cancel or join it so delayed duplicate IMAP work cannot escape the idle gate.
+
+List/read route caches are owner/account-aware. Helper-side summary, AI-reply, tag, calendar-extraction, urgency-alert, and learned sender-signature tables carry owner columns and owner clauses. Thread-boundary rows are still keyed by message shape rather than a full owner/account/mailbox key, so they remain cross-owner audit points when identical messages appear in multiple mailboxes.
+
+## Attachments And Signed Replies
+
+Compose uploads live under `ODYSSEUS_MAIL_ATTACHMENTS_DIR`; missing staged files are skipped with warnings. Attachment-to-document supports PDF, DOCX, TXT, and MD. DOCX depends on `python-docx`; PDF form/open-in-doc flows can depend on optional PyMuPDF.
+
+Email attachment-as-document flows stamp `Document.source_email_*` provenance. `GET /api/email/attachments-download/{uid}` builds an owner-scoped ZIP of visible non-signature attachments using safe names. `compose-from-odysseus` and `compose-from-odysseus-zip` can stage owner-visible documents and gallery images as compose uploads, preserving legacy session fallback only where the source object remains visible to the owner. `prepare-signed-reply` verifies document ownership, reconstructs reply headers, flattens/stages signed PDFs as compose uploads, and leaves final send/draft review to the compose flow.
+
+Email bodies and attachments are untrusted model context.
+
+## Threading And Rendering
+
+`src.email_thread_parser` owns splitting plaintext/HTML email threads into quoted conversation parts. Frontend email library modules own reply-recipient logic, signature folding, local state, and rendering behavior. Bulk selections are cleared when folder/account loads, search text, search pills, or result scope changes so actions cannot carry stale UIDs into a different visible context. `static/js/emailShared.js` owns shared email UI helpers used across inbox/library surfaces.
+
+Remote inbound email HTML is sanitized by frontend email-library utilities before `innerHTML` insertion. Server-side email routes sanitize composed/generated outbound HTML with an allowlist before draft/send, dropping scripts/styles and unsafe attributes. Both sides are part of the rendering invariant.
+
+When the email reader is active, browser chat sends selected-message metadata. `src.tool_implementations` stores that request-local active email reference, `src.agent_loop` injects it as protected untrusted context, and `static/js/chatStream.js` handles `ui_control open_email_reply` so default reply/draft behavior opens the selected message's compose flow instead of a generic new document.
+
+## MCP Email
+
+`mcp_servers/email_server.py` exposes email tools for MCP/agent use. It has its own account discovery, IMAP/SMTP, attachment, cache, and send paths, but account visibility now mirrors the HTTP owner policy. The active owner comes from a hidden `_odysseus_owner` argument when the caller provides one, or from `ODYSSEUS_MCP_EMAIL_OWNER` / `ODYSSEUS_EMAIL_OWNER`. If any enabled account is owner-scoped and no current/configured owner exists, email MCP returns an owner-scope error instead of listing global accounts.
+
+MCP email account filtering includes owner-owned rows and legacy ownerless rows
+whose mailbox/from-address matches the owner. Confirmation-first `send_email`
+resolves the selected account before stashing an `agent_draft`, so drafts cannot
+be staged against another owner's account. MCP-created draft documents use the
+resolved hidden/configured owner when available, with `ODYSSEUS_DOCUMENT_OWNER`
+and single-admin fallback only as document-visibility compatibility.
+
+MCP email send behavior is confirmation-first by default: `send_email` and reply send paths stash a `scheduled_emails` row with `status='agent_draft'` when `agent_email_confirm` is true, and browser routes expose pending drafts for approval or cancellation. Separate MCP draft tools create Odysseus compose documents for user review without sending.
+
+MCP email remains a separate local/admin trust boundary. Public and non-admin users must not see or execute email MCP tools. It still needs route-helper parity audits for attachment path containment, sanitization, transport behavior, and pending-draft result text, but global all-account behavior is no longer the current owner model.
+
+## Contacts
+
+`routes.contacts.contacts_routes` owns global/admin contacts and CardDAV behavior. The top-level `routes.contacts_routes` module is a compatibility shim. The canonical package supports local contacts, CardDAV config, list/search/add/update/delete, VCF/CSV import/export, and clear.
+
+Contact runtime behavior:
+
+- contacts routes are admin-gated;
+- local `data/contacts.json` is used when CardDAV is unconfigured;
+- import paths tolerate malformed or non-string contact bodies by skipping invalid rows instead of crashing the import;
+- configured CardDAV uses REPORT with GET fallback and a short in-memory cache;
+- configured-but-offline CardDAV can return cached reads but writes fail instead of falling back to local JSON;
+- CardDAV config reads mask the password, settings-stored passwords are encrypted with `src.secret_storage`, omitted password updates preserve the existing secret, and an explicit empty password clears it;
+- the native contacts CLI is CardDAV-oriented and does not fully match web JSON fallback behavior;
+- agent contact tools reuse helper functions in-process because the HTTP routes require browser/admin auth.
+
+Contacts are global admin-only data today. There is no per-user contact sharing model unless a future spec defines one.
+
+## Security Policy
+
+Email HTTP access is owner-scoped, including account selection, scheduled email rows, and attachment routes. Null-owner/single-user compatibility paths are security-sensitive and must not allow cross-user mailbox access.
+
+Codex email routes are the scoped bearer-token email API. They enforce `email:read`, `email:draft`, and `email:send` scopes and use token-owner attribution before borrowing email route handlers.
+
+Known security policy details:
+
+- decrypted email credentials stay process-local;
+- account/config reads mask passwords and expose only OAuth status fields, not access or refresh token values;
+- SMTP/IMAP security mode behavior is part of the credential contract;
+- Google OAuth state and callback owner checks are part of the account-boundary contract;
+- scheduled emails must remain owner-scoped;
+- email pre-retrieval contacts context is allowed only for admin/single-user situations;
+- MCP attachment downloads need route-level path-containment parity; current MCP paths are separate from the HTTP compose/attachment helper path.
+
+CardDAV credentials and URLs are security-sensitive. CardDAV URL setup and derived href writes/deletes pass through outbound URL validation; absolute hrefs from a CardDAV server are constrained back to the configured origin before credentials are reused. CardDAV passwords in settings are encrypted and masked on read; environment-sourced legacy password values are used as supplied.
+
+## Degraded Behavior
+
+- IMAP/SMTP providers can be slow or inconsistent; folder resolution, pooled connections, and reconnect behavior should fail with clear errors.
+- Google OAuth requires external Google endpoints plus configured `GOOGLE_OAUTH_CLIENT_ID`/`GOOGLE_OAUTH_CLIENT_SECRET`; missing client credentials or refresh failures degrade to reconnect-required or generic OAuth error paths.
+- Scheduled email delivery depends on `scheduled_emails.db`, poller runtime, and configured SMTP.
+- Attachment handling must tolerate missing staged files, unsupported formats, and inaccessible remote messages.
+- CardDAV local fallback applies only when CardDAV is unconfigured; configured CardDAV outages are not treated as local-write mode.
+- Multi-account list/search behavior can be sequential and cache-sensitive.
+
+## Testing Coverage
+
+Existing coverage includes header/envelope/IMAP/SMTP behavior, serialized default accounts, Google OAuth state/callback/token-refresh/XOAUTH2/redirect/settings behavior, shared-adapter summaries, authoritative read/mark-seen and frontend dedup, idle prewarm, UID-only mutations, scheduled-email claims and urgency checkpoint transactions, MCP full-message/owner behavior, owner scope/caches/signatures, thread/sanitizer behavior, CardDAV password encryption, mail CLI behavior, contacts basics, and selected frontend/security regressions.
+
+Route-level and duplicate-path coverage is still thin for email list/read/search/mutations, account CRUD/security outside the OAuth path, send/draft security, attachments, scheduled-poller failures, contacts admin/CardDAV routes, MCP account/scope behavior, CardDAV degraded mode, and executable frontend behavior.
+
+## Current Gaps
+
+- Owner-keyed cache policy still needs an explicit decision for thread boundaries, plus continued migration/query audits for every email side table.
+- CardDAV still needs redirect/proxy policy and broader route-level tests for URL validation, private-address blocking configuration, and same-origin href enforcement.
+- MCP email needs continued route-helper parity for attachment path containment,
+  sanitization, transport behavior, and pending-draft result text.
+- Empty-owner route compatibility and ownerless email cache rows need
+  end-to-end owner-boundary tests.
+- CLI send/contact paths need parity decisions for SMTP security, recipient parsing, local fallback, and normalized contact shapes.
+- Email HTTP route coverage is concentrated in scheduling/account-test helpers rather than full list/read/search/mutation/send/draft/account/attachment flows.
+- Contacts coverage lacks admin-gate, config masking, import/export, CardDAV fallback, and CardDAV write-failure tests.
+- Multi-account performance and cache staleness remain known audit areas.

specs/frontend.md

@@ -0,0 +1,158 @@
+# Frontend
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+This spec covers the current browser app in:
+
+- static serving and SPA routes in `app.py`;
+- CSP/security headers in `core/middleware.py`;
+- `static/index.html`;
+- `static/login.html`;
+- `static/app.js`;
+- `static/style.css`;
+- `static/js/*.js` and `static/js/*/*.js`;
+- vendor libraries under `static/lib/*`;
+- custom fonts and static assets under `static/fonts/*`;
+- `static/sw.js` and `static/manifest.json`;
+- frontend-oriented tests in `tests/*_js.py`, `tests/*.mjs`, `tests/bombadil-spec.ts`, static DOM/CSS/source-shape tests, and app/static tests such as `tests/test_app_static_mime.py`.
+
+`/backgrounds` currently targets `static/backgrounds.html`; if that route remains, the file must exist or the route should be removed.
+
+`static/manifest.json` and `static/index.html` reference PWA icon files under `static/icons/`; the current 192px, 512px, and maskable icon files exist and should stay aligned with those references.
+
+## Current Call Sites Include
+
+- `static/index.html` script tags and modulepreloads;
+- `static/sw.js` `PRECACHE`;
+- app-owned SPA deep links for notes, calendar, cookbook, email, memory, gallery, tasks, and library;
+- `/login` and app-owned static/HTML routes;
+- `/api/activity/heartbeat` browser visibility pings used by the foreground activity gate;
+- `static/app.js` route opener/sidebar/tool-window wiring;
+- frontend JS helper tests and static HTML/CSS/source-shape regressions;
+- CDN dependencies, local vendor libraries, service worker, and PWA manifest.
+
+## Runtime Shape
+
+The frontend is a raw static SPA served by FastAPI. There is no Vite, React, TypeScript, bundler, or generated build output.
+
+`app.py` owns:
+
+- stable `.js`/`.mjs` MIME registration;
+- the `/static` mount;
+- no-cache headers for `.js`, `.css`, and `.html` static source files;
+- nonce-injected SPA/login HTML serving;
+- SPA deep-link routes.
+
+`static/index.html` owns the DOM shell and script loading order. It loads browser ES modules directly. Current boot order includes nonce-bearing inline boot scripts, self-hosted highlight.js, modulepreloads, ordered module script tags, `static/app.js`, `static/js/init.js`, `static/js/a11y.js`, workspace/chat helpers, provider device-flow helpers, and service-worker registration. KaTeX and Mermaid are vendored under `static/lib` and injected only on first real math/diagram use rather than loading in the initial HTML.
+
+The two first-paint Fira Code faces are preloaded so the shell does not wait for later CSS discovery. `static/js/startupShell.js` lets the visible shell initialize before session loading completes; session/transcript hydration is deferred and coordinated by `static/js/sessions.js` plus history/session routes rather than blocking first paint.
+
+Exact script URL identity matters. Versioned script tags, unversioned imports, and service-worker precache entries must stay aligned. `static/sw.js` deliberately separates first-paint `PRECACHE` from lazy `PANEL_PRECACHE`; the latter currently contains the image-editor module graph so an editor never opened online can still open offline. KaTeX scripts/styles/fonts are also precached. Current service-worker coverage is not a generated full module-graph manifest, so changes still need direct verification.
+
+## Security Policy
+
+`core/middleware.py` owns CSP and security headers. `app.py` injects the per-request nonce into served HTML. New inline scripts or external scripts/styles/images/media must fit the CSP contract or explicitly update it.
+
+`/static/*` is public/auth-exempt. Frontend privilege gates are display-only; backend routes enforce authorization.
+
+XSS/DOM policy:
+
+- prefer DOM construction, `textContent`, and shared escaping helpers;
+- Markdown raw HTML preservation must remain constrained through sanitizer helpers;
+- remote email `body_html` must pass through the email-library sanitizer before insertion;
+- Mermaid, code-runner iframe `srcdoc`, visual reports, remote media, and scattered `innerHTML` templates require explicit review.
+- Visual report Markdown HTML is server-rendered and should be treated as security-sensitive alongside frontend entry points and remote media.
+
+Storage/secrets policy:
+
+- localStorage/sessionStorage are for preferences, UI state, offline caches, and user-switch sentinels;
+- `static/js/init.js` owns user-switch storage cleanup;
+- raw API tokens, provider keys, HF tokens, and other credentials must not be persisted in browser storage unless a feature documents masking/stripping and backend storage ownership.
+
+## Service Worker And PWA
+
+`static/sw.js` owns PWA cache behavior:
+
+- API and non-GET requests are bypassed;
+- root navigation uses stale-while-revalidate;
+- JS/CSS use network-first behavior;
+- other static assets use cache-first with background refresh;
+- `CACHE_NAME` bumps and `PRECACHE` updates must accompany cache policy or shell asset changes.
+
+`static/manifest.json` owns default PWA metadata. Route-specific manifests can be generated as Blob URLs when supported. Current default icon references must match real files under `static/icons/`.
+
+KaTeX and Mermaid are self-hosted and lazy-loaded through memoized, retry-after-failure promises in `static/js/markdown.js`; math placeholders preserve source until KaTeX arrives, detached PDF export renders its own container, and Mermaid fetches only when a diagram exists. Pyodide remains a jsDelivr-loaded optional runtime, so offline/PWA behavior is not fully self-contained.
+
+## Module Ownership
+
+Current major frontend areas include:
+
+- chat, stream handling, rendering, sessions, markdown, uploads, voice recorder, TTS, and keyboard shortcuts;
+- models, provider setup, pure model-key matching helpers, model picker, presets, search, RAG, settings, and admin;
+- settings shell modules under `static/js/settings/`: registry metadata, navigation, finder search, lifecycle/docking, DOM helpers, and persisted sidebar collapse/resize behavior;
+- compare modules under `static/js/compare/`, including sanitized popup/search/image handling;
+- document editor/library in `static/js/document.js` and `static/js/documentLibrary.js`;
+- image editor integration in `static/js/galleryEditor.js` plus leaves under `static/js/editor/`;
+- gallery, email inbox/library, calendar, research panel/jobs/synapse, notes/tasks, assistant, memory/skills, Cookbook/HW Fit, workspace picker, provider device flow, composer ArrowUp recall, theme, modal/window utilities, storage, and accessibility helpers.
+
+Coordinator ownership:
+
+- `static/app.js` owns late orchestration, global fetch 401 redirects, sidebar/tool route wiring, and many `window.*` compatibility bridges;
+- `static/js/init.js` owns post-load cleanup, user-switch storage wipe, and cosmetic privilege gates;
+- `static/js/storage.js` owns shared key constants and safe JSON helpers;
+- feature modules own feature state where possible.
+
+`static/js/appConfig.js` owns one invalidatable promise cache for `GET /api/auth/settings` and `GET /api/tools`, including one-shot login-page settings prefetch, retry after rejected fetches, and explicit invalidation after settings/tool writes. Consumers treat resolved objects as read-only. `static/js/panels.js` owns memoized first-use panel imports; its current registry contains the image editor, shares in-flight imports, and evicts failed imports so a later online retry can succeed.
+
+`static/js/MODULE_SUMMARY.md` is a refreshed ownership/navigation map for the no-build frontend. The current `static/js/` tree, `static/app.js`, `static/index.html`, and executable behavior remain the authority when the summary drifts.
+
+Current small frontend helper contracts include `static/js/model/matchKey.js` for longest-substring model info/pricing matches, `static/js/models.js` for in-flight `/api/models` request sharing, `static/js/providerDeviceFlow.js` for Copilot/ChatGPT Subscription device-flow polling UI, `static/js/composerArrowUpRecall.js` for prompt recall from an empty composer, `static/js/fileHandler.js` for capped pending-file state and collapsed attachment-chip display, `static/js/streamingSegmenter.js` for incremental markdown/code-fence segmentation, `static/js/emojiShortcodes.js` for shortcode replacement, `static/js/documentLibrary.js` for keeping document counters/language chips in sync after archive/delete, `static/js/keyboard-shortcuts.js` for rejecting empty or non-string persisted keybinds before combo parsing, `static/js/modalSnap.js` for reusable desktop modal edge docking, `static/js/toolWindowZOrder.js` for shared portal/window z-index allocation, and `static/js/emailShared.js` for common email UI helpers.
+
+Recent browser behavior contracts include mobile chat Enter inserting newlines while desktop Enter submits; ArrowUp recall only consuming a truly empty composer with the caret at the top, not an unsent multiline prompt; queued prompts preserving mobile behavior; regenerate-from-here versus resend; AI-message delete confirmation; native document tool results opening/updating the editor; and exact tool-approval cards that expose the sealed action/effects/workspace/document identity and submit only opaque task-scope/chat-session-scope/deny decisions without writing synthetic composer text. Chat rendering hides leaked tool JSON/document fences, no longer strips the ordinary word “assistant,” and batches live-thinking DOM updates with bounded timers. Markdown editing/restoration preserves extracted code/math blocks verbatim, including replacement-string `$&` and `$$` text and triple-backtick fences. Session URL hashes are restored, minimized sidebar icon state follows per-tab visibility, detached terminal dots remain centered, and spinner animation starts only when attached.
+
+The Settings finder and navigation are registry-backed, hide admin-only destinations from non-admin users, lazy-load admin panels, and keep the registry synchronized with DOM panels. Email OAuth connect preserves SMTP security and reopens the settings surface; unread message opens use one authoritative backend read/mark-seen request with stale-response guards; email-library prewarm is idle-only, single-flight, bounded to the initial page, and cancelled around visible foreground work.
+
+## UI Policy
+
+- New code must run as browser ES modules without a build step.
+- Reuse existing CSS variables, modal/window patterns, icon style, storage helpers, and route conventions.
+- Custom font handling includes bundled OpenDyslexic assets plus user-supplied fonts exposed through `/api/fonts/custom`; font and text-size settings must stay coordinated between settings UI, theme helpers, and CSS variables.
+- Avoid relying on stale module summaries.
+- API shape changes must update the owning JS module and tests.
+- Add behavior to large coordinators such as `static/app.js`, `static/js/chat.js`, `static/js/document.js`, or `static/js/settings.js` only when it matches their existing wiring ownership.
+
+## Degraded And Platform Behavior
+
+- Server no-cache applies to `.js`, `.css`, and `.html` source files, not every static asset.
+- Service-worker cache changes can affect frontend behavior even when source files revalidate.
+- Mobile behavior uses separate CSS/media/hover/safe-area/`100dvh` handling and JS layout code; check it directly.
+- Browser APIs such as service workers, Blob route manifests, Web Speech, `getUserMedia`, visual viewport, and storage can be absent or restricted.
+- Local libraries and CDN globals degrade differently; document, markdown, math, diagrams, and code runner flows should handle missing globals where possible.
+- localStorage migrations and cross-user cleanup are part of compatibility.
+
+## Testing Coverage
+
+Existing frontend coverage is a mix of Node-executed helper tests, `.mjs` tests, static DOM/CSS/source-shape tests, browser exploration specs, and app/static tests. Many tests are useful source-shape regressions but do not replace browser/module-graph execution.
+
+Recent focused coverage includes model-key matching under Node, document-library counters, chat resend/delete/mobile Enter/ArrowUp, scoped approval continuation and compare routing, route provenance, live-thinking throttling, startup shell/history hydration, shared app-config caching/invalidation, settings registry/navigation/finder/lifecycle, lazy panel loading/offline editor precache, vendored lazy KaTeX/Mermaid rendering, email read dedup/prewarm, Markdown restoration, malformed keybinds, currency-safe inline math, notes/calendar/modal/manifest/admin-log behavior, Markdown XSS helpers, and CardDAV unchanged-password handling.
+
+Missing coverage includes:
+
+- SPA route/static auth and no-cache headers;
+- CSP header contents and nonce injection for `/` and `/login`;
+- service-worker API/non-GET bypass and cache strategy;
+- service-worker precache versus `index.html` script/module tags, including query strings;
+- ongoing manifest/icon reference drift;
+- module graph/load-order validation;
+- degraded vendor-library/browser API behavior, including Pyodide's remaining CDN path.
+
+## Current Gaps
+
+- `static/style.css` and large coordinators remain high-risk owners: `static/js/document.js`, `static/js/settings.js`, `static/js/chat.js`, and `static/app.js`.
+- There is no build-time type checking, module graph validation, script-order validation, or service-worker precache validation.
+- Frontend state is mostly module/global/localStorage driven, so cross-session and cross-user behavior needs explicit care.
+- `window.*` compatibility bridges remain widespread.
+- PWA/static-serving behavior may deserve a separate spec if service worker, manifests, route-specific icons, and cache policy keep growing.
+- A static asset/route manifest regression should verify files referenced by `index.html`, `manifest.json`, `sw.js`, and app-owned HTML routes actually exist.

specs/gallery-editor-media.md

@@ -0,0 +1,165 @@
+# Gallery, Editor, And Media
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+This spec covers media surfaces in:
+
+- app route registration and generated-file serving in `app.py`;
+- canonical models in `core/database.py`, with `src.database` as a compatibility import path;
+- canonical route package `routes/gallery/gallery_routes.py` and `routes/gallery/gallery_helpers.py`, with top-level `routes/gallery_routes.py` and `routes/gallery_helpers.py` compatibility shims;
+- generated-image writers in `src/ai_interaction.py` and `mcp_servers/image_gen_server.py`;
+- local MLX image compatibility server `scripts/mlx_image_server.py`;
+- image tool schemas/dispatch/implementations in `src/tool_schemas.py`, `src/tool_execution.py`, and `src/tool_implementations.py`;
+- `routes/editor_draft_routes.py`;
+- `routes/signature_routes.py` and document signature consumers in canonical `routes/document/document_routes.py`;
+- `routes/emoji_routes.py`;
+- `routes/font_routes.py`;
+- `src/generated_images.py`;
+- `src/visual_report.py` plus research image hide/unhide routes;
+- database models `GalleryImage`, `GalleryAlbum`, `EditorDraft`, and `Signature`;
+- generated files under `data/generated_images`;
+- frontend modules `static/js/gallery.js`, `static/js/galleryEditor.js`, `static/js/editor/*`, `static/js/signature.js`, `static/js/emojiPicker.js`, `static/js/chatRenderer.js`, `static/js/document.js`, `static/js/markdown.js`, and `static/js/theme.js`;
+- CLI surfaces `scripts/odysseus-gallery` and `scripts/odysseus-signature`;
+- tests covering gallery helpers/routes, generated-image serving, editor drafts, signatures, visual reports, fonts, upload limits, and image endpoint security.
+
+## Current Call Sites Include
+
+- gallery upload, library, album, tag, favorite, ZIP, delete, and saved-project views;
+- chat-generated image rendering/edit/delete bubbles;
+- agent `generate_image` and stale `edit_image` tool paths;
+- MCP image-generation rows/files;
+- image editor AI tools and model endpoint pickers;
+- document PDF signing with stored signatures;
+- visual-report hero/section image insertion and research hide/unhide controls;
+- emoji picker/markdown emoji SVG proxy calls;
+- theme custom-font loading;
+- local gallery/signature CLI inspection.
+
+## Gallery
+
+`routes.gallery.gallery_routes` owns gallery upload/import/library/editor transform behavior: upload dedupe, image/video extension handling, EXIF extraction for images, albums, favorites, tags, generated media metadata, search/filter/sort, owner filtering, ZIP downloads, soft delete, disk cleanup, and chat-history cleanup after image delete. Top-level `routes.gallery_routes` is a `sys.modules` compatibility shim to the canonical module.
+
+Frontend gallery behavior includes upload progress, folder-drop album import, stale-while-revalidate cards, saved editor projects, detail actions, bulk delete/download, and cache-busted image refreshes.
+
+Album assignment and gallery image detail/update endpoints enforce owner scope and fail closed when no authenticated owner is available instead of falling back to broad access.
+
+Generated media provenance:
+
+- generated filenames are opaque hex-like media names, not trusted content hashes;
+- upload `file_hash` is a separate metadata field;
+- generated files live under `data/generated_images`;
+- chat image generation writes files and inserts `GalleryImage` rows through `src.ai_interaction`;
+- MCP image generation can create ownerless rows/files;
+- generated-but-not-yet-imported images can have no gallery row;
+- once a gallery row exists, owner checks decide visibility where the route enforces them.
+
+`app.py` owns direct `/api/generated-image/{filename}` serving through `src.generated_images.resolve_generated_image_path()`. It validates hex-like image/video filenames, rejects path escape and missing files, serves rowless generated files, checks row owner when a row exists, allows null-owner compatibility rows, and uses immutable/nosniff cache headers. Gallery replace/rotate/save/delete/ZIP paths also resolve filenames through a shared generated-image path helper so database filenames cannot escape `data/generated_images`. Replace/rotate/save-over-original flows can mutate bytes under the same filename, so frontend cache busting matters.
+
+## Image Tools And Providers
+
+Gallery/editor image transforms are split across:
+
+- `/api/gallery/ai-upscale` and `/api/gallery/style-transfer`;
+- `/api/image/inpaint`;
+- `/api/image/harmonize`;
+- `/api/image/sharpen`;
+- `/api/image/denoise`;
+- `/api/image/upscale-local`;
+- `/api/image/remove-bg`;
+- `/api/image/enhance-face`.
+
+AI image endpoints mostly require image-generation privilege in the gallery route layer. The sharpen route is explicitly auth-gated; utility routes that live outside gallery still need their own route-level gate checks rather than assuming a shared decorator. The chat image-generation session path calls `do_generate_image()` separately and has its own privilege/tool-listing behavior.
+
+Provider behavior:
+
+- OpenAI image edits use multipart `/images/edits`, mask conversion, size coercion, model restrictions, and source compositing where needed;
+- diffusion/self-hosted paths use JSON APIs such as inpaint, img2img, variations, harmonize, or A1111-compatible fallbacks;
+- client-supplied endpoint URLs on selected routes must pass outbound endpoint validation; DB-selected image endpoints should be resolved through owner-visible endpoint queries before decrypted headers/keys are used;
+- provider-returned image result URLs are validated with `src.url_safety.check_outbound_url()` before server-side download, with private-IP blocking controlled by image-route settings;
+- AI endpoint path suffixes are allowlisted before proxy/download use so arbitrary endpoint paths cannot be selected through gallery/editor requests;
+- editor model pickers load `/api/model-endpoints` and classify image-capable endpoints.
+
+Optional dependency behavior:
+
+- Pillow-backed paths are effectively core for EXIF, rotate, sharpen, and image preparation;
+- Real-ESRGAN powers denoise/upscale when installed and otherwise returns install guidance; import-time torchvision compatibility patches run before Real-ESRGAN imports;
+- remove-bg tries `rembg`, then transformers-style fallback, then an error;
+- face enhancement falls back from GFPGAN/OpenCV toward PIL behavior;
+- video uploads intentionally skip EXIF/ffprobe metadata today.
+- grounding and mask model inputs cast only `float64` tensors to `float32` before transfer to Apple's MPS backend, because MPS rejects float64; integer/other tensors and non-tensor processor values preserve their normal device-transfer behavior.
+
+## Editor Drafts
+
+`routes.editor_draft_routes` owns server-backed image editor project payloads. `EditorDraft` rows store title, payload JSON, thumbnail, source image, timestamps, and owner.
+
+Frontend editor behavior is split across `static/js/editor/*` and `static/js/galleryEditor.js`: canvas state, layer panel, masks, history, snapping, stroke pipeline, inpaint/rembg/harmonize tools, AI tool runner, model pickers, an AI edit command box that routes natural-language edit requests into existing inpaint/remove/upscale/background/style actions where possible, import wiring, topbar controls, auto-save, resume by draft ID or source image, draft-only open, and cleanup after close. `static/js/panels.js` loads this module graph on first editor use, shares concurrent imports, retries failed loads, and `static/sw.js` keeps the lazy graph in a separate offline panel precache.
+
+Draft compatibility behavior:
+
+- v2 server drafts store payloads and thumbnails server-side;
+- legacy/local raw payloads can still be restored by the frontend;
+- PUT 404 can recreate a missing draft row;
+- broken image drafts can fall back to the source image;
+- final close persist is best-effort.
+
+## Signatures, Emoji, Fonts
+
+`routes.signature_routes` owns reusable signature/stamp rows. Signature image payloads are normalized to bounded PNG base64, encrypted at rest, and owner-filtered; SVG signature input is not preserved. Document PDF render/export paths owner-filter signature IDs before stamping.
+
+`routes.emoji_routes` owns same-origin OpenMoji black SVG proxy/caching. It validates codepoint filenames, caches SVGs under `data/emoji_cache`, and returns transparent no-store SVGs for invalid, unknown, or unreachable codepoints. `static/js/emojiPicker.js` is a curated inline monochrome picker.
+
+`routes.font_routes` owns deriving available custom font family names from static font files under `static/fonts/custom`.
+
+## Visual Reports
+
+`src.visual_report` owns generated research/report HTML image behavior: HTTPS Open Graph image filtering, hero images, section images, icon/logo filtering, hide/reroll client controls, and inline JSON escaping for scripts.
+
+Research routes and handler code own hidden-image persistence. Visual reports render model/source-influenced Markdown to HTML, so raw HTML/link/image sanitization remains security-sensitive.
+
+## Security Policy
+
+Media routes are cookie/current-user surfaces unless they explicitly implement token owner/scope handling. Bearer-token callers that arrive as synthetic `api` users should not be treated as owner-scoped media API clients without explicit policy.
+
+Known boundaries:
+
+- image-generation routes require `can_generate_images`;
+- image proxy/editor endpoints currently resolve client-selected, DB-selected, or fallback image model endpoints without full owner-scoped endpoint-key policy or uniform outbound revalidation;
+- generated-file serving allows rowless files and null-owner compatibility rows;
+- uploads are byte-limited and extension-gated, with content sniffing available through `UploadHandler.detect_content_type()` when `python-magic`/`libmagic` is installed;
+- several base64 JSON editor routes accept large decoded image payloads and need route-level size discipline;
+- gallery DB filenames should be joined through shared generated-media path helpers before filesystem operations;
+- editor draft source image IDs, payloads, and thumbnails are owner-scoped by draft owner but do not fully validate source-gallery ownership or payload size;
+- emoji proxy constrains codepoint filenames and degrades invalid, unknown, or unreachable SVGs to transparent no-store placeholders, but remote SVG content still deserves security review;
+- visual report Markdown HTML/link/image output needs continued sanitization coverage.
+- `scripts/mlx_image_server.py` pins generation/edit routing to the process-start model and ignores request-selected model names, preventing unauthenticated callers from selecting a local model directory/repository whose model-specific script or bridge would execute.
+
+## Degraded And Compatibility Behavior
+
+- Uploaded images record display dimensions with EXIF orientation when possible; EXIF failures warn/degrade.
+- Video uploads skip EXIF and have no metadata extraction yet.
+- Missing generated files are skipped in ZIP downloads; if all are missing, the route returns no files found.
+- Soft delete commits the gallery row state before removing the disk file, so a failed DB write does not orphan a missing image row.
+- AI tagging can fail when disk files are missing.
+- Static JS/CSS/HTML assets revalidate because there is no frontend build/versioning.
+- Gallery/editor frontend state includes stale-while-revalidate and listener cleanup to avoid stale handlers.
+- `edit_image` tool schema/implementation currently appears stale against implemented `/api/image/*` and `/api/gallery/*` routes.
+
+## Testing Coverage
+
+Existing tests cover EXIF dimensions, owner-filter helper behavior, direct upload limits, image-generation privilege source shape, sharpen auth, gallery null-user denial, endpoint SSRF/source checks, editor draft payload validation, lazy editor loading/offline precache, MLX request-model pinning, font family derivation, visual-report helper behavior, gallery CLI previews, and selected security regressions.
+
+Route-level coverage is thin for full gallery CRUD/album/tag/download/delete flows, generated-image serving, editor draft owner CRUD, signature owner CRUD, emoji proxy/cache behavior, image-tool degraded responses, optional dependency fallbacks, and frontend editor behavior.
+
+## Current Gaps
+
+- Owner-scoped endpoint-key resolution is needed for image proxy/editor routes.
+- Media routes need a clear API-token policy: reject token callers, or implement owner/scope handling.
+- Generated-image serving needs live route tests for invalid filenames, rowless files, owned rows, null-owner rows, MIME/cache headers, and cross-owner behavior.
+- Mutable generated filenames plus immutable cache headers need cache-busting tests for replace/save-over-original flows.
+- Base64 JSON editor payload size limits need hardening; upload content sniffing should keep native/Docker parity coverage as dependencies change.
+- MCP image generation needs an owner attribution decision or explicit admin-only documentation.
+- `edit_image` tool route mapping appears stale.
+- Emoji SVG proxy/cache and visual-report raw HTML/link sanitization need stronger tests.
+- Optional image dependency fallbacks are mostly untested.

specs/integrations.md

@@ -0,0 +1,197 @@
+# Integrations
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+This spec covers external integration surfaces in:
+
+- `routes/codex_routes.py`;
+- `integrations/codex/*` and `integrations/claude/*`;
+- `routes/api_token_routes.py` and bearer-token handling in `app.py`;
+- `routes/auth_routes.py` integration CRUD/test routes;
+- `src/integrations.py` and `data/integrations.json`;
+- canonical `routes/webhook/webhook_routes.py` plus its top-level compatibility shim, and `src/webhook_manager.py`;
+- task webhook generation/triggering in canonical `routes/task/task_routes.py`, its top-level compatibility shim, `app.py`, `static/js/tasks.js`, and `scripts/odysseus-webhook`;
+- companion/mobile pairing in `companion/routes.py` and `companion/pairing.py`;
+- provider OAuth/device-flow endpoint links in `routes/copilot_routes.py`, `routes/chatgpt_subscription_routes.py`, `routes/device_flow.py`, and `ProviderAuthSession` rows;
+- integration UI surfaces in `static/js/settings.js` and `static/js/admin.js`;
+- database models `ApiToken` and `Webhook`.
+
+The SQLAlchemy `Integration` model exists in `core/database.py`, but current Settings generic integration CRUD uses `src/integrations.py` and `data/integrations.json`.
+
+## Scoped Agent Runtime
+
+`/api/codex/*` is the canonical scoped HTTP surface for external coding agents. Claude Code uses the same runtime endpoints; `/api/claude/plugin.zip` only delivers the Claude skill bundle.
+
+`routes.codex_routes` owns:
+
+- `/api/codex/capabilities`;
+- todos list/manage through `do_manage_notes()`;
+- email list/read/draft/send;
+- memory list/add/delete;
+- calendar list/create/delete;
+- document list/read/create/delete;
+- Cookbook task/server/output/cached-model/preset/serve/adopt/stop controls.
+
+`_scope_owner()` owns scope checks and token-owner resolution. `_as_owner()` temporarily runs borrowed route handlers as the scoped owner and restores request state afterward. Borrowed email, memory, calendar, and document route handlers own their domain behavior; Codex routes only adapt them behind scoped access.
+
+Runtime behavior:
+
+- missing scopes return 403;
+- invalid payloads return 400;
+- unavailable borrowed route surfaces return 503;
+- capabilities expose scope-derived booleans and partial availability flags;
+- email send and destructive actions remain described as confirmation-required behavior in bundled agent instructions.
+- Cookbook adopt/stop paths validate stored remote SSH host and port before interpolating them into SSH commands.
+
+The local integration skill/helper files require `ODYSSEUS_URL` and `ODYSSEUS_API_TOKEN`. They must use `/api/codex/*` and must not bypass Settings/token scopes through SSH, Docker, direct DB access, local files, MCP internals, or app imports. Helper scripts refuse non-`/api/codex/*` paths.
+
+## Bundle Distribution
+
+`/api/codex/plugin.zip` ships the Codex plugin tree from `integrations/codex/`. `/api/claude/plugin.zip` ships only the Claude `skills/` subtree from `integrations/claude/skills/`. These routes require an authenticated browser/user request and do not embed an API token.
+
+Setup instructions are duplicated in integration READMEs and `static/js/settings.js`; they need to stay aligned with live route surfaces and `/api/codex/capabilities`.
+
+## API Tokens
+
+`routes.api_token_routes` owns token profiles, allowed scopes, scope normalization, token creation/update/revocation, and profile metadata shown in Settings. Partial updates preserve existing scopes unless new scopes are supplied, owner checks apply to update/delete, and write scopes auto-include their read scope where applicable.
+
+`app.py` owns bearer-token validation. It accepts `Bearer ody_...`, checks a bcrypt hash through a prefix cache, updates `last_used_at` asynchronously, and stamps:
+
+- `request.state.current_user = "api"`;
+- `request.state.api_token = True`;
+- `request.state.api_token_owner`;
+- `request.state.api_token_scopes`.
+
+The raw token is returned only on creation. Stored state is hash, prefix, owner, scopes, active flag, and timestamps. Token create/update/delete invalidates the auth middleware cache. Companion pairing also mints chat-scoped `ApiToken` rows and invalidates that cache.
+
+Current API-token consumers include:
+
+- `/api/codex/*` scoped agent routes;
+- `/api/v1/chat` synchronous external chat;
+- `/api/models` catalog reads for `chat`-scoped token owners;
+- companion read endpoints;
+- selected session and owner-attribution helpers described in `auth-security.md`.
+
+The Cookbook scoped-agent surface currently exposes `cookbook:read` and `cookbook:launch` in Settings and checks them in Codex routes; those scope names must stay reconciled with `routes.api_token_routes.ALLOWED_SCOPES`.
+
+## Generic API Integrations
+
+`src.integrations` owns generic API integration presets, `data/integrations.json`, API-key encryption/decryption, secret masking, plaintext-key migration, enabled integration prompt text, and `execute_api_call()`.
+
+`routes.auth_routes` owns admin-only HTTP CRUD/test routes for these integrations. Presets are public metadata. The ntfy test route is special: it publishes a real test notification to the configured reminder topic instead of only probing server health.
+
+`api_call` is the agent/tool execution path for configured integrations. It is blocked for non-admin/public users by tool security, accepts only relative paths, uses the admin-configured base URL/auth settings, and returns truncated external responses to the model, including a sentinel when long JSON lists are shortened. Admin-authored integration descriptions are prompt context; external responses remain untrusted data.
+
+`execute_api_call()` normalizes base URLs to HTTP(S) scheme, hostname, and
+path-only values, rejects request paths that are not relative absolute paths
+(`/...`) or that carry schemes/fragments, treats `/` as the base URL without
+appending an extra slash, and checks the final URL through `src.url_safety`.
+Link-local/metadata targets are always rejected; setting
+`INTEGRATION_API_BLOCK_PRIVATE_IPS=true` also rejects loopback/RFC1918/private
+addresses for operators who do not need LAN integrations.
+
+After validation, `execute_api_call()` pins the outbound connection to the validated IP snapshot while preserving the configured URL, Host header, TLS server name, and redirect policy. DNS cannot select a different destination between SSRF validation and transport.
+
+Current call sites include:
+
+- `src.agent_loop` injecting enabled integration descriptions;
+- `src.tool_implementations.do_api_call()`;
+- task scheduler discovery/check-ins;
+- note reminder delivery through ntfy integrations and the generic webhook reminder channel.
+
+## Webhooks And External Chat
+
+Outgoing webhooks are admin-managed `Webhook` rows. `routes.webhook_routes` owns CRUD/test/toggle/delete and `/api/v1/chat`. `src.webhook_manager` owns allowed event validation, public URL validation, delivery-time URL revalidation, DNS-rebinding-safe pinned-IP delivery, HMAC signing, fire-and-forget delivery, in-flight task references, and delivery status/error persistence. Sanitized delivery errors redact IPv6-style address details.
+
+Allowed outgoing events are:
+
+- `session.created`;
+- `chat.message`;
+- `chat.completed`;
+- `webhook.test`.
+
+Current webhook event emitters include session creation, chat message/completion paths, and `/api/v1/chat` completion.
+
+`/api/v1/chat` is an inbound external chat endpoint. It requires a `chat` API token, checks session ownership before resume, can create a session from a direct API key, and otherwise falls back to the first owner-visible enabled model endpoint. Token-supplied direct `base_url` values use public-URL validation; configured endpoints remain admin-trusted. Logs and delivery/error text that include endpoint URLs should pass through URL redaction helpers before persistence or diagnostics.
+
+## Task Webhooks And Event Triggers
+
+Task webhook triggers are separate inbound webhooks. `app.py` exempts only `/api/tasks/{task_id}/webhook/{token}` from normal auth so external callers can trigger tasks without cookies. `routes.task.task_routes` owns token generation/regeneration and validates task id, token, and active status before queueing a run; the top-level route module is a compatibility alias.
+
+`static/js/tasks.js` displays the live task webhook URL. `scripts/odysseus-webhook url` now emits the same route with percent-encoded task/token path segments; the CLI still reads and mutates task rows directly for list/show/rotate/revoke rather than delegating to HTTP route policy.
+
+Event-triggered tasks use `src.event_bus`; task execution and scheduling ownership lives in `calendar-tasks-notes.md`.
+
+## Companion Pairing
+
+`companion.routes` owns companion/mobile HTTP routes:
+
+- `/api/companion/ping`;
+- `/api/companion/info`;
+- `/api/companion/models`;
+- `/api/companion/pair`.
+
+Read endpoints accept session or bearer-token callers and resolve the effective owner for visible rows. Model responses omit API keys. Pairing `GET` renders the admin form; pairing `POST` is admin-cookie only, mints a normal chat-scoped API token, invalidates the auth token cache, and returns a host/port/token payload as HTML or JSON.
+
+`companion.pairing` owns LAN host detection, pairing payload shape, token minting, and optional QR generation. QR rendering depends on optional `qrcode`; if unavailable or failing, pairing still returns the text payload.
+
+When `COMPANION_BASE_URL` is set, pairing advertises that validated operator-selected v1 address instead of container/request auto-detection. The accepted form is a canonical ASCII `http://` LAN/Tailscale IPv4, single-label hostname, or `*.local` origin with optional valid port and no credentials/path/query/fragment; HTTPS, public/misleading numeric host spellings, percent/backslash/control characters, and unsupported hosts fail closed. Auth-disabled model inventory retains the normal single-user all-endpoints view instead of filtering every ownerless request to legacy-null rows.
+
+## Unified Settings Surface
+
+The Settings Integrations view aggregates several subsystem surfaces:
+
+- generic API integrations;
+- Codex/Claude agent token setup;
+- CalDAV, CardDAV, email accounts including Google Workspace/.edu OAuth connect flows, MCP/OAuth links, provider device-flow links, and agent tokens.
+- provider-auth backed model endpoints such as ChatGPT Subscription and Copilot, where device-flow credentials live in provider auth rows rather than endpoint API-key fields.
+
+Vault and companion/mobile setup are separate settings/route surfaces today, not entries in the unified add-integration list.
+
+This spec owns the cross-integration framing and agent/token/webhook surfaces. Domain internals stay with their subsystem specs: calendar, email/contacts, shell-MCP, vault/auth, and settings-admin.
+
+## Degraded And Compatibility Behavior
+
+- 403 from scoped APIs means a settings/scope restriction.
+- 503 from Codex borrowed routes means the domain route surface is unavailable.
+- Missing or corrupt `data/integrations.json` loads as an empty list; non-object rows are ignored.
+- Plaintext generic integration API keys migrate to encrypted storage on load.
+- Webhook delivery has no retry/backoff queue; the persisted state is last status or sanitized last error.
+- Webhook URLs are validated at create and delivery time, redirects are disabled,
+  and delivery connects to the IP set validated immediately before the request.
+- Companion LAN detection is best-effort and falls back to local host/port defaults unless a valid `COMPANION_BASE_URL` is configured.
+- `ODYSSEUS_URL` must be reachable from the external coding agent; no Docker/native URL rewrite is performed.
+
+## Security And Provenance
+
+- API-token routes must either enforce a relevant scope or document an explicit exception.
+- Codex/Claude plugin zips must not expose secrets beyond source instructions and helper files.
+- Webhook list responses expose `has_secret`, not the secret value.
+- Webhook secrets are encrypted when an API key manager is available; plaintext fallback is legacy/degraded behavior.
+- Outgoing webhook signatures use `X-Odysseus-Signature`.
+- Generic integration API keys are encrypted at rest and masked in API responses.
+- Generic integration base URLs are admin-configured and not the same public-only policy as webhook URLs.
+- `api_call` output and remote integration responses are untrusted model context.
+- Pairing payloads expose the raw chat token once through HTML/JSON/QR; persisted token storage is hash/prefix only.
+
+## Testing Notes
+
+Current targeted coverage includes API-token CRUD basics, chat-scoped `/api/models` token access, companion pairing/read-only owner scoping, webhook SSRF validation, webhook auth-exempt source checks, webhook CLI token masking, integration-store shape/encryption migration, Google email OAuth route/helper behavior, Cookbook API-token scopes, Cookbook adopt SSH host validation, and `/api/v1/chat` base-url/fallback owner scoping.
+
+The integration audit also ran the targeted venv subset covering those areas with 52 passing tests and one warning.
+
+## Current Gaps
+
+- Codex/Claude scoped routes, owner restoration, degraded 503 behavior, plugin zip contents, and helper-script path refusal need focused regression tests.
+- Token profile/update behavior and Settings agent-token scope toggles need direct coverage.
+- Codex Cookbook scopes need continued Settings, route-check, and `ALLOWED_SCOPES` regression coverage.
+- Generic integration HTTP CRUD/test routes, `execute_api_call()` auth modes, response shaping, and frontend Settings/Admin flows need direct coverage.
+- `do_manage_tokens()` does not match `/api/tokens` semantics for `ody_` prefix, owner, scopes, and cache invalidation.
+- `do_manage_webhooks()` bypasses route behavior and does not cover signing-secret parity.
+- Companion read endpoints should either require `chat` scope or be documented as an explicit scope-policy exception.
+- Decide whether webhook secret plaintext fallback should remain accepted when the API key manager is unavailable.
+- Decide whether generic integration base URLs should stay LAN-capable by default or make `INTEGRATION_API_BLOCK_PRIVATE_IPS=true` the default.
+- Admin-authored integration descriptions and `api_call` results enter the untrusted-result/gated-action pipeline, but their product-level trust presentation still needs continued review.
+- The dormant SQLAlchemy `Integration` model should be removed, migrated into use, or documented as legacy.

specs/llm-models.md

@@ -0,0 +1,153 @@
+# LLM Models And Endpoints
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+This spec covers model/provider behavior in:
+
+- `src/llm_core.py`;
+- `src/endpoint_resolver.py`;
+- `src/foreground_model_routing.py`;
+- `src/model_discovery.py`;
+- `src/model_context.py`;
+- `src/model_capabilities.py`;
+- `src/model_capability_readers/`;
+- `src/task_endpoint.py`;
+- `src/tls_overrides.py`;
+- `src/copilot.py`;
+- `routes/copilot_routes.py`;
+- `routes/chatgpt_subscription_routes.py` and `routes/device_flow.py`;
+- `routes/model_routes.py`;
+- `routes/session_routes.py`;
+- `routes/cookbook_routes.py`, `routes/hwfit_routes.py`, and `services/hwfit/`;
+- `src/settings.py`;
+- `core/database.py` model `ModelEndpoint`;
+- frontend modules `static/js/models.js`, `static/js/modelPicker.js`, `static/js/model/matchKey.js`, `static/js/providers.js`, `static/js/settings.js`, `static/js/admin.js`, `static/js/compare/`, and Cookbook model-serving modules;
+- chat, compare, research, STT/TTS, and utility-model call sites.
+
+## Provider Calls
+
+`src.llm_core` owns provider-call mechanics. It handles OpenAI-compatible calls, Ollama normalization, Anthropic payload conversion, GitHub Copilot and ChatGPT Subscription provider detection/header injection, NVIDIA provider routing, streaming, fallback calls, upstream error formatting, async/streaming host liveness caching, configured model-list cache reads, tool-call sanitization, reasoning/thinking stream routing, and provider-specific parameter rules. GitHub Copilot OAuth/device-flow orchestration lives in `routes/copilot_routes.py` and `src/copilot.py`; ChatGPT Subscription device flow uses `routes/chatgpt_subscription_routes.py`, shared device-flow helpers, and `ProviderAuthSession` rows.
+
+`llm_core` owns payload shape. Route files and chat/agent code should request a call; they should not duplicate provider-specific payload quirks.
+
+Kimi Code User-Agent discovery has both sync and async implementations. Async
+post and stream paths probe `/models` through their existing async client and
+await each candidate, so header negotiation does not block the event loop; both
+paths share the accepted-value cache and 403 fallback policy.
+
+Provider-specific behavior is part of this layer: `LLM_CONNECT_TIMEOUT` controls the connect budget for sync and streaming calls, Kimi Code endpoints retry a small whitelisted User-Agent set on 403 and cache the accepted value, official Moonshot/Kimi Code and Anthropic Opus 4.7+ payloads omit sampling controls where required, and major-only Opus IDs such as `claude-opus-5` also omit temperature instead of falling through numeric minor-version parsing. Reasoning models omit or clamp unsupported temperature values, while self-hosted compatible endpoints keep normal parameters unless detected otherwise. Mistral structured content is normalized in async utility calls as well as stream/chat paths, and Mistral/Moonshot/Kimi reasoning content, `gpt-oss` harmony output, DeepSeek V4 thinking identifiers, and native/OpenAI-compatible Ollama thinking formats keep hidden reasoning separate from visible text. Tool names that collide with GPT-OSS built-ins are aliased on the provider boundary and mapped back before execution. Copilot request metadata remains defensive against malformed `request_flags`.
+
+## Canonical Provider And Model Shape
+
+`src.model_capabilities` owns canonical model family, task, modality,
+capability, limit, evidence, assertion, deterministic-control, probe-result,
+reasoning-control token, and display-query values.
+`src.model_capability_readers` owns endpoint-scoped stable identity, lightweight
+provider detection, record serialization, and normalization of already-fetched
+provider payloads. Readers do no network I/O. Model-specific observations are
+kept in `model-quirks.md`, not a runtime registry without a consumer.
+
+Provider support and model support are different facts. A provider may expose
+tools, reasoning, vision, or multiple APIs while individual models differ.
+Provider-native readers describe where model evidence can appear. Current
+concrete readers cover generic OpenAI-compatible identity, OpenAI, OpenRouter,
+Google, Ollama, LM Studio, and llama.cpp. Identity-only model lists remain
+unknown.
+
+Reader dispatch uses an explicit vendor first, then endpoint kind, label-bounded hostname suffix, and common local-port hints. Generic payload handling accepts `data[]`
+or `models[]` items with `id`, `name`, or `model`; it does not accept a bare
+list and never promotes capability-looking fields. Unknown fields remain in
+the in-memory raw record. See [model-capability-canonical.md](model-capability-canonical.md),
+[model-quirks.md](model-quirks.md), and the
+[provider map](model-providers/_readme.md).
+
+This canonical layer is currently exercised by focused unit tests but is not
+wired into runtime discovery, endpoint resolution, model context, request
+shaping, or frontend pickers. `routes/model_routes.py` model probes continue to
+return model IDs through their existing runtime path.
+
+Route-level probe helpers in `routes/model_routes.py` are the current exception: they build minimal provider-specific probe payloads using `llm_core` detection helpers. Keep probe behavior aligned with `llm_core` provider adapters. LLM provider HTTP clients and endpoint probes share `src.tls_overrides.llm_verify()`, which can add an operator-provided `LLM_CA_BUNDLE` on top of normal certificate verification without turning verification off or widening that trust to arbitrary URL fetches.
+
+## Endpoint Resolution
+
+`src.endpoint_resolver` owns endpoint normalization and URL construction:
+
+- base URL normalization;
+- chat and model-list URL construction;
+- endpoint ID resolution;
+- chat, utility, and vision fallback candidate selection;
+- Tailscale hostname resolution where available.
+
+OpenAI-compatible model-list URL construction preserves `/v1` bases and inserts `/v1/models` for bare local bases such as LM Studio `http://localhost:1234`.
+
+`routes/model_routes.py` owns model endpoint CRUD, admin provider discovery/probing, visible/hidden/pinned model lists, endpoint kind and refresh policy, curated/extra model partitioning, `/api/models` catalog caching, Docker loopback rewriting, tool-support probing, provider-auth linkage, endpoint-dependent settings cleanup, and owner filtering. Endpoint dedupe allows the same base URL under different API keys and surfaces API-key fingerprints/key presence without returning secrets.
+
+`routes/session_routes.py` owns binding sessions to endpoint IDs, owner-scoped header construction, raw-endpoint rejection for non-admin users, model validation, and persisted session headers. Compare panes and normal chat session creation use this path.
+
+`ModelEndpoint` rows own API keys, base URLs, cached/hidden/pinned models, model type, endpoint kind, refresh mode/interval/timeout, supports-tools state, nullable owner, optional provider-auth linkage, and provider metadata. `owner = NULL` means legacy/shared; non-null rows are private to that owner, while admins can see all. Secret fields must remain encrypted and scrubbed in responses.
+
+Decrypted endpoint headers can be copied into session metadata for chat use. Endpoint deletion must clear dependent settings and copied session headers.
+
+## Model Discovery And Lists
+
+`src.model_discovery` owns host/env/Tailscale/local-port scanning for model servers. Admin `/api/providers` and `/api/discover` use that scanner; endpoint CRUD, test, refresh, and hidden-model controls are frontend-owned by `static/js/admin.js`.
+
+`/api/models` is the normal picker/catalog surface. It is auth/owner scoped, per-user/admin-flag cached briefly, can trigger background refresh, preserves offline endpoint rows, filters hidden models, and preserves pinned model IDs for UI selection. API-token callers must carry `chat` scope and a token owner before they can list models. API/proxy endpoint inventory is visible by default until an explicit `pinned_models` allow-list is saved; an explicit empty list means show none, and legacy hidden-list state is upgraded to the equivalent pins so endpoint settings, picker checkboxes, and chat agree. Proxy/API endpoints can be marked cached-first/manual so large upstream catalogs are not repeatedly probed, while explicit refresh paths use longer manual timeouts. Local endpoints get cheap reachability probes before expensive refreshes where possible, and endpoint responses can include explicit `supports_tools` state for schema-emission heuristics. Google Gemini API endpoints use the native paginated `generativelanguage.googleapis.com/v1beta/models` catalog, send API keys in `x-goog-api-key`, retain only content-generation model IDs, and default to manual refresh unless the caller explicitly chooses another mode. Probe failure returns no curated Google fallback. `static/js/models.js` and `static/js/modelPicker.js` own the sidebar/picker catalog; `static/js/model/matchKey.js` owns longest-substring model-info/pricing key matching; `static/js/settings.js` owns default, utility, vision, image, TTS, STT, and fallback selectors.
+
+`src.task_endpoint` owns background-task endpoint/model resolution for task routes and scheduler callers. It resolves `task_endpoint_id`/`task_model` through the normal endpoint resolver with owner context.
+
+Cookbook and HWFit own local model download, serve, ranking, and auto-registration flows. They can create LLM or image `ModelEndpoint` rows, but provider dispatch remains owned by `llm_core`/endpoint resolution.
+
+## Context Length
+
+`src.model_context` owns model context-length lookup/query and token estimation. Cache keys include endpoint plus model so identical model names on different endpoints do not bleed context-window data. Unknown proxy/API models can pick up real context windows from endpoint catalog metadata such as `context_length`; otherwise unknown lengths stay explicit unknowns rather than default values. Known lengths feed chat/agent token-budget scaling through `src.context_budget`. Token estimation counts assistant `tool_calls` arguments so compaction sees tool-only turns instead of underestimating them. Chat/agent context budgeting should call this layer instead of hardcoding model windows.
+
+## Runtime Fallback And Routing
+
+`src.foreground_model_routing` owns foreground Chat/Agent fallback policy. Selected models are strict by default. Fallback requires owner-scoped `foreground_fallback_enabled=true` and an ordered `foreground_model_fallbacks` list; the old `default_model_fallbacks` setting is retired, ignored, and not migrated into consent. Named users never inherit a legacy flat/single-user fallback choice, candidate lists are capped at ten exact owner-visible models, and caller-provided allowed-model restrictions remain authoritative.
+
+Only eligible availability failures before substantive output can fall through. Default eligible statuses are 408, 425, 429, 500, 502, 503, 504, 507, 508, and 529. Missing endpoint/configuration, provider/schema/request errors, empty completions, and post-content failures do not silently change routes. A candidate commits after non-empty visible/reasoning text or a tool call; the answering route is then pinned. Foreground routing carries model and endpoint descriptors together, shapes context/compaction route-neutrally across candidates, persists only answering-route compaction, and records requested/actual/per-round route provenance plus cost attribution. Utility/background and vision fallbacks remain separate policies.
+
+Model selection has three layers: endpoint resolver hidden-model and first-chat-model selection, `/api/default-chat` per-user default/fallback resolution, and frontend picker auto-selection for empty sessions.
+
+Image routing uses model-name prefixes and `ModelEndpoint.model_type == "image"` to bypass text chat and generate media. Vision analysis uses configured vision models and `vision_model_fallbacks`; image and vision endpoint lifecycle changes should update chat, document processing, Cookbook, and settings UI together.
+
+Provider tool calls are untrusted requests, not authorization. `supports_tools` controls schema emission only; `llm_core` normalizes provider tool-call payloads, while execution authority remains in `src.tool_execution`, `src.tool_security`, and agent-tool policy.
+
+## Degraded And Platform Behavior
+
+- Provider offline or probe failures should surface actionable errors without crashing the app. Async calls retry transient 429/502/503/504 responses before failing.
+- Docker deployments may need loopback URL rewriting from `127.0.0.1` to host-accessible addresses.
+- Foreground fallback selection must preserve endpoint identity, explicit owner consent, allowed-model policy, and owner scope. User/API-token LLM dispatch that can carry configured endpoint keys must pass the effective owner into resolver calls.
+- Async and streaming calls use dead-host cooldown; sync utility/vision calls do not have identical cooldown coverage.
+- llama.cpp slot-affinity routing is local-endpoint behavior only and must not be applied to cloud/provider endpoints.
+- Hidden, pinned, cached, endpoint-kind, refresh-policy, and offline model state are UI/runtime compatibility data. Pinned models may not participate in every resolver auto-pick path unless code explicitly includes them.
+- SSE/stream parsers tolerate null choice/usage/tool-call entries and null streaming tool-call arguments; provider events should degrade to empty text or shaped stream errors instead of crashing the chat loop.
+- Provider adapters carry small model-specific quirks: Opus 4.7+ and official Kimi/Moonshot code payloads omit `temperature`, Kimi/Moonshot/Mistral reasoning content is preserved separately, ChatGPT Subscription refreshes bearer credentials, native Ollama can handle multimodal content, and Ollama `/v1` responses for Qwen3/Gemma4-style thinking can suppress thinking text when requested.
+
+## Security Policy
+
+- Endpoint API keys are encrypted in `ModelEndpoint.api_key` and never returned by endpoint APIs; admin surfaces return key presence only.
+- Endpoint CRUD, probes, provider discovery, and most endpoint configuration are admin-cookie or internal-tool gated.
+- `/api/models` is auth/owner scoped for configured deployments; API-token access requires `chat` scope and token-owner attribution.
+- Admin-created model endpoints may target local/LAN servers. Non-admin chat session creation must use registered endpoint IDs. API-token `/api/v1/chat` requires `chat` scope and validates direct `base_url` with public-only URL checks.
+
+## Current Call Sites Include
+
+- chat streaming and non-streaming calls;
+- agent loop calls with optional tool schemas;
+- compare pane calls;
+- research synthesis/probe calls;
+- utility model fallbacks for summarization/extraction;
+- frontend Settings and model picker endpoint management.
+
+## Current Gaps
+
+- Runtime provider detection, model curation, and frontend logos are still split across `llm_core`, `model_routes`, and `providers.js`; the canonical reader package has no production consumer yet.
+- Provider-specific behavior is concentrated in `llm_core.py`, which is large and easy to regress.
+- Several runtime request builders still use model-name heuristics. They should migrate only after endpoint/provider code supplies structured identity and a real consumer contract; the canonical catalog does not add a parallel quirk matcher.
+- Endpoint identity and fallback behavior need careful review when new OAuth/subscription providers are added.
+- Owner must continue to be threaded through new utility/research/default endpoint-resolution call sites so provider keys stay isolated.
+- `/api/models` owner-scoped listing/cache behavior, shared/private endpoint dedupe, endpoint-kind refresh policy, fallback-chain owner scope, and image endpoint create/list/update lifecycle need stronger route-level regression coverage.

specs/memory-skills.md

@@ -0,0 +1,118 @@
+# Memory And Skills
+
+Last updated: dev@2e2bb52 | 2026-08-16
+
+## Scope
+
+This spec covers persistent memory and user skills in:
+
+- app wiring in `app.py` and `src/app_initializer.py`;
+- active legacy memory managers `src/memory.py` and `src/memory_vector.py`;
+- canonical memory routes in `routes/memory/memory_routes.py`, with `routes/memory_routes.py` as a compatibility shim;
+- chat memory/skill gating in `routes/chat_helpers.py`;
+- memory compatibility modules in `services/memory/memory.py`, `services/memory/memory_vector.py`, and `services/memory/service.py`;
+- provider abstractions in `src/memory_provider.py`;
+- LLM extraction/audit in `services/memory/memory_extractor.py`;
+- skill storage, format, import, and extraction in `services/memory/skills.py`, `services/memory/skill_format.py`, `services/memory/skill_importer.py`, and `services/memory/skill_extractor.py`;
+- skill routes in `routes/skills_routes.py`;
+- prompt/tool call sites in `src/chat_processor.py`, `src/agent_loop.py`, `src/ai_interaction.py`, `src/tool_implementations.py`, `src/tool_execution.py`, `src/tool_schemas.py`, and `src/tool_security.py`;
+- MCP and Codex surfaces in `mcp_servers/memory_server.py` and `routes/codex_routes.py`;
+- backup/admin/CLI surfaces in `routes/backup_routes.py`, canonical `routes/admin_wipe/admin_wipe_routes.py` plus its shim, `scripts/odysseus-memory`, `scripts/odysseus-skills`, and `scripts/odysseus-backup`;
+- frontend modules `static/js/memory.js` and `static/js/skills.js`;
+- tests under `tests/test_memory_*`, `tests/test_builtin_memory_consolidation.py`, `tests/test_skill_*`, and `tests/test_skills_*`.
+
+## Memory Runtime
+
+`src.app_initializer.initialize_managers()` creates the active `src.memory.MemoryManager` and `src.memory_vector.MemoryVectorStore` used by app startup. `routes.memory.memory_routes` imports through `services.memory` but is passed the startup manager instances; top-level `routes.memory_routes` is a `sys.modules` compatibility shim.
+
+`MemoryManager` owns JSON-backed memory storage in `data/memory.json`, validation, owner fields, pinned state, use counts, and text/keyword similarity. Read-only `load_all()` remains lenient and can degrade an unreadable store to no memories. Mutating read-modify-write paths use `load_all_for_update()`, which raises `MemoryStoreUnreadable` rather than letting a corrupt or unreadable file be overwritten with an empty list. Agent/MCP/native-provider adds, extraction, backup import, and owner migration preserve that distinction; legacy `memory.txt` migration remains allowed. `MemoryVectorStore` owns semantic lookup when Chroma and embeddings are reachable.
+
+Chat memory behavior:
+
+- chat preferences and incognito state gate memory preface use;
+- pinned memories are loaded for the owner;
+- retrieved memories use keyword matching plus optional vector scoring;
+- inserted memory is wrapped as untrusted context;
+- memory use counts are incremented after insertion.
+
+`services/memory/memory_extractor.py` owns LLM-assisted extraction, audit, and validation flows. It requests model behavior and writes through the memory manager; it does not own chat session persistence.
+
+Extraction handles reasoning-model response shapes and records explicit dislike/drop preferences as `dislikes` rather than losing them to generic fact handling.
+
+## Skills Runtime
+
+`services/memory/skills.py` owns disk-backed skill storage under `data/skills/<category>/<name>/SKILL.md`, plus `_usage.json` usage/audit sidecars. Legacy `data/skills.json` is a read-only fallback/import source, not the current write shape.
+
+`services/memory/skill_format.py` owns frontmatter/body parsing and emission. Quoted scalar parsing/emission is symmetric: JSON escapes decode once, UTF-8/non-ASCII stays intact, emitted values escape line separators safely, and invalid JSON-style escapes fall back to literal text instead of compounding backslashes on every save. `services/memory/skill_importer.py` resolves public GitHub/skills URLs, fetches bundle files with strict public-network URL safety, and chooses/imports `SKILL.md`. Import disables automatic redirects, follows at most five hops, validates and resolves each hop, then connects only to the validated IP snapshot through a pinned transport while preserving URL, Host, and TLS identity; GitHub final-host checks and file/size limits still apply. `routes/skills_routes.py` owns CRUD/search/index/import, owner filtering, skill test/audit jobs, and admin-gated built-in tool instruction overrides.
+
+Skill extraction is owned by `services/memory/skill_extractor.py`. It can suggest or save skills from conversations, tries valid brace-delimited JSON candidates with `JSONDecoder.raw_decode()`, rejects ambiguous multiple top-level JSON objects instead of guessing, and saved skills remain user-editable data.
+
+Agent skill behavior:
+
+- matched skills are owner-scoped, confidence-gated, usage-counted, and wrapped as untrusted context;
+- `index_for()` exposes published skills plus teacher-escalation drafts gated by platform and toolsets; `active_toolsets=None` means the caller has no explicit toolset knowledge and does not hide `requires_toolsets` skills, while an explicit list applies the gate;
+- user prefs such as skills enabled, auto-approve, and max injected skills shape runtime insertion;
+- the level-0 base skill index currently calls `index_for(owner=None)`, so it is not fully owner-scoped.
+- skill tests use the configured utility model rather than the chat default and wrap user-editable skill text as untrusted context; approval continuation for a test or teacher-generated skill uses the same exact-action gate as the normal agent loop.
+
+## Tools, MCP, And Backup
+
+Native `manage_memory` and `manage_skills` tool paths pass owner context and use in-process policy gates. `manage_skills` requires an explicit action instead of silently defaulting a malformed call. Manual memory add can choose a category, and route-side manual add validates the source session owner before attaching session-derived memories. `mcp_servers/memory_server.py` lazy-initializes `src` managers and exposes list/add/edit/delete/search. It can scope to `ODYSSEUS_MCP_MEMORY_OWNER` or `ODYSSEUS_MEMORY_OWNER`; if the JSON store contains owner-bearing entries and no owner env is configured, it returns an owner-scope error instead of listing or mutating across owners. Ownerless stores remain ownerless compatibility mode.
+
+The direct `odysseus-memory add` CLI tolerates non-object legacy/corrupt rows
+when checking whether its newly added entry is already present; it ignores
+those rows instead of calling mapping methods on them and crashing the add.
+
+`/api/export` owner-filters memories and skills. `/api/import` imports skills through current disk-backed `SkillsManager` APIs, stamping missing owners to the importer and preserving supported skill metadata. Full data snapshots through `scripts/odysseus-backup` preserve on-disk skill trees, memory JSON, and caches differently from JSON import/export.
+
+## Compatibility State
+
+Memory and skills are partially migrated:
+
+- app startup, MCP, and some tools still use `src.memory*`;
+- services memory modules remain relevant for imports/tests, with memory and vector modules re-exporting canonical `src` implementations;
+- `services/memory/service.py` is a compatibility facade around the canonical managers, but it remains ownerless and should not be assumed equivalent to route/tool owner policy;
+- skills are service-owned and disk-backed, while backup import and some compatibility paths still expect older JSON/list shapes.
+
+## Degraded Vector Memory
+
+Chroma is an external HTTP service. Native defaults use `localhost:8100`; Docker uses `chromadb:8000`. Embeddings prefer configured HTTP endpoints and can fall back to local FastEmbed.
+
+Startup can degrade to keyword-only memory when vector initialization fails. Extraction/audit paths catch vector failures and continue with text/JSON behavior. Vector dedup is checked against the current owner before suppressing a candidate, and audit rebuilds preserve other owners' vector rows. Chat retrieval assumes a healthy startup vector store remains usable, so post-start vector failures can still break memory retrieval unless handled by the caller.
+
+Admin wipe currently has a vector cleanup compatibility gap because it imports a nonexistent helper before attempting vector clearing.
+
+## Policy
+
+Saved memories and skills are untrusted source data when shown to the model. A stored skill may contain useful instructions, but it is still user-editable content and must be framed consistently with prompt-injection policy.
+
+Owner isolation is surface-specific:
+
+- HTTP memory and skills routes are expected to owner-filter normal user data;
+- native memory/skill tools are expected to pass owner context;
+- Codex exposes scoped token memory behavior separately;
+- normal memory/skills routes are cookie/current-user surfaces, not scoped token APIs;
+- MCP memory uses an environment-configured owner for owner-scoped stores, while the agent level-0 skill index currently has ownerless/global behavior;
+- vector dedup during memory extraction suppresses only same-owner or legacy-ownerless vector matches.
+
+Skill test/audit flows intentionally run user-editable `SKILL.md` content as instructions inside controlled jobs. Those jobs rely on route owner checks, admin gates where applicable, and tool execution policy.
+
+Skill import is admin-gated defense-in-depth, but imported URLs are still untrusted network input. Initial and redirected targets must remain public, automatic redirects stay disabled, and the connection must use only the IP set validated for that hop so DNS rebinding cannot change the destination between validation and transport.
+
+User rename flows update skill frontmatter owner fields and `_usage.json` owner keys alongside memory/upload/research ownership migrations.
+
+## Testing Coverage
+
+Existing tests cover memory extraction/degraded vectors, owner isolation, unreadable-store mutation refusal, MCP memory shape/scope, skill owner update/delete, prompt-injection wrapping and approval continuation, utility-model selection, toolset gating, frontmatter escape round trips, skill-import redirect and DNS-rebinding/SSRF defenses, CLI non-object rows, and selected route owner checks.
+
+Route-level memory CRUD/security, skills route security, MCP memory behavior, vector degraded writes, compatibility facade owner behavior, backup skill import, admin vector cleanup, and frontend endpoint wiring need broader coverage.
+
+## Current Gaps
+
+- `services/memory/service.py` needs an explicit owner-scope/support decision before it is treated as a public memory API.
+- The agent level-0 skill index should thread owner or be documented as an intentional local/global index.
+- MCP memory still needs a deliberate multi-user UX/config decision, but current behavior avoids cross-owner access when owner-bearing rows exist without an explicit MCP owner env.
+- Memory JSON import does not rebuild vector indexes.
+- Admin wipe vector clearing is currently ineffective.
+- Chat memory retrieval needs a graceful path for vector failures after startup.
+- Route-level memory and skills security coverage is incomplete.

specs/model-capability-canonical.md

@@ -0,0 +1,178 @@
+# Canonical Provider And Model Capability Layer
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+This spec covers the implementation introduced on current `dev` in:
+
+- canonical model values and query helpers in `src/model_capabilities.py`;
+- record, identity, and provider-detection helpers in
+  `src/model_capability_readers/base.py`;
+- reader dispatch in `src/model_capability_readers/__init__.py`;
+- concrete readers for generic OpenAI-compatible, OpenAI, OpenRouter, Google,
+  Ollama, LM Studio, and llama.cpp payloads;
+- regression coverage in `tests/test_model_capabilities.py` and
+  `tests/test_model_capability_readers.py`.
+
+The layer normalizes already-fetched JSON-compatible values. It performs no
+network I/O, does not shape provider requests, does not persist its output, and
+does not authorize model or tool use. No production caller currently consumes
+the canonical records outside this package; runtime integration remains later
+work.
+
+There is no `src/provider_capability_schemas.py`, capability-specific
+diagnostics module, or runtime model-quirk registry on current `dev`.
+
+## Layer Boundaries
+
+- `src.model_capabilities` defines normalized families, tasks, modalities,
+  capabilities, evidence sources/confidence, assertion states, deterministic
+  controls, probe results, reasoning-control tokens, and display-surface
+  queries.
+- `ModelCapability` owns family, primary task, input/output modalities,
+  capability tokens, limits, source, and confidence.
+- `CapabilityAssertion` records claimed, verified, unsupported, or unknown
+  status for one capability. Missing evidence is not an unsupported claim.
+- `DeterministicControl` records support evidence for controls such as
+  temperature, top-p, seed, tool choice, or prompt caching. A supported
+  request control is not itself a model capability.
+- `CapabilityProbeResult` is an in-memory evidence shape that converts pass,
+  fail, or partial probe state into an assertion. No current runtime probe
+  stores or merges these objects.
+- `CapabilityQuery` and `display_surfaces_for()` map a normalized capability
+  into candidate surfaces such as chat, vision chat, image generation,
+  embeddings, or reranking. They are not wired into current pickers.
+- Reader `ModelCapabilityRecord` binds a vendor/model identity to the nested
+  capability object, assertions, deterministic controls, and optional raw
+  provider evidence.
+
+Provider transport support and per-model support are separate facts. Request
+and response adapters remain in `src.llm_core` and related provider modules.
+Model-specific observations remain in [model-quirks.md](model-quirks.md).
+
+## Current Serialized Shapes
+
+`ModelCapability.to_dict()` emits the nested capability shape:
+
+```json
+{
+  "family": "chat",
+  "primary_task": "chat.completions",
+  "modalities": {
+    "input": ["text", "image"],
+    "output": ["text"]
+  },
+  "capabilities": ["tool_call", "vision"],
+  "limits": {"context_tokens": 131072},
+  "source": "provider_reader",
+  "confidence": "provider_reported"
+}
+```
+
+`ModelCapabilityRecord.to_dict()` wraps that value with `vendor`, `model_id`,
+`stable_model_id`, `display_name`, `capability_assertions`, and
+`deterministic_controls`. It does not currently emit a schema version or the
+flat `provider`/`model`/`features`/`controls` shape. Raw provider fields are
+included only when the caller passes `include_raw=True`.
+
+Endpoint configuration can explicitly map `model_type=llm` to chat and
+`model_type=image` to image generation. Missing or unrecognized endpoint types
+stay unknown rather than silently becoming chat-capable in this schema layer.
+
+## Identity And Reader Dispatch
+
+`records_from_payload()` selects a reader from an explicit `vendor`, or from
+`detect_vendor(base_url, endpoint_kind)` when no vendor is supplied.
+
+Current detection order and behavior are:
+
+1. a recognized explicit endpoint kind;
+2. label-bounded hostname checks for OpenRouter, OpenAI, Anthropic, Google APIs, and Ollama Cloud;
+3. common local ports: `11434` for Ollama, `1234` for LM Studio, `8000` for vLLM, and `30000` for SGLang;
+4. generic OpenAI-compatible for any other parsed host, otherwise unknown.
+
+These are normalization hints, not authorization. Hostname checks accept an exact domain or its dot-delimited subdomains after lowercasing and removing a trailing dot, so names such as `notopenai.com` do not match `openai.com`; local-port mappings remain intentionally covered by tests. Callers must not treat any result as proof of endpoint trust.
+
+Implemented reader modules are `generic_openai`, `openai`, `openrouter`,
+`google`, `llamacpp`, `ollama`, and `lmstudio`. Anthropic, Hugging Face,
+SGLang, and vLLM have placeholder vendor IDs but currently dispatch through the
+generic identity-only reader. Other explicitly supplied vendor strings are
+also preserved while using that generic reader.
+
+Stable model identity is scoped in this order:
+
+- explicit endpoint ID;
+- a short hash of normalized base URL when an endpoint ID is absent;
+- `global` when neither endpoint identity is supplied.
+
+## Generic Identity-Only Contract
+
+The generic reader accepts mapping payloads containing `data[]` or `models[]`.
+Each item must itself be a mapping and provide `id`, `name`, or `model`.
+Bare-list payloads and `key`/`slug`-only items are not accepted by the current
+implementation.
+
+The reader deliberately returns unknown family, modalities, capabilities, and
+controls. It preserves the raw item on the in-memory record but does not parse
+type/task fields, descriptions, ownership, supported-parameter lists,
+capability-looking booleans, or token limits.
+
+## Provider-Native Readers
+
+- OpenAI keeps the official Models API identity-only.
+- OpenRouter maps explicit architecture modalities, supported parameters,
+  limits, voices, and default parameters into family/capability/control state.
+- Google maps the native Models resource. Embedding-only methods map to the
+  embedding family; content-generation methods do not prove modality or chat
+  family. Explicit thinking, limits, sampling fields, caching, and batch
+  methods are retained without parsing product names.
+- Ollama treats `/api/tags` as identity-only and maps selected-model
+  `/api/show` capability tokens. Context can come from structured fields or a
+  parsed `num_ctx` line in the serialized `parameters` value.
+- LM Studio maps native v1 `models[]` and v0-style `data[]` fields. A plain
+  OpenAI-compatible list without native type/capability fields stays unknown.
+- llama.cpp can merge `/v1/models`, `/props`, and `/slots` evidence for one
+  served model. It records tool/streaming claims, explicit unsupported
+  vision/audio assertions, controls, and runtime/training/size limits.
+
+Readers tolerate non-object entries and unknown fields where their helpers
+permit it. They do not infer authoritative capability from model IDs or display
+names.
+
+## Evidence Semantics
+
+The canonical vocabulary includes admin override, endpoint configuration,
+provider reader, Cookbook/Hugging Face, maintained registries, heuristic,
+probe, and unknown sources. It also defines explicit, provider-reported,
+registry, heuristic, and unknown confidence values.
+
+Those tokens make evidence representable; current `dev` does not implement a
+global precedence, merge, expiry, or conflict-resolution engine. Assertions
+generated by readers are usually `claimed`; a `CapabilityProbeResult` maps pass
+to verified, fail to unsupported, and partial to claimed at the scope carried
+by that object.
+
+## Tests
+
+Focused tests pin:
+
+- endpoint-kind, host, and common-port vendor detection;
+- endpoint/base-URL-scoped stable IDs;
+- unknown behavior for generic and official OpenAI lists;
+- canonical normalization and display-surface matching;
+- assertion, deterministic-control, and probe-result shapes;
+- OpenRouter, Google, Ollama, LM Studio, and llama.cpp mappings;
+- negative cases that avoid name-based media/capability inference.
+
+## Current Gaps
+
+- Canonical records are not yet used by runtime discovery, endpoint resolution, model context, request shaping, or frontend pickers.
+- Reader output is not persisted, refreshed, merged, or expired.
+- Provider detection still uses common-port hints; consumers must not promote normalization hints into trust decisions.
+- Only seven concrete readers exist; placeholder and other providers use the
+  identity-only generic reader.
+- Generic fallback does not accept bare-list or `key`/`slug`-only payloads.
+- There is no capability-specific diagnostic/logging path.
+- Runtime request builders still contain model-name heuristics outside this
+  canonical layer.

specs/model-providers/_readme.md

@@ -0,0 +1,100 @@
+# Provider Capability Specs
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+This directory maps serving-provider observations and current model-catalog
+normalization into the canonical layer defined by
+[model-capability-canonical.md](../model-capability-canonical.md). It records
+current Odysseus implementation evidence, merged fixes, reproducible user
+observations, and provider documentation without treating any single source as
+global model truth.
+
+## General To Specific Resolution
+
+Read specs in this order:
+
+1. [openai-compatible.md](openai-compatible.md) for the conservative general
+   identity-only reader;
+2. the serving-provider file for native endpoints, headers, request/response
+   observations, and catalog fields;
+3. [model-quirks.md](../model-quirks.md) for model-specific observations.
+
+Provider files document transport; runtime adapters still own it. Model quirks
+record only deviations and are not a second runtime matcher. Shared model facts
+must not be copied into every provider file. An OpenAI-compatible provider is
+not OpenAI: an explicitly supplied vendor string is preserved even when it uses
+the generic reader.
+
+Current reader dispatch does not infer a provider from payload shape. It uses an explicit vendor, then endpoint kind, label-bounded hostname matches, and common local-port hints. The port hints map 11434 to Ollama, 1234 to LM Studio, 8000 to vLLM, and 30000 to SGLang. Those hints are normalization behavior, not endpoint trust.
+
+## Provider Map
+
+### Implemented canonical readers
+
+- [openai.md](openai.md): identity-only Models API plus Chat/Responses dialects.
+- [openai-compatible.md](openai-compatible.md): generic compatible catalog and runtime dialect boundaries.
+- [openrouter.md](openrouter.md): rich architecture, modalities, parameters, and limits.
+- [google.md](google.md): native paginated Gemini Models API and GenerateContent.
+- [ollama.md](ollama.md): `/api/tags`, `/api/show`, native chat, and OpenAI compatibility.
+- [lm-studio.md](lm-studio.md): native v1 catalog/chat, explicit v0 compatibility, and OpenAI compatibility.
+- [llama-cpp.md](llama-cpp.md): `/props`, `/slots`, OpenAI/Responses/Anthropic surfaces.
+
+### Placeholder identities using the generic reader
+
+- [anthropic.md](anthropic.md): identity-only Models API and native Messages runtime adapter.
+- [vllm.md](vllm.md): common-port identity hint; deployment capability remains unknown.
+- [sglang.md](sglang.md): common-port identity hint; parser/config-dependent capability remains unknown.
+- [hugging-face.md](hugging-face.md): Hub observations and download/fit metadata without a canonical reader.
+
+### Provider observations without a dedicated canonical reader
+
+- [mistral.md](mistral.md): rich model cards, reasoning controls, and structured runtime content.
+- [github-copilot.md](github-copilot.md): account model-list observations and required runtime headers.
+- [chatgpt-subscription.md](chatgpt-subscription.md): Codex model identity and Responses event shape.
+- [cohere.md](cohere.md): native endpoint/catalog observations; not currently normalized.
+
+### Other provider identity and general/identity-only observations
+
+- [moonshot-kimi.md](moonshot-kimi.md)
+- [deepseek.md](deepseek.md)
+- [groq.md](groq.md)
+- [nvidia-nim.md](nvidia-nim.md)
+- [cerebras.md](cerebras.md)
+- [together.md](together.md)
+- [fireworks.md](fireworks.md)
+- [xai.md](xai.md)
+- [zai.md](zai.md)
+- [opencode.md](opencode.md)
+- [perplexity.md](perplexity.md)
+- [github-models.md](github-models.md)
+- [venice.md](venice.md)
+- [azure-openai.md](azure-openai.md)
+- [bedrock.md](bedrock.md)
+- [cloudflare-workers-ai.md](cloudflare-workers-ai.md)
+- [atlas-cloud.md](atlas-cloud.md)
+- [siliconflow.md](siliconflow.md)
+- [minimax.md](minimax.md)
+
+### Other local/proxy serving identities
+
+- [local-compatible-engines.md](local-compatible-engines.md): MLX LM, TGI,
+  LMDeploy, LiteLLM, and unknown compatible deployments.
+
+## Provider Spec Template
+
+Each provider file records:
+
+- provider identity and API dialects;
+- latest observed native catalog endpoint/envelope and capability-bearing fields;
+- whether current source has a dedicated reader or only generic fallback;
+- observed request, tool, text, reasoning, and control paths owned by runtime
+  adapters rather than the catalog reader;
+- what remains per-model/unknown;
+- Odysseus evidence and regressions;
+- fallback/safety behavior and current gaps.
+
+Marketing capability lists and curated picker lists may guide research but do
+not automatically become model claims. Provider-returned false values can be
+negative evidence only at the same provider/endpoint/model scope.

specs/model-providers/anthropic.md

@@ -0,0 +1,39 @@
+# Anthropic Provider Shape
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+Canonical placeholder vendor ID `anthropic`; Anthropic Messages runtime
+adapter in `src/llm_core.py`. There is no dedicated Anthropic capability-reader
+module; explicit/auto-detected Anthropic payloads use the generic identity-only
+reader.
+
+## Catalog Shape
+
+`GET /v1/models` returns `data[]` model resources with `id`, `type: model`,
+`display_name`, and `created_at`, plus pagination metadata. These fields prove
+identity/availability only. Do not assume all listed Claude models share
+vision, tools, reasoning, sampling, or context limits.
+
+## Request And Response Shape
+
+Native Messages uses a top-level `system`, alternating `messages`, content
+blocks, `tools[].input_schema`, `tool_use` assistant blocks, and `tool_result`
+user blocks. Text, thinking, signatures, server-tool blocks, and tool calls are
+typed content rather than OpenAI roles/fields. Preserve block IDs/signatures
+needed for continuation.
+
+Sampling and thinking support can be version/model specific. The Opus 4.7+ sampling omission is a model-scoped runtime observation, not an Anthropic-wide rule. Runtime version parsing accepts explicit major/minor IDs and later major-only IDs such as `claude-opus-5`, treats a missing minor as `.0`, caps both components so date stamps cannot be misread as versions, and keeps legacy Claude 3 Opus sampling intact. Anthropic-compatible proxies are Anthropic dialect only when configured or their exact payload/endpoint shape proves it (#3110).
+
+## Fallback And Safety
+
+Runtime and canonical reader detection use label-bounded Anthropic host matching or an explicit endpoint kind. A provider using Anthropic Messages through another host must be explicit. Identity-only model cards remain unknown.
+
+## Current Gaps
+
+- The public model list does not provide per-model canonical capability data.
+- There is no dedicated Anthropic canonical reader; only `id`, `name`, or
+  `model` identity survives generic normalization.
+- Runtime model-version parsing needs structured identity before a later
+  consumer can centralize sampling exceptions without another name matcher.

specs/model-providers/atlas-cloud.md

@@ -0,0 +1,21 @@
+# Atlas Cloud Provider Shape
+
+Last updated: dev@28d27ee | 2026-07-17
+
+## Scope
+
+Canonical provider ID `atlas_cloud`; OpenAI-compatible provider proposed in
+#5566 with live `/v1/models` observations for current Qwen/DeepSeek offerings.
+
+## Shape
+
+Treat the observed list as identity-only. Even capability-looking item fields
+remain raw until an Atlas-specific discriminating shape intentionally maps
+them. The model IDs observed by a PR demonstrate availability at that time,
+not permanent capability or a reason to hardcode family-name behavior.
+
+## Fallback And Current Gaps
+
+Exact Atlas Cloud host or explicit kind preserves identity; otherwise use the
+inventory fallback. The provider work is open/unmerged and has no independently
+versioned rich catalog schema, so evidence remains provisional.

specs/model-providers/azure-openai.md

@@ -0,0 +1,26 @@
+# Azure OpenAI Provider Shape
+
+Last updated: dev@28d27ee | 2026-07-17
+
+## Scope
+
+Canonical provider ID `azure_openai`; Azure deployment-scoped OpenAI dialects;
+custom endpoints use explicit configuration.
+
+## Shape
+
+Azure commonly identifies deployments rather than globally stable model IDs.
+Preserve endpoint, deployment ID, API version, and underlying model/version as
+separate structured identity when returned. A standard OpenAI-compatible model
+list is identity-only until an Azure-specific reader intentionally maps its
+deployment fields.
+
+Request paths and authentication can be deployment/API-version specific; do
+not blindly append public OpenAI paths or copy provider quirks. Capability and
+limits are deployment scoped.
+
+## Fallback And Current Gaps
+
+Known `*.openai.azure.com` hosts select Azure OpenAI; other Azure gateways need
+explicit kind. Odysseus lacks a native Azure deployment catalog reader and
+structured API-version persistence in the canonical record.

specs/model-providers/bedrock.md

@@ -0,0 +1,23 @@
+# AWS Bedrock Provider Shape
+
+Last updated: dev@28d27ee | 2026-07-17
+
+## Scope
+
+Canonical provider ID `bedrock`; UI/provider mapping currently recognizes AWS
+Bedrock, but the canonical layer has no native Bedrock runtime reader.
+
+## Shape
+
+Bedrock is not generally an OpenAI-compatible host: model IDs, inference
+profiles, request/response unions, signing, and per-family payloads differ.
+Only an explicitly configured OpenAI/Anthropic-compatible gateway may use those
+dialects. Native Bedrock capability must come from a versioned Bedrock model
+catalog plus exact foundation-model/inference-profile identity.
+
+## Fallback And Current Gaps
+
+Do not classify all `amazonaws.com` hosts as Bedrock; use explicit kind or a
+future region-aware exact host/path shape. General fallback is safe only behind
+an explicitly compatible gateway. Native signing, catalogs, and family payload
+mappings remain unimplemented.

specs/model-providers/cerebras.md

@@ -0,0 +1,23 @@
+# Cerebras Provider Shape
+
+Last updated: dev@28d27ee | 2026-07-17
+
+## Scope
+
+Canonical provider ID `cerebras`; OpenAI-compatible cloud transport; runtime
+provider detection and cache-affinity safeguards in `src/llm_core.py`.
+
+## Shape And Observations
+
+Model lists use the general identity-only inventory reader. Cerebras rejects
+llama.cpp-only `session_id` and `cache_prompt` fields (#4640), so cloud identity
+must suppress local slot-affinity extensions. Current regressions pin this
+provider boundary.
+
+Tool, reasoning, structured output, and limits remain per model. Do not promote
+them from the fact that the API accepts OpenAI Chat.
+
+## Fallback And Current Gaps
+
+Exact `*.cerebras.ai` selects provider identity. Compatible proxies require
+explicit configuration. No rich per-model Cerebras catalog reader is present.

specs/model-providers/chatgpt-subscription.md

@@ -0,0 +1,47 @@
+# ChatGPT Subscription Provider Shape
+
+Last updated: dev@e57f60b | 2026-07-20
+
+## Scope
+
+Canonical provider ID `chatgpt_subscription`; Codex Responses transport;
+auth and runtime code in `src/chatgpt_subscription.py`,
+`routes/chatgpt_subscription_routes.py`, and `src/llm_core.py`.
+There is no dedicated ChatGPT Subscription canonical reader on current `dev`.
+
+## Catalog Shape
+
+The account-scoped Codex models endpoint returns root `models[]`; `slug` is the
+request identity and `visibility`/`priority` control availability/order. These
+fields do not prove tools, reasoning, vision, or context. Null/malformed model
+lists fail soft rather than crashing discovery (#5280/#5281).
+
+The canonical generic reader does not accept `slug`-only items, so this runtime
+catalog is not currently normalized into `ModelCapabilityRecord` values.
+
+## Request And Response Shape
+
+Transport uses a ChatGPT backend Responses endpoint, `input` items, flattened
+function tools, streamed function-call argument events, exact `call_id`, and
+`function_call_output` continuation. Parallel calls and encrypted reasoning
+continuity require preserving typed output/history rather than coercing all
+roles to text. This shape is supported by the existing adapter and the focused
+tool-calling follow-up evidence in #5490; unmerged observations remain claimed
+until integrated/reproduced.
+
+OAuth/device credentials and refresh are provider-session behavior. Expired
+credentials should return an actionable reconnect error, not generic model
+failure.
+
+## Fallback And Safety
+
+Only the explicit internal base/ChatGPT host selects this provider. Never send
+subscription credentials to a custom OpenAI-compatible URL. Catalog slugs stay
+identity-only unless account-scoped fields or probes supply capability.
+
+## Current Gaps
+
+- Comprehensive Responses tool/reasoning parity is still evolving.
+- Account model slugs are not consumed by the canonical reader package.
+- The account catalog does not currently provide a complete canonical
+  capability card for every slug.

specs/model-providers/cloudflare-workers-ai.md

@@ -0,0 +1,21 @@
+# Cloudflare Workers AI Provider Shape
+
+Last updated: dev@28d27ee | 2026-07-17
+
+## Scope
+
+Canonical provider ID `cloudflare_workers_ai`; OpenAI-compatible Workers AI
+endpoint observations in #5175; explicit provider configuration required.
+
+## Shape
+
+Cloudflare account/path identity is part of the endpoint. Use the general
+OpenAI-compatible inventory reader for returned model cards, preserving full
+model IDs but no capability fields.
+Do not identify the provider from broad `api.cloudflare.com` alone or infer
+capability from Workers AI catalog prose.
+
+## Fallback And Current Gaps
+
+Provider identity must be explicit until a narrow account/AI path matcher is
+implemented. There is no rich normalized capability catalog reader.

specs/model-providers/cohere.md

@@ -0,0 +1,56 @@
+# Cohere Provider Shape
+
+Last updated: dev@e57f60b | 2026-07-20
+
+## Scope
+
+Documented provider identity `cohere`; native Chat v2 plus the OpenAI
+Compatibility API. Current `dev` has no dedicated Cohere capability reader or
+direct Cohere request adapter; compatible endpoints use the general runtime
+path when explicitly configured.
+
+## Catalog Shape
+
+`GET /v1/models` returns a paginated `models[]` envelope. Each model can carry
+`name`, `endpoints`, `default_endpoints`, `context_length`, `features`, and
+`sampling_defaults`; the root can carry `next_page_token`.
+
+These are candidate fields for a future dedicated reader:
+
+- a single canonical family from `endpoints`: `chat`/`generate`, `embed`,
+  `rerank`, or `classify`;
+- `context_length` to the endpoint/model context limit;
+- known sampling-default keys to deterministic controls.
+
+Current canonical normalization does not map them. When the generic reader is
+explicitly selected with vendor `cohere`, it preserves only item identity plus
+the raw item; family, context, features, and sampling controls stay unknown.
+
+## Request And Response Shape
+
+Native `POST /v2/chat` uses `messages`, structured content blocks, tools,
+`response_format`, sampling fields, and an optional structured `thinking`
+object. Text lives in `message.content[type=text].text`; reasoning-capable
+models use `message.content[type=thinking].thinking`. Streaming uses typed
+events rather than one generic text delta.
+
+The OpenAI compatibility base is `/compatibility/v1`. Its current chat subset
+includes tools, structured output, sampling, and `reasoning_effort`, but model
+support remains per-model. In the compatibility dialect only `none` and `high`
+currently map to native thinking off/on; do not assume low/medium support.
+
+## Fallback And Safety
+
+No Cohere host or payload-shape detection exists in the canonical reader
+registry. The caller must supply provider/endpoint configuration. Marketing
+pages and provider-wide endpoint features do not grant every listed model
+tools, vision, or reasoning.
+
+## Evidence And Gaps
+
+- Official List/Get Models resources define the catalog fields.
+- Official Chat v2, Reasoning, and Compatibility API resources define the
+  transport and thinking controls.
+- Odysseus has no direct Cohere request adapter, canonical reader, or sanitized
+  canonical fixtures yet; both normalization and runtime integration remain
+  follow-up work.

specs/model-providers/deepseek.md

@@ -0,0 +1,30 @@
+# DeepSeek Provider Shape
+
+Last updated: dev@2e2bb52 | 2026-08-16
+
+## Scope
+
+Canonical provider ID `deepseek`; official cloud OpenAI-compatible API;
+curation/detection in `routes/model_routes.py` and runtime reasoning handling in
+`src/llm_core.py`.
+
+## Shape And Observations
+
+Use the general model-list inventory shape; capability-looking fields remain
+unknown until a DeepSeek-native reader maps them. Cloud response history can use
+`reasoning_content`; preserve it structurally for reasoning turns and tool
+continuation (#968, #3152). `deepseek-chat`, reasoning models, distilled local
+variants, and future V4 models do not share one capability record.
+
+Cloud endpoint evidence can support tools while a local DeepSeek-R1 deployment
+may not have a working tool parser. Existing tool-support tests intentionally
+separate official host from local engine/model-name heuristics.
+
+Current runtime thinking-pattern detection includes DeepSeek V4 identifiers so their structured reasoning channel is handled like the other supported DeepSeek reasoning families. This name-level compatibility rule is not canonical capability evidence and does not make every V4-labelled local deployment tool-capable.
+
+## Fallback And Current Gaps
+
+Exact `*.deepseek.com` selects provider identity; self-hosted checkpoints use
+Ollama/vLLM/SGLang/llama.cpp identity. Curated model IDs and pricing/context
+tables are compatibility data, not authoritative capability. A rich official
+model-card reader is still absent.

specs/model-providers/fireworks.md

@@ -0,0 +1,22 @@
+# Fireworks AI Provider Shape
+
+Last updated: dev@28d27ee | 2026-07-17
+
+## Scope
+
+Canonical provider ID `fireworks`; OpenAI-compatible cloud transport with path
+prefixes such as `/inference/v1`; curation and URL handling in
+`routes/model_routes.py` and `src/endpoint_resolver.py`.
+
+## Shape
+
+Use the general identity-only inventory reader. Fireworks IDs can contain
+account/model paths; preserve the full request ID and endpoint scope. Item
+modalities, supported parameters, task/type, and limits require a
+Fireworks-native mapped shape before promotion.
+
+## Fallback And Current Gaps
+
+Exact `*.fireworks.ai` preserves provider identity and its configured path
+prefix. Do not normalize account-qualified IDs by taking the last path segment.
+No verified rich Fireworks capability catalog is currently mapped.

specs/model-providers/github-copilot.md

@@ -0,0 +1,46 @@
+# GitHub Copilot Provider Shape
+
+Last updated: dev@e57f60b | 2026-07-20
+
+## Scope
+
+Canonical provider ID `copilot`; OpenAI-compatible chat with Copilot headers
+and OAuth; runtime adapter `src/copilot.py` and routes in
+`routes/copilot_routes.py`. There is no dedicated Copilot canonical reader on
+current `dev`.
+
+## Catalog Shape
+
+The observed Copilot `/models` response uses `data[]` entries with:
+
+- `id`;
+- `model_picker_enabled`;
+- `capabilities.supports.tool_calls` and `.vision`;
+- optional limit/family metadata.
+
+Runtime model discovery uses picker state for availability. The canonical
+reader package does not map the nested support fields; an explicitly supplied
+`copilot` vendor currently uses generic identity-only normalization, and
+`model_picker_enabled` does not become canonical capability.
+
+## Request And Response Shape
+
+Chat is OpenAI-compatible but requires Copilot/GitHub API version, editor/plugin
+identity, intent, integration, and initiator headers; image requests add the
+vision request flag. Header derivation must tolerate malformed message entries.
+OAuth token exchange and access policies are provider authentication, not model
+capability.
+
+## Fallback And Safety
+
+Use exact GitHub Copilot host or explicit kind, including the constrained
+enterprise `copilot-api.*.ghe.com` form. Do not treat arbitrary `ghe.com` hosts
+as Copilot. Official model availability tables are useful registry context but
+do not replace the account-scoped catalog response.
+
+## Current Gaps
+
+- The catalog shape is implementation-observed and needs ongoing fixture
+  comparison with current Copilot clients.
+- Copilot catalog capability fields are not normalized by current `dev`.
+- Account/plan/policy availability must remain endpoint-user scoped.

specs/model-providers/github-models.md

@@ -0,0 +1,21 @@
+# GitHub Models Provider Shape
+
+Last updated: dev@28d27ee | 2026-07-17
+
+## Scope
+
+Canonical provider ID `github_models`; OpenAI-compatible GitHub Models/Azure
+inference endpoint observed in #2995; distinct from GitHub Copilot.
+
+## Shape
+
+Use general identity-only inventory. Deployment IDs and account access
+can differ from upstream model IDs. Do not copy Copilot picker metadata,
+headers, plan rules, or capabilities into GitHub Models; they are separate
+providers despite shared GitHub branding.
+
+## Fallback And Current Gaps
+
+The known `models.inference.ai.azure.com` host selects GitHub Models. Other
+Azure deployment hosts require explicit provider configuration. No rich
+account-scoped capability catalog is currently mapped.

specs/model-providers/google.md

@@ -0,0 +1,54 @@
+# Google Gemini Provider Shape
+
+Last updated: dev@e71f8ce | 2026-08-25
+
+## Scope
+
+Canonical provider ID `google`; native GenerateContent plus optional Google
+OpenAI-compatible chat; readers `google.py` and
+`google_ai_studio_mapping.py`; catalog/probe ownership in
+`routes/model_routes.py`.
+
+## Catalog Shape
+
+Use the native paginated `GET /v1beta/models` endpoint, including
+`nextPageToken`, with `x-goog-api-key` when configured. `models[]` can contain:
+
+- `name`, `baseModelId`, `version`, and `displayName`;
+- `inputTokenLimit` and `outputTokenLimit`;
+- `supportedGenerationMethods`;
+- `thinking`, `temperature`, `maxTemperature`, `topP`, and `topK`.
+
+Embedding-only methods map to embedding. Generation methods prove a native
+method, not chat/image/video/audio modality, so those records remain unknown
+unless stronger structured evidence exists. `thinking: true` and explicit
+sampling fields map to a reasoning claim and controls. Model IDs such as
+Imagen, Veo, or TTS names are not parsed.
+
+## Request And Response Shape
+
+Native generation uses `contents`, `systemInstruction`,
+`generationConfig`, `tools[].functionDeclarations`, and
+`models/{model}:generateContent|streamGenerateContent`. Responses use
+`candidates[].content.parts[]` for `text`, `functionCall`, `functionResponse`,
+`thought`, and `thoughtSignature`; token accounting is in `usageMetadata`.
+Native Google tool/thought continuity must not be flattened through an
+OpenAI-only history shape.
+
+## Fallback And Safety
+
+Prefer native model metadata even when chat is configured through Google's
+OpenAI compatibility URL. Pagination parameters must remain stable between
+pages. The route probe activates only for the exact
+`generativelanguage.googleapis.com` hostname, filters the picker list to
+content-generation methods, returns no curated fallback after probe failure,
+and defaults those endpoints to manual catalog refresh unless explicitly
+overridden. The canonical Google reader is not yet called by that probe.
+Unknown methods and fields stay raw; unrecognized prediction models remain
+unknown.
+
+## Current Gaps
+
+- The Models resource does not expose full modalities for every Google media
+  family.
+- Native Gemini request/response support is not yet the only runtime path.

specs/model-providers/groq.md

@@ -0,0 +1,24 @@
+# Groq Provider Shape
+
+Last updated: dev@28d27ee | 2026-07-17
+
+## Scope
+
+Canonical provider ID `groq`; OpenAI-compatible cloud transport; detection and
+request behavior in `src/llm_core.py`.
+
+## Shape
+
+Model discovery falls back to the general `data[].id` identity shape. Richer
+fields require a Groq-native mapped shape even when the payload happens to
+supply modalities, supported parameters, or limits. Groq transport may accept OpenAI-style tools and streaming extensions,
+but support remains per model and account.
+
+Runtime currently exempts Groq/OpenRouter from some parameter stripping paths;
+that is transport compatibility, not a provider-wide model capability claim.
+
+## Fallback And Current Gaps
+
+Exact `*.groq.com` preserves Groq identity. Do not infer Llama/Gemma model
+capabilities from IDs. There is no canonical rich Groq model-card reader or
+freshness policy yet.

specs/model-providers/hugging-face.md

@@ -0,0 +1,41 @@
+# Hugging Face Provider And Registry Shape
+
+Last updated: dev@e57f60b | 2026-07-20
+
+## Scope
+
+Canonical placeholder vendor ID `huggingface`; download/fit metadata in
+`services/hwfit/`; OpenAI-compatible inference providers/TGI handled as their
+serving dialect. There is no dedicated Hugging Face canonical reader on
+current `dev`.
+
+## Hub Model Shape
+
+Hub model info can provide `modelId`/`id`, `pipeline_tag`, `tags`, `config`, and
+card metadata. Current canonical normalization does not map `pipeline_tag`,
+`config.model_type`, or Hub task/modality fields. An explicitly selected
+Hugging Face vendor uses generic identity-only normalization.
+
+This source is `cookbook_hf`/registry confidence, not live endpoint truth.
+Free-form tags, README/card prose, repository names, and architecture names do
+not automatically claim capability. A serving engine can load a model with
+missing projection, different template, or disabled parser.
+
+## Serving Shape
+
+Hugging Face routed inference and TGI can expose OpenAI-compatible endpoints;
+their model list may be identity-only. Keep Hub identity separate from the
+serving endpoint and merge only when exact revision/model identity is known.
+
+## Fallback And Safety
+
+Hub metadata can fill a scoped registry record after provider payload fields
+and probes, but must not overwrite fresh endpoint-negative evidence. Treat
+remote code, model cards, and repository files as untrusted content.
+
+## Current Gaps
+
+- Revision/digest linkage between downloads, Hub records, and serving
+  endpoints is incomplete.
+- Hub task/family metadata is not consumed by the canonical reader package.
+- Pipeline tags can be missing or overly broad; unknown stays unknown.

specs/model-providers/llama-cpp.md

@@ -0,0 +1,47 @@
+# llama.cpp Provider Shape
+
+Last updated: dev@e57f60b | 2026-07-20
+
+## Scope
+
+Canonical provider ID `llamacpp`; OpenAI Chat/Responses and Anthropic Messages
+compatibility plus native server metadata; reader
+`src/model_capability_readers/llamacpp.py`.
+
+## Metadata Shapes
+
+`/v1/models` provides served identity and can include server model entries;
+native `/props` is authoritative for the running model/server combination:
+
+- `model_alias`/`model_path`;
+- `default_generation_settings.n_ctx` and sampling `params`;
+- `total_slots` and optional `/slots[].n_ctx` fallback;
+- `chat_template_caps` for tools/system role;
+- `modalities.vision|audio`;
+- current server/build state.
+
+Capability depends on weights, projection/model assets, chat template, parser,
+and launch flags. It is endpoint evidence, not a checkpoint-name claim.
+`/props` and `/v1/models` can be merged only for the same served identity.
+
+## Request And Response Shape
+
+llama-server supports several OpenAI-compatible tasks and native extensions.
+Do not infer embeddings/rerank/chat solely from the OpenAI model card; use an
+explicit server model capability field or endpoint configuration. Tool and
+reasoning correctness can depend on selected chat template and parser.
+
+## Fallback And Safety
+
+The registry selects llama.cpp through an explicit vendor or endpoint kind; it
+does not auto-detect `/props` from payload shape. Port 8000 currently maps to
+the vLLM placeholder, while 8080 falls through to generic OpenAI-compatible.
+llama.cpp-only `session_id` and `cache_prompt` affinity fields must remain local
+endpoint behavior and never leak to strict cloud providers (#4640 and current
+affinity tests).
+
+## Current Gaps
+
+- Multi-model routing requires per-served-model `/props` association.
+- Parser/template configuration is not yet fully represented in canonical
+  endpoint metadata.

specs/model-providers/lm-studio.md

@@ -0,0 +1,45 @@
+# LM Studio Provider Shape
+
+Last updated: dev@e57f60b | 2026-07-20
+
+## Scope
+
+Canonical provider ID `lmstudio`; native LM Studio v1 plus OpenAI Chat and
+Responses compatibility; reader `src/model_capability_readers/lmstudio.py`.
+
+## Catalog Shapes
+
+Preferred shape is `GET /api/v1/models` with root `models[]`. Current fields
+include `key`, `type` (`llm` or `embedding`), display/publisher data,
+`architecture`, quantization/format/size, `max_context_length`,
+`loaded_instances[].config.context_length`, and a capability object containing
+`vision`, `trained_for_tool_use`, and reasoning options/defaults.
+
+Compatibility shape `GET /api/v0/models` uses `data[]` with `id`, `type`
+(`llm`, `vlm`, or embeddings), `arch`, `compatibility_type`, state, and
+context metadata. It is an explicit older shape, not a loose fallback.
+OpenAI `/v1/models` is identity-only when native endpoints are unavailable.
+
+Loaded-instance context is the effective runtime context; maximum context is a
+separate limit. Model type maps family, explicit capability booleans map
+vision/tools/reasoning, and architecture is provider-reported model family.
+
+## Request And Response Shape
+
+Native v1 chat is `/api/v1/chat` and can expose stateful/MCP-oriented output;
+LM Studio also supports OpenAI Chat and Responses compatibility. Keep dialect
+selection explicit because tool/MCP features differ between native and
+compatible paths.
+
+## Fallback And Safety
+
+Current reader detection identifies port 1234 as LM Studio. Prefer pathless
+native `/api/v1/models` discovery where configured (#1122, #3615), then v0,
+then general identity. The port mapping is a normalization hint, not endpoint
+trust. An error object from an unsupported native route is not a model list.
+
+## Current Gaps
+
+- Runtime discovery does not yet persist native capability records.
+- LM Studio API capabilities continue to evolve; each new native version needs
+  an explicit shape fixture before promotion.

specs/model-providers/local-compatible-engines.md

@@ -0,0 +1,37 @@
+# Other Local And Proxy Compatible Engines
+
+Last updated: dev@28d27ee | 2026-07-17
+
+## Scope
+
+Canonical explicit identities `mlx_lm`, `text_generation_inference`,
+`lmdeploy`, and `litellm`, plus unknown OpenAI-compatible deployments not
+covered by the native Ollama, LM Studio, llama.cpp, vLLM, or SGLang specs.
+
+## Shape
+
+Use explicit endpoint kind when known; otherwise use only the general model
+list envelopes for inventory identity. Capability-looking structural fields
+remain raw. Local host and port do not distinguish these engines.
+MLX/Cookbook launch recipes, TGI task configuration, LMDeploy
+adapters, and LiteLLM upstream routing can all change capability independently
+of the model ID.
+
+Proxy model aliases are endpoint scoped. A proxy may return richer fields, but
+unknown keys remain raw until a versioned shape is added. Provider-specific
+headers/extensions must not be applied based on a port or upstream model name.
+
+## Fallback And Safety
+
+Discovery can probe cheap native identity endpoints when available, but
+capability probes execute only explicit bounded test contracts. Never read
+broad server/environment dumps as ordinary model metadata. Unknown compatible
+servers should still list identities and make conservative text calls where
+explicitly configured, without appearing on capability-gated surfaces.
+
+## Current Gaps
+
+- These engines need individual safe metadata fixtures before they can graduate
+  from general fallback.
+- Gateway upstream identity and effective downstream model capability are not
+  yet represented as a chain.

specs/model-providers/minimax.md

@@ -0,0 +1,48 @@
+# MiniMax Provider Shape
+
+Last updated: dev@28d27ee | 2026-07-17
+
+## Scope
+
+Canonical provider ID `minimax`; international host `api.minimax.io`, China
+host `api.minimaxi.com`; current OpenAI-compatible and recommended
+Anthropic-compatible text transports. Odysseus contains MiniMax-oriented tool
+output handling and local-serving guidance but no dedicated catalog reader.
+
+## Catalog Shape
+
+Current `GET /v1/models` is an OpenAI-compatible identity list:
+`object: list`, `data[]`, and model cards containing `id`, `object: model`,
+`created`, and `owned_by: minimax`. The `owned_by` discriminator identifies the
+provider shape, but the card exposes no per-model capability or modality
+fields. Keep these records unknown and preserve raw identity metadata.
+
+Do not backfill current model capabilities, token limits, or modalities from
+the platform overview into this list response. Those tables are useful scoped
+registry evidence only after model/version identity and freshness are carried
+explicitly.
+
+## Request And Response Shape
+
+- OpenAI compatibility uses `/v1/chat/completions` and structured
+  `reasoning_content` alongside normal message content.
+- Anthropic compatibility uses `/anthropic/v1/messages`; the current M2.7
+  family supports typed thinking blocks and interleaved thinking, making this
+  the preferred reasoning/tool-continuation transport in provider guidance.
+- Native audio, image, video, music, and file endpoints are separate product
+  shapes. They must not be inferred from presence in the text model list.
+
+## Local Deployments
+
+The current provider guide documents vLLM, SGLang, and MLX deployment. Those
+instances retain serving-engine identity and configuration-derived capability;
+the checkpoint name alone does not turn a vLLM/SGLang card into the hosted
+MiniMax provider shape.
+
+## Fallback And Current Gaps
+
+Exact MiniMax hosts or the discriminating `owned_by: minimax` model-list shape
+select provider identity. Unknown compatible proxies retain the general shape.
+The identity list does not safely distinguish M2 reasoning behavior from
+speech/image/video/music products, so exact model quirks remain documentation
+until structured model-version evidence reaches runtime request builders.