Add real-world schema crash test against APIs.guru directory (#3826)

Integration test that runs json_schema_to_type against 232K schemas
from 4,120 real-world OpenAPI specs (APIs.guru openapi-directory).
Snapshots crash counts as regression baselines so future changes
can't silently increase the crash rate.

Current baseline (openapi-directory@f7207cf0):
  TypeErrors:   2,342 (datetime serialization)
  SchemaErrors:   273 (invalid regexes in specs)
  Timeouts:         0
  Other:            0

Skipped unless openapi-directory is cloned locally.
Run with: pytest -m integration tests/.../test_real_world_schemas.py

🤖 Generated with Claude Code

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Bill Easton 2026-04-11 10:23:55 -05:00 committed by GitHub
commit 468559978a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 463 additions and 3 deletions

View file

@ -0,0 +1,55 @@
name: Schema Crash Test
on:
push:
branches: ["main"]
paths:
- "src/fastmcp/utilities/json_schema_type.py"
- "src/fastmcp/utilities/json_schema.py"
- "src/fastmcp/utilities/openapi/**"
- "src/fastmcp/server/providers/openapi/**"
- "src/fastmcp/client/mixins/tools.py"
- "tests/utilities/json_schema_type/test_real_world_schemas.py"
- ".github/workflows/run-schema-crash-test.yml"
pull_request:
paths:
- "src/fastmcp/utilities/json_schema_type.py"
- "src/fastmcp/utilities/json_schema.py"
- "src/fastmcp/utilities/openapi/**"
- "src/fastmcp/server/providers/openapi/**"
- "src/fastmcp/client/mixins/tools.py"
- "tests/utilities/json_schema_type/test_real_world_schemas.py"
- ".github/workflows/run-schema-crash-test.yml"
workflow_dispatch:
permissions:
contents: read
jobs:
schema_crash_test:
name: "Real-world schema crash test (232K schemas)"
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v6
- name: Set up Python
run: uv python install 3.12
- name: Install dependencies
run: uv sync
- name: Clone openapi-directory
run: git clone --depth 1 https://github.com/APIs-guru/openapi-directory.git /tmp/openapi-directory
- name: Run schema crash test
env:
RUN_REAL_WORLD_SCHEMA_TEST: "1"
OPENAPI_DIRECTORY_PATH: /tmp/openapi-directory
run: uv run pytest tests/utilities/json_schema_type/test_real_world_schemas.py -m integration -v -s --timeout-method=thread