mirror of
https://github.com/PrefectHQ/fastmcp.git
synced 2026-08-09 07:09:11 +02:00
Add real-world schema crash test against APIs.guru directory (#3826)
Integration test that runs json_schema_to_type against 232K schemas
from 4,120 real-world OpenAPI specs (APIs.guru openapi-directory).
Snapshots crash counts as regression baselines so future changes
can't silently increase the crash rate.
Current baseline (openapi-directory@f7207cf0):
TypeErrors: 2,342 (datetime serialization)
SchemaErrors: 273 (invalid regexes in specs)
Timeouts: 0
Other: 0
Skipped unless openapi-directory is cloned locally.
Run with: pytest -m integration tests/.../test_real_world_schemas.py
🤖 Generated with Claude Code
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
eec52b1f02
commit
468559978a
3 changed files with 463 additions and 3 deletions
55
.github/workflows/run-schema-crash-test.yml
vendored
Normal file
55
.github/workflows/run-schema-crash-test.yml
vendored
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
name: Schema Crash Test
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: ["main"]
|
||||
paths:
|
||||
- "src/fastmcp/utilities/json_schema_type.py"
|
||||
- "src/fastmcp/utilities/json_schema.py"
|
||||
- "src/fastmcp/utilities/openapi/**"
|
||||
- "src/fastmcp/server/providers/openapi/**"
|
||||
- "src/fastmcp/client/mixins/tools.py"
|
||||
- "tests/utilities/json_schema_type/test_real_world_schemas.py"
|
||||
- ".github/workflows/run-schema-crash-test.yml"
|
||||
|
||||
pull_request:
|
||||
paths:
|
||||
- "src/fastmcp/utilities/json_schema_type.py"
|
||||
- "src/fastmcp/utilities/json_schema.py"
|
||||
- "src/fastmcp/utilities/openapi/**"
|
||||
- "src/fastmcp/server/providers/openapi/**"
|
||||
- "src/fastmcp/client/mixins/tools.py"
|
||||
- "tests/utilities/json_schema_type/test_real_world_schemas.py"
|
||||
- ".github/workflows/run-schema-crash-test.yml"
|
||||
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
schema_crash_test:
|
||||
name: "Real-world schema crash test (232K schemas)"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 45
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
|
||||
- name: Set up Python
|
||||
run: uv python install 3.12
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync
|
||||
|
||||
- name: Clone openapi-directory
|
||||
run: git clone --depth 1 https://github.com/APIs-guru/openapi-directory.git /tmp/openapi-directory
|
||||
|
||||
- name: Run schema crash test
|
||||
env:
|
||||
RUN_REAL_WORLD_SCHEMA_TEST: "1"
|
||||
OPENAPI_DIRECTORY_PATH: /tmp/openapi-directory
|
||||
run: uv run pytest tests/utilities/json_schema_type/test_real_world_schemas.py -m integration -v -s --timeout-method=thread
|
||||
Loading…
Add table
Add a link
Reference in a new issue