f2d047ff06
Addresses a perf issue found while demoing the github_repo seed reader: Scraper is too slow at scale. The PRs GraphQL query pulls deeply nested fields (reviewThreads, reviews, commits, timelineItems, etc.) so the page size was pinned at 3 to stay under GitHub's node-count ceiling. 100 PRs meant 34 serial round trips. Added lighter query variants (PRS_PAGE_QUERY_LIGHT, ISSUES_PAGE_QUERY_LIGHT) that drop the fields the Studio flatten layer does not use (it only reads title, body, state, author, labels, comments). With the light query PR pages can safely go to 25 per page and issues to 50. The plugin scraper now passes light=True to RepoScraper so Studio always uses the fast path; the heavy query remains available for other callers. Recipe defaults are now demo-ready with production knobs called out: - max_parallel_requests: 1 and max_tokens: 800 so small local models stay stable when running the support_answer structured column. - support_answer prompt trimmed to 80-200 words so gemma-4-E2B GGUF can actually comply with the schema. The canonical 150-300 word codex prompt is still documented in the node3 markdown note for production upgrades. |
||
|---|---|---|
| .. | ||
| data-designer-github-repo-seed | ||
| data-designer-unstructured-seed | ||
| __init__.py | ||