f2d047ff06
Addresses a perf issue found while demoing the github_repo seed reader: Scraper is too slow at scale. The PRs GraphQL query pulls deeply nested fields (reviewThreads, reviews, commits, timelineItems, etc.) so the page size was pinned at 3 to stay under GitHub's node-count ceiling. 100 PRs meant 34 serial round trips. Added lighter query variants (PRS_PAGE_QUERY_LIGHT, ISSUES_PAGE_QUERY_LIGHT) that drop the fields the Studio flatten layer does not use (it only reads title, body, state, author, labels, comments). With the light query PR pages can safely go to 25 per page and issues to 50. The plugin scraper now passes light=True to RepoScraper so Studio always uses the fast path; the heavy query remains available for other callers. Recipe defaults are now demo-ready with production knobs called out: - max_parallel_requests: 1 and max_tokens: 800 so small local models stay stable when running the support_answer structured column. - support_answer prompt trimmed to 80-200 words so gemma-4-E2B GGUF can actually comply with the schema. The canonical 150-300 word codex prompt is still documented in the node3 markdown note for production upgrades. |
||
|---|---|---|
| .. | ||
| src/data_designer_github_repo_seed | ||
| pyproject.toml | ||
| README.md | ||
data-designer-github-repo-seed
A Data Designer seed-reader plugin for Unsloth Studio that scrapes real GitHub data (issues, pull requests, commits) from one or more repositories and hands it to the recipe pipeline as a seed dataset.
Designed to ship with Studio as a default seed source so any user with a GitHub token can build training datasets straight from live repos.
What it does
Given a list of owner/name repos, a GitHub token, and a per-resource
limit, the plugin uses GitHub's GraphQL API to fetch issues, pull
requests, and/or commits, with labels, state, authors, and the first N
comments of each item, and materialises a single JSONL with uniform
columns so the rest of the recipe (LLM text / LLM structured / processors)
can treat it like any other seed table.
| Column | Description |
|---|---|
item_type |
issue / pull / commit |
repo |
owner/name |
number |
Issue/PR number, or commit SHA |
title |
Title (or commit message headline) |
body |
Issue/PR body (or full commit message) |
state |
OPEN / CLOSED / MERGED (empty for commit) |
author |
GitHub login of the author |
created_at |
ISO8601 |
closed_at |
ISO8601 (empty for commits) |
url |
Permalink |
labels |
List of label names |
comments |
First N comments concatenated |
Usage in a recipe
{
"seed_config": {
"source": {
"seed_type": "github_repo",
"repos": ["unslothai/unsloth", "unslothai/unsloth-zoo"],
"token": "",
"item_types": ["issues", "pulls"],
"limit": 100,
"include_comments": true,
"max_comments_per_item": 30
},
"sampling_strategy": "shuffle",
"selection_strategy": null
}
}
Leave token empty to fall back to the server's GH_TOKEN / GITHUB_TOKEN
environment variable, useful when the recipe is published and shouldn't
carry a secret.
Auth
A GitHub personal access token with public_repo scope is enough for public
repositories; repo scope is required for private ones. GraphQL requests
are rate-limit aware: the client inspects x-ratelimit-* headers and
sleeps until reset when the budget drops below a safety threshold.
Install
Shipped as a default Studio plugin. For development:
pip install -e .
Registered automatically via the data_designer.plugins entry point.