Studio: add github_repo seed reader and GitHub Support Bot recipe

Adds a first-party Data Designer seed reader that scrapes GitHub issues,
pull requests, and commits from one or more repositories via the GraphQL
API, and a learning recipe (GitHub Support Bot) that turns those rows into
synthetic support Q&A pairs for fine-tuning.

Backend (new plugin studio/backend/plugins/data-designer-github-repo-seed):
* GitHubRepoSeedSource config: repos, token (falls back to GH_TOKEN /
  GITHUB_TOKEN env var), item_types (issues / pulls / commits),
  per-resource limit (0 means all), max_comments_per_item.
* Rate-limit-aware GraphQL client (GitHubClient + RepoScraper) shared
  across repos; flattens each item into a uniform row with columns
  item_type, repo, number, title, body, state, author, created_at,
  closed_at, url, labels, comments.
* Registered via the data_designer.plugins entry point.

Frontend:
* New seed_github block variant so the seed node card shows
  "GitHub repositories" instead of the generic "Document file"
  placeholder, with its own icon and inline summary (repo count +
  item-type list).
* Rewritten seed dialog github_repo form: repos textarea pre-filled with
  unslothai/unsloth + unslothai/unsloth-zoo, password input for the GH
  token, items-per-repo number with an "All" toggle, and the noisier
  options (item types, max comments, include comments) tucked under an
  Advanced collapsible.
* Local model auto-load on Run: if a recipe uses an is_local provider
  and the inference server is not already serving that model, the
  executions hook calls /api/inference/load first. Removes the "open
  /chat to load a model" prerequisite that users kept tripping on.
* Honor the recipe's run.rows value in the Run dialog (previously the
  store reset to 5 regardless of what the template shipped).

Recipe (studio/frontend/src/features/data-recipes/learning-recipes/
github-support-bot.json):
* Defaults to the Local Model provider + unsloth/gemma-4-E2B-it-GGUF.
* Scrapes unslothai/unsloth and unslothai/unsloth-zoo, issues and pulls,
  up to 100 items per resource.
* Two LLM blocks: normalized_question (llm-text) rewrites each thread
  into a clean support question, support_answer (llm-structured)
  produces JSON with answer / diagnosis_questions / cites / confidence.
* Run defaults to 10 rows for a quick smoke test.

Verified end-to-end on a running Studio: card renders, source-data
dialog is pre-populated, All toggle disables the limit input, the
recipe executes and produces rows against a loaded local GGUF.
This commit is contained in:
Daniel Han 2026-04-24 13:44:30 +00:00
commit d8cd2f8693
25 changed files with 2426 additions and 33 deletions

View file

@ -0,0 +1,73 @@
# data-designer-github-repo-seed
A Data Designer seed-reader plugin for **Unsloth Studio** that scrapes real
GitHub data (issues, pull requests, commits) from one or more repositories
and hands it to the recipe pipeline as a seed dataset.
Designed to ship with Studio as a default seed source so any user with a
GitHub token can build training datasets straight from live repos.
## What it does
Given a list of `owner/name` repos, a GitHub token, and a per-resource
`limit`, the plugin uses GitHub's GraphQL API to fetch issues, pull
requests, and/or commits, with labels, state, authors, and the first N
comments of each item, and materialises a single JSONL with uniform
columns so the rest of the recipe (LLM text / LLM structured / processors)
can treat it like any other seed table.
| Column | Description |
|---------------|------------------------------------------------|
| `item_type` | `issue` / `pull` / `commit` |
| `repo` | `owner/name` |
| `number` | Issue/PR number, or commit SHA |
| `title` | Title (or commit message headline) |
| `body` | Issue/PR body (or full commit message) |
| `state` | `OPEN` / `CLOSED` / `MERGED` (empty for commit)|
| `author` | GitHub login of the author |
| `created_at` | ISO8601 |
| `closed_at` | ISO8601 (empty for commits) |
| `url` | Permalink |
| `labels` | List of label names |
| `comments` | First N comments concatenated |
## Usage in a recipe
```json
{
"seed_config": {
"source": {
"seed_type": "github_repo",
"repos": ["unslothai/unsloth", "unslothai/unsloth-zoo"],
"token": "",
"item_types": ["issues", "pulls"],
"limit": 100,
"include_comments": true,
"max_comments_per_item": 30
},
"sampling_strategy": "shuffle",
"selection_strategy": null
}
}
```
Leave `token` empty to fall back to the server's `GH_TOKEN` / `GITHUB_TOKEN`
environment variable, useful when the recipe is published and shouldn't
carry a secret.
## Auth
A GitHub personal access token with `public_repo` scope is enough for public
repositories; `repo` scope is required for private ones. GraphQL requests
are rate-limit aware: the client inspects `x-ratelimit-*` headers and
sleeps until reset when the budget drops below a safety threshold.
## Install
Shipped as a default Studio plugin. For development:
```bash
pip install -e .
```
Registered automatically via the `data_designer.plugins` entry point.

View file

@ -0,0 +1,24 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "data-designer-github-repo-seed"
version = "0.1.0"
description = "Unsloth Studio seed plugin that scrapes GitHub issues, PRs, and commits."
requires-python = ">=3.11"
dependencies = [
"requests>=2.31",
]
[project.entry-points."data_designer.plugins"]
github_repo_seed = "data_designer_github_repo_seed.plugin:github_repo_seed_plugin"
[tool.setuptools]
package-dir = {"" = "src"}
[tool.setuptools.packages.find]
where = ["src"]

View file

@ -0,0 +1,7 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
# Intentionally empty. Data-designer loads submodules lazily via qualified names
# (impl_qualified_name / config_qualified_name in plugin.py), so importing this
# package must NOT touch modules that depend on data_designer.engine.* during
# Studio's bootstrap (circular import).

View file

@ -0,0 +1,64 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
from __future__ import annotations
from typing import Literal
from pydantic import Field, field_validator, model_validator
from data_designer.config.seed_source import SeedSource
class GitHubRepoSeedSource(SeedSource):
seed_type: Literal["github_repo"] = "github_repo"
repos: list[str] = Field(
default_factory=list,
description="List of GitHub repositories to scrape, each in `owner/name` form.",
)
token: str = Field(
default="",
description="Personal access token. Leave blank to read GH_TOKEN / GITHUB_TOKEN from env at run time.",
)
item_types: list[Literal["issues", "pulls", "commits"]] = Field(
default=["issues", "pulls"],
description="Which GitHub item types to fetch per repo.",
)
limit: int = Field(
default=100,
ge=1,
le=5000,
description="Maximum items per repo per item type (e.g. limit=100 + ['issues','pulls'] => up to 200 items per repo).",
)
include_comments: bool = Field(
default=True,
description="Fetch the first N comments of each issue/PR and include them in the `comments` column.",
)
max_comments_per_item: int = Field(default=30, ge=0, le=200)
@field_validator("repos")
@classmethod
def _validate_repos(cls, v: list[str]) -> list[str]:
out: list[str] = []
for r in v or []:
r = r.strip()
if not r:
continue
if r.count("/") != 1 or not all(r.split("/")):
raise ValueError(f"Each repo must be `owner/name`; got {r!r}")
out.append(r)
return out
@field_validator("item_types")
@classmethod
def _validate_item_types(cls, v: list[str]) -> list[str]:
if not v:
raise ValueError("item_types must not be empty")
return list(dict.fromkeys(v))
@model_validator(mode="after")
def _ensure_repos(self) -> "GitHubRepoSeedSource":
if not self.repos:
raise ValueError("At least one repo is required")
return self

View file

@ -0,0 +1,31 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
from __future__ import annotations
import tempfile
from pathlib import Path
import data_designer.lazy_heavy_imports as lazy
from data_designer.engine.resources.seed_reader import SeedReader
from .config import GitHubRepoSeedSource
from .scraper import ScrapeConfig, materialize_to_jsonl
class GitHubRepoSeedReader(SeedReader[GitHubRepoSeedSource]):
def create_duckdb_connection(self):
return lazy.duckdb.connect()
def get_dataset_uri(self) -> str:
out_dir = Path(tempfile.gettempdir()) / "studio-github-repo-seed"
cfg = ScrapeConfig(
repos=list(self.source.repos),
token=self.source.token,
item_types=list(self.source.item_types),
limit=self.source.limit,
include_comments=self.source.include_comments,
max_comments_per_item=self.source.max_comments_per_item,
)
path = materialize_to_jsonl(cfg, out_dir)
return str(path)

View file

@ -0,0 +1,10 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
from data_designer.plugins.plugin import Plugin, PluginType
github_repo_seed_plugin = Plugin(
impl_qualified_name="data_designer_github_repo_seed.impl.GitHubRepoSeedReader",
config_qualified_name="data_designer_github_repo_seed.config.GitHubRepoSeedSource",
plugin_type=PluginType.SEED_READER,
)

View file

@ -0,0 +1,209 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""Multi-repo GitHub scraper for the Studio seed plugin.
Drives the GraphQL-based scraper in `scraper_impl/` per repo. Each repo is
scraped with a trial_limits cap so we stop at `limit` items per resource.
After scraping, we read the per-resource JSONL shards and flatten them into
a single unified JSONL with stable columns (`item_type`, `repo`, `number`,
`title`, `body`, ...).
"""
from __future__ import annotations
import json
import os
import sys
import time
from dataclasses import dataclass
from pathlib import Path
# The scraper_impl gh_client reads GH_TOKEN at import time and raises if
# missing, so we defer the imports until `scrape()` runs with a resolved token.
_IMPL_DIR = Path(__file__).parent / "scraper_impl"
def _ensure_impl_on_path() -> None:
if str(_IMPL_DIR) not in sys.path:
sys.path.insert(0, str(_IMPL_DIR))
def _load_impl():
_ensure_impl_on_path()
import importlib
gh_client = importlib.import_module("gh_client") # type: ignore
scraper_mod = importlib.import_module("scraper") # type: ignore
return gh_client.GitHubClient, scraper_mod.RepoScraper
@dataclass
class ScrapeConfig:
repos: list[str]
token: str
item_types: list[str]
limit: int
include_comments: bool
max_comments_per_item: int
def _resolve_token(token: str) -> str:
tok = token or os.environ.get("GH_TOKEN", "") or os.environ.get("GITHUB_TOKEN", "")
if not tok:
raise ValueError(
"GitHub token is required. Set it in the recipe config or the GH_TOKEN / GITHUB_TOKEN env var."
)
return tok
def _read_jsonl(path: Path, max_rows: int | None = None):
if not path.exists():
return
with path.open(encoding="utf-8") as f:
for i, line in enumerate(f):
if not line.strip():
continue
if max_rows is not None and i >= max_rows:
return
try:
yield json.loads(line)
except json.JSONDecodeError:
continue
def _flatten_issue_row(r: dict, repo: str, include_comments: bool, max_c: int) -> dict:
labels = [l.get("name") for l in (r.get("labels", {}) or {}).get("nodes", []) if l.get("name")]
comments_nodes = (r.get("comments") or {}).get("nodes") or []
comments_text = ""
if include_comments and comments_nodes:
kept = comments_nodes[:max_c]
comments_text = "\n\n".join(
f"[{(c.get('author') or {}).get('login', '?')}]: {c.get('body') or ''}"
for c in kept
)
return {
"item_type": "issue",
"repo": repo,
"number": r.get("number"),
"title": r.get("title") or "",
"body": r.get("body") or "",
"state": r.get("state") or "",
"author": (r.get("author") or {}).get("login", ""),
"created_at": r.get("createdAt") or "",
"closed_at": r.get("closedAt") or "",
"url": r.get("url") or r.get("permalink") or "",
"labels": labels,
"comments": comments_text,
}
def _flatten_pr_row(r: dict, repo: str, include_comments: bool, max_c: int) -> dict:
labels = [l.get("name") for l in (r.get("labels", {}) or {}).get("nodes", []) if l.get("name")]
comments_nodes = (r.get("comments") or {}).get("nodes") or []
comments_text = ""
if include_comments and comments_nodes:
kept = comments_nodes[:max_c]
comments_text = "\n\n".join(
f"[{(c.get('author') or {}).get('login', '?')}]: {c.get('body') or ''}"
for c in kept
)
return {
"item_type": "pull",
"repo": repo,
"number": r.get("number"),
"title": r.get("title") or "",
"body": r.get("body") or "",
"state": r.get("state") or "",
"author": (r.get("author") or {}).get("login", ""),
"created_at": r.get("createdAt") or "",
"closed_at": r.get("closedAt") or "",
"url": r.get("url") or r.get("permalink") or "",
"labels": labels,
"comments": comments_text,
}
def _flatten_commit_row(r: dict, repo: str) -> dict:
msg = r.get("messageHeadline") or r.get("message") or ""
body = r.get("messageBody") or r.get("message") or msg
author = (r.get("author") or {})
return {
"item_type": "commit",
"repo": repo,
"number": r.get("oid") or r.get("sha") or "",
"title": msg,
"body": body,
"state": "",
"author": (author.get("user") or {}).get("login") or author.get("name", ""),
"created_at": (author.get("date") or r.get("committedDate") or ""),
"closed_at": "",
"url": r.get("url") or "",
"labels": [],
"comments": "",
}
def scrape(cfg: ScrapeConfig, base_dir: Path):
token = _resolve_token(cfg.token)
os.environ["GH_TOKEN"] = token
GitHubClient, RepoScraper = _load_impl()
client = GitHubClient()
base_dir.mkdir(parents=True, exist_ok=True)
# Per-resource trial limits. limit <= 0 means "all": use a very large cap.
effective_limit = cfg.limit if cfg.limit and cfg.limit > 0 else 1_000_000
trial_limits: dict[str, int] = {}
if "issues" in cfg.item_types:
trial_limits["issues"] = effective_limit
if "pulls" in cfg.item_types:
trial_limits["pull_requests"] = effective_limit
if "commits" in cfg.item_types:
trial_limits["commits"] = effective_limit
all_rows: list[dict] = []
for repo in cfg.repos:
owner, name = repo.split("/", 1)
scraper = RepoScraper(
owner=owner,
name=name,
base_dir=base_dir,
client=client,
trial_limits=trial_limits,
)
try:
scraper.scrape_repo_meta()
if "issues" in cfg.item_types:
scraper.scrape_issues()
if "pulls" in cfg.item_types:
scraper.scrape_prs()
if "commits" in cfg.item_types:
scraper.scrape_commits()
finally:
scraper.close()
read_cap = cfg.limit if cfg.limit and cfg.limit > 0 else None
repo_dir = base_dir / f"{owner}__{name}"
if "issues" in cfg.item_types:
for row in _read_jsonl(repo_dir / "issues.jsonl", read_cap):
all_rows.append(_flatten_issue_row(row, repo, cfg.include_comments, cfg.max_comments_per_item))
if "pulls" in cfg.item_types:
for row in _read_jsonl(repo_dir / "pull_requests.jsonl", read_cap):
all_rows.append(_flatten_pr_row(row, repo, cfg.include_comments, cfg.max_comments_per_item))
if "commits" in cfg.item_types:
for row in _read_jsonl(repo_dir / "commits.jsonl", read_cap):
all_rows.append(_flatten_commit_row(row, repo))
return all_rows
def materialize_to_jsonl(cfg: ScrapeConfig, out_dir: Path) -> Path:
out_dir.mkdir(parents=True, exist_ok=True)
tag = "-".join(r.replace("/", "__") for r in cfg.repos)[:120]
kinds = "-".join(cfg.item_types)
fname = f"github_{tag}__{kinds}__{cfg.limit}_{int(time.time())}.jsonl"
out = out_dir / fname
rows = scrape(cfg, out_dir / "raw")
with out.open("w", encoding="utf-8") as f:
for r in rows:
f.write(json.dumps(r, ensure_ascii=False) + "\n")
return out

View file

@ -0,0 +1,4 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0

View file

@ -0,0 +1,225 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""GitHub API client with rate-limit awareness, retry, and dual REST/GraphQL support."""
from __future__ import annotations
import json
import os
import time
import logging
from typing import Any, Dict, Iterable, Iterator, List, Optional
import requests
log = logging.getLogger("gh_client")
GH_TOKEN = os.environ.get("GH_TOKEN") or os.environ.get("GITHUB_TOKEN")
if not GH_TOKEN:
raise RuntimeError("GH_TOKEN not set in environment")
GRAPHQL_URL = "https://api.github.com/graphql"
REST_BASE = "https://api.github.com"
DEFAULT_HEADERS = {
"Authorization": f"Bearer {GH_TOKEN}",
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
"User-Agent": "github-data-gatherer/1.0",
}
class RateLimitError(Exception):
pass
class GitHubClient:
def __init__(self, min_remaining_graphql: int = 100, min_remaining_rest: int = 100):
self.session = requests.Session()
self.session.headers.update(DEFAULT_HEADERS)
self.min_remaining_graphql = min_remaining_graphql
self.min_remaining_rest = min_remaining_rest
self.graphql_remaining: Optional[int] = None
self.graphql_reset: Optional[int] = None
self.rest_remaining: Optional[int] = None
self.rest_reset: Optional[int] = None
self.calls_graphql = 0
self.calls_rest = 0
self.retry_count = 0
def _sleep_until(self, reset_ts: int, buffer_s: int = 10) -> None:
now = int(time.time())
wait = max(0, reset_ts - now) + buffer_s
log.warning("Rate limit hit. Sleeping %ds until reset.", wait)
time.sleep(wait)
def _check_rate_and_wait(self, kind: str) -> None:
if kind == "graphql":
remaining = self.graphql_remaining
reset = self.graphql_reset
min_remaining = self.min_remaining_graphql
else:
remaining = self.rest_remaining
reset = self.rest_reset
min_remaining = self.min_remaining_rest
if remaining is not None and remaining < min_remaining:
if reset:
self._sleep_until(reset)
# Reset remaining so we don't spin
if kind == "graphql":
self.graphql_remaining = None
else:
self.rest_remaining = None
def graphql(self, query: str, variables: Optional[Dict[str, Any]] = None,
max_retries: int = 20) -> Dict[str, Any]:
self._check_rate_and_wait("graphql")
backoff = 2
last_err = None
for attempt in range(max_retries):
try:
r = self.session.post(
GRAPHQL_URL,
json={"query": query, "variables": variables or {}},
timeout=120,
)
self.calls_graphql += 1
# Update rate info from response headers
rem = r.headers.get("X-RateLimit-Remaining")
rst = r.headers.get("X-RateLimit-Reset")
if rem is not None:
try:
self.graphql_remaining = int(rem)
except ValueError:
pass
if rst is not None:
try:
self.graphql_reset = int(rst)
except ValueError:
pass
if r.status_code in (502, 503, 504):
log.warning("GraphQL %s transient, retrying", r.status_code)
time.sleep(backoff)
backoff = min(backoff * 2, 60)
continue
if r.status_code == 403 or r.status_code == 429:
# Check for secondary/abuse
retry_after = r.headers.get("Retry-After")
if retry_after:
t = int(retry_after)
log.warning("Secondary rate limit. Sleep %ds.", t)
time.sleep(t + 2)
continue
if self.graphql_reset:
self._sleep_until(self.graphql_reset)
continue
time.sleep(60)
continue
r.raise_for_status()
data = r.json()
if "errors" in data and data["errors"]:
# Surface errors but allow partial data
errs = data["errors"]
# Retry on RATE_LIMITED
for e in errs:
if e.get("type") == "RATE_LIMITED":
self._sleep_until((self.graphql_reset or int(time.time()) + 60))
break
else:
# No rate-limit error, log and return partial
log.warning("GraphQL errors: %s", json.dumps(errs)[:400])
return data
continue
return data
except requests.RequestException as e:
last_err = e
log.warning("GraphQL network error: %s. Retry.", e)
time.sleep(backoff)
backoff = min(backoff * 2, 60)
raise RuntimeError(f"GraphQL failed after {max_retries} retries: {last_err}")
def rest(self, method: str, path: str, params: Optional[Dict[str, Any]] = None,
json_body: Optional[Dict[str, Any]] = None, max_retries: int = 6) -> requests.Response:
self._check_rate_and_wait("rest")
if path.startswith("http"):
url = path
else:
url = REST_BASE + path
backoff = 2
last_err = None
for attempt in range(max_retries):
try:
r = self.session.request(method, url, params=params, json=json_body, timeout=120)
self.calls_rest += 1
rem = r.headers.get("X-RateLimit-Remaining")
rst = r.headers.get("X-RateLimit-Reset")
if rem is not None:
try:
self.rest_remaining = int(rem)
except ValueError:
pass
if rst is not None:
try:
self.rest_reset = int(rst)
except ValueError:
pass
if r.status_code in (502, 503, 504):
log.warning("REST %s transient, retrying", r.status_code)
time.sleep(backoff)
backoff = min(backoff * 2, 60)
continue
if r.status_code in (403, 429):
retry_after = r.headers.get("Retry-After")
if retry_after:
t = int(retry_after)
log.warning("Secondary rate limit on REST. Sleep %ds.", t)
time.sleep(t + 2)
continue
# Check if primary rate
if self.rest_remaining == 0 and self.rest_reset:
self._sleep_until(self.rest_reset)
continue
log.warning("REST 403/429, sleep 60")
time.sleep(60)
continue
return r
except requests.RequestException as e:
last_err = e
log.warning("REST network error: %s. Retry.", e)
time.sleep(backoff)
backoff = min(backoff * 2, 60)
raise RuntimeError(f"REST failed after {max_retries} retries: {last_err}")
def rest_paginate(self, path: str, params: Optional[Dict[str, Any]] = None,
per_page: int = 100) -> Iterator[dict]:
params = dict(params or {})
params.setdefault("per_page", per_page)
url = path
while True:
r = self.rest("GET", url, params=params if url == path else None)
if r.status_code != 200:
log.error("REST paginate got %s at %s: %s", r.status_code, url, r.text[:200])
return
items = r.json()
if isinstance(items, dict):
# Some endpoints return dict with list field
items = items.get("items", [])
for it in items:
yield it
# Follow link header
link = r.headers.get("Link", "")
nxt = None
for part in link.split(","):
if 'rel="next"' in part:
nxt = part.split(";")[0].strip().strip("<>")
break
if not nxt:
return
url = nxt
params = None
def rate_snapshot(self) -> Dict[str, Any]:
r = self.rest("GET", "/rate_limit")
if r.status_code == 200:
return r.json()
return {}

View file

@ -0,0 +1,592 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""GraphQL queries for GitHub data scraping.
GitHub's GraphQL rejects queries that define unused fragments, so each query
only includes the fragments it actually references.
"""
# ---- Fragments (kept as raw strings, composed per query) ----
F_ACTOR = """
fragment ActorFields on Actor {
__typename
login
url
avatarUrl
... on User { id databaseId name }
... on Bot { id databaseId }
... on Organization { id databaseId name }
}
"""
F_LABEL = """
fragment LabelFields on Label {
id
name
color
description
createdAt
}
"""
F_TIMELINE = """
fragment TimelineItem on IssueTimelineItems {
__typename
... on Node { id }
... on AddedToProjectEvent { createdAt actor { ...ActorFields } }
... on AssignedEvent { createdAt actor { ...ActorFields } assignee { __typename ... on User { login } ... on Bot { login } } }
... on ClosedEvent { createdAt actor { ...ActorFields } stateReason closer { __typename ... on Commit { oid url } ... on PullRequest { number url } } }
... on CommentDeletedEvent { createdAt actor { ...ActorFields } }
... on ConnectedEvent { createdAt actor { ...ActorFields } source { __typename ... on Issue { number url repository { nameWithOwner } } ... on PullRequest { number url repository { nameWithOwner } } } subject { __typename ... on Issue { number url } ... on PullRequest { number url } } }
... on ConvertedNoteToIssueEvent { createdAt actor { ...ActorFields } }
... on CrossReferencedEvent { createdAt actor { ...ActorFields } isCrossRepository willCloseTarget source { __typename ... on Issue { number url repository { nameWithOwner } title } ... on PullRequest { number url repository { nameWithOwner } title } } }
... on DemilestonedEvent { createdAt actor { ...ActorFields } milestoneTitle }
... on DisconnectedEvent { createdAt actor { ...ActorFields } subject { __typename ... on Issue { number url } ... on PullRequest { number url } } source { __typename ... on Issue { number url } ... on PullRequest { number url } } }
... on IssueComment { id databaseId createdAt updatedAt author { ...ActorFields } body url reactionGroups { content reactors { totalCount } } }
... on LabeledEvent { createdAt actor { ...ActorFields } label { name color } }
... on LockedEvent { createdAt actor { ...ActorFields } lockReason }
... on MarkedAsDuplicateEvent { createdAt actor { ...ActorFields } canonical { __typename ... on Issue { number url } ... on PullRequest { number url } } }
... on MentionedEvent { createdAt actor { ...ActorFields } }
... on MilestonedEvent { createdAt actor { ...ActorFields } milestoneTitle }
... on MovedColumnsInProjectEvent { createdAt actor { ...ActorFields } }
... on PinnedEvent { createdAt actor { ...ActorFields } }
... on ReferencedEvent { createdAt actor { ...ActorFields } commit { oid url } commitRepository { nameWithOwner } }
... on RemovedFromProjectEvent { createdAt actor { ...ActorFields } }
... on RenamedTitleEvent { createdAt actor { ...ActorFields } previousTitle currentTitle }
... on ReopenedEvent { createdAt actor { ...ActorFields } }
... on SubscribedEvent { createdAt actor { ...ActorFields } }
... on TransferredEvent { createdAt actor { ...ActorFields } fromRepository { nameWithOwner } }
... on UnassignedEvent { createdAt actor { ...ActorFields } assignee { __typename ... on User { login } ... on Bot { login } } }
... on UnlabeledEvent { createdAt actor { ...ActorFields } label { name color } }
... on UnlockedEvent { createdAt actor { ...ActorFields } }
... on UnmarkedAsDuplicateEvent { createdAt actor { ...ActorFields } }
... on UnpinnedEvent { createdAt actor { ...ActorFields } }
... on UnsubscribedEvent { createdAt actor { ...ActorFields } }
... on UserBlockedEvent { createdAt actor { ...ActorFields } blockDuration }
}
"""
F_PR_TIMELINE = """
fragment PRTimelineItem on PullRequestTimelineItems {
__typename
... on Node { id }
... on AssignedEvent { createdAt actor { ...ActorFields } assignee { __typename ... on User { login } ... on Bot { login } } }
... on AutoMergeDisabledEvent { createdAt actor { ...ActorFields } reason }
... on AutoMergeEnabledEvent { createdAt actor { ...ActorFields } }
... on AutoRebaseEnabledEvent { createdAt actor { ...ActorFields } }
... on AutoSquashEnabledEvent { createdAt actor { ...ActorFields } }
... on AutomaticBaseChangeFailedEvent { createdAt actor { ...ActorFields } oldBase newBase }
... on AutomaticBaseChangeSucceededEvent { createdAt actor { ...ActorFields } oldBase newBase }
... on BaseRefChangedEvent { createdAt actor { ...ActorFields } previousRefName currentRefName }
... on BaseRefDeletedEvent { createdAt actor { ...ActorFields } baseRefName }
... on BaseRefForcePushedEvent { createdAt actor { ...ActorFields } beforeCommit { oid } afterCommit { oid } ref { name } }
... on ClosedEvent { createdAt actor { ...ActorFields } stateReason }
... on CommentDeletedEvent { createdAt actor { ...ActorFields } }
... on ConnectedEvent { createdAt actor { ...ActorFields } source { __typename ... on Issue { number url } ... on PullRequest { number url } } subject { __typename ... on Issue { number url } ... on PullRequest { number url } } }
... on ConvertToDraftEvent { createdAt actor { ...ActorFields } }
... on CrossReferencedEvent { createdAt actor { ...ActorFields } isCrossRepository willCloseTarget source { __typename ... on Issue { number url repository { nameWithOwner } title } ... on PullRequest { number url repository { nameWithOwner } title } } }
... on DemilestonedEvent { createdAt actor { ...ActorFields } milestoneTitle }
... on DeployedEvent { createdAt actor { ...ActorFields } }
... on DeploymentEnvironmentChangedEvent { createdAt actor { ...ActorFields } }
... on DisconnectedEvent { createdAt actor { ...ActorFields } subject { __typename ... on Issue { number url } ... on PullRequest { number url } } source { __typename ... on Issue { number url } ... on PullRequest { number url } } }
... on HeadRefDeletedEvent { createdAt actor { ...ActorFields } headRefName }
... on HeadRefForcePushedEvent { createdAt actor { ...ActorFields } beforeCommit { oid } afterCommit { oid } ref { name } }
... on HeadRefRestoredEvent { createdAt actor { ...ActorFields } }
... on IssueComment { id databaseId createdAt updatedAt author { ...ActorFields } body url reactionGroups { content reactors { totalCount } } }
... on LabeledEvent { createdAt actor { ...ActorFields } label { name color } }
... on LockedEvent { createdAt actor { ...ActorFields } lockReason }
... on MarkedAsDuplicateEvent { createdAt actor { ...ActorFields } canonical { __typename ... on Issue { number url } ... on PullRequest { number url } } }
... on MentionedEvent { createdAt actor { ...ActorFields } }
... on MergedEvent { createdAt actor { ...ActorFields } commit { oid url } mergeRefName }
... on MilestonedEvent { createdAt actor { ...ActorFields } milestoneTitle }
... on MovedColumnsInProjectEvent { createdAt actor { ...ActorFields } }
... on PinnedEvent { createdAt actor { ...ActorFields } }
... on PullRequestCommit { commit { oid url message author { user { login } date } committedDate } }
... on PullRequestCommitCommentThread { commit { oid } }
... on PullRequestReview { id databaseId createdAt submittedAt author { ...ActorFields } body state url reactionGroups { content reactors { totalCount } } }
... on PullRequestReviewThread { id isResolved isOutdated path line diffSide }
... on PullRequestRevisionMarker { createdAt lastSeenCommit { oid } }
... on ReadyForReviewEvent { createdAt actor { ...ActorFields } }
... on ReferencedEvent { createdAt actor { ...ActorFields } commit { oid url } commitRepository { nameWithOwner } }
... on RenamedTitleEvent { createdAt actor { ...ActorFields } previousTitle currentTitle }
... on ReopenedEvent { createdAt actor { ...ActorFields } }
... on ReviewDismissedEvent { createdAt actor { ...ActorFields } dismissalMessage previousReviewState }
... on ReviewRequestRemovedEvent { createdAt actor { ...ActorFields } requestedReviewer { __typename ... on User { login } ... on Team { name } } }
... on ReviewRequestedEvent { createdAt actor { ...ActorFields } requestedReviewer { __typename ... on User { login } ... on Team { name } } }
... on SubscribedEvent { createdAt actor { ...ActorFields } }
... on TransferredEvent { createdAt actor { ...ActorFields } fromRepository { nameWithOwner } }
... on UnassignedEvent { createdAt actor { ...ActorFields } assignee { __typename ... on User { login } ... on Bot { login } } }
... on UnlabeledEvent { createdAt actor { ...ActorFields } label { name color } }
... on UnlockedEvent { createdAt actor { ...ActorFields } }
... on UnmarkedAsDuplicateEvent { createdAt actor { ...ActorFields } }
... on UnpinnedEvent { createdAt actor { ...ActorFields } }
... on UnsubscribedEvent { createdAt actor { ...ActorFields } }
... on UserBlockedEvent { createdAt actor { ...ActorFields } blockDuration }
}
"""
def _q(parts: list[str], body: str) -> str:
return "\n".join(parts + [body])
ISSUES_PAGE_QUERY = _q([F_ACTOR, F_LABEL, F_TIMELINE], """
query IssuesPage($owner: String!, $name: String!, $first: Int!, $after: String) {
repository(owner: $owner, name: $name) {
issues(first: $first, after: $after, orderBy: {field: CREATED_AT, direction: ASC}) {
pageInfo { hasNextPage endCursor }
totalCount
nodes {
id databaseId number title body state stateReason
createdAt updatedAt closedAt
url
author { ...ActorFields }
editor { ...ActorFields }
labels(first: 50) { nodes { ...LabelFields } }
assignees(first: 20) { nodes { login id } }
milestone { title number state dueOn }
reactionGroups { content reactors { totalCount } }
comments(first: 100) {
totalCount
pageInfo { hasNextPage endCursor }
nodes {
id databaseId createdAt updatedAt url body
author { ...ActorFields }
editor { ...ActorFields }
reactionGroups { content reactors { totalCount } }
}
}
timelineItems(first: 100) {
totalCount
pageInfo { hasNextPage endCursor }
nodes { ...TimelineItem }
}
trackedInIssues(first: 20) { totalCount nodes { number url repository { nameWithOwner } } }
trackedIssues(first: 20) { totalCount nodes { number url repository { nameWithOwner } } }
}
}
}
rateLimit { cost remaining resetAt }
}
""")
PRS_PAGE_QUERY = _q([F_ACTOR, F_LABEL, F_PR_TIMELINE], """
query PRsPage($owner: String!, $name: String!, $first: Int!, $after: String) {
repository(owner: $owner, name: $name) {
pullRequests(first: $first, after: $after, orderBy: {field: CREATED_AT, direction: ASC}) {
pageInfo { hasNextPage endCursor }
totalCount
nodes {
id databaseId number title body state isDraft
createdAt updatedAt closedAt mergedAt
url
headRefName headRefOid
baseRefName baseRefOid
additions deletions changedFiles
mergeable merged mergeStateStatus
author { ...ActorFields }
editor { ...ActorFields }
mergedBy { ...ActorFields }
labels(first: 50) { nodes { ...LabelFields } }
assignees(first: 20) { nodes { login id } }
milestone { title number state dueOn }
reactionGroups { content reactors { totalCount } }
closingIssuesReferences(first: 20) { totalCount nodes { number url repository { nameWithOwner } title } }
comments(first: 100) {
totalCount
pageInfo { hasNextPage endCursor }
nodes {
id databaseId createdAt updatedAt url body
author { ...ActorFields }
editor { ...ActorFields }
reactionGroups { content reactors { totalCount } }
}
}
reviewThreads(first: 50) {
totalCount
pageInfo { hasNextPage endCursor }
nodes {
id isResolved isOutdated path line diffSide
comments(first: 50) {
totalCount
pageInfo { hasNextPage endCursor }
nodes {
id databaseId createdAt updatedAt url body path diffHunk
author { ...ActorFields }
editor { ...ActorFields }
position originalPosition line originalLine
commit { oid }
reactionGroups { content reactors { totalCount } }
}
}
}
}
reviews(first: 50) {
totalCount
pageInfo { hasNextPage endCursor }
nodes {
id databaseId state createdAt submittedAt body url
author { ...ActorFields }
reactionGroups { content reactors { totalCount } }
}
}
commits(first: 100) {
totalCount
pageInfo { hasNextPage endCursor }
nodes {
commit {
oid
message
messageHeadline
committedDate
authoredDate
author { name email user { login } date }
committer { name email user { login } date }
additions deletions changedFilesIfAvailable
parents(first: 3) { nodes { oid } }
}
}
}
files(first: 100) {
totalCount
pageInfo { hasNextPage endCursor }
nodes {
path additions deletions changeType
}
}
timelineItems(first: 100) {
totalCount
pageInfo { hasNextPage endCursor }
nodes { ...PRTimelineItem }
}
}
}
}
rateLimit { cost remaining resetAt }
}
""")
ISSUE_COMMENTS_QUERY = _q([F_ACTOR], """
query IssueComments($owner: String!, $name: String!, $number: Int!, $after: String) {
repository(owner: $owner, name: $name) {
issueOrPullRequest(number: $number) {
__typename
... on Issue {
comments(first: 100, after: $after) {
pageInfo { hasNextPage endCursor }
nodes {
id databaseId createdAt updatedAt url body
author { ...ActorFields }
editor { ...ActorFields }
reactionGroups { content reactors { totalCount } }
}
}
}
... on PullRequest {
comments(first: 100, after: $after) {
pageInfo { hasNextPage endCursor }
nodes {
id databaseId createdAt updatedAt url body
author { ...ActorFields }
editor { ...ActorFields }
reactionGroups { content reactors { totalCount } }
}
}
}
}
}
rateLimit { cost remaining resetAt }
}
""")
ISSUE_TIMELINE_QUERY = _q([F_ACTOR, F_TIMELINE], """
query IssueTimeline($owner: String!, $name: String!, $number: Int!, $after: String) {
repository(owner: $owner, name: $name) {
issue(number: $number) {
timelineItems(first: 100, after: $after) {
pageInfo { hasNextPage endCursor }
nodes { ...TimelineItem }
}
}
}
rateLimit { cost remaining resetAt }
}
""")
PR_TIMELINE_QUERY = _q([F_ACTOR, F_PR_TIMELINE], """
query PRTimeline($owner: String!, $name: String!, $number: Int!, $after: String) {
repository(owner: $owner, name: $name) {
pullRequest(number: $number) {
timelineItems(first: 100, after: $after) {
pageInfo { hasNextPage endCursor }
nodes { ...PRTimelineItem }
}
}
}
rateLimit { cost remaining resetAt }
}
""")
PR_COMMITS_QUERY = """
query PRCommits($owner: String!, $name: String!, $number: Int!, $after: String) {
repository(owner: $owner, name: $name) {
pullRequest(number: $number) {
commits(first: 100, after: $after) {
pageInfo { hasNextPage endCursor }
nodes {
commit {
oid message messageHeadline committedDate authoredDate
author { name email user { login } date }
committer { name email user { login } date }
additions deletions changedFilesIfAvailable
parents(first: 3) { nodes { oid } }
}
}
}
}
}
rateLimit { cost remaining resetAt }
}
"""
PR_FILES_QUERY = """
query PRFiles($owner: String!, $name: String!, $number: Int!, $after: String) {
repository(owner: $owner, name: $name) {
pullRequest(number: $number) {
files(first: 100, after: $after) {
pageInfo { hasNextPage endCursor }
nodes { path additions deletions changeType }
}
}
}
rateLimit { cost remaining resetAt }
}
"""
PR_REVIEW_THREADS_QUERY = _q([F_ACTOR], """
query PRReviewThreads($owner: String!, $name: String!, $number: Int!, $after: String) {
repository(owner: $owner, name: $name) {
pullRequest(number: $number) {
reviewThreads(first: 50, after: $after) {
pageInfo { hasNextPage endCursor }
nodes {
id isResolved isOutdated path line diffSide
comments(first: 50) {
totalCount
nodes {
id databaseId createdAt updatedAt url body path diffHunk
author { ...ActorFields }
editor { ...ActorFields }
position originalPosition line originalLine
commit { oid }
reactionGroups { content reactors { totalCount } }
}
}
}
}
}
}
rateLimit { cost remaining resetAt }
}
""")
DISCUSSIONS_PAGE_QUERY = _q([F_ACTOR, F_LABEL], """
query DiscussionsPage($owner: String!, $name: String!, $first: Int!, $after: String) {
repository(owner: $owner, name: $name) {
discussions(first: $first, after: $after, orderBy: {field: CREATED_AT, direction: ASC}) {
pageInfo { hasNextPage endCursor }
totalCount
nodes {
id databaseId number title body
createdAt updatedAt url
author { ...ActorFields }
editor { ...ActorFields }
locked
answerChosenAt
closed closedAt
category { id name emoji description isAnswerable }
labels(first: 30) { nodes { ...LabelFields } }
upvoteCount
answer { id databaseId body author { ...ActorFields } createdAt url }
reactionGroups { content reactors { totalCount } }
comments(first: 50) {
totalCount
pageInfo { hasNextPage endCursor }
nodes {
id databaseId body createdAt updatedAt url
author { ...ActorFields }
editor { ...ActorFields }
upvoteCount
isAnswer
reactionGroups { content reactors { totalCount } }
replies(first: 50) {
totalCount
pageInfo { hasNextPage endCursor }
nodes {
id databaseId body createdAt updatedAt url
author { ...ActorFields }
editor { ...ActorFields }
reactionGroups { content reactors { totalCount } }
}
}
}
}
}
}
}
rateLimit { cost remaining resetAt }
}
""")
DISCUSSION_COMMENTS_QUERY = _q([F_ACTOR], """
query DiscussionComments($owner: String!, $name: String!, $number: Int!, $after: String) {
repository(owner: $owner, name: $name) {
discussion(number: $number) {
comments(first: 50, after: $after) {
pageInfo { hasNextPage endCursor }
nodes {
id databaseId body createdAt updatedAt url
author { ...ActorFields }
editor { ...ActorFields }
upvoteCount
isAnswer
reactionGroups { content reactors { totalCount } }
replies(first: 50) {
totalCount
nodes {
id databaseId body createdAt updatedAt url
author { ...ActorFields }
editor { ...ActorFields }
reactionGroups { content reactors { totalCount } }
}
}
}
}
}
}
rateLimit { cost remaining resetAt }
}
""")
DISCUSSION_REPLIES_QUERY = _q([F_ACTOR], """
query DiscussionReplies($commentId: ID!, $after: String) {
node(id: $commentId) {
... on DiscussionComment {
replies(first: 50, after: $after) {
pageInfo { hasNextPage endCursor }
nodes {
id databaseId body createdAt updatedAt url
author { ...ActorFields }
editor { ...ActorFields }
reactionGroups { content reactors { totalCount } }
}
}
}
}
rateLimit { cost remaining resetAt }
}
""")
COMMITS_PAGE_QUERY = """
query CommitsPage($owner: String!, $name: String!, $first: Int!, $after: String, $branch: String!) {
repository(owner: $owner, name: $name) {
ref(qualifiedName: $branch) {
target {
... on Commit {
history(first: $first, after: $after) {
pageInfo { hasNextPage endCursor }
totalCount
nodes {
oid
message
messageHeadline
committedDate
authoredDate
url
additions deletions changedFilesIfAvailable
author { name email date user { login id } }
committer { name email date user { login id } }
parents(first: 3) { nodes { oid } }
associatedPullRequests(first: 5) { nodes { number url state } }
}
}
}
}
}
}
rateLimit { cost remaining resetAt }
}
"""
RELEASES_QUERY = _q([F_ACTOR], """
query Releases($owner: String!, $name: String!, $first: Int!, $after: String) {
repository(owner: $owner, name: $name) {
releases(first: $first, after: $after, orderBy: {field: CREATED_AT, direction: ASC}) {
pageInfo { hasNextPage endCursor }
nodes {
id databaseId name tagName description
createdAt publishedAt updatedAt
isDraft isPrerelease isLatest
url
author { ...ActorFields }
tagCommit { oid url }
reactionGroups { content reactors { totalCount } }
releaseAssets(first: 50) {
nodes { name contentType size downloadUrl createdAt updatedAt }
}
}
}
}
rateLimit { cost remaining resetAt }
}
""")
LABELS_QUERY = _q([F_LABEL], """
query LabelsList($owner: String!, $name: String!, $first: Int!, $after: String) {
repository(owner: $owner, name: $name) {
labels(first: $first, after: $after) {
pageInfo { hasNextPage endCursor }
nodes { ...LabelFields }
}
}
rateLimit { cost remaining resetAt }
}
""")
MILESTONES_QUERY = """
query Milestones($owner: String!, $name: String!, $first: Int!, $after: String) {
repository(owner: $owner, name: $name) {
milestones(first: $first, after: $after) {
pageInfo { hasNextPage endCursor }
nodes {
id number title description state
createdAt updatedAt closedAt dueOn
creator { login }
}
}
}
rateLimit { cost remaining resetAt }
}
"""
REPO_META_QUERY = """
query RepoMeta($owner: String!, $name: String!) {
repository(owner: $owner, name: $name) {
id databaseId name nameWithOwner description url
createdAt updatedAt pushedAt
isArchived isDisabled isFork isPrivate
primaryLanguage { name }
languages(first: 20, orderBy: {field: SIZE, direction: DESC}) {
edges { size node { name } }
totalSize
}
stargazerCount forkCount watchers { totalCount }
diskUsage
licenseInfo { key name }
homepageUrl
defaultBranchRef { name }
}
rateLimit { cost remaining resetAt }
}
"""

View file

@ -0,0 +1,526 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""Main scraper orchestration. Collects issues, PRs, discussions, commits, releases, etc.
Resumable via state file. Writes JSONL shards under data/{repo}/{resource}.jsonl.
"""
from __future__ import annotations
import argparse
import json
import logging
import os
import subprocess
import sys
import time
from pathlib import Path
from typing import Any, Dict, Iterable, List, Optional, Tuple
# Allow running as a module or script
THIS_DIR = Path(__file__).resolve().parent
if str(THIS_DIR) not in sys.path:
sys.path.insert(0, str(THIS_DIR))
from gh_client import GitHubClient
from state_store import JsonlWriter, StateStore
import queries as Q
log = logging.getLogger("scraper")
def ts() -> str:
return time.strftime("%Y-%m-%d %H:%M:%S")
class RepoScraper:
def __init__(self, owner: str, name: str, base_dir: Path, client: GitHubClient,
trial_limits: Optional[Dict[str, int]] = None):
self.owner = owner
self.name = name
self.base_dir = base_dir
self.client = client
self.trial_limits = trial_limits or {}
self.repo_dir = base_dir / f"{owner}__{name}"
self.repo_dir.mkdir(parents=True, exist_ok=True)
self.state = StateStore(base_dir / "state" / f"{owner}__{name}.json")
# Writers
self.writers: Dict[str, JsonlWriter] = {}
for key in ("issues", "pull_requests", "discussions",
"commits", "releases", "labels", "milestones",
"pr_extra_comments", "pr_extra_timeline", "pr_extra_reviews",
"issue_extra_comments", "issue_extra_timeline",
"discussion_extra_comments", "discussion_extra_replies",
"repo_meta"):
self.writers[key] = JsonlWriter(self.repo_dir / f"{key}.jsonl")
# ----- helpers -----
def _trial_stop(self, key: str, counter: int) -> bool:
lim = self.trial_limits.get(key)
if lim is None:
return False
return counter >= lim
def _log_rate(self, where: str, data: Dict[str, Any]) -> None:
rl = data.get("data", {}).get("rateLimit") if isinstance(data.get("data"), dict) else None
if rl:
log.debug("[%s] rate cost=%s remaining=%s resetAt=%s",
where, rl.get("cost"), rl.get("remaining"), rl.get("resetAt"))
# ----- repo meta -----
def scrape_repo_meta(self) -> None:
data = self.client.graphql(Q.REPO_META_QUERY, {"owner": self.owner, "name": self.name})
self._log_rate("repo_meta", data)
repo = data.get("data", {}).get("repository") or {}
repo["_fetchedAt"] = ts()
self.writers["repo_meta"].write(repo)
# ----- issues -----
def scrape_issues(self) -> int:
key = "issues"
cursor = self.state.get(f"{key}_cursor")
done = self.state.get(f"{key}_done", False)
if done:
log.info("%s/%s issues already complete", self.owner, self.name)
return 0
total_new = 0
page = 0
per_page = 15 # conservative for heavy nested query
while True:
page += 1
vars_ = {"owner": self.owner, "name": self.name, "first": per_page, "after": cursor}
data = self.client.graphql(Q.ISSUES_PAGE_QUERY, vars_)
self._log_rate("issues", data)
repo = (data.get("data") or {}).get("repository") or {}
issues = (repo.get("issues") or {})
nodes = issues.get("nodes") or []
for it in nodes:
it["_owner"] = self.owner
it["_repo"] = self.name
it["_fetchedAt"] = ts()
# Paginate nested comments/timeline if more exist
if it.get("comments", {}).get("pageInfo", {}).get("hasNextPage"):
self._paginate_issue_comments(it["number"], it["comments"]["pageInfo"]["endCursor"])
if it.get("timelineItems", {}).get("pageInfo", {}).get("hasNextPage"):
self._paginate_issue_timeline(it["number"], it["timelineItems"]["pageInfo"]["endCursor"])
if self.writers[key].write(it):
total_new += 1
info = issues.get("pageInfo") or {}
cursor = info.get("endCursor")
self.state.set(f"{key}_cursor", cursor)
log.info("[%s/%s] issues page %d (+%d) cursor=%s remaining=%s",
self.owner, self.name, page, len(nodes), str(cursor)[:20],
self.client.graphql_remaining)
if self._trial_stop(key, total_new):
log.info("Trial limit reached for issues (%d)", total_new)
return total_new
if not info.get("hasNextPage"):
self.state.set(f"{key}_done", True)
break
return total_new
def _paginate_issue_comments(self, number: int, after: str) -> None:
cur = after
while cur:
vars_ = {"owner": self.owner, "name": self.name, "number": number, "after": cur}
data = self.client.graphql(Q.ISSUE_COMMENTS_QUERY, vars_)
item = (((data.get("data") or {}).get("repository") or {}).get("issueOrPullRequest") or {})
comments = (item.get("comments") or {})
for c in comments.get("nodes") or []:
c["_owner"] = self.owner
c["_repo"] = self.name
c["_issueNumber"] = number
self.writers["issue_extra_comments"].write(c)
info = comments.get("pageInfo") or {}
cur = info.get("endCursor") if info.get("hasNextPage") else None
def _paginate_issue_timeline(self, number: int, after: str) -> None:
cur = after
while cur:
vars_ = {"owner": self.owner, "name": self.name, "number": number, "after": cur}
data = self.client.graphql(Q.ISSUE_TIMELINE_QUERY, vars_)
item = (((data.get("data") or {}).get("repository") or {}).get("issue") or {})
tl = (item.get("timelineItems") or {})
for ev in tl.get("nodes") or []:
ev["_owner"] = self.owner
ev["_repo"] = self.name
ev["_issueNumber"] = number
self.writers["issue_extra_timeline"].write(ev)
info = tl.get("pageInfo") or {}
cur = info.get("endCursor") if info.get("hasNextPage") else None
# ----- PRs -----
def scrape_prs(self) -> int:
key = "pull_requests"
cursor = self.state.get(f"{key}_cursor")
done = self.state.get(f"{key}_done", False)
if done:
log.info("%s/%s PRs already complete", self.owner, self.name)
return 0
total_new = 0
page = 0
per_page = 3 # PR query is heavy; keep small so huge PRs don't OOM GraphQL
while True:
page += 1
vars_ = {"owner": self.owner, "name": self.name, "first": per_page, "after": cursor}
data = self.client.graphql(Q.PRS_PAGE_QUERY, vars_)
self._log_rate("prs", data)
repo = (data.get("data") or {}).get("repository") or {}
prs = (repo.get("pullRequests") or {})
nodes = prs.get("nodes") or []
for pr in nodes:
pr["_owner"] = self.owner
pr["_repo"] = self.name
pr["_fetchedAt"] = ts()
num = pr["number"]
if pr.get("comments", {}).get("pageInfo", {}).get("hasNextPage"):
self._paginate_pr_comments(num, pr["comments"]["pageInfo"]["endCursor"])
if pr.get("timelineItems", {}).get("pageInfo", {}).get("hasNextPage"):
self._paginate_pr_timeline(num, pr["timelineItems"]["pageInfo"]["endCursor"])
if pr.get("commits", {}).get("pageInfo", {}).get("hasNextPage"):
self._paginate_pr_commits(num, pr["commits"]["pageInfo"]["endCursor"])
if pr.get("files", {}).get("pageInfo", {}).get("hasNextPage"):
self._paginate_pr_files(num, pr["files"]["pageInfo"]["endCursor"])
if pr.get("reviewThreads", {}).get("pageInfo", {}).get("hasNextPage"):
self._paginate_pr_review_threads(num, pr["reviewThreads"]["pageInfo"]["endCursor"])
if self.writers[key].write(pr):
total_new += 1
info = prs.get("pageInfo") or {}
cursor = info.get("endCursor")
self.state.set(f"{key}_cursor", cursor)
log.info("[%s/%s] PRs page %d (+%d) cursor=%s remaining=%s",
self.owner, self.name, page, len(nodes), str(cursor)[:20],
self.client.graphql_remaining)
if self._trial_stop(key, total_new):
log.info("Trial limit reached for PRs (%d)", total_new)
return total_new
if not info.get("hasNextPage"):
self.state.set(f"{key}_done", True)
break
return total_new
def _paginate_pr_comments(self, number: int, after: str) -> None:
cur = after
while cur:
vars_ = {"owner": self.owner, "name": self.name, "number": number, "after": cur}
data = self.client.graphql(Q.ISSUE_COMMENTS_QUERY, vars_)
item = (((data.get("data") or {}).get("repository") or {}).get("issueOrPullRequest") or {})
comments = (item.get("comments") or {})
for c in comments.get("nodes") or []:
c["_owner"] = self.owner
c["_repo"] = self.name
c["_prNumber"] = number
self.writers["pr_extra_comments"].write(c)
info = comments.get("pageInfo") or {}
cur = info.get("endCursor") if info.get("hasNextPage") else None
def _paginate_pr_timeline(self, number: int, after: str) -> None:
cur = after
while cur:
vars_ = {"owner": self.owner, "name": self.name, "number": number, "after": cur}
data = self.client.graphql(Q.PR_TIMELINE_QUERY, vars_)
item = (((data.get("data") or {}).get("repository") or {}).get("pullRequest") or {})
tl = (item.get("timelineItems") or {})
for ev in tl.get("nodes") or []:
ev["_owner"] = self.owner
ev["_repo"] = self.name
ev["_prNumber"] = number
self.writers["pr_extra_timeline"].write(ev)
info = tl.get("pageInfo") or {}
cur = info.get("endCursor") if info.get("hasNextPage") else None
def _paginate_pr_commits(self, number: int, after: str) -> None:
cur = after
out_key = "pr_extra_commits"
if out_key not in self.writers:
self.writers[out_key] = JsonlWriter(self.repo_dir / f"{out_key}.jsonl")
while cur:
vars_ = {"owner": self.owner, "name": self.name, "number": number, "after": cur}
data = self.client.graphql(Q.PR_COMMITS_QUERY, vars_)
item = (((data.get("data") or {}).get("repository") or {}).get("pullRequest") or {})
cc = (item.get("commits") or {})
for c in cc.get("nodes") or []:
c["_owner"] = self.owner
c["_repo"] = self.name
c["_prNumber"] = number
self.writers[out_key].write(c)
info = cc.get("pageInfo") or {}
cur = info.get("endCursor") if info.get("hasNextPage") else None
def _paginate_pr_files(self, number: int, after: str) -> None:
cur = after
out_key = "pr_extra_files"
if out_key not in self.writers:
self.writers[out_key] = JsonlWriter(self.repo_dir / f"{out_key}.jsonl")
while cur:
vars_ = {"owner": self.owner, "name": self.name, "number": number, "after": cur}
data = self.client.graphql(Q.PR_FILES_QUERY, vars_)
item = (((data.get("data") or {}).get("repository") or {}).get("pullRequest") or {})
ff = (item.get("files") or {})
for f in ff.get("nodes") or []:
f["_owner"] = self.owner
f["_repo"] = self.name
f["_prNumber"] = number
# files don't have id, synthesize one
f["_syntheticId"] = f"{self.owner}/{self.name}#{number}:{f.get('path')}"
self.writers[out_key].write(f)
info = ff.get("pageInfo") or {}
cur = info.get("endCursor") if info.get("hasNextPage") else None
def _paginate_pr_review_threads(self, number: int, after: str) -> None:
cur = after
out_key = "pr_extra_review_threads"
if out_key not in self.writers:
self.writers[out_key] = JsonlWriter(self.repo_dir / f"{out_key}.jsonl")
while cur:
vars_ = {"owner": self.owner, "name": self.name, "number": number, "after": cur}
data = self.client.graphql(Q.PR_REVIEW_THREADS_QUERY, vars_)
item = (((data.get("data") or {}).get("repository") or {}).get("pullRequest") or {})
rt = (item.get("reviewThreads") or {})
for th in rt.get("nodes") or []:
th["_owner"] = self.owner
th["_repo"] = self.name
th["_prNumber"] = number
self.writers[out_key].write(th)
info = rt.get("pageInfo") or {}
cur = info.get("endCursor") if info.get("hasNextPage") else None
# ----- Discussions -----
def scrape_discussions(self) -> int:
key = "discussions"
cursor = self.state.get(f"{key}_cursor")
done = self.state.get(f"{key}_done", False)
if done:
log.info("%s/%s discussions already complete", self.owner, self.name)
return 0
total_new = 0
page = 0
per_page = 15
while True:
page += 1
vars_ = {"owner": self.owner, "name": self.name, "first": per_page, "after": cursor}
data = self.client.graphql(Q.DISCUSSIONS_PAGE_QUERY, vars_)
self._log_rate("discussions", data)
repo = (data.get("data") or {}).get("repository") or {}
dd = (repo.get("discussions") or {})
nodes = dd.get("nodes") or []
for d in nodes:
d["_owner"] = self.owner
d["_repo"] = self.name
d["_fetchedAt"] = ts()
num = d["number"]
if d.get("comments", {}).get("pageInfo", {}).get("hasNextPage"):
self._paginate_discussion_comments(num, d["comments"]["pageInfo"]["endCursor"])
# paginate replies per comment if needed
for c in d.get("comments", {}).get("nodes", []) or []:
if c.get("replies", {}).get("pageInfo", {}).get("hasNextPage"):
self._paginate_discussion_replies(c["id"], c["replies"]["pageInfo"]["endCursor"], num)
if self.writers[key].write(d):
total_new += 1
info = dd.get("pageInfo") or {}
cursor = info.get("endCursor")
self.state.set(f"{key}_cursor", cursor)
log.info("[%s/%s] discussions page %d (+%d) cursor=%s remaining=%s",
self.owner, self.name, page, len(nodes), str(cursor)[:20],
self.client.graphql_remaining)
if self._trial_stop(key, total_new):
return total_new
if not info.get("hasNextPage"):
self.state.set(f"{key}_done", True)
break
return total_new
def _paginate_discussion_comments(self, number: int, after: str) -> None:
cur = after
while cur:
vars_ = {"owner": self.owner, "name": self.name, "number": number, "after": cur}
data = self.client.graphql(Q.DISCUSSION_COMMENTS_QUERY, vars_)
disc = (((data.get("data") or {}).get("repository") or {}).get("discussion") or {})
cc = (disc.get("comments") or {})
for c in cc.get("nodes") or []:
c["_owner"] = self.owner
c["_repo"] = self.name
c["_discussionNumber"] = number
self.writers["discussion_extra_comments"].write(c)
info = cc.get("pageInfo") or {}
cur = info.get("endCursor") if info.get("hasNextPage") else None
def _paginate_discussion_replies(self, comment_id: str, after: str, disc_number: int) -> None:
cur = after
while cur:
vars_ = {"owner": self.owner, "name": self.name, "commentId": comment_id, "after": cur}
data = self.client.graphql(Q.DISCUSSION_REPLIES_QUERY, vars_)
node = (data.get("data") or {}).get("node") or {}
replies = node.get("replies") or {}
for r in replies.get("nodes") or []:
r["_owner"] = self.owner
r["_repo"] = self.name
r["_discussionNumber"] = disc_number
r["_commentId"] = comment_id
self.writers["discussion_extra_replies"].write(r)
info = replies.get("pageInfo") or {}
cur = info.get("endCursor") if info.get("hasNextPage") else None
# ----- Commits -----
def scrape_commits(self, branch: str = "refs/heads/main") -> int:
key = "commits"
cursor = self.state.get(f"{key}_cursor")
done = self.state.get(f"{key}_done", False)
if done:
return 0
total_new = 0
page = 0
per_page = 100
while True:
page += 1
vars_ = {"owner": self.owner, "name": self.name,
"first": per_page, "after": cursor, "branch": branch}
data = self.client.graphql(Q.COMMITS_PAGE_QUERY, vars_)
self._log_rate("commits", data)
ref = (((data.get("data") or {}).get("repository") or {}).get("ref") or {})
tgt = ref.get("target") or {}
hist = tgt.get("history") or {}
nodes = hist.get("nodes") or []
for c in nodes:
c["_owner"] = self.owner
c["_repo"] = self.name
c["_fetchedAt"] = ts()
if self.writers[key].write(c):
total_new += 1
info = hist.get("pageInfo") or {}
cursor = info.get("endCursor")
self.state.set(f"{key}_cursor", cursor)
log.info("[%s/%s] commits page %d (+%d) remaining=%s",
self.owner, self.name, page, len(nodes), self.client.graphql_remaining)
if self._trial_stop(key, total_new):
return total_new
if not info.get("hasNextPage"):
self.state.set(f"{key}_done", True)
break
return total_new
# ----- Releases/Labels/Milestones -----
def scrape_releases(self) -> int:
return self._scrape_simple("releases", Q.RELEASES_QUERY, "releases")
def scrape_labels(self) -> int:
return self._scrape_simple("labels", Q.LABELS_QUERY, "labels")
def scrape_milestones(self) -> int:
return self._scrape_simple("milestones", Q.MILESTONES_QUERY, "milestones")
def _scrape_simple(self, key: str, query: str, field: str) -> int:
cursor = self.state.get(f"{key}_cursor")
done = self.state.get(f"{key}_done", False)
if done:
return 0
total_new = 0
while True:
vars_ = {"owner": self.owner, "name": self.name, "first": 50, "after": cursor}
data = self.client.graphql(query, vars_)
repo = (data.get("data") or {}).get("repository") or {}
col = (repo.get(field) or {})
for it in col.get("nodes") or []:
it["_owner"] = self.owner
it["_repo"] = self.name
it["_fetchedAt"] = ts()
if self.writers[key].write(it):
total_new += 1
info = col.get("pageInfo") or {}
cursor = info.get("endCursor")
self.state.set(f"{key}_cursor", cursor)
if self._trial_stop(key, total_new):
return total_new
if not info.get("hasNextPage"):
self.state.set(f"{key}_done", True)
break
log.info("[%s/%s] %s done +%d", self.owner, self.name, key, total_new)
return total_new
def close(self) -> None:
for w in self.writers.values():
try:
w.close()
except Exception:
pass
def setup_logging(log_file: Path) -> None:
log_file.parent.mkdir(parents=True, exist_ok=True)
fmt = "%(asctime)s %(levelname)s [%(name)s] %(message)s"
handlers = [
logging.StreamHandler(sys.stdout),
logging.FileHandler(log_file, mode="a", encoding="utf-8"),
]
logging.basicConfig(level=logging.INFO, format=fmt, handlers=handlers, force=True)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--base-dir", default="/mnt/disks/unslothai/ubuntu/workspace_34/github_scraper")
ap.add_argument("--repos", nargs="+", default=["unslothai/unsloth", "unslothai/unsloth-zoo"])
ap.add_argument("--trial", action="store_true", help="Small trial run")
ap.add_argument("--only", nargs="+", default=None,
help="Only run these resource keys: issues,pulls,discussions,commits,releases,labels,milestones,meta")
ap.add_argument("--hf-upload-interval", type=int, default=900,
help="Seconds between HF uploads (0 to disable)")
args = ap.parse_args()
base = Path(args.base_dir)
data_dir = base / "data"
data_dir.mkdir(parents=True, exist_ok=True)
setup_logging(base / "logs" / f"scraper_{time.strftime('%Y%m%d_%H%M%S')}.log")
log.info("Scraper starting: repos=%s trial=%s", args.repos, args.trial)
client = GitHubClient(min_remaining_graphql=80, min_remaining_rest=80)
rl = client.rate_snapshot()
log.info("Rate limit snapshot: %s", json.dumps(rl.get("resources", {}), default=str)[:400])
# Start HF uploader in background if requested
uploader = None
if args.hf_upload_interval > 0:
from hf_uploader import HFUploader
uploader = HFUploader(data_dir, interval_s=args.hf_upload_interval)
uploader.start()
trial_limits = None
if args.trial:
trial_limits = {"issues": 5, "pull_requests": 5, "discussions": 3,
"commits": 20, "releases": 3, "labels": 20, "milestones": 20}
only = set(args.only or [])
try:
for repo_spec in args.repos:
owner, name = repo_spec.split("/")
scraper = RepoScraper(owner, name, data_dir, client, trial_limits)
try:
if not only or "meta" in only:
scraper.scrape_repo_meta()
if not only or "labels" in only:
scraper.scrape_labels()
if not only or "milestones" in only:
scraper.scrape_milestones()
if not only or "releases" in only:
scraper.scrape_releases()
if not only or "discussions" in only:
scraper.scrape_discussions()
if not only or "issues" in only:
scraper.scrape_issues()
if not only or "pulls" in only:
scraper.scrape_prs()
if not only or "commits" in only:
scraper.scrape_commits()
finally:
scraper.close()
finally:
if uploader:
log.info("Stopping uploader and final sync...")
uploader.stop(final_upload=True)
log.info("Scraper complete. GraphQL calls=%d REST calls=%d",
client.calls_graphql, client.calls_rest)
if __name__ == "__main__":
main()

View file

@ -0,0 +1,103 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""Checkpoint state management for resumable scraping."""
from __future__ import annotations
import json
import os
import threading
from pathlib import Path
from typing import Any, Dict
class StateStore:
def __init__(self, path: str | Path):
self.path = Path(path)
self.path.parent.mkdir(parents=True, exist_ok=True)
self._lock = threading.Lock()
self._data: Dict[str, Any] = {}
if self.path.exists():
try:
with self.path.open() as f:
self._data = json.load(f)
except Exception:
self._data = {}
def get(self, key: str, default: Any = None) -> Any:
with self._lock:
return self._data.get(key, default)
def set(self, key: str, value: Any) -> None:
with self._lock:
self._data[key] = value
self._flush()
def update(self, key: str, **kwargs) -> None:
with self._lock:
sub = dict(self._data.get(key, {}))
sub.update(kwargs)
self._data[key] = sub
self._flush()
def all(self) -> Dict[str, Any]:
with self._lock:
return dict(self._data)
def _flush(self) -> None:
tmp = self.path.with_suffix(self.path.suffix + ".tmp")
with tmp.open("w") as f:
json.dump(self._data, f, indent=2, default=str)
os.replace(tmp, self.path)
class JsonlWriter:
"""Append-only JSONL writer, thread-safe, with line buffering."""
def __init__(self, path: str | Path):
self.path = Path(path)
self.path.parent.mkdir(parents=True, exist_ok=True)
self._lock = threading.Lock()
self._fh = self.path.open("a", buffering=1)
self._count_seen_keys: set[str] = set()
# Preload seen keys if file exists (for dedup across resumes)
if self.path.exists() and self.path.stat().st_size > 0:
try:
with self.path.open() as f:
for line in f:
try:
obj = json.loads(line)
k = self._key(obj)
if k is not None:
self._count_seen_keys.add(k)
except Exception:
pass
except Exception:
pass
def _key(self, obj: dict) -> str | None:
for k in ("id", "node_id", "number", "sha", "url"):
if k in obj:
return f"{k}:{obj[k]}"
return None
def has(self, key: str) -> bool:
return key in self._count_seen_keys
def write(self, obj: dict) -> bool:
"""Return True if newly written, False if already present."""
k = self._key(obj)
with self._lock:
if k is not None and k in self._count_seen_keys:
return False
if k is not None:
self._count_seen_keys.add(k)
self._fh.write(json.dumps(obj, default=str, ensure_ascii=False))
self._fh.write("\n")
self._fh.flush()
return True
def close(self) -> None:
try:
self._fh.close()
except Exception:
pass

View file

@ -0,0 +1,126 @@
{
"recipe": {
"model_providers": [
{
"name": "Local Model",
"endpoint": "",
"provider_type": "openai",
"api_key": "",
"is_local": true,
"extra_headers": {},
"extra_body": {}
}
],
"mcp_providers": [],
"model_configs": [
{
"alias": "model_1",
"model": "unsloth/gemma-4-E2B-it-GGUF",
"provider": "Local Model",
"inference_parameters": {
"temperature": 0.4,
"max_tokens": 1500
}
}
],
"tool_configs": [],
"seed_config": {
"source": {
"seed_type": "github_repo",
"repos": ["unslothai/unsloth", "unslothai/unsloth-zoo"],
"token": "",
"item_types": ["issues", "pulls"],
"limit": 100,
"include_comments": true,
"max_comments_per_item": 20
},
"sampling_strategy": "shuffle",
"selection_strategy": null
},
"columns": [
{
"column_type": "llm-text",
"name": "normalized_question",
"drop": false,
"model_alias": "model_1",
"prompt": "Rewrite the following GitHub {{ item_type }} into a single, standalone Unsloth support question a user would ask on Discord. Keep the key technical details (code snippets, tracebacks, versions) verbatim. Keep it 1-5 sentences. Output ONLY the rewritten question.\n\n--- INPUT ---\nRepo: {{ repo }}\nTitle: {{ title }}\nBody:\n{{ body }}\n\nFirst comments:\n{{ comments }}",
"system_prompt": "You rewrite real GitHub issue / PR threads into concise support questions. Preserve technical fidelity: do not invent facts.",
"with_trace": "none"
},
{
"column_type": "llm-structured",
"name": "support_answer",
"drop": false,
"model_alias": "model_1",
"prompt": "You are the Unsloth support assistant. Answer the user's question grounded in how Unsloth actually works. Produce structured JSON.\n\nSource issue/PR:\n- Repo: {{ repo }}\n- Title: {{ title }}\n- URL: {{ url }}\n- State: {{ state }}\n- Labels: {{ labels }}\n\nNormalized user question:\n{{ normalized_question }}\n\nRules:\n- `answer` is 3-8 sentences of Markdown. Include a python code block when showing usage.\n- `diagnosis_questions` is 1-4 extra questions the user should answer if the info is insufficient (versions, GPU, full traceback). Empty list if the answer is complete.\n- `cites` is a list of references the answer is grounded in (doc paths, GitHub URLs). Always include the source `url` above.\n- `confidence` is one of `high` / `medium` / `low`. Use `low` when ambiguous or out of scope.\n- Never invent APIs. If uncertain, set confidence=`low` and ask diagnosis_questions rather than fabricating.",
"output_format": {
"type": "object",
"properties": {
"answer": {"type": "string", "minLength": 1},
"diagnosis_questions": {"type": "array", "items": {"type": "string"}, "maxItems": 4},
"cites": {"type": "array", "items": {"type": "string"}, "maxItems": 6},
"confidence": {"type": "string", "enum": ["high", "medium", "low"]}
},
"required": ["answer", "diagnosis_questions", "cites", "confidence"],
"additionalProperties": false
}
}
],
"processors": []
},
"run": {
"rows": 10,
"preview": true,
"output_formats": ["jsonl"]
},
"ui": {
"nodes": [
{"id": "Local Model", "x": -1056, "y": 520, "width": 400},
{"id": "model_1", "x": -544, "y": 488, "width": 400},
{"id": "seed", "x": 0, "y": 140, "width": 400},
{"id": "normalized_question", "x": 0, "y": 440, "width": 400},
{"id": "support_answer", "x": 0, "y": 740, "width": 400},
{
"id": "note_1",
"x": 480,
"y": 120,
"width": 400,
"node_type": "markdown_note",
"name": "note_1",
"markdown": "### GitHub Support Bot\nReal GitHub data to synthetic Q&A pairs for support-bot fine-tuning.\n\n**1. Click the Source Data node (top-left of the canvas) to enter:**\n- **Repos**: one `owner/name` per line\n- **GitHub token**: your `GH_TOKEN` (or leave blank to use the server's env var)\n- Item types (issues / pulls / commits) and per-resource limit\n\n**2. Click `Run` below and set the number of rows** (defaults to 10 for a quick test).\n\nThe built-in `github_repo` seed reader does rate-limit-aware GraphQL scraping. Each scraped item becomes a row with `title`, `body`, `comments`, `labels`, etc. Two LLM blocks turn it into `{normalized_question, support_answer}`. Answers are structured JSON with `answer` / `diagnosis_questions` / `cites` / `confidence`.",
"note_color": "#E0F2FE",
"note_opacity": "35"
},
{
"id": "note_2",
"x": 480,
"y": 420,
"width": 400,
"node_type": "markdown_note",
"name": "note_2",
"markdown": "The **normalize** step turns a raw GitHub thread (which often has title/body/comments scattered) into a single clean support question.\n\nTweak the prompt to:\n- always include the traceback\n- drop off-topic chitchat\n- target a specific user persona",
"note_color": "#E0F2FE",
"note_opacity": "35"
},
{
"id": "note_3",
"x": 480,
"y": 740,
"width": 400,
"node_type": "markdown_note",
"name": "note_3",
"markdown": "The **answer** block enforces a JSON shape that maps directly onto the grounded RAG answerer format.\n\nFields:\n- `answer` (Markdown with code)\n- `diagnosis_questions` (follow-ups)\n- `cites` (doc + GitHub URLs)\n- `confidence` (high/medium/low)\n\nThis is key: uniform schema = less cleanup later.",
"note_color": "#E0F2FE",
"note_opacity": "35"
}
],
"edges": [
{"from": "seed", "to": "normalized_question", "type": "canvas", "source_handle": "data-out-bottom", "target_handle": "data-in-top"},
{"from": "normalized_question", "to": "support_answer", "type": "canvas", "source_handle": "data-out-bottom", "target_handle": "data-in-top"},
{"from": "Local Model", "to": "model_1", "type": "semantic", "source_handle": "semantic-out", "target_handle": "semantic-in"},
{"from": "model_1", "to": "normalized_question", "type": "semantic", "source_handle": "semantic-out", "target_handle": "data-in"},
{"from": "model_1", "to": "support_answer", "type": "semantic", "source_handle": "semantic-out-bottom", "target_handle": "data-in"}
],
"layout_direction": "LR"
}
}

View file

@ -19,6 +19,10 @@ const ocrDocumentExtractionUrl = new URL(
"./ocr-document-extraction.json",
import.meta.url,
).href;
const githubSupportBotUrl = new URL(
"./github-support-bot.json",
import.meta.url,
).href;
function isRecord(value: unknown): value is Record<string, unknown> {
return !!value && typeof value === "object" && !Array.isArray(value);
@ -137,4 +141,11 @@ export const LEARNING_RECIPES: LearningRecipeDef[] = [
"Use image context to generate OCR-style document extraction output.",
loadPayload: () => loadPayloadFromUrl(ocrDocumentExtractionUrl),
},
{
id: "github-support-bot",
title: "GitHub Support Bot",
description:
"Generate realistic Unsloth support Q&A (question + structured answer with citations) for fine-tuning a support assistant.",
loadPayload: () => loadPayloadFromUrl(githubSupportBotUrl),
},
];

View file

@ -35,6 +35,7 @@ import {
Delete02Icon,
DocumentAttachmentIcon,
FunctionIcon,
GithubIcon,
Plant01Icon,
PlusSignIcon,
} from "@hugeicons/core-free-icons";
@ -162,6 +163,22 @@ const TEMPLATE_CARDS: TemplateCard[] = [
],
learningRecipeId: "structured-outputs-jinja",
},
{
title: "GitHub Support Bot",
description:
"Scrape real GitHub issues / PRs / commits (multi-repo) and turn each into a normalized question + structured answer for fine-tuning a support assistant.",
icon: GithubIcon,
difficulty: "Intermediate",
learningBadges: ["GitHub", "LLM Text", "Structured LLM"],
surfaceClassName:
"from-slate-500/15 via-zinc-500/5 to-transparent dark:from-slate-400/30 dark:via-zinc-400/14 dark:to-slate-950/16",
shineColor: [
"rgb(71 85 105 / 0.45)",
"rgb(100 116 139 / 0.4)",
"rgb(148 163 184 / 0.45)",
],
learningRecipeId: "github-support-bot",
},
];
const LEARNING_RECIPE_BY_ID = new Map(

View file

@ -12,6 +12,7 @@ import {
EqualSignIcon,
FingerPrintIcon,
FunctionIcon,
GithubIcon,
Plug01Icon,
Parabola02Icon,
PencilEdit02Icon,
@ -58,11 +59,16 @@ export type BlockType =
| "seed_hf"
| "seed_local"
| "seed_unstructured"
| "seed_github"
| "model_provider"
| "model_config"
| "tool_config";
export type SeedBlockType = "seed_hf" | "seed_local" | "seed_unstructured";
export type SeedBlockType =
| "seed_hf"
| "seed_local"
| "seed_unstructured"
| "seed_github";
type IconType = typeof CodeIcon;
@ -169,6 +175,15 @@ const BLOCK_DEFINITIONS: BlockDefinition[] = [
dialogKey: "seed",
createConfig: (id, existing) => makeSeedConfig(id, existing, "unstructured"),
},
{
kind: "seed",
type: "seed_github",
title: "GitHub repositories",
description: "Scrape issues, pull requests, and commits from one or more GitHub repos.",
icon: GithubIcon,
dialogKey: "seed",
createConfig: (id, existing) => makeSeedConfig(id, existing, "github_repo"),
},
{
kind: "sampler",
type: "category",
@ -388,6 +403,7 @@ export function getBlockDefinitionForConfig(
hf: "seed_hf",
local: "seed_local",
unstructured: "seed_unstructured",
github_repo: "seed_github",
};
return getBlockDefinition("seed", seedType[config.seed_source_type ?? "hf"]);
}

View file

@ -1,7 +1,12 @@
// SPDX-License-Identifier: AGPL-3.0-only
// Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
import { DocumentAttachmentIcon, DocumentCodeIcon, Plant01Icon } from "@hugeicons/core-free-icons";
import {
DocumentAttachmentIcon,
DocumentCodeIcon,
GithubIcon,
Plant01Icon,
} from "@hugeicons/core-free-icons";
import { HugeiconsIcon } from "@hugeicons/react";
import type { ReactElement } from "react";
import type { SeedConfig } from "../../types";
@ -16,6 +21,38 @@ type InlineSeedProps = {
export function InlineSeed({ config, onUpdate }: InlineSeedProps): ReactElement {
const mode = config.seed_source_type ?? "hf";
if (mode === "github_repo") {
const repos = (config.github_repo_slug ?? "")
.split(/\r?\n/)
.map((r) => r.trim())
.filter(Boolean);
const summary =
repos.length === 0
? "No repositories"
: repos.length === 1
? repos[0]
: `${repos.length} repositories`;
const items = config.github_item_types ?? [];
const itemsLabel = items.length ? items.join(" · ") : "issues · pulls";
return (
<div className="corner-squircle flex items-center gap-2 rounded-md border border-border/60 bg-muted/30 px-2 py-2">
<div className="corner-squircle rounded-md bg-primary/10 p-1.5 text-primary">
<HugeiconsIcon icon={GithubIcon} className="size-3.5" />
</div>
<div className="min-w-0">
<p className="truncate text-xs font-medium">{summary}</p>
<p className="truncate text-[11px] text-muted-foreground">
{itemsLabel} · configure in dialog
</p>
</div>
<HugeiconsIcon
icon={Plant01Icon}
className="ml-auto size-3.5 text-muted-foreground/60"
/>
</div>
);
}
if (mode === "hf") {
return (
<div className="space-y-2">

View file

@ -75,6 +75,137 @@ type SeedDialogProps = {
open: boolean;
};
function GithubRepoSeedForm({
config,
onUpdate,
}: {
config: SeedConfig;
onUpdate: (patch: Partial<SeedConfig>) => void;
}): ReactElement {
const [advancedOpen, setAdvancedOpen] = useState(false);
const limitStr = (config.github_limit ?? "100").trim();
const allMode = limitStr === "" || limitStr === "0";
return (
<div className="space-y-3 rounded-xl corner-squircle border border-border/60 p-3">
<div className="grid gap-1.5">
<FieldLabel
label="GitHub repositories"
hint="One owner/name per line. Defaults to the two Unsloth repos."
/>
<textarea
className="nodrag min-h-20 w-full resize-y rounded-md border border-border/60 bg-background px-2 py-1.5 text-xs font-mono"
value={config.github_repo_slug ?? ""}
onChange={(e) => onUpdate({ github_repo_slug: e.target.value })}
placeholder="unslothai/unsloth&#10;unslothai/unsloth-zoo"
/>
</div>
<div className="grid gap-1.5">
<FieldLabel
label="GitHub token"
hint="Personal access token with repo scope. Leave blank to use the server's GH_TOKEN / GITHUB_TOKEN env var."
/>
<Input
type="password"
className="nodrag"
value={config.github_token ?? ""}
onChange={(e) => onUpdate({ github_token: e.target.value })}
placeholder="ghp_... (optional if server has GH_TOKEN)"
/>
</div>
<div className="grid gap-1.5">
<FieldLabel
label="Items per repo"
hint="How many issues/PRs/commits to fetch from each repo. Toggle All to scrape everything."
/>
<div className="flex items-center gap-2">
<Input
type="number"
className="nodrag flex-1"
min={1}
max={100000}
disabled={allMode}
value={allMode ? "" : limitStr}
onChange={(e) => onUpdate({ github_limit: e.target.value })}
placeholder={allMode ? "All" : "100"}
/>
<label className="flex cursor-pointer items-center gap-1.5 text-xs">
<Checkbox
checked={allMode}
onCheckedChange={(v) =>
onUpdate({ github_limit: v === true ? "0" : "100" })
}
/>
<span>All</span>
</label>
</div>
</div>
<Collapsible open={advancedOpen} onOpenChange={setAdvancedOpen}>
<CollapsibleTrigger asChild={true}>
<button
type="button"
className="text-xs font-semibold uppercase tracking-wide text-muted-foreground hover:text-foreground"
>
{advancedOpen ? "Hide advanced" : "Advanced"}
</button>
</CollapsibleTrigger>
<CollapsibleContent className="mt-2 space-y-3">
<div className="grid gap-1.5">
<FieldLabel
label="Item types"
hint="Which GitHub resources to scrape per repo."
/>
<div className="flex flex-wrap gap-3 text-xs">
{(["issues", "pulls", "commits"] as const).map((kind) => {
const current = config.github_item_types ?? ["issues", "pulls"];
const checked = current.includes(kind);
return (
<label key={kind} className="flex cursor-pointer items-center gap-1.5">
<Checkbox
checked={checked}
onCheckedChange={(v) => {
const next = v === true
? Array.from(new Set([...current, kind]))
: current.filter((k) => k !== kind);
onUpdate({ github_item_types: next.length ? next : ["issues"] });
}}
/>
<span>{kind}</span>
</label>
);
})}
</div>
</div>
<div className="grid gap-1.5">
<FieldLabel
label="Max comments / item"
hint="Comments are concatenated into the comments column."
/>
<Input
type="number"
className="nodrag"
min={0}
max={200}
value={config.github_max_comments_per_item ?? "30"}
onChange={(e) => onUpdate({ github_max_comments_per_item: e.target.value })}
/>
</div>
<label className="flex cursor-pointer items-center gap-1.5 text-xs">
<Checkbox
checked={config.github_include_comments ?? true}
onCheckedChange={(v) => onUpdate({ github_include_comments: v === true })}
/>
<span>Include comments</span>
</label>
</CollapsibleContent>
</Collapsible>
<p className="text-xs text-muted-foreground">
Backed by Studio's built-in <code>github_repo</code> seed reader: a
rate-limit-aware GraphQL scraper for issues, pull requests, and commits.
</p>
</div>
);
}
function getErrorMessage(error: unknown, fallback: string): string {
if (error instanceof Error && error.message) {
return error.message;
@ -600,9 +731,13 @@ export function SeedDialog({ config, onUpdate, open }: SeedDialogProps): ReactEl
/>
)}
{mode === "github_repo" && (
<GithubRepoSeedForm config={config} onUpdate={onUpdate} />
)}
{inspectError && <p className="text-xs text-red-600">{inspectError}</p>}
{mode !== "unstructured" && (
{mode !== "unstructured" && mode !== "github_repo" && (
<div className="space-y-2 rounded-xl corner-squircle border border-border/60 p-3">
<FieldLabel
label="Drop specific seed columns"

View file

@ -3,7 +3,12 @@
import { useCallback, useEffect, useState } from "react";
import { useShallow } from "zustand/react/shallow";
import { toast } from "sonner";
import { toastError } from "@/shared/toast";
import {
getInferenceStatus,
loadModel,
} from "@/features/chat/api/chat-api";
import {
cancelRecipeJob,
createRecipeJob,
@ -39,10 +44,84 @@ import {
} from "../stores/recipe-executions";
import type { RecipePayload, RecipePayloadResult } from "../utils/payload/types";
/**
* Auto-load the local model before running a recipe that uses it.
*
* Looks at payload.recipe.model_providers for any provider with is_local=true,
* finds the bound model_configs and asks the backend to load whichever model
* the first local-bound model_config points at. Skips when the inference
* server already has that exact model active. This removes the "open /chat
* first" prerequisite that users kept tripping on.
*/
async function ensureLocalModelLoaded(
payload: RecipePayload,
): Promise<string | null> {
const providers = Array.isArray(payload.recipe.model_providers)
? (payload.recipe.model_providers as Array<Record<string, unknown>>)
: [];
const localProviderNames = new Set<string>();
for (const p of providers) {
if (p.is_local === true && typeof p.name === "string") {
localProviderNames.add(p.name);
}
}
if (localProviderNames.size === 0) {
return null;
}
const modelConfigs = Array.isArray(payload.recipe.model_configs)
? (payload.recipe.model_configs as Array<Record<string, unknown>>)
: [];
const boundConfig = modelConfigs.find(
(c) => typeof c.provider === "string" && localProviderNames.has(c.provider),
);
const target =
typeof boundConfig?.model === "string" ? boundConfig.model.trim() : "";
if (!target) {
return null;
}
try {
const status = await getInferenceStatus();
if (
status.active_model &&
status.active_model.toLowerCase() === target.toLowerCase()
) {
return null;
}
} catch {
// Fall through to load attempt; the backend will re-error if needed.
}
const toastId = toast.loading(`Loading ${target}`, {
description: "Starting the local inference server for this recipe.",
});
try {
const isGguf = /gguf/i.test(target);
await loadModel({
model_path: target,
hf_token: null,
max_seq_length: isGguf ? 0 : 4096,
load_in_4bit: true,
is_lora: false,
gguf_variant: null,
trust_remote_code: false,
chat_template_override: null,
cache_type_kv: null,
speculative_type: null,
});
toast.success(`Loaded ${target}`, { id: toastId, duration: 2000 });
return null;
} catch (error) {
toast.dismiss(toastId);
return error instanceof Error ? error.message : String(error);
}
}
type UseRecipeExecutionsParams = {
recipeId: string;
currentSignature: string;
payloadResult: RecipePayloadResult;
initialRunRows?: number | null;
onExecutionStart?: () => void;
onPreviewSuccess?: () => void;
};
@ -101,6 +180,7 @@ export function useRecipeExecutions({
recipeId,
currentSignature,
payloadResult,
initialRunRows,
onExecutionStart,
onPreviewSuccess,
}: UseRecipeExecutionsParams): UseRecipeExecutionsResult {
@ -181,6 +261,19 @@ export function useRecipeExecutions({
resetForRecipe();
// Seed previewRows from the recipe's original run.rows (read from the
// loaded JSON, not the rebuilt payload (the builder hardcodes 5).
// Templates ship their own suggested preview size (e.g. GitHub Support
// Bot: 10); we honor it so users don't see a surprise 5.
if (
typeof initialRunRows === "number" &&
Number.isFinite(initialRunRows) &&
initialRunRows > 0 &&
initialRunRows !== 5
) {
setPreviewRows(Math.floor(initialRunRows));
}
async function hydrate(): Promise<void> {
try {
const records = await loadSortedRecipeExecutions(recipeId);
@ -216,10 +309,12 @@ export function useRecipeExecutions({
cancelled = true;
};
}, [
initialRunRows,
onPreviewSuccess,
recipeId,
resetForRecipe,
setExecutions,
setPreviewRows,
setRunErrors,
upsertAndPersist,
]);
@ -340,6 +435,13 @@ export function useRecipeExecutions({
return false;
}
const localLoadError = await ensureLocalModelLoaded(payload);
if (localLoadError) {
setRunErrors([localLoadError]);
toastError("Local model failed to load", localLoadError);
return false;
}
const normalizedRows = sanitizeExecutionRows(rows, kind);
const executionPayload = buildExecutionPayload({
payload,

View file

@ -112,6 +112,10 @@ export function useRecipeStudioActions({
recipeId,
currentSignature: persistence.currentSignature,
payloadResult,
initialRunRows:
typeof initialPayload?.run?.rows === "number"
? initialPayload.run.rows
: null,
onExecutionStart,
onPreviewSuccess,
});

View file

@ -38,7 +38,10 @@ export type LayoutDirection = "LR" | "TB";
export type SeedSamplingStrategy = "ordered" | "shuffle";
export type SeedSelectionType = "none" | "index_range" | "partition_block";
export type SeedSourceType = "hf" | "local" | "unstructured";
export type SeedSourceType = "hf" | "local" | "unstructured" | "github_repo";
export type GithubItemType = "issues" | "pulls" | "commits";
export type GithubStateFilter = "all" | "open" | "closed";
export const INFRA_NODE_KINDS = new Set([
"model_provider",
"model_config",
@ -338,6 +341,20 @@ export type SeedConfig = {
unstructured_file_ids?: string[];
unstructured_file_names?: string[];
unstructured_file_sizes?: number[];
// biome-ignore lint/style/useNamingConvention: api schema
github_repo_slug?: string;
// biome-ignore lint/style/useNamingConvention: api schema
github_token?: string;
// biome-ignore lint/style/useNamingConvention: api schema
github_limit?: string;
// biome-ignore lint/style/useNamingConvention: api schema
github_item_types?: GithubItemType[];
// biome-ignore lint/style/useNamingConvention: api schema
github_state?: GithubStateFilter;
// biome-ignore lint/style/useNamingConvention: api schema
github_include_comments?: boolean;
// biome-ignore lint/style/useNamingConvention: api schema
github_max_comments_per_item?: string;
resolved_paths?: string[];
// ui-only
seed_preview_rows?: Record<string, unknown>[];

View file

@ -80,6 +80,12 @@ function parseSeedSettings(seedConfigRaw: unknown): Partial<SeedConfig> {
let resolved_paths: string[] = [];
let unstructured_chunk_size = "1200";
let unstructured_chunk_overlap = "200";
let github_repo_slug = "";
let github_token = "";
let github_limit = "100";
let github_item_types: ("issues" | "pulls" | "commits")[] = ["issues", "pulls"];
let github_include_comments = true;
let github_max_comments_per_item = "30";
const sourceRaw = seedConfigRaw.source;
if (isRecord(sourceRaw)) {
const seedType = readString(sourceRaw.seed_type);
@ -107,6 +113,26 @@ function parseSeedSettings(seedConfigRaw: unknown): Partial<SeedConfig> {
unstructuredFileNames = [];
unstructured_chunk_size = readNumberString(sourceRaw.chunk_size) || "1200";
unstructured_chunk_overlap = readNumberString(sourceRaw.chunk_overlap) || "200";
} else if (seedType === "github_repo") {
seed_source_type = "github_repo";
const rawRepos = Array.isArray(sourceRaw.repos) ? sourceRaw.repos : [];
const repos = rawRepos.filter((r): r is string => typeof r === "string");
github_repo_slug = repos.join("\n");
github_token = readString(sourceRaw.token) ?? "";
github_limit = readNumberString(sourceRaw.limit) || "100";
const rawItems = Array.isArray(sourceRaw.item_types) ? sourceRaw.item_types : [];
const validItems = rawItems.filter(
(t): t is "issues" | "pulls" | "commits" =>
t === "issues" || t === "pulls" || t === "commits",
);
if (validItems.length > 0) {
github_item_types = validItems;
}
if (typeof sourceRaw.include_comments === "boolean") {
github_include_comments = sourceRaw.include_comments;
}
github_max_comments_per_item =
readNumberString(sourceRaw.max_comments_per_item) || "30";
}
}
@ -147,6 +173,12 @@ function parseSeedSettings(seedConfigRaw: unknown): Partial<SeedConfig> {
resolved_paths,
unstructured_chunk_size,
unstructured_chunk_overlap,
github_repo_slug,
github_token,
github_limit,
github_item_types,
github_include_comments,
github_max_comments_per_item,
sampling_strategy,
selection_type,
selection_start,

View file

@ -70,7 +70,9 @@ export function nodeDataFromConfig(
? "Hugging Face dataset"
: seedSourceType === "local"
? "CSV or JSON file"
: "Document file";
: seedSourceType === "github_repo"
? "GitHub repositories"
: "Document file";
return {
title: "Source data",
kind: "seed",

View file

@ -56,33 +56,59 @@ export function buildSeedConfig(
selectionStrategy = { index, num_partitions: numPartitions };
}
const source =
seedSourceType === "hf"
? {
// biome-ignore lint/style/useNamingConvention: api schema
seed_type: "hf",
path,
token,
endpoint,
}
: seedSourceType === "unstructured"
? (() => {
const { chunkSize, chunkOverlap } = resolveChunking(config);
return {
// biome-ignore lint/style/useNamingConvention: api schema
seed_type: "unstructured",
paths: config.resolved_paths?.length ? config.resolved_paths : [config.hf_path],
// biome-ignore lint/style/useNamingConvention: api schema
chunk_size: chunkSize,
// biome-ignore lint/style/useNamingConvention: api schema
chunk_overlap: chunkOverlap,
};
})()
: {
// biome-ignore lint/style/useNamingConvention: api schema
seed_type: "local",
path,
};
let source: Record<string, unknown>;
if (seedSourceType === "hf") {
source = {
// biome-ignore lint/style/useNamingConvention: api schema
seed_type: "hf",
path,
token,
endpoint,
};
} else if (seedSourceType === "unstructured") {
const { chunkSize, chunkOverlap } = resolveChunking(config);
source = {
// biome-ignore lint/style/useNamingConvention: api schema
seed_type: "unstructured",
paths: config.resolved_paths?.length ? config.resolved_paths : [config.hf_path],
// biome-ignore lint/style/useNamingConvention: api schema
chunk_size: chunkSize,
// biome-ignore lint/style/useNamingConvention: api schema
chunk_overlap: chunkOverlap,
};
} else if (seedSourceType === "github_repo") {
const repos = (config.github_repo_slug ?? "")
.split(/[\n,]/)
.map((r) => r.trim())
.filter(Boolean);
if (repos.length === 0) {
errors.push(`Seed ${config.name}: at least one repo is required.`);
return undefined;
}
const limitNum = parseIntStrict(config.github_limit ?? "100");
const maxCommentsNum = parseIntStrict(config.github_max_comments_per_item ?? "30");
source = {
// biome-ignore lint/style/useNamingConvention: api schema
seed_type: "github_repo",
repos,
token: (config.github_token ?? "").trim(),
// biome-ignore lint/style/useNamingConvention: api schema
item_types: config.github_item_types?.length
? config.github_item_types
: ["issues", "pulls"],
limit: limitNum !== null ? limitNum : 100,
// biome-ignore lint/style/useNamingConvention: api schema
include_comments: config.github_include_comments ?? true,
// biome-ignore lint/style/useNamingConvention: api schema
max_comments_per_item: maxCommentsNum !== null ? maxCommentsNum : 30,
};
} else {
source = {
// biome-ignore lint/style/useNamingConvention: api schema
seed_type: "local",
path,
};
}
return {
source,

View file

@ -57,7 +57,7 @@ export type RecipePayload = {
// ui-only: graph orientation
layout_direction?: "LR" | "TB";
// ui-only, used to preserve seed block mode across imports/refresh
seed_source_type?: "hf" | "local" | "unstructured";
seed_source_type?: "hf" | "local" | "unstructured" | "github_repo";
// ui-only, persisted aux node positions by llm name + aux key
aux_nodes?: Array<{
llm: string;