History

Daniel Han 3fd948eb95 Pin utf-8 on shipping-code text I/O instead of the operator locale (#7486 ) * Pin utf-8 on shipping-code text I/O instead of the operator locale 113 read_text/write_text/open call sites across unsloth, studio and unsloth_cli let locale.getencoding() decide the encoding. That is utf-8 on the Linux and macOS runners and cp1252 on a stock Windows install, so the same file decodes differently for a Windows user and silently produces mojibake or raises UnicodeDecodeError. Adds tests/test_runtime_text_encoding.py to keep it that way. It resolves openers through each file's own imports rather than a fixed list of module names, so an aliased tarfile.open or a local from PIL.Image import open is not asked for an encoding it does not take. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scan tracked files only and resolve the unbound Path calling forms * Honour PEP 263 when scanning sources and migrate a legacy JSONL before appending * Scope guard imports lexically and only migrate a legacy file when it round-trips * Leave a legacy JSONL untouched and resolve path aliases in the foreign-opener check * Tighten comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>		2026-07-27 02:14:20 -07:00
..
src/data_designer_github_repo_seed	Pin utf-8 on shipping-code text I/O instead of the operator locale (#7486 )	2026-07-27 02:14:20 -07:00
pyproject.toml	Studio: add github_repo seed reader and GitHub Support Bot recipe (#5169 )	2026-04-24 12:02:03 -07:00
README.md	Replace standalone Studio wording with Unsloth (#7221 )	2026-07-19 00:47:04 -07:00

README.md

data-designer-github-repo-seed

A Data Designer seed-reader plugin for Unsloth Studio that scrapes real GitHub data (issues, pull requests, commits) from one or more repositories and hands it to the recipe pipeline as a seed dataset.

Designed to ship with Unsloth as a default seed source so any user with a GitHub token can build training datasets straight from live repos.

What it does

Given a list of owner/name repos, a GitHub token, and a per-resource limit, the plugin uses GitHub's GraphQL API to fetch issues, pull requests, and/or commits, with labels, state, authors, and the first N comments of each item, and materialises a single JSONL with uniform columns so the rest of the recipe (LLM text / LLM structured / processors) can treat it like any other seed table.

Column	Description
`item_type`	`issue` / `pull` / `commit`
`repo`	`owner/name`
`number`	Issue/PR number, or commit SHA
`title`	Title (or commit message headline)
`body`	Issue/PR body (or full commit message)
`state`	`OPEN` / `CLOSED` / `MERGED` (empty for commit)
`author`	GitHub login of the author
`created_at`	ISO8601
`closed_at`	ISO8601 (empty for commits)
`url`	Permalink
`labels`	List of label names
`comments`	First N comments concatenated

Usage in a recipe

{
  "seed_config": {
    "source": {
      "seed_type": "github_repo",
      "repos": ["unslothai/unsloth", "unslothai/unsloth-zoo"],
      "token": "",
      "item_types": ["issues", "pulls"],
      "limit": 100,
      "include_comments": true,
      "max_comments_per_item": 30
    },
    "sampling_strategy": "shuffle",
    "selection_strategy": null
  }
}

Leave token empty to fall back to the server's GH_TOKEN / GITHUB_TOKEN environment variable, useful when the recipe is published and shouldn't carry a secret.

Auth

A GitHub personal access token with public_repo scope is enough for public repositories; repo scope is required for private ones. GraphQL requests are rate-limit aware: the client inspects x-ratelimit-* headers and sleeps until reset when the budget drops below a safety threshold.

Install

Shipped as a default Unsloth plugin. For development:

pip install -e .

Registered automatically via the data_designer.plugins entry point.