History

Daniel Han b09aa82a3a Studio: add github_repo seed reader and GitHub Support Bot recipe (#5169 ) * Studio: add github_repo seed reader and GitHub Support Bot recipe Adds a first-party Data Designer seed reader that scrapes GitHub issues, pull requests, and commits from one or more repositories via the GraphQL API, and a learning recipe (GitHub Support Bot) that turns those rows into synthetic support Q&A pairs for fine-tuning. Backend (new plugin studio/backend/plugins/data-designer-github-repo-seed): * GitHubRepoSeedSource config: repos, token (falls back to GH_TOKEN / GITHUB_TOKEN env var), item_types (issues / pulls / commits), per-resource limit (0 means all), max_comments_per_item. * Rate-limit-aware GraphQL client (GitHubClient + RepoScraper) shared across repos; flattens each item into a uniform row with columns item_type, repo, number, title, body, state, author, created_at, closed_at, url, labels, comments. * Registered via the data_designer.plugins entry point. Frontend: * New seed_github block variant so the seed node card shows "GitHub repositories" instead of the generic "Document file" placeholder, with its own icon and inline summary (repo count + item-type list). * Rewritten seed dialog github_repo form: repos textarea pre-filled with unslothai/unsloth + unslothai/unsloth-zoo, password input for the GH token, items-per-repo number with an "All" toggle, and the noisier options (item types, max comments, include comments) tucked under an Advanced collapsible. * Local model auto-load on Run: if a recipe uses an is_local provider and the inference server is not already serving that model, the executions hook calls /api/inference/load first. Removes the "open /chat to load a model" prerequisite that users kept tripping on. * Honor the recipe's run.rows value in the Run dialog (previously the store reset to 5 regardless of what the template shipped). Recipe (studio/frontend/src/features/data-recipes/learning-recipes/ github-support-bot.json): * Defaults to the Local Model provider + unsloth/gemma-4-E2B-it-GGUF. * Scrapes unslothai/unsloth and unslothai/unsloth-zoo, issues and pulls, up to 100 items per resource. * Two LLM blocks: normalized_question (llm-text) rewrites each thread into a clean support question, support_answer (llm-structured) produces JSON with answer / diagnosis_questions / cites / confidence. * Run defaults to 10 rows for a quick smoke test. Verified end-to-end on a running Studio: card renders, source-data dialog is pre-populated, All toggle disables the limit input, the recipe executes and produces rows against a loaded local GGUF. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: improve GitHub recipe support * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: speed up GitHub scraper and harden the support-bot recipe Addresses a perf issue found while demoing the github_repo seed reader: Scraper is too slow at scale. The PRs GraphQL query pulls deeply nested fields (reviewThreads, reviews, commits, timelineItems, etc.) so the page size was pinned at 3 to stay under GitHub's node-count ceiling. 100 PRs meant 34 serial round trips. Added lighter query variants (PRS_PAGE_QUERY_LIGHT, ISSUES_PAGE_QUERY_LIGHT) that drop the fields the Studio flatten layer does not use (it only reads title, body, state, author, labels, comments). With the light query PR pages can safely go to 25 per page and issues to 50. The plugin scraper now passes light=True to RepoScraper so Studio always uses the fast path; the heavy query remains available for other callers. Recipe defaults are now demo-ready with production knobs called out: - max_parallel_requests: 1 and max_tokens: 800 so small local models stay stable when running the support_answer structured column. - support_answer prompt trimmed to 80-200 words so gemma-4-E2B GGUF can actually comply with the schema. The canonical 150-300 word codex prompt is still documented in the node3 markdown note for production upgrades. * Studio: rename GitHub recipe to 'GitHub Scraper' and add Easy mode Changes the recipe framing from a single-purpose 'Support Bot' pipeline to a general-purpose scraper that produces {user_request, grounded_response} training pairs. Aligns with the canonical github_data_gatherer dataset (11 enrichment tasks mirrored in pr_requests_20 / issue_requests_20 on the input side and explain_pr / issue_fix_plan / issue_solution on the output side). Recipe JSON changes: - columns[0] renamed normalized_question -> user_request, prompt now inverts a GitHub thread into a realistic user ask instead of normalising it. - columns[1] renamed support_answer -> coauthor_response, emits {response, followups, cites, task, confidence} and branches on issue vs PR thread type. - Notes rewritten to document the 11-task catalog and the canonical production prompt to paste in for a full dataset backfill. Frontend: Easy mode for github_repo recipes. The drag-and-drop canvas is hidden behind an 'Advanced' tab; Easy mode is the default for any recipe whose seed_source_type is github_repo. The Easy form reuses the existing GithubRepoSeedForm (promoted to exported), adds a rows input bound to previewRows, a model field bound to the model_config, and a single Run button that calls runPreview() directly (no modal). Non-github recipes see the same Editor / Runs tabs as before. View mode persists per-recipe-id in localStorage under recipe-studio:view-mode:<recipeId>. * Studio: auto-detect server GH_TOKEN and widen Easy-mode detection The GitHub seed form now fetches /api/data-recipe/seed/github/env-token on mount and, when the server exposes a GH_TOKEN / GITHUB_TOKEN env var and the token field is blank, shows a small 'Using server env var' badge and swaps the placeholder text. The token value itself is never returned to the UI. Widens Easy-mode detection in recipe-studio-page.tsx so that recipes saved before ui.seed_source_type was persisted also get the Easy tab: falls back to recipe.seed_config.source.seed_type, which is always present for github_repo seeds. * fix: polish GitHub recipe UI * Studio: default llama-server --threads to -1 (auto) Previously we passed --threads only when the caller set an explicit value, which meant llama-server fell back to its internal default. That default has varied across llama.cpp builds (some versions use hardware concurrency including hyperthreads, which hurts throughput on CPU-heavy inference). Always passing --threads -1 pins the behaviour to llama.cpp's auto-detect (physical cores). Caller-supplied n_threads still wins when non-None. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch Easy mode to Runs pane on run start Easy mode had no progress island or canvas overlay, so after clicking Run the only visible state was the button label flipping to "Running..." while the screen otherwise stayed identical. This reads as stuck even though the job is progressing. Wire an onExecutionStart callback from recipe-studio-page.tsx through to useRecipeExecutions so that when a run is kicked off from easy mode, the page flips to the executions view where the Runs sidebar, progress bar, rate/ETA panel, and live log are rendered. Advanced/editor mode keeps its existing behavior and stays on the canvas (it already has the floating ExecutionProgressIsland). * fix: clean up GitHub scraper layout * Studio: forward llm-structured output_format as llama-server response_format Local GGUF runs of llm-structured columns used to generate the full max_tokens budget before the prompt-level "return JSON in a ```json fence" instruction got parsed. Small models (e.g. gemma-4-E2B-it) routinely broke format, so each row took ~65s and frequently failed with "No parsable JSON structure within ```json markdown fence". For any local-provider model_config referenced by an llm-structured column, clone the model_config and inject response_format into the clone's inference_parameters. Uses llama.cpp server's flat shape (tools/server/README.md): {"type": "json_schema", "schema": <output_format>} Not the OpenAI-nested form; data_designer's OpenAI adapter forwards response_format verbatim via facade._COMPLETION_REQUEST_FIELDS, and llama-server's documented schema path expects the flat variant. The clone is per (model_alias, column) so: - llm-text / llm-judge columns that share the same alias keep free-form sampling. - Each structured column gets its own schema, so columns with different output_formats don't collide. Effect on gemma-4-E2B-it demos: every row parses cleanly, and the model terminates immediately after the closing brace instead of running to max_tokens. Net wall-clock is usually faster even though grammar-constrained sampling is slightly slower per token. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: flip Easy to Runs pane before validation scrape, not after Previously onExecutionStart fired inside runExecution, which runs AFTER validateRecipe() -- and validation re-invokes the seed reader. For the github_repo reader that is a full GraphQL scrape, so the user sat on a "Running..." button with an otherwise unchanged Easy form for 10-15s before anything moved. Call onExecutionStart at the top of runWithValidation, right after we have a payload to send. The view flips immediately; ensureLocalModelLoaded + validateRecipe now run against the Runs pane instead of a frozen Easy form. runExecution still calls onExecutionStart downstream, but the callback is idempotent (the page's easy -> executions guard skips the second call), so no behaviour change for runs that pass validation. If validation fails the toast + runErrors path still fires; the Easy form's error banner still reads runErrors when the user switches back. * Studio: unify data-recipe workflow auth on sk-unsloth-* keys The previous commit (`a61b4cc9`) assumed storage.create_api_key(..., internal=True) and storage.revoke_internal_api_key(key_id) existed, but those helpers were only in the working tree, never committed. Recipe runs in local-model mode were therefore crashing with 500 when _inject_local_providers tried to mint a workflow key. This commit ships the missing pieces. auth/storage.py: - api_keys schema gains is_internal INTEGER DEFAULT 0 (with a guarded ALTER TABLE migration so existing auth.db files upgrade in place). - create_api_key takes an internal=False kwarg; internal keys are flagged so they can be hidden from user-facing listings. - list_api_keys takes include_internal=False so UIs never see workflow keys. - New revoke_internal_api_key(key_id): id-only revoke for keys minted by non-user subjects (the JobManager does not know a username). core/data_recipe/jobs/manager.py: - JobManager.start accepts internal_api_key_id and stores it on Job so lifecycle handlers can revoke eagerly. - _handle_event revokes on EVENT_JOB_COMPLETED / _ERROR / _CANCELLED. - _pump_loop subprocess-died fallback also retires the key so a crashed worker cannot leak a live sk-unsloth-* beyond its TTL. - Revocation is best-effort (swallow exceptions) -- the 24h TTL is the safety net if storage hiccups. core/data_recipe/jobs/types.py: - Job dataclass gains internal_api_key_id: int \| None = None. Replaces the bespoke 24h JWT path that jobs.py used to mint for local providers. One mint/revoke/verify surface for every API key the server issues, and revocation is now eager (seconds, not 24h) instead of TTL-only. * Studio: plug workflow-key leak on unexpected create_job errors Review follow-up on the sk-unsloth-* workflow-key lifecycle in create_job. Previously the revoke handlers wrapped mgr.start(...) but only caught RuntimeError and ValueError, and get_job_manager() sat outside the try block entirely. Any other exception type (TypeError from a mismatched kwarg, OSError from the queue write, etc.) would bubble up to FastAPI and leave the minted key live until its 24h TTL. Fix: one try block covers both get_job_manager() and mgr.start(), with a trailing except Exception that revokes and re-raises. The RuntimeError -> 409 and ValueError -> 400 paths are unchanged so specific client-facing status codes still surface. Revocation is still best-effort (_revoke_internal_api_key_safe swallows errors) because we never want revoke failures to mask the original crash. Severity is low -- the key can't bootstrap longer access and the 24h TTL bounds the window -- but the reviewer's point stands: eager revoke on every failure path is the right invariant. * Studio: nest response_format under extra_body so pydantic accepts it The previous commit dropped response_format at the top level of a cloned model_config's inference_parameters, which BuilderConfig rejected with: ValidationError: Extra inputs are not permitted [type=extra_forbidden] data_designer.model_configs.1.inference_parameters.response_format data_designer's BaseInferenceParams is a pydantic model with extra=forbid and only a fixed set of fields (temperature, top_p, max_tokens, max_parallel_requests, timeout, extra_body). The pass-through path for anything the schema doesn't know about is `extra_body`, which the OpenAI SDK spreads into the chat-completions request body at the top level -- which is exactly where llama-server reads response_format from. Inject under extra_body (merging with any existing extra_body contents) so the clone validates. llama-server still receives {"type": "json_schema", "schema": <output_format>} at the top level of the request body, which is the flat shape llama.cpp's server expects. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: forward response_format to llama-server and fence-wrap the reply Two-part fix for the llm-structured data-recipe path: (1) The /v1/chat/completions proxy was dropping response_format. The route's passthrough branch only triggered on tools / tool messages, so requests carrying a JSON schema fell into the non-passthrough GGUF path which calls generate_chat_completion (no response_format kwarg). The schema never reached llama-server, so guided decoding was a no-op and the model emitted free-form text that happened to parse a fraction of the time. Widen the passthrough trigger and teach _build_passthrough_payload to forward response_format so llama-server's GBNF grammar actually runs. Guided decoding does not require supports_tools, so split the condition: a request is now passthrough-routed if it carries tools/tool messages (existing behavior) OR carries response_format (new). The vision guard, streaming fork, and tools-choice defaulting are unchanged. (2) data_designer's llm-structured parser looks for a ```json ... ``` markdown fence and discards anything else. Guided decoding emits only the JSON object (the GBNF grammar has no fence tokens), so a 100%-valid schema-constrained run still ended up 0 ok / N failed with "No parsable JSON structure within ```json markdown fence". In _openai_passthrough_non_streaming, wrap each choice's content in the expected fence when the caller asked for guided decoding. Already-fenced content is left alone so other clients that prefer raw JSON are not affected; the wrap is scoped to requests that carried response_format. Net effect on the GitHub Support Bot recipe on a local GGUF: schema actually binds during sampling, content arrives wrapped in the fence data_designer expects, and generation terminates immediately after the closing brace instead of running out to max_tokens. * Studio: Easy mode runs a full run, capped at the user's row count Easy mode used to call runPreview, which produces a test run: no artifact persisted, reduced progress tracking, and framed in the Runs pane as "Test run". The whole point of the form is to let a user kick off a real dataset build with one click, so wire it to runFull instead and bind the Rows input to fullRows (not previewRows). runFull requires a non-empty fullRunName. The Easy form has no run-name input, so seed a default on mount whenever Easy is active and fullRunName is still empty. Uses `<recipe name> <iso-timestamp>` so each Easy run gets a stable-ish default that still sorts chronologically in the Runs pane. User can override it from the Advanced run dialog before clicking Run. Rename GithubScraperEasyView's rows props from previewRows/setPreviewRows to rows/setRows so the view stays agnostic to which hook state the page chooses to bind. Loading indicator now follows fullLoading. * Studio: clamp GitHub scrape page size and memoize the materialization Two wins for the "before Generating fires" gap on small previews: (1) scrape_{issues,prs,commits} hardcoded per_page (50 / 25 / 100) and only checked the trial limit AFTER the page was written, so a 1-row Easy run still asked GitHub for a full 50-issue + 25-PR page, wrote them all to JSONL, and then stopped because total_new already exceeded the trial cap. Cap per_page at min(page_cap, trial_limit) so github_limit=1 actually asks for first:1. (2) GitHubRepoSeedReader.get_dataset_uri used to scrape fresh on every invocation. data_designer calls the seed reader multiple times per recipe job (validation, preview, per-column sampling), so a 2-repo Easy preview ran the full GraphQL scrape three times back-to-back, burning ~15s of dead air before any LLM generation began. Added a module-level in-process cache keyed on (repos, item_types, limit, include_comments, max_comments_per_item, sha256(token)[:16]) that stores the JSONL path of the first materialization. Subsequent calls with the same signature return the cached path, guarded by a staleness check that drops the entry if the file was tmp-cleaned. Raw token values never land in the key. Net effect on a 1-row Easy run, 2 repos, limit=1: 2 GraphQL round trips instead of ~12, and the first-to-Generating gap collapses from ~15s to roughly 2-3s. * Studio: make Easy mode Rows input editable instead of snapping to 1 The Rows to generate input used type="number" with value bound directly to the rows state and an onChange that coerced any non-positive parse result back to 1. The moment the user pressed backspace to clear the field, the parent re-rendered with value=1 and the caret jumped, making it impossible to change the value without arrowing the browser's +/- spinner. Switch to a text input with inputMode="numeric" and pattern="[0-9]" (so mobile still shows a numeric keyboard, and the browser drops the spinner buttons the user did not want). Add a local rowsText buffer so the field can hold transient empty / partial digit strings while editing without fighting the parent state; the canonical rows value only advances when the buffer parses to a valid integer in [1, 10000], and onBlur clamps back to 1 or 10000 if the user left it out of range. No behavior change for valid numeric edits - the downstream runFull() still sees a clean positive integer. Studio: expand dataset cells horizontally by column on click Click a long cell to expand that whole column. Click again to collapse. Replaces the prior row-level vertical expansion which made it hard to compare cells across columns. State is scoped per execution and per column; the row itself is no longer a click target. * Studio: force expanded dataset column to grow wide enough to read * Studio: disable thinking for local recipe inference and plumb the kwarg Reasoning-capable models (gemma-3n, qwen3.5, etc.) emit a <think>...</think> preamble ahead of the answer by default, which roughly doubles the generated token count per row on a local GGUF and pushes the actual answer past data_designer's json-fence regex on llm-structured columns. Recipes want the terse answer, not the scratchpad. Two halves of the fix: (1) routes/data_recipe/jobs.py: when _inject_local_providers walks the recipe's model_configs to point them at the local endpoint, also stash chat_template_kwargs={"enable_thinking": false} under each config's inference_parameters.extra_body. OpenAI SDK spreads extra_body into the top-level request body, so llama-server and the Studio /v1/chat/completions route both see it. (2) routes/inference.py: the chat-completions route previously dropped chat_template_kwargs on the floor because the whitelist body builder only forwarded known fields. - At the top of openai_chat_completions, lift chat_template_kwargs.enable_thinking from payload.model_extra onto the typed payload.enable_thinking field when the caller did not set the latter, so the non-passthrough GGUF path's generate_chat_completion(...) call honors the override. - Teach _build_passthrough_payload to forward a chat_template_kwargs dict, and have _build_openai_passthrough_body derive that dict from payload.enable_thinking so response_format requests (structured columns) also land at llama-server with the reasoning preamble suppressed. Net effect on a 10-row support-bot run with gemma-4-E2B-it-GGUF: responses arrive without <think> tags, wall-clock per call drops roughly in half, and structured columns stop leaking reasoning tokens through the GBNF-constrained output. * Studio: update GitHub Support Bot learning recipe with maintainer layout Replace the template with the hand-laid-out export from the maintainer so note nodes ship with real x/y positions (scattered around the graph instead of all stacked at x=480) and the edges / canvas pan look correct on first load. Also picks up the maintainer's prompt tweaks and output schema names (coauthor_response / user_request / followups / task / cites / confidence). Diff is mostly ui.nodes positions and prompt bodies; runtime shape is unchanged (seed_config / columns still target model_1 against the Local Model provider). * Studio: auto-size dataset sample columns; wide text gets a wide column Drop the per-column click-to-expand toggle and the 180-char truncation. Every column now renders its full value. Columns with long text get a min-w of 48rem so the text is readable without wrapping into a tall block; narrow-content columns get a 12rem min-w. The table wrapper already has overflow-x-auto, so wide-column totals cause a horizontal scrollbar instead of cramming everything into the viewport. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix GitHub scrape progress * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add resetApiBase export for test setup * Studio: rename github-support-bot output columns to User / Assistant Previously emitted user_request and coauthor_response, which did not match the canonical User / Assistant chat-pair shape that downstream SFT consumers expect. Renamed the columns in the recipe JSON (columns, UI node ids, edges, notes, prompt Jinja refs) and the matching copy in the learning-recipes index, data-recipes-page, and easy view. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: wasimysaid <wasimysdev@gmail.com>		2026-04-24 12:02:03 -07:00
..
src/data_designer_github_repo_seed	Studio: add github_repo seed reader and GitHub Support Bot recipe (#5169 )	2026-04-24 12:02:03 -07:00
pyproject.toml	Studio: add github_repo seed reader and GitHub Support Bot recipe (#5169 )	2026-04-24 12:02:03 -07:00
README.md	Studio: add github_repo seed reader and GitHub Support Bot recipe (#5169 )	2026-04-24 12:02:03 -07:00

README.md

data-designer-github-repo-seed

A Data Designer seed-reader plugin for Unsloth Studio that scrapes real GitHub data (issues, pull requests, commits) from one or more repositories and hands it to the recipe pipeline as a seed dataset.

Designed to ship with Studio as a default seed source so any user with a GitHub token can build training datasets straight from live repos.

What it does

Given a list of owner/name repos, a GitHub token, and a per-resource limit, the plugin uses GitHub's GraphQL API to fetch issues, pull requests, and/or commits, with labels, state, authors, and the first N comments of each item, and materialises a single JSONL with uniform columns so the rest of the recipe (LLM text / LLM structured / processors) can treat it like any other seed table.

Column	Description
`item_type`	`issue` / `pull` / `commit`
`repo`	`owner/name`
`number`	Issue/PR number, or commit SHA
`title`	Title (or commit message headline)
`body`	Issue/PR body (or full commit message)
`state`	`OPEN` / `CLOSED` / `MERGED` (empty for commit)
`author`	GitHub login of the author
`created_at`	ISO8601
`closed_at`	ISO8601 (empty for commits)
`url`	Permalink
`labels`	List of label names
`comments`	First N comments concatenated

Usage in a recipe

{
  "seed_config": {
    "source": {
      "seed_type": "github_repo",
      "repos": ["unslothai/unsloth", "unslothai/unsloth-zoo"],
      "token": "",
      "item_types": ["issues", "pulls"],
      "limit": 100,
      "include_comments": true,
      "max_comments_per_item": 30
    },
    "sampling_strategy": "shuffle",
    "selection_strategy": null
  }
}

Leave token empty to fall back to the server's GH_TOKEN / GITHUB_TOKEN environment variable, useful when the recipe is published and shouldn't carry a secret.

Auth

A GitHub personal access token with public_repo scope is enough for public repositories; repo scope is required for private ones. GraphQL requests are rate-limit aware: the client inspects x-ratelimit-* headers and sleeps until reset when the budget drops below a safety threshold.

Install

Shipped as a default Studio plugin. For development:

pip install -e .

Registered automatically via the data_designer.plugins entry point.