* Studio: harden the data-recipe and inference consumer loops against pump death Follow-up to #6643. The same single-unsupervised-consumer pattern the training pump had lives in two sibling loops, with the same failure mode: one bad event kills the only thread that updates the in-memory state every UI surface reads, while the worker subprocess keeps running. - data_recipe JobManager._pump_loop: a malformed worker log line that makes parse_log_message raise no longer kills the pump. Guard _handle_event, the queue read, and the worker-exit finalize, and broaden _drain_queue so a drain error still finalizes the job instead of leaving it wedged "active" (which also leaked the workflow-scoped API key until its 24h expiry). - inference InferenceOrchestrator._dispatcher_loop: guard the routing body so a malformed response or a mailbox put error can't kill the dispatcher and hang every in-flight generation (callers key liveness on the subprocess, not on this thread). Adds regression tests for both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: extend consumer-loop hardening to RAG, hub, auth, and stream-reader paths Continuation of the data-recipe and inference pump hardening: the same "background producer updates in-memory state that a single unsupervised consumer surfaces to the UI" pattern shows up in several more Studio paths, each able to silently freeze a UI surface while the worker keeps running. RAG ingestion SSE (core/rag/ingestion.py): - job_events polled the queue with a blocking get and never noticed client disconnect or a dead worker, so a closed tab or a producer that died without emitting a terminal event left the stream hanging. It now polls with a timeout, emits heartbeats, ends on terminal job status, caps idle time, and always pops the job registry in finally. - Added _reap_finished_jobs() and call it from start_ingestion so finished job state does not accumulate. Startup reconcile (storage/rag_db.py, main.py): - reconcile_orphaned_ingestion_jobs() marks ingestion jobs (and their documents) that were left non-terminal by a previous crash as failed, so the UI does not show jobs stuck "running" forever after a restart. Wired in at startup next to cleanup_orphaned_runs(). Hub download watcher (hub/services/download_lifecycle.py): - _watch() could leave a job pinned "running" if finalize raised. Body is now guarded: on failure it logs and sets the job to error, and always invalidates the hf cache scan in finally. External provider stream (core/inference/external_provider.py): - read timeout was None (no stall ceiling); set to 300s so a wedged upstream surfaces as an error instead of an indefinitely hung stream. Auth store (auth/storage.py): - Enable WAL + busy_timeout on the auth DB so token validation (read on every request) and login writes stop serialising on the rollback journal. Matches studio_db / rag_db / providers_db. Login rate limiter (routes/auth.py): - _LOGIN_IP_BUCKETS could grow unbounded under spoofed-IP traffic; cap it and prune stale buckets, mirroring the per-account bucket handling. Training progress SSE (routes/training.py): - Break promptly on client disconnect instead of waiting for the next yield to fail on a closed socket, matching the export / data-recipe SSE routes. llama-server stdout drain (core/inference/llama_cpp.py): - Broaden the drain guard so an unexpected decode/read error logs at debug and stops the drainer cleanly instead of escaping the thread. Frontend stream readers (chat-api.ts, rag-api.ts): - Wrap the SSE read loops in try/finally + reader.cancel() so early return ([DONE]), thrown errors, and consumer aborts release the reader lock instead of holding it until GC. Tests: - test_training_progress_stream_nan: fake request now implements the async is_disconnected() the route polls, matching the other SSE route fakes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address Codex review feedback on the consumer-loop hardening Four follow-ups from the automated review, all on code this PR introduced: - Data-recipe pump (manager.py): a queue read that keeps raising an error outside the read's narrow catch set (e.g. a broken queue pipe after the child died) hit the `continue` guard and skipped the dead-worker finalize below, spinning forever and leaving the job wedged "active" with its workflow key unretired. On a read failure, fall through to finalize when the worker is no longer alive. Added a regression test. - RAG ingestion SSE (ingestion.py): the 5-minute idle cap could end the stream while the job was still pending/running (a large document spends minutes in embedding/storing with no per-batch progress event). The route then sends [DONE], and the client treats a no-terminal-frame end as completion, marking the document indexed mid-ingestion. Drop the idle cap: while the worker is alive and non-terminal we keep heartbeating; the stream ends only on terminal DB status, the None sentinel, or client disconnect. - Login rate limiter (auth.py): the per-IP path pruned but then added the new IP unconditionally, so a spoofed-source-IP spray kept _LOGIN_IP_BUCKETS unbounded and made every new IP pay a full-dict prune scan. Gate the add on the cap, mirroring the account path. - Hub download watcher (download_lifecycle.py): if finalize raised before it reaped (proc.wait) and dropped the worker (e.g. an I/O error draining stderr), the crash path published a terminal state while the live Popen stayed registered and kept writing the cache, and the terminal set_job let claim() admit a retry on the same repo. Terminate + drop the worker before setting the terminal state. * Studio: keep login throttling working when the per-IP bucket dict saturates Review follow-up. The previous cap fix skipped creating a bucket for a new IP once _LOGIN_IP_BUCKETS was full, returning ip_fails=0. Under a sustained spray that also fills the account dict, every failure from such an IP then looked first-seen and _login_blocked had no bucket to enforce, so the cap effectively disabled throttling once saturated. Bound the dict with a FIFO eviction instead: if the IP is new and the dict is full, reclaim expired buckets (rate-limited so a burst of distinct IPs can't make each failure an O(n) sweep) and, if still full, evict the oldest-inserted IP. The new IP always gets a real bucket, so a saturating (e.g. spoofed X-Forwarded-For) spray stays throttled while memory stays bounded. Added a regression test that saturates the dict and asserts a later IP is still blocked. * Studio: address Codex review (RAG queue lifecycle, stream error, orphan chunks) Three follow-ups on the Phase 6 changes: - RAG ingestion SSE (ingestion.py): job_events removed the per-job queue in its finally on ANY exit, including an early client disconnect while the worker is still running. That dropped the worker's later events (the queue is the only one _emit writes to) and made a reconnect find no queue and receive only [DONE], which the client treats as completion. Only drop the queue on a terminal exit (None sentinel / terminal DB status); leftover terminal queues are still swept by _reap_finished_jobs. Added queue-lifecycle tests. - External provider stream (routes/inference.py): once the 300s read timeout can fire, the stream's except path failed the monitor but ended without an error frame or [DONE], so the chat client saw a bare EOF and saved the timed-out answer as a successful partial with no error. Emit an SSE error frame (and [DONE]) on stream failure so the client surfaces it. - RAG startup reconcile (storage/rag_db.py): marking a half-ingested document failed left its chunks/fts/vec rows intact, and retrieval filters by scope not status, so a failed document could still be retrieved and cited. Purge the document's chunks when reconciling it to failed (the doc row stays for re-ingest). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: release the remaining SSE stream readers (training, data-recipe, export) reviewer.py follow-up. The chat and RAG SSE readers were wrapped in try/finally + reader.cancel(), but the other three readers built on the same response.body.getReader() pattern were left without it: streamTrainingProgress, streamRecipeJobEvents, and streamExportLogs leak the ReadableStreamDefaultReader lock (held until GC) when the consumer aborts, returns early, or a parse/callback throws. Wrap each in try/finally + reader.cancel() (export already had a try/catch, so it only needed the finally). All five frontend SSE readers now release the reader symmetrically. * Tighten resilience comments and docstrings Condense the verbose explanatory comments and internal-helper docstrings added in this branch to shorter, clearer forms. Comment/whitespace only; verified no code changed via AST diff. No behaviour change. * Studio: keep chunks for completed docs during ingestion reconcile Startup reconciliation flips orphaned (non-terminal) ingestion jobs to failed and purges the document's chunks so a failed source can't be retrieved. But it dropped the chunks unconditionally, so a document the worker had already committed as 'completed' before the crash (only its job row left non-terminal) lost every chunk while still reporting 'completed'. That leaves an empty source that retrieval can't return and dedup (status != 'failed') blocks from re-ingest. Only purge chunks when the document UPDATE actually transitions it to failed; an already-completed document keeps its chunks. Adds reconcile regression tests for both the completed-doc and genuine in-flight-orphan cases. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: drop a finished RAG job's queue when the client disconnects job_events kept the per-job queue until it consumed the None sentinel, so a UI that stops on the terminal event (its reader.cancel aborts the stream before [DONE]) left the queue registered until the next _reap_finished_jobs sweep; a batch of uploads followed by idling retained them all. _run writes the terminal DB status before emitting the terminal event, so on generator exit, drop the queue when the job's DB row is already terminal (worker done, nothing to resume) and keep it only while the worker is still running. Adds a disconnect-after-terminal-event regression test. * Remove stray async task output files committed by mistake * Studio: harden login IP throttle and end progress stream on disconnect Two Codex review items: Login per-IP throttle: when the per-IP bucket dict saturated, FIFO eviction could drop a still-hot (blocked) bucket, so an IP could flood the dict with distinct (or spoofed) source IPs to push out its own bucket and retry as first-seen. Stop evicting hot buckets; a new IP that can't fit now shares a bounded overflow counter that still trips the per-IP threshold, so a saturating spray stays throttled and no live counter is reset. Progress SSE: on client disconnect the polling loop only broke and fell through to the unconditional final 'complete' frame, so a buffered or proxying consumer could read a still-active run as completed. Return from the generator instead. Adds regression tests for both (spray cannot reset a hot bucket; disconnect while active emits no complete frame). * Studio: shard the login overflow counter and stop cancelling chat stream after [DONE] Two Codex review items: Login throttle overflow: the single shared overflow counter meant that once a saturating spray pushed it past the per-IP threshold, _login_blocked returned 429 for every new unbucketed source IP, before credentials were checked -- a global login denial. Shard the overflow into a fixed array of counters keyed by hash(ip), so a hot shard only throttles the IPs that map to it while a single source's repeated failures still concentrate in one shard and stay throttled. Memory stays bounded and no live bucket is evicted. Adds a regression test that a hot overflow shard does not block an unrelated IP. Chat stream: the reader.cancel() in the SSE finally fired even after a natural [DONE]/EOF. The backend finalizes its api-monitor entry right after yielding the sentinel (the local pass-through finishes after the last yield), so a client cancel there can be observed as a disconnect and mark a completed request as cancelled. Track natural completion and only cancel on an early/abnormal exit. (No frontend unit test: the Studio frontend has no test harness.) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give prep-timeout test fakes an is_disconnected method The progress stream now ends on client disconnect (await request.is_disconnected() before falling through to the terminal frame). After merging that into the prep-timeout tests added later on main, their _FakeRequest/_ReconnectRequest must provide is_disconnected or the generator raises AttributeError under CI. * Studio: keep the login overflow throttle when bucket capacity frees up _login_blocked only consulted the per-IP overflow shard while the bucket dict was still at capacity. If a slot freed before the 60s window expired (e.g. another IP's successful login calls _clear_login_bucket), a source counted in a hot shard stopped being blocked and its next failure got a fresh per-IP bucket, resetting the throttle the overflow path exists to preserve. Always max in the IP's shard (shards are empty outside saturation, so it is a no-op in the common case). Adds a regression test that a hot source stays throttled after a bucket frees. * Studio: clear a login IP's overflow throttle on successful login _clear_login_bucket reset the per-IP and per-account buckets on a successful login but not the overflow shard, so after the dict saturated and an IP was counted in overflow, a later successful login left those entries behind and the next failed attempt could immediately return 429. Store overflow entries as (timestamp, ip) so a source is throttled by its own count within the shard (also removing cross-IP collateral within a shard), and drop just that IP's entries in _clear_login_bucket. Adds a regression test that a successful login clears the overflow throttle. * Studio: bound the login overflow shard memory under high-cardinality spray The per-IP overflow tracked failures in a time-pruned deque of (timestamp, ip) tuples, so a spoofed-X-Forwarded-For spray of distinct one-off IPs grew memory and the per-check scan with request cardinality for the whole window -- undermining the bucket cap that exists to bound memory. Replace each shard with a fixed- capacity dict (ip -> [count, window_start]): O(1) lookups, and when a shard is full a one-off IP evicts the lowest-count entry (Space-Saving) so memory is hard- bounded while a persistent attacker keeps a high count and is never evicted. Adds a regression test that shards stay within the per-shard cap under a 5000-IP spray. * Studio: purge chunks for already-failed docs during ingestion reconcile The reconcile chunk-purge was gated on the documents UPDATE actually flipping a non-terminal doc to failed. A doc the worker had already marked 'failed' before the crash (job row left non-terminal) was not re-flipped, so its committed chunks were kept and stayed retrievable/citable, since retrieval filters by scope not status. Purge chunks whenever the document is not 'completed' (failed, in-flight, or gone), preserving the completed-doc carve-out. Adds a regression test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: don't inherit an evicted IP's count onto a new overflow source When a full overflow shard evicted the lowest-count entry, the new source inherited that count (Space-Saving base + 1). If a shard was saturated with hot entries, an unrelated new IP could land at/over the threshold and be 429'd after a single attempt -- cross-IP collateral despite the per-source-isolation intent. New entries now start clean at count 1; the only cost is that a heavy hitter that is the lowest-count entry in a fully saturated shard can briefly reset, which is preferable to blocking a bystander. Adds a regression test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: carry overflow failures into a new IP bucket on transition _login_blocked took max(per-IP bucket, overflow shard) rather than combining them, so a source could log (threshold-1) failures in overflow during saturation and, once a bucket slot freed, another (threshold-1) in a fresh bucket within the same window -- roughly doubling the per-IP limit. When a saturated-era IP first gets a real bucket, migrate its windowed overflow count into that bucket (and drop the overflow entry) so the combined failures throttle at the intended limit. Adds a regression test. * Studio: reconcile a completed doc's orphaned job to completed, not failed When a crash left an ingestion job non-terminal after its document was already committed as completed, reconcile marked the job failed. After restart the upload UI has no in-memory SSE queue and falls back to getJob(), which treats a failed job as an indexing failure and removes/toasts a document that is actually searchable. Mark the job completed (keeping its chunks) when its document is completed. Extends the completed-doc reconcile test to assert the job status. * Studio: clamp the overflow failure count migrated into a login bucket A saturated source could accrue an unbounded overflow count, then materialize one deque entry per recorded failure when a bucket slot freed, allocating an arbitrarily large deque under the login lock. Only at-or-above the per-IP threshold matters for blocking, so cap the count there at the record and take sites; the migration is now bounded without weakening the limit. * Studio: keep the RAG job stream alive on a transient status read The heartbeat poll read the job row unguarded; a momentarily-locked DB would raise out of job_events, which the SSE route turns into a terminal error frame, and the UI drops a document whose worker is still running. Treat a failed status read as non-terminal: heartbeat and retry, and keep the queue so a reconnect can resume. * Studio: set busy_timeout before journal_mode on the auth DB Switching journal_mode needs a lock, so if a refresh-token write already holds one, journal_mode=WAL raises SQLITE_BUSY and the shared try leaves the connection on SQLite's default zero lock wait. Set busy_timeout first so the switch waits instead of failing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
575 lines
22 KiB
Python
575 lines
22 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Authentication API routes."""
|
|
|
|
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
|
|
|
|
import base64
|
|
import ipaddress
|
|
import os
|
|
import shlex
|
|
import sys
|
|
import threading
|
|
import time
|
|
from collections import deque
|
|
from datetime import datetime, timedelta, timezone
|
|
|
|
from models.auth import (
|
|
ApiKeyListResponse,
|
|
ApiKeyResponse,
|
|
AuthLoginRequest,
|
|
AuthStatusResponse,
|
|
ChangePasswordRequest,
|
|
CreateApiKeyRequest,
|
|
CreateApiKeyResponse,
|
|
DesktopLoginRequest,
|
|
RefreshTokenRequest,
|
|
)
|
|
from models.users import Token
|
|
from auth import storage, hashing
|
|
from auth.authentication import (
|
|
create_access_token,
|
|
create_refresh_token,
|
|
get_current_subject,
|
|
get_current_subject_allow_password_change,
|
|
refresh_access_token,
|
|
)
|
|
|
|
router = APIRouter()
|
|
|
|
|
|
def _reset_password_command() -> str:
|
|
"""Shell command shown in the 'incorrect password' hint.
|
|
|
|
Prefer the absolute path to this install's ``unsloth`` launcher (sibling of
|
|
the running interpreter) so the hint works even when its dir isn't on PATH.
|
|
|
|
POSIX paths are shell-quoted. On Windows we use the bare absolute path only
|
|
when it has no spaces (a quoted path differs between cmd and PowerShell);
|
|
otherwise, or if the launcher can't be located, fall back to the PATH form.
|
|
"""
|
|
try:
|
|
bin_dir = os.path.dirname(os.path.abspath(sys.executable))
|
|
if os.name == "nt":
|
|
exe = os.path.join(bin_dir, "unsloth.exe")
|
|
if os.path.isfile(exe) and " " not in exe:
|
|
return f"{exe} studio reset-password"
|
|
else:
|
|
exe = os.path.join(bin_dir, "unsloth")
|
|
if os.path.isfile(exe):
|
|
return f"{shlex.quote(exe)} studio reset-password"
|
|
except Exception:
|
|
pass
|
|
return "unsloth studio reset-password"
|
|
|
|
|
|
# Per-(ip, username) bucket + per-IP aggregate. Account bucket stops one user's
|
|
# typos from blocking others; the aggregate stops username-rotation spray.
|
|
# Single-process only; multi-worker deployments need a shared store.
|
|
_LOGIN_BUCKETS: dict[tuple[str, str], deque] = {}
|
|
_LOGIN_IP_BUCKETS: dict[str, deque] = {}
|
|
_LOGIN_BUCKETS_LOCK = threading.Lock()
|
|
_LOGIN_WINDOW_SECONDS = 60.0
|
|
_LOGIN_MAX_FAILS = 5
|
|
_LOGIN_IP_MAX_FAILS = 30
|
|
_LOGIN_LOCKOUT_SECONDS = 60
|
|
# Bucket-dict cap. On overflow, reclaim expired buckets; a new IP that still can't
|
|
# fit falls back to a sharded overflow rather than evicting a hot bucket.
|
|
_LOGIN_MAX_BUCKETS = 4096
|
|
# Last full stale-sweep time; rate-limits the O(n) sweep under a burst of new IPs.
|
|
_LAST_IP_PRUNE = 0.0
|
|
# Sharded overflow for per-IP failures that can't get their own bucket while the
|
|
# dict is saturated. Each shard is a small fixed-capacity dict ``ip -> [count,
|
|
# window_start]``: a per-IP count (so a source is throttled, and cleared on
|
|
# success, by its own failures -- no cross-IP collateral) with hard-bounded
|
|
# memory and O(1) lookups. When a shard is full a new IP evicts the lowest-count
|
|
# entry (and starts clean, never inheriting its count) rather than growing without
|
|
# bound, so a high-cardinality spray can't blow memory/CPU the way a per-failure
|
|
# deque could; a persistent attacker keeps a high count and is never the one
|
|
# evicted.
|
|
_LOGIN_IP_OVERFLOW_SHARDS = 256
|
|
_LOGIN_IP_OVERFLOW_MAX = 64 # distinct IPs tracked per shard
|
|
_LOGIN_IP_OVERFLOW: list[dict] = [dict() for _ in range(_LOGIN_IP_OVERFLOW_SHARDS)]
|
|
|
|
|
|
def _overflow_shard(ip: str) -> dict:
|
|
return _LOGIN_IP_OVERFLOW[hash(ip) % _LOGIN_IP_OVERFLOW_SHARDS]
|
|
|
|
|
|
def _overflow_record(ip: str, now: float) -> int:
|
|
"""Record an overflow failure for ``ip`` and return its windowed count."""
|
|
shard = _overflow_shard(ip)
|
|
entry = shard.get(ip)
|
|
if entry is not None:
|
|
if now - entry[1] > _LOGIN_WINDOW_SECONDS:
|
|
entry[0], entry[1] = 1, now
|
|
else:
|
|
# Only "at or above the per-IP threshold" matters for blocking, so cap
|
|
# the count there. This also keeps the migration into a per-IP bucket
|
|
# bounded -- without the cap a saturated source could accrue an
|
|
# unbounded count, then materialize one deque entry per failure
|
|
# (``[start] * carried``) on the next attempt, allocating an arbitrarily
|
|
# large deque while holding the login lock.
|
|
entry[0] = min(entry[0] + 1, _LOGIN_IP_MAX_FAILS)
|
|
return entry[0]
|
|
if len(shard) >= _LOGIN_IP_OVERFLOW_MAX:
|
|
# Make room by dropping the lowest-count entry, but the new source starts
|
|
# clean -- never inherit the evicted IP's failures, or an unrelated source
|
|
# could be 429'd after one attempt. Worst case under a saturated shard is
|
|
# that a heavy hitter briefly resets, not that a bystander is blocked.
|
|
del shard[min(shard, key = lambda k: shard[k][0])]
|
|
shard[ip] = [1, now]
|
|
return 1
|
|
|
|
|
|
def _overflow_blocked(ip: str, now: float) -> int:
|
|
"""Seconds this IP is throttled by its own overflow count, or 0."""
|
|
shard = _overflow_shard(ip)
|
|
entry = shard.get(ip)
|
|
if entry is None:
|
|
return 0
|
|
if now - entry[1] > _LOGIN_WINDOW_SECONDS:
|
|
del shard[ip]
|
|
return 0
|
|
if entry[0] >= _LOGIN_IP_MAX_FAILS:
|
|
return max(1, int(_LOGIN_WINDOW_SECONDS - (now - entry[1])))
|
|
return 0
|
|
|
|
|
|
def _overflow_take(ip: str, now: float) -> tuple[int, float]:
|
|
"""Pop ip's overflow entry, returning its ``(count, window_start)`` so the
|
|
count can migrate into a fresh per-IP bucket. ``(0, now)`` if none/expired."""
|
|
entry = _overflow_shard(ip).pop(ip, None)
|
|
if entry is None or now - entry[1] > _LOGIN_WINDOW_SECONDS:
|
|
return 0, now
|
|
# Cap the carried count so the bucket migration never allocates more than the
|
|
# per-IP threshold worth of deque entries (defensive; _overflow_record already
|
|
# clamps, but keep the bound at the consumption site too).
|
|
return min(entry[0], _LOGIN_IP_MAX_FAILS), entry[1]
|
|
|
|
|
|
# Unrepresentable as a real username (leading NUL); folds unknown-user attempts
|
|
# into one slot so attacker cardinality can't blow the bucket dict.
|
|
_UNKNOWN_LOGIN_USER = "\x00unknown-user"
|
|
|
|
|
|
def _trust_forwarded_for() -> bool:
|
|
"""Honour X-Forwarded-For only when UNSLOTH_STUDIO_TRUST_FORWARDED is set.
|
|
|
|
Off by default so a direct caller can't spoof the header.
|
|
"""
|
|
return os.environ.get("UNSLOTH_STUDIO_TRUST_FORWARDED", "").lower() in (
|
|
"1",
|
|
"true",
|
|
"yes",
|
|
)
|
|
|
|
|
|
def _normalize_forwarded_addr(value: str) -> str:
|
|
"""Parse an XFF / Forwarded `for=` value into a bare IP (port-stripped)."""
|
|
value = (value or "").strip().strip('"')
|
|
if not value or value.lower() == "unknown":
|
|
return ""
|
|
if value.startswith("["):
|
|
# Bracketed IPv6, optionally with port.
|
|
end = value.find("]")
|
|
if end <= 0:
|
|
return ""
|
|
host = value[1:end]
|
|
elif value.count(":") == 1:
|
|
# IPv4:port. Bare IPv6 has multiple colons → else branch.
|
|
head, _, tail = value.rpartition(":")
|
|
host = head if tail.isdigit() and head else value
|
|
else:
|
|
host = value
|
|
try:
|
|
return str(ipaddress.ip_address(host))
|
|
except ValueError:
|
|
return ""
|
|
|
|
|
|
def _forwarded_for_from_element(element: str) -> str:
|
|
"""Pick the `for=` token out of a single ``Forwarded`` element."""
|
|
for tok in element.split(";"):
|
|
key, sep, val = tok.strip().partition("=")
|
|
if sep and key.lower() == "for":
|
|
return _normalize_forwarded_addr(val)
|
|
return ""
|
|
|
|
|
|
def _client_ip(request: Request | None) -> str:
|
|
if request is None:
|
|
return "_unknown"
|
|
if _trust_forwarded_for():
|
|
xff = request.headers.get("x-forwarded-for", "")
|
|
if xff:
|
|
# First entry is the originating client.
|
|
normalized = _normalize_forwarded_addr(xff.split(",", 1)[0])
|
|
if normalized:
|
|
return normalized
|
|
fwd = request.headers.get("forwarded", "")
|
|
if fwd:
|
|
# First element only; multi-element headers can't fork buckets.
|
|
normalized = _forwarded_for_from_element(fwd.split(",", 1)[0])
|
|
if normalized:
|
|
return normalized
|
|
return (request.client.host if request.client else None) or "_unknown"
|
|
|
|
|
|
def _bucket_key(request: Request | None, username: str) -> tuple[str, str]:
|
|
return (_client_ip(request), (username or "").casefold())
|
|
|
|
|
|
def _unknown_user_key(request: Request | None) -> tuple[str, str]:
|
|
return (_client_ip(request), _UNKNOWN_LOGIN_USER)
|
|
|
|
|
|
def _prune_bucket(bucket: deque, now: float) -> None:
|
|
while bucket and now - bucket[0] > _LOGIN_WINDOW_SECONDS:
|
|
bucket.popleft()
|
|
|
|
|
|
def _prune_stale_buckets(now: float) -> None:
|
|
"""Drop empty / expired account buckets to bound memory under spray."""
|
|
stale: list[tuple[str, str]] = []
|
|
for key, bucket in _LOGIN_BUCKETS.items():
|
|
_prune_bucket(bucket, now)
|
|
if not bucket:
|
|
stale.append(key)
|
|
for key in stale:
|
|
_LOGIN_BUCKETS.pop(key, None)
|
|
|
|
|
|
def _prune_stale_ip_buckets(now: float) -> None:
|
|
"""Drop empty / expired per-IP buckets to bound memory under spray.
|
|
|
|
The dict is otherwise reclaimed only on a successful login, so a failure-only
|
|
spray from many (or spoofed) IPs would grow it without bound.
|
|
"""
|
|
stale: list[str] = []
|
|
for bucket_ip, bucket in _LOGIN_IP_BUCKETS.items():
|
|
_prune_bucket(bucket, now)
|
|
if not bucket:
|
|
stale.append(bucket_ip)
|
|
for bucket_ip in stale:
|
|
_LOGIN_IP_BUCKETS.pop(bucket_ip, None)
|
|
|
|
|
|
def _record_login_failure(key: tuple[str, str]) -> int:
|
|
global _LAST_IP_PRUNE
|
|
now = time.monotonic()
|
|
ip, _username = key
|
|
with _LOGIN_BUCKETS_LOCK:
|
|
# Keep the dict bounded without disabling throttling and without letting a
|
|
# spray reset a hot bucket: for a new IP at the cap, reclaim expired buckets
|
|
# (rate-limited) to make room.
|
|
ip_bucket = _LOGIN_IP_BUCKETS.get(ip)
|
|
if ip_bucket is None and len(_LOGIN_IP_BUCKETS) >= _LOGIN_MAX_BUCKETS:
|
|
if now - _LAST_IP_PRUNE >= 1.0:
|
|
_prune_stale_ip_buckets(now)
|
|
_LAST_IP_PRUNE = now
|
|
if ip_bucket is None and len(_LOGIN_IP_BUCKETS) >= _LOGIN_MAX_BUCKETS:
|
|
# Still full -- every bucket is hot. Count this failure in the IP's
|
|
# bounded overflow shard instead of evicting a live one, so the spray
|
|
# stays throttled but can't push out (and reset) any IP's own counter.
|
|
ip_fails = _overflow_record(ip, now)
|
|
else:
|
|
if ip_bucket is None:
|
|
ip_bucket = _LOGIN_IP_BUCKETS[ip] = deque()
|
|
# Carry over any overflow failures this IP accrued while the dict
|
|
# was saturated, so straddling the overflow -> bucket transition
|
|
# can't double the effective per-IP limit.
|
|
carried, start = _overflow_take(ip, now)
|
|
ip_bucket.extend([start] * carried)
|
|
_prune_bucket(ip_bucket, now)
|
|
ip_bucket.append(now)
|
|
ip_fails = len(ip_bucket)
|
|
|
|
if key not in _LOGIN_BUCKETS and len(_LOGIN_BUCKETS) >= _LOGIN_MAX_BUCKETS:
|
|
_prune_stale_buckets(now)
|
|
if key in _LOGIN_BUCKETS or len(_LOGIN_BUCKETS) < _LOGIN_MAX_BUCKETS:
|
|
account_bucket = _LOGIN_BUCKETS.setdefault(key, deque())
|
|
_prune_bucket(account_bucket, now)
|
|
account_bucket.append(now)
|
|
return len(account_bucket)
|
|
# Both dicts at cap (sustained spray): fall back to the per-IP count.
|
|
return ip_fails
|
|
|
|
|
|
def _blocked_for(bucket: deque | None, now: float, max_fails: int) -> int:
|
|
if not bucket:
|
|
return 0
|
|
_prune_bucket(bucket, now)
|
|
if len(bucket) >= max_fails:
|
|
return max(1, int(_LOGIN_WINDOW_SECONDS - (now - bucket[0])))
|
|
return 0
|
|
|
|
|
|
def _login_blocked(key: tuple[str, str]) -> int:
|
|
"""Return seconds until the next attempt is allowed, or 0."""
|
|
now = time.monotonic()
|
|
ip, _username = key
|
|
with _LOGIN_BUCKETS_LOCK:
|
|
# Honor the IP's overflow shard regardless of current dict capacity: a
|
|
# source counted there during saturation must stay throttled until those
|
|
# failures age out, even if a bucket later frees up -- otherwise a fresh
|
|
# bucket would reset it. Shards are empty outside saturation, so this is a
|
|
# no-op in the common case.
|
|
ip_blocked = max(
|
|
_blocked_for(_LOGIN_IP_BUCKETS.get(ip), now, _LOGIN_IP_MAX_FAILS),
|
|
_overflow_blocked(ip, now),
|
|
)
|
|
return max(_blocked_for(_LOGIN_BUCKETS.get(key), now, _LOGIN_MAX_FAILS), ip_blocked)
|
|
|
|
|
|
def _clear_login_bucket(key: tuple[str, str]) -> None:
|
|
ip, _username = key
|
|
with _LOGIN_BUCKETS_LOCK:
|
|
_LOGIN_BUCKETS.pop(key, None)
|
|
_LOGIN_IP_BUCKETS.pop(ip, None)
|
|
# A successful login resets the IP's throttle, including any overflow it
|
|
# accumulated during saturation (drop only this IP's entry, so a
|
|
# shard-mate's throttle is untouched).
|
|
_overflow_shard(ip).pop(ip, None)
|
|
|
|
|
|
# Sync def (not async): compute_identity_proof touches SQLite on the first call,
|
|
# so FastAPI runs it in the threadpool rather than blocking the event loop.
|
|
@router.get("/identity")
|
|
def identity(nonce: str, request: Request) -> dict:
|
|
"""Challenge-response proof this is the real local Studio: caller sends a nonce,
|
|
gets HMAC(install identity secret, nonce, connection address + port).
|
|
Unauthenticated and side-effect free; a process that can't read the same-user
|
|
secret can't forge a proof, and binding to the address/port the connection
|
|
landed on stops a squatter relaying a proof from the real Studio elsewhere."""
|
|
try:
|
|
raw = base64.urlsafe_b64decode(nonce)
|
|
except Exception:
|
|
raise HTTPException(
|
|
status_code = status.HTTP_400_BAD_REQUEST, detail = "nonce must be base64url"
|
|
)
|
|
if not 16 <= len(raw) <= 128:
|
|
raise HTTPException(
|
|
status_code = status.HTTP_400_BAD_REQUEST, detail = "nonce must decode to 16-128 bytes"
|
|
)
|
|
# The address + port the connection actually landed on, from the socket
|
|
# (request.scope is getsockname, so it is the real local address even when
|
|
# bound to 0.0.0.0), never the client-controlled Host header.
|
|
server = request.scope.get("server") or ("", 0)
|
|
host = server[0] or ""
|
|
port = server[1] if server[1] is not None else 0
|
|
return {"proof": storage.compute_identity_proof(raw, host, port)}
|
|
|
|
|
|
@router.get("/status", response_model = AuthStatusResponse)
|
|
async def auth_status() -> AuthStatusResponse:
|
|
"""Auth initialization state; ``default_username`` is exposed for first-boot UI prefill only."""
|
|
return AuthStatusResponse(
|
|
initialized = storage.is_initialized(),
|
|
default_username = storage.DEFAULT_ADMIN_USERNAME,
|
|
requires_password_change = storage.requires_password_change(storage.DEFAULT_ADMIN_USERNAME)
|
|
if storage.is_initialized()
|
|
else True,
|
|
)
|
|
|
|
|
|
@router.post("/login", response_model = Token)
|
|
async def login(payload: AuthLoginRequest, request: Request) -> Token:
|
|
"""Login with username/password. Per-account + per-IP rate-limited."""
|
|
key = _bucket_key(request, payload.username)
|
|
unknown_key = _unknown_user_key(request)
|
|
blocked_for = max(_login_blocked(key), _login_blocked(unknown_key))
|
|
if blocked_for > 0:
|
|
raise HTTPException(
|
|
status_code = status.HTTP_429_TOO_MANY_REQUESTS,
|
|
# IP not interpolated into the body; behind a proxy/NAT it's
|
|
# misleading or an info leak.
|
|
detail = (f"Too many failed login attempts. " f"Try again in {blocked_for} seconds."),
|
|
headers = {"Retry-After": str(blocked_for)},
|
|
)
|
|
|
|
record = storage.get_user_and_secret(payload.username)
|
|
if record is None:
|
|
# Record under one sentinel key per IP so attacker-controlled username
|
|
# cardinality can't allocate unbounded buckets.
|
|
_record_login_failure(unknown_key)
|
|
raise HTTPException(
|
|
status_code = status.HTTP_401_UNAUTHORIZED,
|
|
detail = f"Incorrect password. To reset it, run this in your terminal: {_reset_password_command()}",
|
|
)
|
|
|
|
salt, pwd_hash, _jwt_secret, must_change_password = record
|
|
if not hashing.verify_password(payload.password, salt, pwd_hash):
|
|
_record_login_failure(key)
|
|
raise HTTPException(
|
|
status_code = status.HTTP_401_UNAUTHORIZED,
|
|
detail = f"Incorrect password. To reset it, run this in your terminal: {_reset_password_command()}",
|
|
)
|
|
|
|
_clear_login_bucket(key)
|
|
_clear_login_bucket(unknown_key)
|
|
access_token = create_access_token(subject = payload.username)
|
|
refresh_token = create_refresh_token(subject = payload.username)
|
|
return Token(
|
|
access_token = access_token,
|
|
refresh_token = refresh_token,
|
|
token_type = "bearer",
|
|
must_change_password = must_change_password,
|
|
)
|
|
|
|
|
|
@router.post("/logout", status_code = status.HTTP_204_NO_CONTENT)
|
|
async def logout(
|
|
request: Request, current_subject: str = Depends(get_current_subject_allow_password_change)
|
|
) -> Response:
|
|
"""Revoke refresh tokens for the subject; the access token is stateless and expires on its own."""
|
|
try:
|
|
storage.revoke_user_refresh_tokens(current_subject)
|
|
except Exception:
|
|
pass
|
|
try:
|
|
request.app.state.bootstrap_password = None
|
|
except AttributeError:
|
|
pass
|
|
return Response(status_code = status.HTTP_204_NO_CONTENT)
|
|
|
|
|
|
@router.post("/desktop-login", response_model = Token)
|
|
async def desktop_login(payload: DesktopLoginRequest) -> Token:
|
|
"""Exchange a local desktop secret for normal admin-subject tokens."""
|
|
username = storage.validate_desktop_secret(payload.secret)
|
|
if username is None:
|
|
raise HTTPException(
|
|
status_code = status.HTTP_401_UNAUTHORIZED,
|
|
detail = "Desktop authentication failed",
|
|
)
|
|
|
|
return Token(
|
|
access_token = create_access_token(subject = username, desktop = True),
|
|
refresh_token = create_refresh_token(subject = username, desktop = True),
|
|
token_type = "bearer",
|
|
must_change_password = False,
|
|
)
|
|
|
|
|
|
@router.post("/refresh", response_model = Token)
|
|
async def refresh(payload: RefreshTokenRequest) -> Token:
|
|
"""Exchange a refresh token for a new access+refresh pair (single-use)."""
|
|
consumed = storage.consume_refresh_token(payload.refresh_token)
|
|
if consumed is None:
|
|
raise HTTPException(
|
|
status_code = status.HTTP_401_UNAUTHORIZED,
|
|
detail = "Invalid or expired refresh token",
|
|
)
|
|
username, is_desktop = consumed
|
|
new_access_token = create_access_token(subject = username, desktop = is_desktop)
|
|
new_refresh_token = create_refresh_token(subject = username, desktop = is_desktop)
|
|
|
|
return Token(
|
|
access_token = new_access_token,
|
|
refresh_token = new_refresh_token,
|
|
token_type = "bearer",
|
|
must_change_password = False if is_desktop else storage.requires_password_change(username),
|
|
)
|
|
|
|
|
|
@router.post("/change-password", response_model = Token)
|
|
async def change_password(
|
|
payload: ChangePasswordRequest,
|
|
request: Request,
|
|
current_subject: str = Depends(get_current_subject_allow_password_change),
|
|
) -> Token:
|
|
"""Allow the authenticated user to replace the default password."""
|
|
record = storage.get_user_and_secret(current_subject)
|
|
if record is None:
|
|
raise HTTPException(
|
|
status_code = status.HTTP_401_UNAUTHORIZED,
|
|
detail = "User session is invalid",
|
|
)
|
|
|
|
salt, pwd_hash, _jwt_secret, _must_change_password = record
|
|
if not hashing.verify_password(payload.current_password, salt, pwd_hash):
|
|
raise HTTPException(
|
|
status_code = status.HTTP_401_UNAUTHORIZED,
|
|
detail = "Current password is incorrect",
|
|
)
|
|
if payload.current_password == payload.new_password:
|
|
raise HTTPException(
|
|
status_code = status.HTTP_400_BAD_REQUEST,
|
|
detail = "New password must be different from the current password",
|
|
)
|
|
|
|
storage.update_password(current_subject, payload.new_password)
|
|
storage.revoke_user_refresh_tokens(current_subject)
|
|
try:
|
|
request.app.state.bootstrap_password = None
|
|
except AttributeError:
|
|
pass
|
|
access_token = create_access_token(subject = current_subject)
|
|
refresh_token = create_refresh_token(subject = current_subject)
|
|
return Token(
|
|
access_token = access_token,
|
|
refresh_token = refresh_token,
|
|
token_type = "bearer",
|
|
must_change_password = False,
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# API key management
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _row_to_api_key_response(row: dict) -> ApiKeyResponse:
|
|
return ApiKeyResponse(
|
|
id = row["id"],
|
|
name = row["name"],
|
|
key_prefix = row["key_prefix"],
|
|
created_at = row["created_at"],
|
|
last_used_at = row.get("last_used_at"),
|
|
expires_at = row.get("expires_at"),
|
|
is_active = bool(row["is_active"]),
|
|
)
|
|
|
|
|
|
@router.post("/api-keys", response_model = CreateApiKeyResponse)
|
|
async def create_api_key(
|
|
payload: CreateApiKeyRequest, current_subject: str = Depends(get_current_subject)
|
|
) -> CreateApiKeyResponse:
|
|
"""Create a new API key. The raw key is returned once and cannot be retrieved later."""
|
|
expires_at = None
|
|
if payload.expires_in_days is not None:
|
|
expires_at = (
|
|
datetime.now(timezone.utc) + timedelta(days = payload.expires_in_days)
|
|
).isoformat()
|
|
|
|
raw_key, row = storage.create_api_key(
|
|
username = current_subject,
|
|
name = payload.name,
|
|
expires_at = expires_at,
|
|
)
|
|
return CreateApiKeyResponse(
|
|
key = raw_key,
|
|
api_key = _row_to_api_key_response(row),
|
|
)
|
|
|
|
|
|
@router.get("/api-keys", response_model = ApiKeyListResponse)
|
|
async def list_api_keys(current_subject: str = Depends(get_current_subject)) -> ApiKeyListResponse:
|
|
"""List all API keys for the authenticated user (raw keys are never exposed)."""
|
|
rows = storage.list_api_keys(current_subject)
|
|
return ApiKeyListResponse(
|
|
api_keys = [_row_to_api_key_response(r) for r in rows],
|
|
)
|
|
|
|
|
|
@router.delete("/api-keys/{key_id}")
|
|
async def revoke_api_key(key_id: int, current_subject: str = Depends(get_current_subject)) -> dict:
|
|
"""Revoke (soft-delete) an API key."""
|
|
if not storage.revoke_api_key(current_subject, key_id):
|
|
raise HTTPException(
|
|
status_code = status.HTTP_404_NOT_FOUND,
|
|
detail = "API key not found",
|
|
)
|
|
return {"detail": "API key revoked"}
|