unsloth/studio/backend/routes/auth.py
Daniel Han 2ef394137a
Studio: harden background consumer loops and streaming paths against silent UI freezes (#6653)
* Studio: harden the data-recipe and inference consumer loops against pump death

Follow-up to #6643. The same single-unsupervised-consumer pattern the training
pump had lives in two sibling loops, with the same failure mode: one bad event
kills the only thread that updates the in-memory state every UI surface reads,
while the worker subprocess keeps running.

- data_recipe JobManager._pump_loop: a malformed worker log line that makes
  parse_log_message raise no longer kills the pump. Guard _handle_event, the
  queue read, and the worker-exit finalize, and broaden _drain_queue so a drain
  error still finalizes the job instead of leaving it wedged "active" (which also
  leaked the workflow-scoped API key until its 24h expiry).
- inference InferenceOrchestrator._dispatcher_loop: guard the routing body so a
  malformed response or a mailbox put error can't kill the dispatcher and hang
  every in-flight generation (callers key liveness on the subprocess, not on
  this thread).

Adds regression tests for both.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: extend consumer-loop hardening to RAG, hub, auth, and stream-reader paths

Continuation of the data-recipe and inference pump hardening: the same
"background producer updates in-memory state that a single unsupervised
consumer surfaces to the UI" pattern shows up in several more Studio paths,
each able to silently freeze a UI surface while the worker keeps running.

RAG ingestion SSE (core/rag/ingestion.py):
- job_events polled the queue with a blocking get and never noticed client
  disconnect or a dead worker, so a closed tab or a producer that died
  without emitting a terminal event left the stream hanging. It now polls
  with a timeout, emits heartbeats, ends on terminal job status, caps idle
  time, and always pops the job registry in finally.
- Added _reap_finished_jobs() and call it from start_ingestion so finished
  job state does not accumulate.

Startup reconcile (storage/rag_db.py, main.py):
- reconcile_orphaned_ingestion_jobs() marks ingestion jobs (and their
  documents) that were left non-terminal by a previous crash as failed, so
  the UI does not show jobs stuck "running" forever after a restart. Wired
  in at startup next to cleanup_orphaned_runs().

Hub download watcher (hub/services/download_lifecycle.py):
- _watch() could leave a job pinned "running" if finalize raised. Body is
  now guarded: on failure it logs and sets the job to error, and always
  invalidates the hf cache scan in finally.

External provider stream (core/inference/external_provider.py):
- read timeout was None (no stall ceiling); set to 300s so a wedged
  upstream surfaces as an error instead of an indefinitely hung stream.

Auth store (auth/storage.py):
- Enable WAL + busy_timeout on the auth DB so token validation (read on
  every request) and login writes stop serialising on the rollback journal.
  Matches studio_db / rag_db / providers_db.

Login rate limiter (routes/auth.py):
- _LOGIN_IP_BUCKETS could grow unbounded under spoofed-IP traffic; cap it
  and prune stale buckets, mirroring the per-account bucket handling.

Training progress SSE (routes/training.py):
- Break promptly on client disconnect instead of waiting for the next
  yield to fail on a closed socket, matching the export / data-recipe SSE
  routes.

llama-server stdout drain (core/inference/llama_cpp.py):
- Broaden the drain guard so an unexpected decode/read error logs at debug
  and stops the drainer cleanly instead of escaping the thread.

Frontend stream readers (chat-api.ts, rag-api.ts):
- Wrap the SSE read loops in try/finally + reader.cancel() so early return
  ([DONE]), thrown errors, and consumer aborts release the reader lock
  instead of holding it until GC.

Tests:
- test_training_progress_stream_nan: fake request now implements the async
  is_disconnected() the route polls, matching the other SSE route fakes.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: address Codex review feedback on the consumer-loop hardening

Four follow-ups from the automated review, all on code this PR introduced:

- Data-recipe pump (manager.py): a queue read that keeps raising an error
  outside the read's narrow catch set (e.g. a broken queue pipe after the
  child died) hit the `continue` guard and skipped the dead-worker finalize
  below, spinning forever and leaving the job wedged "active" with its
  workflow key unretired. On a read failure, fall through to finalize when
  the worker is no longer alive. Added a regression test.

- RAG ingestion SSE (ingestion.py): the 5-minute idle cap could end the
  stream while the job was still pending/running (a large document spends
  minutes in embedding/storing with no per-batch progress event). The route
  then sends [DONE], and the client treats a no-terminal-frame end as
  completion, marking the document indexed mid-ingestion. Drop the idle cap:
  while the worker is alive and non-terminal we keep heartbeating; the stream
  ends only on terminal DB status, the None sentinel, or client disconnect.

- Login rate limiter (auth.py): the per-IP path pruned but then added the
  new IP unconditionally, so a spoofed-source-IP spray kept _LOGIN_IP_BUCKETS
  unbounded and made every new IP pay a full-dict prune scan. Gate the add on
  the cap, mirroring the account path.

- Hub download watcher (download_lifecycle.py): if finalize raised before it
  reaped (proc.wait) and dropped the worker (e.g. an I/O error draining
  stderr), the crash path published a terminal state while the live Popen
  stayed registered and kept writing the cache, and the terminal set_job let
  claim() admit a retry on the same repo. Terminate + drop the worker before
  setting the terminal state.

* Studio: keep login throttling working when the per-IP bucket dict saturates

Review follow-up. The previous cap fix skipped creating a bucket for a new IP
once _LOGIN_IP_BUCKETS was full, returning ip_fails=0. Under a sustained spray
that also fills the account dict, every failure from such an IP then looked
first-seen and _login_blocked had no bucket to enforce, so the cap effectively
disabled throttling once saturated.

Bound the dict with a FIFO eviction instead: if the IP is new and the dict is
full, reclaim expired buckets (rate-limited so a burst of distinct IPs can't
make each failure an O(n) sweep) and, if still full, evict the oldest-inserted
IP. The new IP always gets a real bucket, so a saturating (e.g. spoofed
X-Forwarded-For) spray stays throttled while memory stays bounded. Added a
regression test that saturates the dict and asserts a later IP is still blocked.

* Studio: address Codex review (RAG queue lifecycle, stream error, orphan chunks)

Three follow-ups on the Phase 6 changes:

- RAG ingestion SSE (ingestion.py): job_events removed the per-job queue in its
  finally on ANY exit, including an early client disconnect while the worker is
  still running. That dropped the worker's later events (the queue is the only
  one _emit writes to) and made a reconnect find no queue and receive only
  [DONE], which the client treats as completion. Only drop the queue on a
  terminal exit (None sentinel / terminal DB status); leftover terminal queues
  are still swept by _reap_finished_jobs. Added queue-lifecycle tests.

- External provider stream (routes/inference.py): once the 300s read timeout can
  fire, the stream's except path failed the monitor but ended without an error
  frame or [DONE], so the chat client saw a bare EOF and saved the timed-out
  answer as a successful partial with no error. Emit an SSE error frame (and
  [DONE]) on stream failure so the client surfaces it.

- RAG startup reconcile (storage/rag_db.py): marking a half-ingested document
  failed left its chunks/fts/vec rows intact, and retrieval filters by scope not
  status, so a failed document could still be retrieved and cited. Purge the
  document's chunks when reconciling it to failed (the doc row stays for
  re-ingest).

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: release the remaining SSE stream readers (training, data-recipe, export)

reviewer.py follow-up. The chat and RAG SSE readers were wrapped in
try/finally + reader.cancel(), but the other three readers built on the same
response.body.getReader() pattern were left without it: streamTrainingProgress,
streamRecipeJobEvents, and streamExportLogs leak the ReadableStreamDefaultReader
lock (held until GC) when the consumer aborts, returns early, or a parse/callback
throws. Wrap each in try/finally + reader.cancel() (export already had a
try/catch, so it only needed the finally). All five frontend SSE readers now
release the reader symmetrically.

* Tighten resilience comments and docstrings

Condense the verbose explanatory comments and internal-helper docstrings added
in this branch to shorter, clearer forms. Comment/whitespace only; verified no
code changed via AST diff. No behaviour change.

* Studio: keep chunks for completed docs during ingestion reconcile

Startup reconciliation flips orphaned (non-terminal) ingestion jobs to failed and
purges the document's chunks so a failed source can't be retrieved. But it dropped
the chunks unconditionally, so a document the worker had already committed as
'completed' before the crash (only its job row left non-terminal) lost every chunk
while still reporting 'completed'. That leaves an empty source that retrieval can't
return and dedup (status != 'failed') blocks from re-ingest.

Only purge chunks when the document UPDATE actually transitions it to failed; an
already-completed document keeps its chunks. Adds reconcile regression tests for
both the completed-doc and genuine in-flight-orphan cases.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: drop a finished RAG job's queue when the client disconnects

job_events kept the per-job queue until it consumed the None sentinel, so a UI
that stops on the terminal event (its reader.cancel aborts the stream before
[DONE]) left the queue registered until the next _reap_finished_jobs sweep; a
batch of uploads followed by idling retained them all.

_run writes the terminal DB status before emitting the terminal event, so on
generator exit, drop the queue when the job's DB row is already terminal (worker
done, nothing to resume) and keep it only while the worker is still running. Adds
a disconnect-after-terminal-event regression test.

* Remove stray async task output files committed by mistake

* Studio: harden login IP throttle and end progress stream on disconnect

Two Codex review items:

Login per-IP throttle: when the per-IP bucket dict saturated, FIFO eviction could
drop a still-hot (blocked) bucket, so an IP could flood the dict with distinct
(or spoofed) source IPs to push out its own bucket and retry as first-seen. Stop
evicting hot buckets; a new IP that can't fit now shares a bounded overflow
counter that still trips the per-IP threshold, so a saturating spray stays
throttled and no live counter is reset.

Progress SSE: on client disconnect the polling loop only broke and fell through
to the unconditional final 'complete' frame, so a buffered or proxying consumer
could read a still-active run as completed. Return from the generator instead.

Adds regression tests for both (spray cannot reset a hot bucket; disconnect while
active emits no complete frame).

* Studio: shard the login overflow counter and stop cancelling chat stream after [DONE]

Two Codex review items:

Login throttle overflow: the single shared overflow counter meant that once a
saturating spray pushed it past the per-IP threshold, _login_blocked returned 429
for every new unbucketed source IP, before credentials were checked -- a global
login denial. Shard the overflow into a fixed array of counters keyed by hash(ip),
so a hot shard only throttles the IPs that map to it while a single source's
repeated failures still concentrate in one shard and stay throttled. Memory stays
bounded and no live bucket is evicted. Adds a regression test that a hot overflow
shard does not block an unrelated IP.

Chat stream: the reader.cancel() in the SSE finally fired even after a natural
[DONE]/EOF. The backend finalizes its api-monitor entry right after yielding the
sentinel (the local pass-through finishes after the last yield), so a client
cancel there can be observed as a disconnect and mark a completed request as
cancelled. Track natural completion and only cancel on an early/abnormal exit.
(No frontend unit test: the Studio frontend has no test harness.)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: give prep-timeout test fakes an is_disconnected method

The progress stream now ends on client disconnect (await request.is_disconnected()
before falling through to the terminal frame). After merging that into the
prep-timeout tests added later on main, their _FakeRequest/_ReconnectRequest must
provide is_disconnected or the generator raises AttributeError under CI.

* Studio: keep the login overflow throttle when bucket capacity frees up

_login_blocked only consulted the per-IP overflow shard while the bucket dict was
still at capacity. If a slot freed before the 60s window expired (e.g. another
IP's successful login calls _clear_login_bucket), a source counted in a hot shard
stopped being blocked and its next failure got a fresh per-IP bucket, resetting
the throttle the overflow path exists to preserve. Always max in the IP's shard
(shards are empty outside saturation, so it is a no-op in the common case). Adds a
regression test that a hot source stays throttled after a bucket frees.

* Studio: clear a login IP's overflow throttle on successful login

_clear_login_bucket reset the per-IP and per-account buckets on a successful
login but not the overflow shard, so after the dict saturated and an IP was
counted in overflow, a later successful login left those entries behind and the
next failed attempt could immediately return 429.

Store overflow entries as (timestamp, ip) so a source is throttled by its own
count within the shard (also removing cross-IP collateral within a shard), and
drop just that IP's entries in _clear_login_bucket. Adds a regression test that a
successful login clears the overflow throttle.

* Studio: bound the login overflow shard memory under high-cardinality spray

The per-IP overflow tracked failures in a time-pruned deque of (timestamp, ip)
tuples, so a spoofed-X-Forwarded-For spray of distinct one-off IPs grew memory and
the per-check scan with request cardinality for the whole window -- undermining
the bucket cap that exists to bound memory. Replace each shard with a fixed-
capacity dict (ip -> [count, window_start]): O(1) lookups, and when a shard is
full a one-off IP evicts the lowest-count entry (Space-Saving) so memory is hard-
bounded while a persistent attacker keeps a high count and is never evicted. Adds
a regression test that shards stay within the per-shard cap under a 5000-IP spray.

* Studio: purge chunks for already-failed docs during ingestion reconcile

The reconcile chunk-purge was gated on the documents UPDATE actually flipping a
non-terminal doc to failed. A doc the worker had already marked 'failed' before
the crash (job row left non-terminal) was not re-flipped, so its committed chunks
were kept and stayed retrievable/citable, since retrieval filters by scope not
status. Purge chunks whenever the document is not 'completed' (failed, in-flight,
or gone), preserving the completed-doc carve-out. Adds a regression test.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: don't inherit an evicted IP's count onto a new overflow source

When a full overflow shard evicted the lowest-count entry, the new source
inherited that count (Space-Saving base + 1). If a shard was saturated with hot
entries, an unrelated new IP could land at/over the threshold and be 429'd after a
single attempt -- cross-IP collateral despite the per-source-isolation intent.
New entries now start clean at count 1; the only cost is that a heavy hitter that
is the lowest-count entry in a fully saturated shard can briefly reset, which is
preferable to blocking a bystander. Adds a regression test.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: carry overflow failures into a new IP bucket on transition

_login_blocked took max(per-IP bucket, overflow shard) rather than combining them,
so a source could log (threshold-1) failures in overflow during saturation and,
once a bucket slot freed, another (threshold-1) in a fresh bucket within the same
window -- roughly doubling the per-IP limit. When a saturated-era IP first gets a
real bucket, migrate its windowed overflow count into that bucket (and drop the
overflow entry) so the combined failures throttle at the intended limit. Adds a
regression test.

* Studio: reconcile a completed doc's orphaned job to completed, not failed

When a crash left an ingestion job non-terminal after its document was already
committed as completed, reconcile marked the job failed. After restart the upload
UI has no in-memory SSE queue and falls back to getJob(), which treats a failed
job as an indexing failure and removes/toasts a document that is actually
searchable. Mark the job completed (keeping its chunks) when its document is
completed. Extends the completed-doc reconcile test to assert the job status.

* Studio: clamp the overflow failure count migrated into a login bucket

A saturated source could accrue an unbounded overflow count, then materialize
one deque entry per recorded failure when a bucket slot freed, allocating an
arbitrarily large deque under the login lock. Only at-or-above the per-IP
threshold matters for blocking, so cap the count there at the record and take
sites; the migration is now bounded without weakening the limit.

* Studio: keep the RAG job stream alive on a transient status read

The heartbeat poll read the job row unguarded; a momentarily-locked DB would
raise out of job_events, which the SSE route turns into a terminal error frame,
and the UI drops a document whose worker is still running. Treat a failed status
read as non-terminal: heartbeat and retry, and keep the queue so a reconnect can
resume.

* Studio: set busy_timeout before journal_mode on the auth DB

Switching journal_mode needs a lock, so if a refresh-token write already holds
one, journal_mode=WAL raises SQLITE_BUSY and the shared try leaves the
connection on SQLite's default zero lock wait. Set busy_timeout first so the
switch waits instead of failing.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-06-26 03:31:33 -07:00

575 lines
22 KiB
Python

# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""Authentication API routes."""
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
import base64
import ipaddress
import os
import shlex
import sys
import threading
import time
from collections import deque
from datetime import datetime, timedelta, timezone
from models.auth import (
ApiKeyListResponse,
ApiKeyResponse,
AuthLoginRequest,
AuthStatusResponse,
ChangePasswordRequest,
CreateApiKeyRequest,
CreateApiKeyResponse,
DesktopLoginRequest,
RefreshTokenRequest,
)
from models.users import Token
from auth import storage, hashing
from auth.authentication import (
create_access_token,
create_refresh_token,
get_current_subject,
get_current_subject_allow_password_change,
refresh_access_token,
)
router = APIRouter()
def _reset_password_command() -> str:
"""Shell command shown in the 'incorrect password' hint.
Prefer the absolute path to this install's ``unsloth`` launcher (sibling of
the running interpreter) so the hint works even when its dir isn't on PATH.
POSIX paths are shell-quoted. On Windows we use the bare absolute path only
when it has no spaces (a quoted path differs between cmd and PowerShell);
otherwise, or if the launcher can't be located, fall back to the PATH form.
"""
try:
bin_dir = os.path.dirname(os.path.abspath(sys.executable))
if os.name == "nt":
exe = os.path.join(bin_dir, "unsloth.exe")
if os.path.isfile(exe) and " " not in exe:
return f"{exe} studio reset-password"
else:
exe = os.path.join(bin_dir, "unsloth")
if os.path.isfile(exe):
return f"{shlex.quote(exe)} studio reset-password"
except Exception:
pass
return "unsloth studio reset-password"
# Per-(ip, username) bucket + per-IP aggregate. Account bucket stops one user's
# typos from blocking others; the aggregate stops username-rotation spray.
# Single-process only; multi-worker deployments need a shared store.
_LOGIN_BUCKETS: dict[tuple[str, str], deque] = {}
_LOGIN_IP_BUCKETS: dict[str, deque] = {}
_LOGIN_BUCKETS_LOCK = threading.Lock()
_LOGIN_WINDOW_SECONDS = 60.0
_LOGIN_MAX_FAILS = 5
_LOGIN_IP_MAX_FAILS = 30
_LOGIN_LOCKOUT_SECONDS = 60
# Bucket-dict cap. On overflow, reclaim expired buckets; a new IP that still can't
# fit falls back to a sharded overflow rather than evicting a hot bucket.
_LOGIN_MAX_BUCKETS = 4096
# Last full stale-sweep time; rate-limits the O(n) sweep under a burst of new IPs.
_LAST_IP_PRUNE = 0.0
# Sharded overflow for per-IP failures that can't get their own bucket while the
# dict is saturated. Each shard is a small fixed-capacity dict ``ip -> [count,
# window_start]``: a per-IP count (so a source is throttled, and cleared on
# success, by its own failures -- no cross-IP collateral) with hard-bounded
# memory and O(1) lookups. When a shard is full a new IP evicts the lowest-count
# entry (and starts clean, never inheriting its count) rather than growing without
# bound, so a high-cardinality spray can't blow memory/CPU the way a per-failure
# deque could; a persistent attacker keeps a high count and is never the one
# evicted.
_LOGIN_IP_OVERFLOW_SHARDS = 256
_LOGIN_IP_OVERFLOW_MAX = 64 # distinct IPs tracked per shard
_LOGIN_IP_OVERFLOW: list[dict] = [dict() for _ in range(_LOGIN_IP_OVERFLOW_SHARDS)]
def _overflow_shard(ip: str) -> dict:
return _LOGIN_IP_OVERFLOW[hash(ip) % _LOGIN_IP_OVERFLOW_SHARDS]
def _overflow_record(ip: str, now: float) -> int:
"""Record an overflow failure for ``ip`` and return its windowed count."""
shard = _overflow_shard(ip)
entry = shard.get(ip)
if entry is not None:
if now - entry[1] > _LOGIN_WINDOW_SECONDS:
entry[0], entry[1] = 1, now
else:
# Only "at or above the per-IP threshold" matters for blocking, so cap
# the count there. This also keeps the migration into a per-IP bucket
# bounded -- without the cap a saturated source could accrue an
# unbounded count, then materialize one deque entry per failure
# (``[start] * carried``) on the next attempt, allocating an arbitrarily
# large deque while holding the login lock.
entry[0] = min(entry[0] + 1, _LOGIN_IP_MAX_FAILS)
return entry[0]
if len(shard) >= _LOGIN_IP_OVERFLOW_MAX:
# Make room by dropping the lowest-count entry, but the new source starts
# clean -- never inherit the evicted IP's failures, or an unrelated source
# could be 429'd after one attempt. Worst case under a saturated shard is
# that a heavy hitter briefly resets, not that a bystander is blocked.
del shard[min(shard, key = lambda k: shard[k][0])]
shard[ip] = [1, now]
return 1
def _overflow_blocked(ip: str, now: float) -> int:
"""Seconds this IP is throttled by its own overflow count, or 0."""
shard = _overflow_shard(ip)
entry = shard.get(ip)
if entry is None:
return 0
if now - entry[1] > _LOGIN_WINDOW_SECONDS:
del shard[ip]
return 0
if entry[0] >= _LOGIN_IP_MAX_FAILS:
return max(1, int(_LOGIN_WINDOW_SECONDS - (now - entry[1])))
return 0
def _overflow_take(ip: str, now: float) -> tuple[int, float]:
"""Pop ip's overflow entry, returning its ``(count, window_start)`` so the
count can migrate into a fresh per-IP bucket. ``(0, now)`` if none/expired."""
entry = _overflow_shard(ip).pop(ip, None)
if entry is None or now - entry[1] > _LOGIN_WINDOW_SECONDS:
return 0, now
# Cap the carried count so the bucket migration never allocates more than the
# per-IP threshold worth of deque entries (defensive; _overflow_record already
# clamps, but keep the bound at the consumption site too).
return min(entry[0], _LOGIN_IP_MAX_FAILS), entry[1]
# Unrepresentable as a real username (leading NUL); folds unknown-user attempts
# into one slot so attacker cardinality can't blow the bucket dict.
_UNKNOWN_LOGIN_USER = "\x00unknown-user"
def _trust_forwarded_for() -> bool:
"""Honour X-Forwarded-For only when UNSLOTH_STUDIO_TRUST_FORWARDED is set.
Off by default so a direct caller can't spoof the header.
"""
return os.environ.get("UNSLOTH_STUDIO_TRUST_FORWARDED", "").lower() in (
"1",
"true",
"yes",
)
def _normalize_forwarded_addr(value: str) -> str:
"""Parse an XFF / Forwarded `for=` value into a bare IP (port-stripped)."""
value = (value or "").strip().strip('"')
if not value or value.lower() == "unknown":
return ""
if value.startswith("["):
# Bracketed IPv6, optionally with port.
end = value.find("]")
if end <= 0:
return ""
host = value[1:end]
elif value.count(":") == 1:
# IPv4:port. Bare IPv6 has multiple colons → else branch.
head, _, tail = value.rpartition(":")
host = head if tail.isdigit() and head else value
else:
host = value
try:
return str(ipaddress.ip_address(host))
except ValueError:
return ""
def _forwarded_for_from_element(element: str) -> str:
"""Pick the `for=` token out of a single ``Forwarded`` element."""
for tok in element.split(";"):
key, sep, val = tok.strip().partition("=")
if sep and key.lower() == "for":
return _normalize_forwarded_addr(val)
return ""
def _client_ip(request: Request | None) -> str:
if request is None:
return "_unknown"
if _trust_forwarded_for():
xff = request.headers.get("x-forwarded-for", "")
if xff:
# First entry is the originating client.
normalized = _normalize_forwarded_addr(xff.split(",", 1)[0])
if normalized:
return normalized
fwd = request.headers.get("forwarded", "")
if fwd:
# First element only; multi-element headers can't fork buckets.
normalized = _forwarded_for_from_element(fwd.split(",", 1)[0])
if normalized:
return normalized
return (request.client.host if request.client else None) or "_unknown"
def _bucket_key(request: Request | None, username: str) -> tuple[str, str]:
return (_client_ip(request), (username or "").casefold())
def _unknown_user_key(request: Request | None) -> tuple[str, str]:
return (_client_ip(request), _UNKNOWN_LOGIN_USER)
def _prune_bucket(bucket: deque, now: float) -> None:
while bucket and now - bucket[0] > _LOGIN_WINDOW_SECONDS:
bucket.popleft()
def _prune_stale_buckets(now: float) -> None:
"""Drop empty / expired account buckets to bound memory under spray."""
stale: list[tuple[str, str]] = []
for key, bucket in _LOGIN_BUCKETS.items():
_prune_bucket(bucket, now)
if not bucket:
stale.append(key)
for key in stale:
_LOGIN_BUCKETS.pop(key, None)
def _prune_stale_ip_buckets(now: float) -> None:
"""Drop empty / expired per-IP buckets to bound memory under spray.
The dict is otherwise reclaimed only on a successful login, so a failure-only
spray from many (or spoofed) IPs would grow it without bound.
"""
stale: list[str] = []
for bucket_ip, bucket in _LOGIN_IP_BUCKETS.items():
_prune_bucket(bucket, now)
if not bucket:
stale.append(bucket_ip)
for bucket_ip in stale:
_LOGIN_IP_BUCKETS.pop(bucket_ip, None)
def _record_login_failure(key: tuple[str, str]) -> int:
global _LAST_IP_PRUNE
now = time.monotonic()
ip, _username = key
with _LOGIN_BUCKETS_LOCK:
# Keep the dict bounded without disabling throttling and without letting a
# spray reset a hot bucket: for a new IP at the cap, reclaim expired buckets
# (rate-limited) to make room.
ip_bucket = _LOGIN_IP_BUCKETS.get(ip)
if ip_bucket is None and len(_LOGIN_IP_BUCKETS) >= _LOGIN_MAX_BUCKETS:
if now - _LAST_IP_PRUNE >= 1.0:
_prune_stale_ip_buckets(now)
_LAST_IP_PRUNE = now
if ip_bucket is None and len(_LOGIN_IP_BUCKETS) >= _LOGIN_MAX_BUCKETS:
# Still full -- every bucket is hot. Count this failure in the IP's
# bounded overflow shard instead of evicting a live one, so the spray
# stays throttled but can't push out (and reset) any IP's own counter.
ip_fails = _overflow_record(ip, now)
else:
if ip_bucket is None:
ip_bucket = _LOGIN_IP_BUCKETS[ip] = deque()
# Carry over any overflow failures this IP accrued while the dict
# was saturated, so straddling the overflow -> bucket transition
# can't double the effective per-IP limit.
carried, start = _overflow_take(ip, now)
ip_bucket.extend([start] * carried)
_prune_bucket(ip_bucket, now)
ip_bucket.append(now)
ip_fails = len(ip_bucket)
if key not in _LOGIN_BUCKETS and len(_LOGIN_BUCKETS) >= _LOGIN_MAX_BUCKETS:
_prune_stale_buckets(now)
if key in _LOGIN_BUCKETS or len(_LOGIN_BUCKETS) < _LOGIN_MAX_BUCKETS:
account_bucket = _LOGIN_BUCKETS.setdefault(key, deque())
_prune_bucket(account_bucket, now)
account_bucket.append(now)
return len(account_bucket)
# Both dicts at cap (sustained spray): fall back to the per-IP count.
return ip_fails
def _blocked_for(bucket: deque | None, now: float, max_fails: int) -> int:
if not bucket:
return 0
_prune_bucket(bucket, now)
if len(bucket) >= max_fails:
return max(1, int(_LOGIN_WINDOW_SECONDS - (now - bucket[0])))
return 0
def _login_blocked(key: tuple[str, str]) -> int:
"""Return seconds until the next attempt is allowed, or 0."""
now = time.monotonic()
ip, _username = key
with _LOGIN_BUCKETS_LOCK:
# Honor the IP's overflow shard regardless of current dict capacity: a
# source counted there during saturation must stay throttled until those
# failures age out, even if a bucket later frees up -- otherwise a fresh
# bucket would reset it. Shards are empty outside saturation, so this is a
# no-op in the common case.
ip_blocked = max(
_blocked_for(_LOGIN_IP_BUCKETS.get(ip), now, _LOGIN_IP_MAX_FAILS),
_overflow_blocked(ip, now),
)
return max(_blocked_for(_LOGIN_BUCKETS.get(key), now, _LOGIN_MAX_FAILS), ip_blocked)
def _clear_login_bucket(key: tuple[str, str]) -> None:
ip, _username = key
with _LOGIN_BUCKETS_LOCK:
_LOGIN_BUCKETS.pop(key, None)
_LOGIN_IP_BUCKETS.pop(ip, None)
# A successful login resets the IP's throttle, including any overflow it
# accumulated during saturation (drop only this IP's entry, so a
# shard-mate's throttle is untouched).
_overflow_shard(ip).pop(ip, None)
# Sync def (not async): compute_identity_proof touches SQLite on the first call,
# so FastAPI runs it in the threadpool rather than blocking the event loop.
@router.get("/identity")
def identity(nonce: str, request: Request) -> dict:
"""Challenge-response proof this is the real local Studio: caller sends a nonce,
gets HMAC(install identity secret, nonce, connection address + port).
Unauthenticated and side-effect free; a process that can't read the same-user
secret can't forge a proof, and binding to the address/port the connection
landed on stops a squatter relaying a proof from the real Studio elsewhere."""
try:
raw = base64.urlsafe_b64decode(nonce)
except Exception:
raise HTTPException(
status_code = status.HTTP_400_BAD_REQUEST, detail = "nonce must be base64url"
)
if not 16 <= len(raw) <= 128:
raise HTTPException(
status_code = status.HTTP_400_BAD_REQUEST, detail = "nonce must decode to 16-128 bytes"
)
# The address + port the connection actually landed on, from the socket
# (request.scope is getsockname, so it is the real local address even when
# bound to 0.0.0.0), never the client-controlled Host header.
server = request.scope.get("server") or ("", 0)
host = server[0] or ""
port = server[1] if server[1] is not None else 0
return {"proof": storage.compute_identity_proof(raw, host, port)}
@router.get("/status", response_model = AuthStatusResponse)
async def auth_status() -> AuthStatusResponse:
"""Auth initialization state; ``default_username`` is exposed for first-boot UI prefill only."""
return AuthStatusResponse(
initialized = storage.is_initialized(),
default_username = storage.DEFAULT_ADMIN_USERNAME,
requires_password_change = storage.requires_password_change(storage.DEFAULT_ADMIN_USERNAME)
if storage.is_initialized()
else True,
)
@router.post("/login", response_model = Token)
async def login(payload: AuthLoginRequest, request: Request) -> Token:
"""Login with username/password. Per-account + per-IP rate-limited."""
key = _bucket_key(request, payload.username)
unknown_key = _unknown_user_key(request)
blocked_for = max(_login_blocked(key), _login_blocked(unknown_key))
if blocked_for > 0:
raise HTTPException(
status_code = status.HTTP_429_TOO_MANY_REQUESTS,
# IP not interpolated into the body; behind a proxy/NAT it's
# misleading or an info leak.
detail = (f"Too many failed login attempts. " f"Try again in {blocked_for} seconds."),
headers = {"Retry-After": str(blocked_for)},
)
record = storage.get_user_and_secret(payload.username)
if record is None:
# Record under one sentinel key per IP so attacker-controlled username
# cardinality can't allocate unbounded buckets.
_record_login_failure(unknown_key)
raise HTTPException(
status_code = status.HTTP_401_UNAUTHORIZED,
detail = f"Incorrect password. To reset it, run this in your terminal: {_reset_password_command()}",
)
salt, pwd_hash, _jwt_secret, must_change_password = record
if not hashing.verify_password(payload.password, salt, pwd_hash):
_record_login_failure(key)
raise HTTPException(
status_code = status.HTTP_401_UNAUTHORIZED,
detail = f"Incorrect password. To reset it, run this in your terminal: {_reset_password_command()}",
)
_clear_login_bucket(key)
_clear_login_bucket(unknown_key)
access_token = create_access_token(subject = payload.username)
refresh_token = create_refresh_token(subject = payload.username)
return Token(
access_token = access_token,
refresh_token = refresh_token,
token_type = "bearer",
must_change_password = must_change_password,
)
@router.post("/logout", status_code = status.HTTP_204_NO_CONTENT)
async def logout(
request: Request, current_subject: str = Depends(get_current_subject_allow_password_change)
) -> Response:
"""Revoke refresh tokens for the subject; the access token is stateless and expires on its own."""
try:
storage.revoke_user_refresh_tokens(current_subject)
except Exception:
pass
try:
request.app.state.bootstrap_password = None
except AttributeError:
pass
return Response(status_code = status.HTTP_204_NO_CONTENT)
@router.post("/desktop-login", response_model = Token)
async def desktop_login(payload: DesktopLoginRequest) -> Token:
"""Exchange a local desktop secret for normal admin-subject tokens."""
username = storage.validate_desktop_secret(payload.secret)
if username is None:
raise HTTPException(
status_code = status.HTTP_401_UNAUTHORIZED,
detail = "Desktop authentication failed",
)
return Token(
access_token = create_access_token(subject = username, desktop = True),
refresh_token = create_refresh_token(subject = username, desktop = True),
token_type = "bearer",
must_change_password = False,
)
@router.post("/refresh", response_model = Token)
async def refresh(payload: RefreshTokenRequest) -> Token:
"""Exchange a refresh token for a new access+refresh pair (single-use)."""
consumed = storage.consume_refresh_token(payload.refresh_token)
if consumed is None:
raise HTTPException(
status_code = status.HTTP_401_UNAUTHORIZED,
detail = "Invalid or expired refresh token",
)
username, is_desktop = consumed
new_access_token = create_access_token(subject = username, desktop = is_desktop)
new_refresh_token = create_refresh_token(subject = username, desktop = is_desktop)
return Token(
access_token = new_access_token,
refresh_token = new_refresh_token,
token_type = "bearer",
must_change_password = False if is_desktop else storage.requires_password_change(username),
)
@router.post("/change-password", response_model = Token)
async def change_password(
payload: ChangePasswordRequest,
request: Request,
current_subject: str = Depends(get_current_subject_allow_password_change),
) -> Token:
"""Allow the authenticated user to replace the default password."""
record = storage.get_user_and_secret(current_subject)
if record is None:
raise HTTPException(
status_code = status.HTTP_401_UNAUTHORIZED,
detail = "User session is invalid",
)
salt, pwd_hash, _jwt_secret, _must_change_password = record
if not hashing.verify_password(payload.current_password, salt, pwd_hash):
raise HTTPException(
status_code = status.HTTP_401_UNAUTHORIZED,
detail = "Current password is incorrect",
)
if payload.current_password == payload.new_password:
raise HTTPException(
status_code = status.HTTP_400_BAD_REQUEST,
detail = "New password must be different from the current password",
)
storage.update_password(current_subject, payload.new_password)
storage.revoke_user_refresh_tokens(current_subject)
try:
request.app.state.bootstrap_password = None
except AttributeError:
pass
access_token = create_access_token(subject = current_subject)
refresh_token = create_refresh_token(subject = current_subject)
return Token(
access_token = access_token,
refresh_token = refresh_token,
token_type = "bearer",
must_change_password = False,
)
# ---------------------------------------------------------------------------
# API key management
# ---------------------------------------------------------------------------
def _row_to_api_key_response(row: dict) -> ApiKeyResponse:
return ApiKeyResponse(
id = row["id"],
name = row["name"],
key_prefix = row["key_prefix"],
created_at = row["created_at"],
last_used_at = row.get("last_used_at"),
expires_at = row.get("expires_at"),
is_active = bool(row["is_active"]),
)
@router.post("/api-keys", response_model = CreateApiKeyResponse)
async def create_api_key(
payload: CreateApiKeyRequest, current_subject: str = Depends(get_current_subject)
) -> CreateApiKeyResponse:
"""Create a new API key. The raw key is returned once and cannot be retrieved later."""
expires_at = None
if payload.expires_in_days is not None:
expires_at = (
datetime.now(timezone.utc) + timedelta(days = payload.expires_in_days)
).isoformat()
raw_key, row = storage.create_api_key(
username = current_subject,
name = payload.name,
expires_at = expires_at,
)
return CreateApiKeyResponse(
key = raw_key,
api_key = _row_to_api_key_response(row),
)
@router.get("/api-keys", response_model = ApiKeyListResponse)
async def list_api_keys(current_subject: str = Depends(get_current_subject)) -> ApiKeyListResponse:
"""List all API keys for the authenticated user (raw keys are never exposed)."""
rows = storage.list_api_keys(current_subject)
return ApiKeyListResponse(
api_keys = [_row_to_api_key_response(r) for r in rows],
)
@router.delete("/api-keys/{key_id}")
async def revoke_api_key(key_id: int, current_subject: str = Depends(get_current_subject)) -> dict:
"""Revoke (soft-delete) an API key."""
if not storage.revoke_api_key(current_subject, key_id):
raise HTTPException(
status_code = status.HTTP_404_NOT_FOUND,
detail = "API key not found",
)
return {"detail": "API key revoked"}