5907d1525c
Phase 1 results (`scripts/benchmarks/results/lora_rollout_baselines.md`):
- vLLM+LoRA: 4581 decode tok/s, 156 GB peak (gold, 100%)
- unsloth_fi_false+LoRA: 641 tok/s, 15.8 GB peak (14%, 7x lower mem)
- CB paged+FA4 persistent+LoRA: 422 tok/s (9.2%)
- CB sdpa_paged persistent+LoRA: 434 tok/s (9.5%)
Headline finding: Unsloth's `fast_inference=False` path (custom HF inference
kernels with cached fp16 LoRA in `fast_linear_forward`) is 1.5x faster than
CB at 1/7th the peak memory. Phase 2 will include it as a first-class backend.
cb_vs_vllm_generation.py:
- New --lora_adapter flag. vLLM uses LoRARequest; tpaged uses
PeftModel.from_pretrained (no merge_adapter so we measure LoRA-active
inference); unsloth_fi_false copies the adapter weights into Unsloth's
get_peft_model wrapper (with key normalization so PEFT's base_model.model.
prefix and Unsloth's .default. wrapper both match).
- New unsloth_fi_false backend: batched generate across all 32 prompts in
a single call after FastLanguageModel.for_inference(model).
- Exposed sampling knobs (temp/top_p/min_p/top_k); defaults are the
equivalence params.
cb_sync_driver.py (Phase 3 scaffold):
- SyncCBDriver owns PagedAttentionCache + ContinuousBatchProcessor +
FIFOScheduler on the main thread. Never calls manager.start() so there is
no background thread.
- slice_inputs=False => fixed-shape buffer views each step => CUDA graph
replay is safe.
- use_cuda_graph=True path: 2-step eager warmup, then capture one decode
step, then replay. `_is_pure_decode()` keeps prefill out of the graphed
path since those have varying shapes.
- Greedy sampling only (CUDA-graph-safe); stochastic sanity checks stay in
the non-graphed path.
- Standalone benchmark harness at the bottom.
qwen3_grpo_unified.py (Phase 4 scaffold):
- Single entrypoint for vllm / unsloth_fi_false / cb_paged / cb_sdpa /
naive_trl backends sharing dataset, reward funcs, sampling, and the
torch_debugging_utils StatisticsCallback.
- New --compile_mode {default,reduce-overhead,max-autotune-no-cudagraphs}
that compiles `trainer.model.forward` and `trainer.ref_model.forward`
after the trainer is built. CompileDebugger tracks graph breaks and
recompiles. Skipped for vLLM since vLLM owns its own compile pipeline.
- Post-warmup median (skip first 3 steps) is computed and saved alongside
the full per-step logs.
make_lora_adapter.py: writes a canonical PEFT adapter to
outputs/lora_rank32_fresh. Re-initializes lora_B with a tiny gaussian so
the adapter isn't a no-op (PEFT's default zero-init would let LoRA kernels
short-circuit).
qwen3_grpo_notebook.py (Phase 0): notebook-to-script port with
StatisticsCallback and equivalence sampling. 10-step reference reported in
scripts/benchmarks/results/notebook_ref_10.md (median step 5.80s,
peak 158.9 GB).
93 lines
3.3 KiB
Python
93 lines
3.3 KiB
Python
"""One-shot: materialize a rank-32 LoRA adapter on `unsloth/Qwen3-4B-Base`.
|
|
|
|
Writes a PEFT-style directory so every backend (vLLM `LoRARequest`,
|
|
`peft.PeftModel.from_pretrained`, Unsloth `FastLanguageModel.get_peft_model`)
|
|
can load the SAME weights. Random-init is fine for throughput measurement --
|
|
the goal is to have LoRA kernels active during generation, not a trained
|
|
model.
|
|
|
|
Run:
|
|
CUDA_VISIBLE_DEVICES=6 python scripts/benchmarks/make_lora_adapter.py \
|
|
--output outputs/lora_rank32_fresh
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import os
|
|
from pathlib import Path
|
|
|
|
|
|
def parse_args():
|
|
p = argparse.ArgumentParser()
|
|
p.add_argument("--model_name", default="unsloth/Qwen3-4B-Base")
|
|
p.add_argument("--output", default="outputs/lora_rank32_fresh")
|
|
p.add_argument("--rank", type=int, default=32)
|
|
return p.parse_args()
|
|
|
|
|
|
def main():
|
|
args = parse_args()
|
|
out_dir = Path(args.output).resolve()
|
|
out_dir.mkdir(parents=True, exist_ok=True)
|
|
|
|
# Use vanilla HF -- PEFT's save_pretrained yields the canonical
|
|
# adapter_config.json + adapter_model.safetensors that vLLM's LoRARequest
|
|
# expects. Loading via Unsloth would leak Unsloth-specific LoRA wrappers.
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
from peft import LoraConfig, get_peft_model
|
|
|
|
tok = AutoTokenizer.from_pretrained(args.model_name)
|
|
if tok.pad_token is None:
|
|
tok.pad_token = tok.eos_token
|
|
|
|
# bf16 base; we only need structure + save. Keep on CPU to avoid a GPU load
|
|
# just for `save_pretrained`.
|
|
print(f"[make_lora_adapter] Loading {args.model_name} on CPU...")
|
|
model = AutoModelForCausalLM.from_pretrained(args.model_name, dtype=torch.bfloat16)
|
|
|
|
peft_cfg = LoraConfig(
|
|
r=args.rank,
|
|
lora_alpha=args.rank * 2,
|
|
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
|
|
"gate_proj", "up_proj", "down_proj"],
|
|
bias="none",
|
|
task_type="CAUSAL_LM",
|
|
lora_dropout=0.0,
|
|
)
|
|
peft_model = get_peft_model(model, peft_cfg)
|
|
peft_model.print_trainable_parameters()
|
|
|
|
# Ensure both A and B matrices are non-zero. PEFT initializes A with
|
|
# kaiming_uniform and B with zeros -- which makes the adapter a no-op and
|
|
# would mask LoRA kernels on some backends. Seed B with tiny random values.
|
|
n_reinit = 0
|
|
with torch.no_grad():
|
|
for name, p in peft_model.named_parameters():
|
|
if "lora_B" in name:
|
|
p.normal_(mean=0.0, std=1e-4)
|
|
n_reinit += 1
|
|
print(f"[make_lora_adapter] Reinitialized {n_reinit} lora_B matrices with tiny gaussian.")
|
|
|
|
peft_model.save_pretrained(str(out_dir))
|
|
tok.save_pretrained(str(out_dir))
|
|
|
|
# Sanity: verify safetensors file present and non-trivial.
|
|
from safetensors import safe_open
|
|
st_path = out_dir / "adapter_model.safetensors"
|
|
n_zero_tensors = 0
|
|
n_tensors = 0
|
|
with safe_open(str(st_path), framework="pt") as f:
|
|
for key in f.keys():
|
|
t = f.get_tensor(key)
|
|
n_tensors += 1
|
|
if (t == 0).all().item():
|
|
n_zero_tensors += 1
|
|
print(f"[make_lora_adapter] Wrote {n_tensors} tensors to {st_path} "
|
|
f"({n_zero_tensors} all-zero).")
|
|
print(f"[make_lora_adapter] Adapter saved to {out_dir}")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|