4717bce97e
Adds --load_in_4bit (+ --model_name_4bit override) to both the flex benchmark script and the vllm/tpaged benchmark script. When set, loads the pre-quantized Unsloth bnb-4bit shard (e.g. unsloth/Qwen3-4B-Base-unsloth-bnb-4bit) and keeps the LoRA adapter as a PEFT wrapper instead of merging, because merging into 4-bit weights is not supported. Ties lm_head.weight to model.embed_tokens.weight post-load in both scripts, because the bnb-4bit shards ship without an lm_head parameter even though tie_word_embeddings is True in the config, so transformers leaves it randomly initialised otherwise (garbage generations). Results at batch 64 + LoRA rank 32: | Backend | tok/s | peak mem | output | |-------------------------------|------:|---------:|-----------| | Unsloth fast_inference (vLLM) | 4515 | 159 GB | coherent | | flex (this PR) | 1738 | 40.6 GB | coherent | | transformers CB (sdpa) | 504 | 124 GB | gibberish | 4-bit costs ~40 % throughput on the vLLM path vs bf16 and ~70 % on flex. flex regresses worse because PEFT-without-merge doubles the matmuls per projection (base + LoRA add) on top of bnb dequant, whereas bf16 flex merges LoRA into the base. Peak memory barely moves for vLLM because KV cache at gpu_memory_utilization=0.8 dominates regardless of base size. transformers CB (generate_batch) at 4-bit + LoRA produces garbage even with lm_head tied. Likely PEFT-over-bnb + batched CB interaction; not debugged further -- it was always the 10 % reference path. Writeup updated in scripts/benchmarks/results/flex_vs_vllm.md with a new "Same workload at load_in_4bit=True" section.
25 lines
No EOL
1.4 KiB
JSON
25 lines
No EOL
1.4 KiB
JSON
{
|
|
"backend": "qwen3_flex",
|
|
"capture_cudagraph": true,
|
|
"lora_adapter": "outputs/lora_rank32_fresh",
|
|
"n_prompts": 64,
|
|
"n_decoded_tokens": 28172,
|
|
"wall_times_s": [
|
|
19.392819664964918,
|
|
14.716534855018836,
|
|
14.195494230021723,
|
|
15.050612487946637,
|
|
16.052564749028534
|
|
],
|
|
"median_wall_s": 15.050612487946637,
|
|
"best_wall_s": 14.195494230021723,
|
|
"decode_tps_median": 1871.8175105871403,
|
|
"decode_tps_best": 1984.5733824765107,
|
|
"max_new_tokens": 512,
|
|
"peak_memory_gb": 40.59123468399048,
|
|
"sample_completions": [
|
|
"Let's denote the length of segment $DB$ as $ units.\n<start_working_out>\nTo solve this problem, we can use the Power of a Point theorem, which states that for a point P inside a circle, the product of the lengths of the segments of any two intersecting chords through P is constant. In this case, we have two intersecting chords: AB and CD. Let",
|
|
"Let $P(x)$ be a monic polynomial of degree $2023$ such that $P(k) = k^{2023}P(1-\\frac{1}{k})$ for every positive integer $1 \\leq k \\leq 2023$. We want to find $P(-1)$.\n\nFirst, we can rewrite the given equation",
|
|
"First, let's consider the cube's edges. A cube has 12 edges. Each edge is parallel to 3 other edges. However, we need to be careful not to double-count the pairs.\n\nLet's count the pairs of parallel edges:\n\n1. Each edge is parallel to 3 other edges, so there are 12 * 3 = 36 pairs.\n2."
|
|
]
|
|
} |