Four integration fixes wired up while bringing Qwen3-30B-A3B-Instruct-2507
green end-to-end on UNSLOTH_FAST_INFERENCE=1:
1. unsloth/models/llama.py patch_peft_model: transformers 5.x reports
model_type as "qwen3_moe" (with underscore); the PR's check was
"qwen3moe" and fell through to NotImplementedError.
2. unsloth/models/llama.py patch_peft_model dense MLP patching: the
fused gate/up/down LoRAMLP swap walks layer.mlp.gate_proj, which is
a Qwen3MoeSparseMoeBlock for MoE and has no gate_proj attribute.
Skip the swap when the MLP does not expose the dense trio; MoE
LoRA is wired through unsloth_zoo/moe_utils anyway.
3. unsloth/inference/flex_qwen3_llama.py flex attention forward:
bnb-4bit Linear compute produces fp32 k / v even under autocast,
which makes the paged KV index_put_ refuse the mixed dtype (bf16
cache, fp32 update). Cast k / v to self._paged_cache.k_cache.dtype
before update. Also benefits the dense path.
4. unsloth/inference/flex_engine.py bind_peft_model: for Qwen3 MoE the
ParamWrapper keeps LoRA un-merged on the stacked expert tensors, so
the training model's expert weights ARE the pristine source. Skip
the pristine-base deep-copy for arch=="qwen3_moe" and point
refresh_moe_lora_merge_from_pristine at the training base directly.
Avoids a third 30-60 GB residency on 30B-A3B.
5. unsloth/inference/flex_moe.py call_moe_model_with_flex_kwargs: lock
activations to the embed dtype across layernorm + MoE MLP; RMSNorm
+ bnb-4bit compute promote activations to fp32 along the MoE path
under autocast. Also force-restore Qwen3MoeSparseMoeBlock.forward
to the stock or unsloth_zoo version if FastQwen3MoeModel.pre_patch
clobbered it with a legacy Qwen3MoeSparseMoeBlock_fast_forward that
expects a flat self.gate_proj (which does not exist on transformers
5.x stacked-expert MoE blocks).
Adds tests/flex_moe_smoke.py: generates 32 tokens twice (cold + warm),
records first-call / warm-call tokens/s, peak VRAM, arch, impl. Writes
async_task_outputs/qwen3_moe_grpo_bench/smoke_A_{4bit,bf16}.json.
Measured on a single B200 (sm_100), Qwen3-30B-A3B-Instruct-2507 +
LoRA rank 16 + grouped_mm MoE backend:
| precision | t_load (s) | peak VRAM (GB) | cold tok/s | warm tok/s |
|-----------|------------|----------------|------------|------------|
| 4bit | 27.0 | 123.4 | 3.7 | 6.9 |
| bf16 | 33.5 | 125.9 | 3.7 | 6.7 |
Both completions coherent ("the lazy dog. ...").
a1aec618ce