* Scope MoE expert LoRA detection to actual MLP projection targets
_moe_target_set_from_string treated any regex containing the substring mlp
or ffn as targeting the expert MLP projections. Unsloth's auto-generated
attention-only regex lists mlp, ffn and feed_forward as allowed intermediate
path segments while its final group matches only q_proj/k_proj/v_proj/o_proj,
so attention-only finetuning on MoE models silently enabled expert LoRA as
well: the experts were trained and every MoE layer paid the extra expert LoRA
grouped matmuls. Detect expert intent from the projection names themselves
(gate_proj/up_proj/down_proj/gate_up_proj) instead of the mlp substring.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
* Detect MoE expert LoRA via mlp path segment, not proj names
The auto-generated target regex always lists every projection leaf
(q/k/v/o and gate/up/down), so keying detection on a proj name mis-fired:
it enabled expert LoRA for attention-only regexes and dropped the
mlp/ffn path regexes. Key on the mlp/ffn/feed_forward/experts path
segment instead, which is present only when the MLP/experts are actually
targeted. Add a regression test for the attention-only case.
* Scope expert LoRA targets to the leaves a regex names
An mlp path alternative with attention-only leaves, for example
(mlp|self_attn).(q_proj|o_proj), no longer enables expert LoRA, and a
regex naming a single expert leaf such as .*experts.*down_proj now
targets only that projection instead of the whole broad set. Generic
mlp projections (.*mlp.*proj) and the auto regex mlp tag block keep the
broad set for fused-expert models whose leaves are plain Parameters.
* Route explicit leaf list into MoE expert detection
An attention-only explicit target_modules list routed through get_peft_regex
for family scoping (e.g. FastVisionModel with vision layers off) yields a
regex carrying the full mlp|feed_forward|ffn|dense component block even though
its leaf group only names q/k/v/o_proj. Keying expert detection on that regex
trained the experts for a language-only/attention-only request. Use the
caller's original leaf list for detection; only the auto path uses the regex,
where the mlp block is the sole MLP-intent signal on fused-expert models.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Respect finetune_mlp_modules and finetune_language_layers scope for MoE expert detection
When an explicit leaf list that names MLP projections (gate_proj/up_proj/down_proj)
is routed through get_peft_regex under finetune_mlp_modules=False, the scoped regex
correctly drops the MLP leaves, but MoE expert detection was still keyed on the
original list and re-added mlp.experts.* via target_parameters, training the experts
the caller had frozen. Same gap for finetune_language_layers=False on vision-only runs.
Prefer the original list only when MLP and language families are both in scope
(preserving the attention-only fix); otherwise honor the scoped result so the frozen
family is respected. Factored the choice into _select_moe_detection_targets with unit
tests over the full selection matrix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
7cc1752a64