feat(moe): mxfp4-resident MoE experts for DeepSeek-V4-Flash LoRA#2548

Open

excepshenal wants to merge 24 commits into

NVIDIA-NeMo:mainfrom

excepshenal:dshen/feat/mxfp4-expert-lora

Commits on Jun 12, 2026

chore(dsv4): fused AdamW + lora attn=tilelang
Daniel
committed
fix(moe): lazy DeepEP buffer alloc (single-node OOM workaround)
Daniel
committed
feat(peft): mxfp4-resident MoE expert base weights for LoRA

Daniel
and
claude
committed

Commits on Jun 13, 2026

feat(peft): mxfp4-resident frozen MoE experts for DeepSeek-V4-Flash LoRA

Daniel
and
claude
committed
feat(moe): init-capable packed-weight registration for mxfp4 experts
Daniel
committed
feat(moe): mxfp4 passthrough load path for DeepSeek-V4 (no bf16 experts)

Daniel
and
claude
committed
refactor(quantization): move mxfp4 primitives into components/quantization

Daniel
and
claude
committed
feat(dsv4): wire mxfp4 passthrough end-to-end + to_hf packed split

Daniel
and
claude
committed
fix(moe): skip random init for mxfp4-resident experts

Daniel
and
claude
committed
fix(moe): set requires_grad at Parameter construction in ExpertParallel

Daniel
and
claude
committed
fix(dsv4): require expert parallelism for mxfp4; guard single-GPU

Daniel
and
claude
committed
fix(recipe): clean process exit + deadlock-safe distributed validation

Daniel
and
claude
committed
perf(mxfp4): drop redundant contiguous copy in expert grouped GEMM

Daniel
and
claude
committed
fix(peft): support mxfp4 passthrough for LoRA-targeted experts

Daniel
and
claude
committed
style(mxfp4): apply ruff format to expert LoRA + DSV4 adapter

Daniel
and
claude
committed
perf(mxfp4): faster, spec-correct expert dequant (~2.4x kernel, ~2.5x e2e)

Daniel
authored and
Daniel Shen
committed

Commits on Jun 14, 2026

Merge branch 'main' into dshen/feat/mxfp4-expert-lora
akoumpa
authored

Commits on Jun 15, 2026

feat(moe): support mxfp4-resident experts under DeepEP dispatch

Daniel Shen
and
claude
committed
feat(peft): enable LoRA on routed+shared experts under mxfp4

Daniel Shen
and
claude
committed
fix(deepseek_v4): cast lm_head weight to activation dtype for fused linear-CE

Daniel Shen
and
claude
committed
chore(deepseek_v4): default the mxfp4 LoRA recipes to FusedLinearCrossEntropy

Daniel Shen
and
claude
committed
chore(deepseek_v4): fp32 LoRA adapters in the mxfp4 recipes

Daniel Shen
and
claude
committed
test(moe): fix stale unit tests broken by lazy DeepEP buffer + packed-param requires_grad

Daniel Shen
and
claude
committed
docs(examples): clarify dequantize_base_checkpoint for mxfp4 passthrough

Daniel
and
claude
committed