Skip to content

Commit c76f87a

Browse files
functionstackxclaude
functionstackx
and
claude
committed
[Klaud Cold] minimaxm3-fp8-h200-vllm-agentic: test CPU KV-offload connector
Test recipe for vLLM v1's native CPU KV-cache offloading connector on MiniMax-M3 MXFP8 (H200), using the agentic-coding scenario (Claude Code trace replay via aiperf inferencex-agentx-mvp) at a single large concurrency. Config (nvidia-master.yaml minimaxm3-fp8-h200-vllm-agentic): TEP8 (TP8 + expert parallel), offloading: cpu, conc 64, duration 1800s, on the day-zero vllm/vllm-openai:minimax-m3 image. New script benchmarks/single_node/agentic/minimaxm3_fp8_h200.sh, modeled on the M2.5 H200 agentic sibling with M3-specific serve flags: - --block-size 128 (mandatory for MSA sparse attention) - --language-model-only (text-only; frees VRAM for KV) - BF16 KV (no --kv-cache-dtype fp8: MXFP8 lacks calibrated KV scales and fp8 KV corrupts output, vllm-project/vllm#45381) - prefix caching ENABLED (coding traces share large prefixes; offloading that cache to CPU is the point of the test) - CPU offload via vLLM native connector: --kv-offloading-backend native --kv-offloading-size $TOTAL_CPU_DRAM_GB --disable-hybrid-kv-cache-manager (TOTAL_CPU_DRAM_GB default 600), same path as the M2.5 H200 agentic recipe Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent e68e746 commit c76f87a

3 files changed

Lines changed: 148 additions & 0 deletions

File tree

‎.github/configs/nvidia-master.yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4902,6 +4902,29 @@ minimaxm3-fp8-h200-vllm:
49024902
- { tp: 8, ep: 8, conc-start: 1, conc-end: 256 }
49034903
- { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 512 }
49044904

4905+
# MiniMax-M3 MXFP8 H200 agentic-coding (Claude Code trace replay) with CPU
4906+
# KV-cache offloading. Test of vLLM v1's native KV-offloading connector on M3 at
4907+
# a single large concurrency: TEP8 (TP8 + expert parallel), offloading: cpu,
4908+
# conc 64. Serve flags mirror the M3 H200 fixed-seq-len recipe (--block-size 128
4909+
# for MSA, --language-model-only, BF16 KV — MXFP8 has no calibrated fp8 KV
4910+
# scales, vllm-project/vllm#45381). Prefix caching stays enabled (coding traces
4911+
# share large prefixes; offloading that cache is the point). CPU offload via
4912+
# --kv-offloading-backend native (TOTAL_CPU_DRAM_GB default 600), matching the
4913+
# M2.5 H200 agentic recipe.
4914+
minimaxm3-fp8-h200-vllm-agentic:
4915+
image: vllm/vllm-openai:minimax-m3
4916+
model: MiniMaxAI/MiniMax-M3-MXFP8
4917+
model-prefix: minimaxm3
4918+
runner: h200
4919+
precision: fp8
4920+
framework: vllm
4921+
multinode: false
4922+
scenarios:
4923+
agentic-coding:
4924+
- duration: 1800
4925+
search-space:
4926+
- { tp: 8, ep: 8, offloading: cpu, conc-list: [ 64 ] }
4927+
49054928
dsr1-fp4-gb200-dynamo-trt:
49064929
image: nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.8.1.post2
49074930
model: nvidia/DeepSeek-R1-0528-NVFP4-v2

‎benchmarks/single_node/agentic/minimaxm3_fp8_h200.sh‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
set -x
4+
5+
# Agentic trace-replay benchmark for MiniMax-M3 MXFP8 on H200 using vLLM, with
6+
# CPU KV-cache offloading. 427B/26B-active MoE with MSA sparse attention.
7+
#
8+
# M3-specific serve flags (vs the M2.5 H200 agentic sibling):
9+
# --block-size 128 mandatory for MSA (sparse_block_size is 128; the
10+
# default 16 misaligns sparse indexing).
11+
# --language-model-only text-only workload; skips the vision encoder, freeing
12+
# VRAM for KV.
13+
# NO --kv-cache-dtype fp8: MiniMax-M3-MXFP8 ships without calibrated KV scales,
14+
# so fp8 KV corrupts output (vllm-project/vllm#45381).
15+
# Keep BF16 KV (matches the M3 H200 fixed-seq-len recipe).
16+
# Prefix caching is left ENABLED (vLLM default): agentic coding traces share
17+
# large prefixes across turns/subagents, so prefix-cache reuse — and offloading
18+
# that cache to the CPU tier — is exactly what this test exercises.
19+
#
20+
# CPU offloading (OFFLOADING=cpu) uses vLLM v1's native KV-offloading connector
21+
# (--kv-offloading-backend native), the same path the M2.5 H200 agentic recipe
22+
# uses, to extend usable concurrency past the on-HBM KV cliff.
23+
#
24+
# Required env vars:
25+
# MODEL, TP, CONC, OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR, DURATION, EP_SIZE
26+
27+
source "$(dirname "$0")/../../benchmark_lib.sh"
28+
29+
check_env_vars MODEL TP CONC OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE
30+
31+
if [ -z "${MAX_MODEL_LEN:-}" ] || [ "$MAX_MODEL_LEN" = "0" ]; then
32+
MAX_MODEL_LEN=131072
33+
fi
34+
35+
if [[ -n "${SLURM_JOB_ID:-}" ]]; then
36+
echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}"
37+
fi
38+
39+
nvidia-smi
40+
41+
# ---- Resolve traces and install deps ----------------------------------------
42+
# M3 serves at max_model_len up to ~250k; use the 256k-capped trace corpus
43+
# (470 traces, max in+out <= 256k) so requests aren't rejected at the cap.
44+
export WEKA_LOADER_OVERRIDE=semianalysis_cc_traces_weka_with_subagents_256k
45+
46+
# ~444 GB of MXFP8 weights off a shared network FS; concurrent day-zero
47+
# downloads hit huggingface_hub's WeakFileLock "[Errno 116] Stale file handle"
48+
# race. Retry the (resumable) download, then serve with HF_HUB_OFFLINE=1 so
49+
# vllm's snapshot_download does a lock-free local-cache read.
50+
SERVE_OFFLINE=()
51+
if [[ "$MODEL" != /* ]]; then
52+
for attempt in 1 2 3 4 5; do
53+
hf download "$MODEL" && break
54+
if [ "$attempt" = 5 ]; then echo "hf download failed after $attempt attempts" >&2; exit 1; fi
55+
echo "hf download attempt $attempt failed; retrying in 60s" >&2
56+
sleep 60
57+
done
58+
SERVE_OFFLINE=(env HF_HUB_OFFLINE=1)
59+
fi
60+
61+
resolve_trace_source
62+
install_agentic_deps
63+
64+
# ---- Server config ----------------------------------------------------------
65+
SERVER_LOG="$RESULT_DIR/server.log"
66+
mkdir -p "$RESULT_DIR"
67+
68+
OFFLOAD_ARGS=""
69+
case "$OFFLOADING" in
70+
none) ;;
71+
cpu)
72+
# vLLM v1 native KV-offloading connector (CPU tier).
73+
export VLLM_USE_SIMPLE_KV_OFFLOAD=1
74+
OFFLOAD_ARGS="--kv_offloading_backend native --kv_offloading_size $TOTAL_CPU_DRAM_GB --disable-hybrid-kv-cache-manager"
75+
;;
76+
*)
77+
echo "Error: unsupported OFFLOADING value '$OFFLOADING' (expected one of: none, cpu)" >&2
78+
exit 1
79+
;;
80+
esac
81+
82+
if [ "$EP_SIZE" -gt 1 ]; then
83+
EP=" --enable-expert-parallel"
84+
else
85+
EP=" "
86+
fi
87+
88+
echo "Starting vllm server..."
89+
export TORCH_CUDA_ARCH_LIST="9.0"
90+
export PYTHONNOUSERSITE=1
91+
# ~444 GB of MXFP8 weights off shared FS; engine startup can exceed the
92+
# default 600s readiness window.
93+
export VLLM_ENGINE_READY_TIMEOUT_S=3600
94+
95+
"${SERVE_OFFLINE[@]}" vllm serve $MODEL \
96+
--host 0.0.0.0 \
97+
--port $PORT \
98+
--tensor-parallel-size=$TP \
99+
$EP \
100+
--gpu-memory-utilization 0.90 \
101+
--max-model-len $MAX_MODEL_LEN \
102+
--block-size 128 \
103+
--language-model-only \
104+
--max-num-seqs $CONC \
105+
--stream-interval 20 \
106+
--trust-remote-code \
107+
$OFFLOAD_ARGS > "$SERVER_LOG" 2>&1 &
108+
SERVER_PID=$!
109+
echo "Server PID: $SERVER_PID"
110+
111+
wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"
112+
113+
# ---- Run benchmark ----------------------------------------------------------
114+
build_replay_cmd "$RESULT_DIR"
115+
116+
run_agentic_replay_and_write_outputs "$RESULT_DIR"

‎perf-changelog.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3814,3 +3814,12 @@
38143814
description:
38153815
- "Extend MiniMax-M3 MXFP8 H100/H200 non-MTP sweeps to concurrency 1 on the latency rows (H100: TP8; H200: TP4 and TP8) and add full TEP coverage from conc 1 to 256 (H100: TP8+EP8; H200: TP4+EP4 and TP8+EP8, incl. a new TP4+EP4 row for 8k1k). H200 TP8+EP8 upper bound moves 512->256 (high concurrency stays covered by the TP8+EP8 dp-attn DEP rows). DEP rows unchanged"
38163816
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1761
3817+
3818+
- config-keys:
3819+
- minimaxm3-fp8-h200-vllm-agentic
3820+
scenario-type:
3821+
- agentic-coding
3822+
description:
3823+
- "Initial submission (test): MiniMax-M3 MXFP8 H200 agentic-coding (Claude Code trace replay) with CPU KV-cache offloading — tests vLLM v1's native KV-offloading connector on M3 at a single large concurrency (TEP8: TP8 + expert parallel, offloading: cpu, conc 64, duration 1800s)"
3824+
- "New agentic recipe benchmarks/single_node/agentic/minimaxm3_fp8_h200.sh: --block-size 128 (MSA), --language-model-only, BF16 KV (no --kv-cache-dtype fp8 — MXFP8 lacks calibrated KV scales, vllm-project/vllm#45381), prefix caching enabled; CPU offload via --kv-offloading-backend native --kv-offloading-size $TOTAL_CPU_DRAM_GB --disable-hybrid-kv-cache-manager (default 600 GB), matching the M2.5 H200 agentic recipe"
3825+
pr-link: TBD

0 commit comments

Comments
 (0)