Which NVFP4 MoE backend actually works on SM121 (it depends on the model)

Three answers, apparently in conflict

Three sources, three different answers to what sounds like one question — which NVFP4 MoE backend should vLLM use on SM121:

Model	Working setting	Evidence
Mistral Small 4 119B NVFP4	--moe-backend cutlass, stable 9+ hours under load. Marlin is FP4→FP16 decompression and very slow; flashinfer_cutlass / auto hit the TMA descriptor crash.	My own testing
Nemotron-3-Super 120B-A12B NVFP4	VLLM_SCALED_MM_BACKEND=marlin on the stock nightly, which flips the NvFp4 MoE backend from FLASHINFER_CUTLASS to MARLIN. Full async speed, 14.1–14.8 tok/s sequential.	My own testing and write-up
Not model-specific	--moe-backend marlin	NVIDIA's official DGX Spark guidance

Read quickly, that looks like it can't all be true. It's all true. Two things are easy to conflate here, and both need to be said plainly: --moe-backend and VLLM_SCALED_MM_BACKEND are different knobs, not two names for the same setting — and the right answer between them is model-dependent, not hardware-dependent. Both rows above ran on the same DGX Spark GB10 hardware. What changed was the model sitting on top of it.

Two knobs, not one disagreement

--moe-backend is an explicit vLLM CLI flag. Set it to cutlass or marlin and vLLM uses that MoE kernel implementation directly — that's the flag in the first and third rows above.

VLLM_SCALED_MM_BACKEND is not that flag. It's an environment variable that governs quantized-GEMM backend selection more broadly, and in the middle row's config — where --moe-backend was never set at all — setting it to marlin had the side effect of flipping the resolved NvFp4 MoE backend from FLASHINFER_CUTLASS to MARLIN. Same destination, different road. The two are easy to conflate because they share a value (marlin) and end up choosing the same kind of thing.

It's worth being concrete about how easy this is to get wrong: the working Nemotron config already had three other Marlin-flavored environment variables set — VLLM_NVFP4_GEMM_BACKEND=marlin, VLLM_TEST_FORCE_FP8_MARLIN=1, VLLM_MARLIN_USE_ATOMIC_ADD=1 — and all three were already in place while the server was still crashing on the third request, every time. None of them touch MoE backend selection. The one that stopped the crashes was VLLM_SCALED_MM_BACKEND=marlin, added on its own.

The diagnostic that ruled out broken kernels

Before that fix was found, the working theory could easily have been "these kernels are fundamentally broken on SM 12.1." The diagnostic that ruled that out was CUDA_LAUNCH_BLOCKING=1, set on a hunch.

That variable forces every CUDA kernel launch to be synchronous — no overlap between streams. With it set, every crash disappeared: ten sequential requests, then a full benchmark including 4- and 8-concurrent runs, zero failures. Throughput dropped from the ~11–13 tok/s seen on the first couple of successful async requests to about 7–8 tok/s — exactly the cost you'd expect from losing kernel overlap. Rock solid, just slower.

That result is the whole diagnosis in miniature. If serializing the kernel launches makes a crash disappear, the kernels are not fundamentally incompatible with the hardware — something about their overlap, on this architecture specifically, is unsafe. Before this test, crash traces had landed in a different place almost every run — FlashInfer attention prefill, scaled_fp8_quant, FlashInferFP8ScaledMMLinearKernel — which looks like three unrelated bugs until you know CUDA reports these errors asynchronously, and any of those call sites could simply be wherever the host happened to check next. CUDA_LAUNCH_BLOCKING=1 turned "three bugs in three places" into "one race condition, observed in three places."

VLLM_SCALED_MM_BACKEND=marlin fixed the actual race at full async speed, so CUDA_LAUNCH_BLOCKING was never needed in production here. But as a diagnostic step, before you know what the fix is, it's the cheapest way I know of to tell "broken kernel" apart from "async race."

Test your own model

I don't have a root cause that explains why Mistral Small 4 wants cutlass and Nemotron-3- Super wants marlin. Nemotron-3-Super is a Mamba-2 + attention hybrid, with about 12B of its 120B parameters active per token through its MoE layer — a different architecture, a different vLLM build, tested days apart. That may be enough to explain the divergence. It may not be. I genuinely don't know yet, which is why this note is filed as "investigating," not "resolved."

What I am confident of is the shape of the mistake to avoid: don't take any single row of that table — mine included — and apply it to a model it wasn't tested on. If you landed here from a search for an SM121 MoE crash, the useful next step isn't copying a flag off this page. It's setting CUDA_LAUNCH_BLOCKING=1 against your own model, confirming whether the crash disappears, and then testing --moe-backend cutlass against VLLM_SCALED_MM_BACKEND=marlin against NVIDIA's --moe-backend marlin on your own hardware — before trusting any of the three, including this one.