Field note · vLLM

Structured output stalls forever once you register 28 tools

vLLM builds a Lark grammar enumerating every registered tool as an alternative. Past roughly 28 tools the guidance parser gives up, and the model generates tokens that never reach the client.

Software
vLLM 0.18.1rc1.dev205+ga6f72a773 · Mistral Small 4 119B NVFP4
Stop: ParserTooComplex

Symptom

With a client registering 28 tools — file and process operations, session management, web fetch, image and PDF handling, scraping, among others — running against Mistral Small 4 with --reasoning-parser mistral --tool-call-parser mistral --enable-auto-tool-choice, requests stop completing. The model keeps generating, visibly, at 15–17 tok/s — it isn't stuck — but the output never reaches the client. The request hangs indefinitely, and the GPU sits busy the whole time producing tokens nobody receives.

Root cause

PR #37081 changed vLLM's guided decoding to generate a Lark grammar enumerating every registered tool as a parser alternative:

fcalls: ((<TOOL_CALLS> SAFE_WS? "read" <ARGS> ...) | (<TOOL_CALLS> SAFE_WS? "edit" <ARGS> ...) | ... 28 tools ...)+

At 28 tools, that grammar is too complex for the llguidance parser: 98,431 cached rows, 53,575 items, and 518,155 trie nodes walked before it gives up.

This isn't specific to DGX Spark, or to this one client. It's a property of how the grammar is built: complexity scales linearly with tool count, on any hardware, for any client that registers enough tools. The PR's own review thread said as much — reviewer sfeng33 noted that "the Lark grammar applies overhead to all tool_choice modes, including none and auto, even when unnecessary." Anyone registering a large tool set against vLLM's guided decoding is exposed to this, independent of what hardware it runs on.

Fix

Drop --reasoning-parser mistral for clients that register a lot of tools — in this deployment, that meant launching with mistral-start-local.sh (no reasoning parser) instead of mistral-start-local-patched.sh, which turns it on. The cost: without the reasoning parser, raw [THINK]...[/THINK] tags show up unparsed in the model's output, so the client has to strip or handle them itself instead of vLLM doing it.