A pre-flight probe reported an exhausted thinking budget as an unconstrained grammar

finish_reason: "length"  content: ""  reasoning_content: 172 chars  (max_tokens=40, thinking ON)

Symptom

An engine-swap pre-flight exists specifically to catch what /v1/models can't — it probes the parameters and features the pipeline actually sends before trusting a new engine with a real edition. On the 2026-08-26 swap to llamacpp/Qwen3.8-Flash-Next-UD-IQ4_XS, it declared the engine NOT SAFE:

NOT SAFE — 1 problem(s):
  - response_format: accepted but the reply does not match the schema — JSON-mode
    beats will run unconstrained and log nothing

That is the exact wording the same tool uses for a real defect it caught on 2026-08-20 — llama.cpp returning 500 for GBNF on the chat endpoint while silently accepting and ignoring response_format. This time the schema bound perfectly. The probe was the problem.

The three-arm test that isolated the cause

The pre-flight's schema-binding probe asks the model to answer {"verdict": "yes"|"no"} inside a max_tokens budget. Holding the budget and the model's thinking mode independent of each other isolates which one actually produced the empty reply:

max_tokens	thinking	result
40	ON	finish_reason: "length", content: "", reasoning_content: 172 chars
512	ON	{"verdict": "yes"} — binds
40	OFF	{"verdict": "yes"} — binds

Either lever alone recovers a verdict. Raise the budget with thinking still on: fine. Turn thinking off and leave the budget at 40: also fine. The only failing cell is the one where a model that reasons before it answers was given a budget too small to finish reasoning and still emit anything — a fact about the budget, not about whether response_format constrains the output.

Root cause

The probe read message.content, parsed it as JSON, and treated any exception as "accepted but not binding":

txt = body["choices"][0]["message"]["content"]
obj = json.loads(txt)
bound = set(obj) == {"verdict"} and obj["verdict"] in ("yes", "no")

At max_tokens=40 with the model's default --reasoning auto, reasoning consumed the entire budget before any content token was emitted. txt was "". json.loads("") raised, the surrounding except caught it, bound stayed False, and the tool printed ACCEPTED BUT NOT BINDING — a verdict about the schema, produced by a failure that had nothing to do with the schema. finish_reason was present in the same response and was never read.

Fix

Two changes, in hn-briefings/tools/preflight_engine.py and hn-briefings/tests/test_preflight_engine.py:

• The probe's own max_tokens moved from 40 to 512 — enough room for the 94 completion tokens the identical request needed with thinking on.
• An empty completion at finish_reason: "length" is now checked for explicitly, before the json.loads call, and reported as its own state — NO VERDICT — thinking exhausted the probe budget — instead of being folded into the schema-binding result. It still exits 1: an unanswerable probe is a real problem, just a different one, and the failure text now says binding is unknown, not disproved.

A second test in the same file had been silently vacuous since it was written: it asserted on max_tokens by scraping inspect.getsource starting from the response_format key, but max_tokens is assigned earlier in the function than that key, so the scrape always read None and the assertion could never fail. It had been passing the entire time the probe's budget was 40. It was rewritten to assert on the payload the probe actually sends.

Why the wrong verdict cost more than a false positive

The test file's own module docstring names the stakes plainly:

It runs at exactly the moment the operator is least able to debug it — right after an engine swap, before the first edition — so a broken diagnostic is worse than none.

This one didn't just misfire quietly. It reused the exact description of a real, previous outage to explain a problem that wasn't that outage. A false alarm wearing a real name is the expensive kind: it sends whoever is on call to re-check something that was never broken, and it spends the credibility the same alarm needs the next time response_format genuinely doesn't bind.

Reproduction

1. Serve a model with reasoning on by default (--reasoning auto in llama.cpp, or equivalent) behind an OpenAI-compatible endpoint.
2. Send a chat completion with a response_format JSON-schema constraint and a max_tokens small enough that reasoning alone can exhaust it — 40 reproduced this on Qwen3.8-Flash-Next (qwen4exp).
3. Confirm the response shows finish_reason: "length" with an empty content and a non-empty reasoning_content.
4. Re-run holding one variable fixed and changing the other — a materially larger max_tokens, or thinking disabled. Either change alone should recover a binding response, confirming the original failure was budget, not grammar.
5. Confirm whatever reads the response checks finish_reason before treating empty content as evidence about the schema.