A quantized KV cache crashes qwen4exp on llama.cpp, in two different ways

Symptom

--cache-type-k q8_0 --cache-type-v q8_0 loads cleanly — the full 93.6 GB across three shards comes up without complaint — and then aborts about 85 seconds later, after the weights are already resident, on the assertion above.

Root cause

The gate lives in src/llama-kv-cache.cpp:319:

attn_rot_k = !attn_rot_disable
          && n_embd_head_k_all > 0
          && ggml_is_quantized(type_k)           // q8_0 true, f16 false
          && hparams.n_embd_head_k() % 64 == 0;  // 256 % 64 == 0 -> true

A quantized K cache turns on a Hadamard rotation of K before it's written to the cache — error mitigation that shipped in PR #21038 — which allocates a self_k_rot buffer. qwen4exp's build_attn_qsa() was never taught to expect that buffer: its own assertion, at qwen4exp.cpp:544, insists self_k_rot and self_v_rot are null. One piece of code turns the rotation on for any quantized K type; the other was written before that rotation existed for this architecture. Neither is wrong in isolation — they simply disagree.

No smaller quant dodges this. The gate is ggml_is_quantized(type_k), true for q8_0, q5_1, q4_0, iq4_nl, and every other quantized KV type alike — this isn't a capacity problem that a smaller cache format works around.

The escape hatch, tested

The same expression is gated by the environment variable LLAMA_ATTN_ROT_DISABLE=1, which forces attn_rot_k false and lets q8_0 build. I ran it to conclusion. It gets further than expected, and it still can't be used.

What worked:

It loads. llama_kv_cache: attention rotation force disabled (LLAMA_ATTN_ROT_DISABLE) appears in the log, and the server reaches READY in 85 seconds — the exact point at which it previously aborted.
It saves 4 GiB — slightly better than the ~3 GiB predicted beforehand: MemAvailable 14 GiB idle versus 10 GiB on f16.

Quality held on every single-stream test. Five greedy prompts (temperature 0, top_k 1, fixed seed) against an f16 control:

prompt	truth	f16	q8_0
needle @ ~7.6K ctx	port 4007	`4007`	`4007` (byte-identical)
needle late @ ~7.6K	rack 17, team-J	correct	correct
count+enumerate over ctx	6: 003…018	correct	correct
multi-step arithmetic	261 km, 99.75 km	correct	correct
generate code + asserts	runs clean	PASS	PASS

Wording diverged between arms on a couple of prompts, which is expected rather than alarming: a tiny logit perturbation flips one token and greedy decoding follows a different but equally valid path from there. Semantics were identical on all five, and both generated-code outputs executed and passed their own asserts.

What killed it

A second, different assertion — one that never fired during sequential testing:

qwen4exp.cpp:284: GGML_ASSERT(mctx_idx->get_n_kv() == inp->mctx->get_attn()->get_n_kv()
                  && "the indexer cache must track the attention cache cell for cell") failed

The QSA indexer cache and the attention cache disagreed on cell count. It didn't fire during the sequential test suite. It fired once real concurrent load arrived — an agentic pipeline driving all four slots alongside the test.

A same-day control run made this look like a q8_0-specific problem:

arm	slots exercised	tasks	asserts	outcome
f16	0,1,2,3	11	0	survived
q8_0 + rot-disable	0,1,2,3	18	1	crashed

"f16 survived, so q8_0 causes it" is the reading most people would stop at here — and it's wrong. The next section shows why, and how that was confirmed.

The honest limit on that conclusion

get_n_kv() pads from used_max_p1() and does not reference the cache type at all (llama-kv-cache.cpp:1235), so the table above never had a direct causal path from quantization to the desync — only a correlation. It was equally consistent with a latent multi-slot bug in llama_memory_hybrid_idx that q8_0 merely perturbed into firing sooner. Eleven tasks with zero asserts on f16 is absence of evidence, not evidence of absence, and that is exactly why the original verdict here was written as f16 "not observed failing under real load" rather than "proven safe."

That hedge was confirmed, not just defensible, later the same day. At 13:22:57, the identical assert fired on f16 — no quantization anywhere in the run — at CTX_SIZE=131072 / UBATCH_SIZE=1024, after just 7 tasks across 3 slots. It was not an OOM this time either: nothing in the kernel log, a clean ggml_abort.

So the finding was never about quantization at all:

llama_memory_hybrid_idx desynchronizes the QSA indexer cache from the attention cache under multi-slot use, on any KV type. q8_0 reached it sooner; f16 reaches it too.

The log lines immediately before the second crash point at slot allocation, not decoding:

slot get_availabl: id  1 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id  1 | task 7225 | processing task, is_child = 0
qwen4exp.cpp:284: GGML_ASSERT(... indexer cache must track the attention cache cell for cell) failed

i.e., it fires when a new sequence starts alongside sequences already running — exactly when the two caches' occupancy can diverge. Practically, --parallel 4 is not safe on this architecture today, on either KV type. --parallel 1 removes cross-slot divergence entirely and is the only configuration not yet observed to hit this — which, per the paragraph above, is a claim this note has already been burned by trusting more than it deserved.

Verdict: f16 with --parallel 1

Neither KV type fixes the multi-slot crash — only reducing slot count does. q8_0 still costs no measurable quality versus f16 and still saves 4 GiB, but that saving no longer buys the concurrency headroom it originally looked like it might, since --parallel 1 is required on both KV types until upstream fixes the desync. f16 remains the safer default: it carries no crash of its own, and it avoids stacking a second, untested source of accuracy loss (the disabled Hadamard rotation) on top of a server already running at reduced concurrency. Whether --parallel 1 actually avoids the indexer desync for good is, itself, unconfirmed — it is only the one configuration not yet observed to hit it, and that was said of f16 under heavier concurrency right up until it was not true.

What's unaffected

--cache-type-* only ever touches the 12 attention layers. The recurrent GatedDeltaNet state is pinned to GGML_TYPE_F32 by the PR itself, and the QSA indexer cache is managed inside llama_memory_hybrid_idx — neither honors this flag, so there's no equivalent trap waiting in the other 36 layers.