Field notes
Things that broke while running language models on hardware I own, and what fixed them. Written for whoever hits the same error next — including me.
DGX Spark
workaroundDefaultOOMScoreAdjust=200 in the systemd user manager kills terminal-launched servers first
A production llama-server process was killed by the kernel with nothing in its own log: no assert, no stack trace, just silence, noticed only because /metrics stopped responding. The killed process carried oom_score_adj:200. The cause was not the coding-agent session that launched it — it was a systemd user-manager default, DefaultOOMScoreAdjust=200, which scores every terminal-launched process to die before system services do, even ones many times smaller. That score cannot be lowered after launch by an unprivileged process, so the fix is to run the server as a systemd system service instead, which defaults to OOMScoreAdjust=0.
August 26, 2026
DGX Spark
resolvedDetecting which inference engine owns a port: vLLM vs. llama.cpp
Detection that infers the engine from the port number breaks the moment either engine moves. Moving llama.cpp onto the port normally used by vLLM mislabeled it, and downstream code silently wrote null. /props is a clean discriminator: llama-server serves it, and the vLLM OpenAI-compatible server has no such route and returns 404. A second gotcha in the same detour: docker ps --filter publish= does not match a container using host networking.
August 26, 2026
agentic-daily
resolvedA pre-flight probe reported an exhausted thinking budget as an unconstrained grammar
An engine-swap safety check asked a reasoning model for a JSON verdict inside a 40-token budget. Reasoning consumed the whole budget, content came back empty, and the probe scored that as a schema that does not bind. The grammar was fine. A three-arm test isolates budget from grammar to show why.
August 26, 2026
llama.cpp
workaroundA quantized KV cache crashes qwen4exp on llama.cpp, in two different ways
A quantized KV cache (--cache-type-k q8_0) crashes qwen4exp twice, not once. The first crash is a Hadamard-rotation gate that qwen4exp does not implement; disabling that rotation produces a server that loads, saves 4 GiB, and matches f16 on every quality test run against it, then fails a second, different assert once real concurrent load arrives across all four slots. That second assert looked quantization-specific at first. It is not: the identical assert was later confirmed on f16 too, with no quantization involved, so the real cause is a multi-slot desync in llama_memory_hybrid_idx that q8_0 merely reaches sooner. Verdict: f16, run with --parallel 1 until upstream fixes the desync.
August 26, 2026
llama.cpp
upstreamThe Qwen3.8-Flash-Next MTP head is real, and it is in none of the GGUFs
The model card advertises a 4B multi-token-prediction head, but the converter shipped with the PR that adds this architecture sets supports_mtp_export = False, no GGUF carries a nextn_predict_layers key, and qwen4exp.cpp has zero nextn references. No speculative decoding is possible today, and two separate upstream changes, not one, would be needed before it is.
August 26, 2026
llama.cpp
workaroundqwen4exp is not qwen3next: llama.cpp master cannot load Qwen3.8-Flash-Next
The GGUF for Qwen3.8-Flash-Next reports its own architecture as qwen4exp, not qwen3next, and llama.cpp master has zero references to that string — it dies on unknown-architecture without ever mentioning a branch. PR #27742 is the only way to load it, and the payoff is a 125B model running at 22-27 tok/s with no speculative decoding at all, close to the 25.5 tok/s this host measures from Qwen3.8-27B on vLLM INT4 with a DFlash2 drafter attached.
August 26, 2026
llama.cpp
resolvedRSS is the wrong instrument for memory on a DGX Spark
The llama-server process shows about 29 GiB of RSS while about 87 GiB of weights for Qwen3.8-Flash-Next sit in CUDA buffers that RSS never counts, and nvidia-smi reports [N/A] for memory on GB10 because there is no separate GPU pool to report. MemAvailable before and after start is the honest measurement, and that holds for every model on a DGX Spark, not just this one. The idle headroom measured at launch was not production headroom either: the same configuration was later killed by the kernel under real load, with nothing in its own log, and working defaults moved to a smaller context size and ubatch afterward.
August 26, 2026
vLLM
investigatingA reasoning parser with no start/end strings silently disables thinking-budget enforcement
Muse-Glimmer delimits reasoning by channel markers rather than a token pair, so vLLM cannot resolve reasoning token ids, so reasoning_config.enabled stays false, so the budget cap never engages — on a model whose chat template reasons by default.
August 21, 2026
vLLM
investigatingMuse-Glimmer returns empty reasoning_content while the parser demonstrably extracts it
Every completion reports reasoning_content: "" and reasoning_tokens: 0, even when a hundred tokens of reasoning were generated. The parser returns the reasoning when called directly. The defect is somewhere between the two, and I have not found it.
August 21, 2026
vLLM
workaroundcache_kernels.cu fails to build on aarch64 with CUDA 13.0
A KV-cache batching optimization calls cuMemcpyBatchAsync with a signature the CUDA 13.0 aarch64 driver API does not have. The build stops before anything runs.
April 3, 2026
vLLM
investigatingWhich NVFP4 MoE backend actually works on SM121 (it depends on the model)
One model needs --moe-backend cutlass and is slow under marlin. Another needs VLLM_SCALED_MM_BACKEND=marlin and crashes without it. NVIDIA officially recommends marlin. All three statements are true.
March 30, 2026
vLLM
workaroundFlashInfer CUTLASS MoE fails TMA descriptor init on SM121
FlashInfer ships a precompiled fused-MoE library built for SM120. Its TMA descriptors do not initialize on SM121, so concurrent MoE batches crash after hours of clean operation. vLLM cannot patch it.
March 29, 2026
vLLM
resolvedEvery FP8 GEMM traps on DGX Spark (SM121)
vLLM guards its FP8 CUTLASS kernels on __CUDA_ARCH__ == 1200 and executes a deliberate trap instruction on anything else. DGX Spark is arch 1210, so every FP8 GEMM crashes. Two lines fix it.
March 28, 2026
vLLM
investigatingTwo ways a DGX Spark inference server dies after hours of clean operation
Mistral NVFP4 traps with an illegal instruction at around ten hours. Qwen3.6 int4 deadlocks silently at around seven, still answering health checks at zero tokens per second. Whether these share a root cause is still open.
March 28, 2026
vLLM
workaroundStructured output stalls forever once you register 28 tools
vLLM builds a Lark grammar enumerating every registered tool as an alternative. Past roughly 28 tools the guidance parser gives up, and the model generates tokens that never reach the client.
March 28, 2026
vLLM
upstreamIndexError in the multimodal processor when vLLM initializes Mistral Small 4
A change to how multimodal processors handle token inputs crashes Pixtral-architecture models during dummy-input construction, before the server ever accepts a request.
March 26, 2026