What I think is happening

1. Evaluation is the bottleneck, not capability

I ran an LLM judge in production for months, trusted its verdicts, and shipped changes on the strength of them. Then I gave it two drafts generated from an identicalconfiguration — same prompt, same model, same settings, differing only by sampling noise — and asked it to pick a winner. It picked one, confidently, with reasons, about a third of the time. Every A/B result I had produced before that test wasn't simply wrong, it was unreadable: it sat on top of noise I had never measured, and a 55% win rate had not beaten anything if the instrument invents a preference out of nothing 33% of the time on its own.

Full account: My LLM judge was flipping a coin a third of the time.

2. Autonomous systems fail in ways that look like success

The Agentic Daily researches, drafts, judges, and publishes an edition every day with nobody holding the merge button. A system like that doesn't announce when a part of it breaks — it keeps shipping, and a green checkmark is not the same claim as a correct one. The clearest example I've measured: constraining a model's output with a decoding grammar took a task from 121.9 seconds to 1.4 seconds on identical input. Both outputs passed schema validation. Only one of them had actually reasoned about the question — the grammar forces the first token, so the model never enters its reasoning phase at all. The JSON stayed green while the answer quietly got worse.

The same shape shows up a layer down, in the inference server itself, on the same class of hardware the newsroom depends on. One completion I captured came back with its full token budget spent, HTTP 200, empty content, and reasoning_tokens pinned at zero — no error, no partial output, nothing in the response a caller would normally trust to say anything had gone wrong. Just nothing, successfully.

More on the newsroom: The Agentic Daily. The completion above: the empty-reasoning field note.

3. The undocumented frontier is closer than it looks

“Frontier” usually means something abstract and far off. On my desk it has meant: I bought current-generation hardware, and the software did not know it existed yet. DGX Spark's GPU identifies as SM121. vLLM's FP8 kernels were guarded by a compile-time check for exactly SM120 — __CUDA_ARCH__ == 1200 — and executed a deliberate trap instruction on anything else, including hardware one number past the one the guard was written for. The fix was two lines, widening an equality check to a range, and nobody had published it yet.

That was one of nine failures I've documented on this machine so far: a build that only fails on aarch64 because a CUDA call was compiled against a signature the driver doesn't have; a structured-output grammar that, past roughly 28 registered tools, keeps a model generating tokens nobody ever receives; a different model on the same box deadlocking silently around seven hours in, still answering health checks while producing zero tokens per second. None of this is exotic, and it isn't all about hardware: the tool-count ceiling scales the same way on any machine — its own note says so outright — while the SM121 trap and the aarch64 build failure are what new hardware concretely costs before the software catches up. Either way, undocumented failure surface is where I've spent most of my actual engineering hours this year.

The running list, updated as I hit the next one: field notes.