Rich Sutton's Bitter Lesson Revisited
Rich Sutton's seminal 2019 essay, The Bitter Lesson, demonstrated that general methods leveraging massive search and learning consistently outperform human-crafted heuristics in AI. But in 2026, we are witnessing an inverse distortion: developers are using massive general-purpose autoregressive models to perform rigid, closed-world mechanical lookups that should never have entered a chat completion loop in the first place.
When an agent wants to know whether src/auth.ts exists, prompting a 100-billion parameter transformer with 64,000 tokens of chat history to emit {"name": "view_file", "arguments": {"path": "src/auth.ts"}} is not a triumph of general intelligence—it is a catastrophic architectural inefficiency.
THE CHAT COMPLETION LOOP BOTTLENECK:
[Agent Runner]
|
+--> [Format JSON Prompt with 200 MCP tool descriptions] (8,000 tokens)
+--> [HTTPS POST to /v1/chat/completions]
+--> [Tokenize entire conversation history]
+--> [Prefill KV Cache: 380ms]
+--> [Decode: 20 tokens @ 35ms/token: 700ms]
+--> [Total Roundtrip: 1,350ms]
|
+--> [Parse JSON from markdown backticks, handle schema errors]
+--> [Execute Local Tool]The Overhead of Conversational Tool Use
Conversational interfaces were designed for humans reading paragraphs on glass screens. Machines communicating with machines do not require conversational formatting. When agent harnesses wrap tool calling inside chat completions, they incur three compounding penalties:
- Syntax Serialization Overhead: Tools are described as JSON Schema objects inside system prompts. A developer with 40 MCP tools easily consumes 12,000 tokens of prompt context on *every single request* before the user even types a character.
2. Decoding Latency: Autoregressive decoding is memory-bandwidth bound. Generating 30 tokens of JSON arguments one token at a time requires 30 sequential GPU memory reads across entire parameter tensors.
3. Schema Hallucination & Repair: Because models generate tokens probabilistically, they frequently emit invalid JSON keys, forgotten closing braces, or invented parameter types, forcing agent frameworks to spend additional turns on JSON repair.
Moving to Machine-Native Gateways
JevProxy re-architects this flow by positioning a machine-native reverse proxy in front of LLMs. Instead of transmitting dozens of tool definitions upstream, JevProxy inspects incoming agent intentions and executes deterministic tool routing at the edge in sub-25ms. When upstream models are invoked, they receive clean, pre-sharded contexts with forced tool choice—reducing token consumption by over 80%.
The future of autonomous software is not bigger chat models answering smaller questions. It is fast, calibrated System One gateways handling the machinery of software development, freeing frontier models to focus exclusively on difficult reasoning.