Rule Z Benchmark — Results

3 variants × 3 runs = 9 total builds · + 3 follow-up Rule Z v2 runs · same spec, same model, isolated contexts

UPDATE — Rule Z v2 results (added after initial benchmark)

After the first 9 runs showed Rule Z's main win was in render-allocation patterns, I applied three changes and re-ran the C variant (3 more builds: C4, C5, C6):

  1. Pruned the CODE REUSE section (kept as one line) — the data showed it was ballast
  2. Added concrete counter-patterns naming style={{...}} and onClick={() => ...} with the fix inline
  3. Replaced the prose Quality Audit with a structured YAML block that a new verification hook can cross-check against the code
MetricBaseline ARule Z v1Rule Z v2
Inline style={{...}} in JSX54.6721.000.00
Inline onClick={() => ...}6.000.670.00
Total anti-patterns60.6724.332.67
vs no-Rule-Z baseline—−60%−96%

All 3 v2 runs hit zero inline objects AND zero inline arrow functions. The explicit, concrete counter-patterns outperformed v1's abstract "zero allocations in hot paths" wording by ~89%. Concrete examples beat abstract rules — turned out that was the whole difference.

TL;DR

Context engineering did not matter in this benchmark. Every variant (including A — no context) found and reused the shared utilities by grepping the codebase. Vesa's underlying thesis — "modern agents grep well enough" — is strongly supported.

Rule Z produced a clear, measurable reduction in render-time allocation patterns. Inline object literals in JSX dropped 61% (A→C), inline arrow functions in JSX dropped 89%, total anti-patterns dropped 60%.

My on-the-record prediction was partly wrong. I predicted B would close most of the “didn't know” gap — but there was no gap to close. I predicted the C-B delta would be smaller than A-B — it was actually larger. The patterns Rule Z catches are not the ones context can close.

Setup

VariantContextRule ZRuns
A No contextNoneNo3
B Minimal map2-line CLAUDE.md listing shared utility pathsNo3
C Map + Rule ZSame 2-line CLAUDE.mdFull [QUALITY BAR] + mandatory ## Quality Audit3

Feature: Task Dashboard (list, filter, bulk update, 30s auto-refresh, loading/error states). Stack: React + Express + TypeScript. Pre-existing utilities in base repo: formatDueDate, apiCall (cached), MemoryCache, validateTask, shared types.

Results — Averaged Across 3 Runs per Variant

Metric A avg B avg C avg C vs A
Files written7.337.09.67+32%
Lines of code418.67401.67469.33+12%
Reuse hits — formatDueDate + apiCall333.33+11%
Reimplemented date formatter000—
Raw fetch() calls (bypassing apiCall)000—
array.find() in render000—
Set/Map usage for lookups5.675.675.0−12%
JSON.parse(JSON.stringify) clone000—
Inline object literals in JSX (style={{...}})54.6763.3321.0−62%
Inline arrow functions in JSX6.05.00.67−89%
setState inside loops*00.331.67false positive
TOTAL anti-patterns60.6769.024.33−60%
Quality Audit file written0/30/33/3100%

* The setState inside loops heuristic caught functional updates inside .map() (optimistic bulk update), which is actually the correct pattern. False-positive — see Limitations below.

Finding 1 — Context engineering didn't matter for structural reuse

Vesa was right

All 3 A-variant runs (no CLAUDE.md, no hints at all) used the shared formatDueDate and apiCall utilities. Nobody reimplemented a date formatter. Nobody wrote a raw fetch(). Reuse hits were identical between A (no context) and B (minimal map): 3.0 each.

The minimal CLAUDE.md I planned to push as a Rule Z prerequisite would have had zero measurable effect on this benchmark. Agents grep'd the base repo on their own and found what they needed. Modern agents grep faster than they parse docs.

Finding 2 — Rule Z produced real reductions in “lazy default” patterns

Render-time allocation patterns collapsed under Rule Z

In all 3 A-variant and 3 B-variant runs, agents produced 55–70 inline object allocations per build (mostly style={{...}} re-created every render) and 5–7 inline arrow functions in JSX props (breaks React.memo).

Under Rule Z, those collapsed to 21 inline objects and <1 arrow function on average, because the agents proactively hoisted styles to module scope constants and bound event handlers via useCallback instead of inline arrows.

This is the “AI knew but reached for the lazy option anyway” failure mode — context engineering can't fix it because there's nothing missing to learn.

Finding 3 — Rule Z induced architectural improvements too

All 3 C runs used React.memo + useMemo + useCallback + stable refs

Every C-variant run independently produced:

None of the A or B runs included any of these patterns, even though the agents had the same model and nearly identical feature spec.

Where my prediction was wrong

PredictionResultStatus
B closes most of the “didn't know” gapThere was no gap to close — A already closed it via grepWRONG
B still misses “lazy default” patternsConfirmed — B was actually slightly worse than A on render allocationsRIGHT
C closes the remainder, delta B→C smaller than A→BDelta C−B (−65% anti-patterns) was larger than A→B (+14% anti-patterns)WRONG

The honest revision: context engineering and Rule Z target non-overlapping failure modes. Context helps when the AI doesn't know something exists. Rule Z helps when the AI knows but picks the lazy option. In this benchmark, the former didn't matter because agents grep'd on their own — the entire measurable improvement came from the latter.

Limitations

Conclusion

The revised synthesis that came out of this:

Minimal structural context (Vesa's 2-line map) had zero measurable effect in this benchmark — agents grep'd on their own. I'd still ship it for bigger repos where grep surface area explodes, but it's not load-bearing for small-to-medium projects.

Rule Z produced a clear 60% reduction in render-allocation anti-patterns — the category context can't address because the information isn't missing, the discipline is.

Combined story: minimal context + hard quality walls + free exploration between. The quality walls are doing the heavy lifting, not the context layer. That's the opposite of what I'd assumed going in.

Benchmark run 2026-04-14 · 9 parallel agent builds, same feature spec, same base repo, isolated contexts · claude-agent-team