Does a written pattern library teach a model how to do the work, or only how this shop does it? This experiment strips the library in stages and scores what falls out. Judgment survived at zero context. Conventions did not. The gap was 11 points, not the 40 the prediction called for.
The pattern library is the document that tells a coding agent how this project builds a data vault: naming, the hashing mandate, layer rules, two-stage staging, materializations, testing minimums. It also contained a Structures section explaining what hubs, links and satellites are, and when each applies. That second part is general discipline knowledge, not a fact about this project, and I suspected it was paying no rent.
Three runs, same frozen source spec, same rubric, same harness. Only the context varied: the full library, a thin version with the Structures section removed, and nothing at all beyond "build a raw vault to good Data Vault 2.0 practice."
The spec carries two deliberate traps. One is a collision case: a licence number that is only unique within a jurisdiction, sitting next to a parcel number that is genuinely global, so the correct design keys them differently. The other is a set of repeating attributes that should become a multi-active satellite rather than a hub and link.
Written down before any of it ran:
And the falsification condition, also written first: if the no-library run fails a trap, the library is carrying judgment, the teaching sections earn their place, and thinning it is dangerous.
| Context | Words | Score | Collision trap | Multi-active trap |
|---|---|---|---|---|
| Full library | 709 | 100 | pass | pass |
| Full library, repeat run | 709 | 100 | pass | pass |
| Thin (Structures section removed) | 592 | 100 | pass | pass |
| Colloquial rewrite (deliberately degraded) | 1,376 | 100 | pass | pass |
| None | 0 | 89 | pass | pass |
Four different context treatments, one score. Full, thin, and a rewrite I degraded on purpose all land on 100, and two independent runs of the identical input agree exactly, dimension for dimension.
Removing the library entirely costs 11 points, and all 11 sit in conventions. The judgment dimensions score 44 out of 44 in every condition, including the one with no context at all.
The library's existence is worth 11 points. Its content and its phrasing are worth zero. What it buys is the part that cannot be inferred from anywhere, and nothing else.
Prediction two said the no-library run would land between 45 and 65. The first grading gave it 62. Inside the range, recorded as confirmed, and quoted for a day as a 34-point context effect.
Then the whole series was re-graded blind, in a shuffled pool with prior scores stripped and a matched effort protocol. The no-library run came back at 89. Nine of the ten cells scored the same or higher, and this was the largest correction in the series at 27 points.
So the prediction was wrong by more than twenty points, and the grading bias hid it. First-pass scoring was systematically harsh rather than randomly noisy, and the error scaled with how bad a submission looked. The worst-looking cell got the worst treatment, which is exactly the cell that decided whether my prediction held.
I would have carried "prediction two confirmed" forward as a fact. That is the argument for re-grading your own instrument before you trust anything it told you, and it is a separate write-up.
Given nothing but the source spec and one line asking for good Data Vault practice, the run independently:
That last one is the part I did not expect. The instruction asking for escalation lives in the library, and escalation behaviour survived its removal.
It is also worth being precise about what the no-library run
lost. It was not sloppy. It used hub_contractor and
sat_ prefixes, single-stage staging, and a
thoughtful but non-matching set of tests. Those are
automate_dv's own documented conventions: coherent,
defensible, and not this project's. The run followed a different
standard rather than no standard.
Running the no-library condition against a mid-tier model changes which half breaks.
| No library | Total | Judgment (44) | Conventions (56) | Multi-active trap |
|---|---|---|---|---|
| Stronger model | 89 | 44 | 45 | pass |
| Mid-tier model | 92.5 | 40 | 52.5 | partial |
Strip the context and the stronger model loses conventions only. The mid-tier model holds its conventions better and loses judgment instead.
Which means the cost of under-specifying is not a fixed number you can budget against. It changes category as you move down the stack. A convention gap is visible in review: wrong prefix, missing test, staging in one step instead of two. A judgment gap looks correct.
The mid-tier model outscores the stronger one, 92.5 to 89, while being worse at the thing that matters. The rubric pays more for convention polish than for judgment, so the more dangerous output wins on the scoreboard.
This is a rubric flaw, and it is also the finding, demonstrated on the instrument built to catch exactly that failure mode. A benchmark designed to detect confidently wrong design ranked the confidently wrong design higher. A human reading a pull request has less to work with than the rubric did.
Two time-boxed blocks and roughly 229,000 tokens for the two parallel cold runs. The re-grade added ten cold graders across the pool.
The expensive part was neither. It was discovering that a day of conclusions rested on grading nobody had controlled, and having to re-run the scoring before any of it could be used.
A context document's job is the un-inferable: conventions, tooling mandates, layer structure, testing minimums, the exceptions specific to this business. Sentences that teach the discipline itself measured zero at every tier tested.
They also carry a risk the un-inferable content does not. A stale or wrong sentence in a teaching section can only override a model that would have been right. Nobody has a competing belief about your naming convention, so that half has no such downside.
The working test for any section of a context document: could a competent practitioner who has never seen this project derive this? If yes, it is a candidate for deletion. Then re-run the ablation rather than trusting the judgment, and grade it blind in a shuffled pool before believing the number.
The frozen spec, the rubric, all ten graded submissions, the
pre-registered hypotheses, and the re-grade protocol live in the
lab repository. Every experiment in that lab runs on public or
synthetic data only, so nothing here is downstream of anyone's
production system.
Related: porting 65 dbt models to Snowflake