Proven on real data. Every figure measured.

Spiderbrain turns a project into a dependency graph and scores every node for importance, blast radius and risk. For those scores to be worth acting on, two things must hold: the ranking has to concentrate what matters, and the engine must never silently corrupt a node's identity or give a different answer on the same input. This report documents both, measured on 14 real datasets: 13 external corpora across six domains, plus Spiderbrain's own codebase.

External corpora validated 2026-05-30 with validate-real-corpora.mjs on decision layer v5.0.7. Self-analysis generated 2026-06-04 with the v5 engine. Consolidated in v5.1.0: 30 gate scripts, 353 assertions green. Nothing here is a target or a projection.

Summary

Across 13 external corpora, signal lift ranged 1.48× to 2.41× (importance concentrated 48% to 141% above a typical item). Identity silent-corruption was zero; 94 real same-hash collision groups were tested with zero silent misattachment; every decision score was finite; and every result was bit-identical under suppression and deterministic under shuffling. One defect was found by this validation and fixed before release. Run on its own codebase, Spiderbrain found 7 masters across 1,525 nodes with clean community structure and provably independent metrics.

† The blind-spot signal lift is not computable on the self-brain (metric variance too low, see Assessment); webscore concentration is shown instead. Identity and collision guarantees apply to the decision-layer mutation tests on the 13 external corpora.

Metrics, names and definitions

Every metric used in this report, in plain language.

The five decision-layer guarantees

Validation log

Per-dataset results, exactly as the harness recorded them. Signal lift is the discriminationRatio; rename-auto is the share of renames auto-resolved (the rest correctly defer); reach and necessity are decision-layer scores on the top-ranked node.

Every corpus also passed, uniformly: silent-corruption 0, decision scores finite, determinism identical ×3, and signal lift bit-identical before and after suppression. Suppression removed exactly 2 adjudicated nodes per corpus without changing the ranking of the rest.

14th dataset: Spiderbrain on Spiderbrain

We run the engine on its own source. This is the brain of the shipping app, generated on 2026-06-04.

Rationale

Why across domains. An importance score that only works on code proves little. Code, people, insurance, medicine, geophysics and materials share nothing except structure, so passing all of them shows the method keys on structure, not on the quirks of one domain.

Why these guarantees. A ranking is only useful if the thing it ranks keeps its identity as the project changes, and if the same input always yields the same answer. So we test identity under churn, the worst case of a deleted content twin, finiteness, determinism, and invariance under suppression.

Why dogfood. We run Spiderbrain on Spiderbrain. If the team would not trust it on its own code, neither should you.

Why determinism is the bar. Agents act on these scores. A score that changes between runs cannot be audited or trusted. Same input, same brain, every time.

Assessment

All 13 external corpora pass every guarantee after one fix, and the self-analysis behaved exactly as designed, including where it declined to score.

The defect this validation caught. The first run was not clean, which is the point of testing on real data. Neurology and insurance each showed one silent misattachment. The root cause: the matcher assumed a coincidental content twin has different neighbors, but on dense record graphs two records can share both the content hash and most of their neighbors. When one was deleted and exactly one twin survived, the judgment auto-applied. The fix captures whether the content hash was unique at the moment of decision; a judgment made on a non-unique hash never auto-applies and always defers to a human. Code brains never hit this because their twins differ structurally, so only the dense real record graphs exposed it.

Honest notes.Across the six clinical areas, signal lift spans 1.48× (vaccines) to 1.60× (diabetes); we show that exact range rather than rounding it to a tidy 1.5× to 1.6×. The self-brain's blind-spot index could not be computed: its variance was too low because the brain was built without commit history and the codebase is structurally uniform, so the engine returned a diagnostic instead of a fabricated number. Rename-auto rates below 60/60 on the code brains (44/60, 52/60, and neurology's 59/60) are collision-group members correctly deferring, not failures: you cannot auto-resolve a rename while an identical twin still exists.

Scope and limits. Mutations are synthetic but applied systematically (60-sample stride, no clock, no randomness). Each corpus was run once, which is sufficient because every result is deterministic and bit-identical under shuffle. Public datasets are modeled as dependency graphs, not used in their original tabular form.

Impact

Most relevance and importance scores are heuristic and drift between runs. Spiderbrain's does not. It is deterministic, it is identity-safe under churn, and it generalizes from source code to clinical trials to earthquakes on the same engine. That is the property an AI agent needs from its memory: a structural map it can act on, replay and audit, that says the same thing today and next week. This is the foundation under context engineering, the difference between a model that re-reads everything and one that knows what matters.

What this means, by domain