Evaluating code intelligence tools is genuinely hard. The obvious metrics, does the completion compile? does the test pass?, capture correctness but miss the structural properties that matter most for production engineering: is the suggested code consistent with the codebase's architecture? Does it respect team boundaries? Does it introduce a dependency that shouldn't exist?
CrossCodeEval is the most carefully constructed benchmark for cross-file code completion in TypeScript (among other languages). It is worth understanding both what it measures well and where it falls short.
What CrossCodeEval measures
CrossCodeEval evaluates a model's ability to complete code that requires understanding cross-file context, definitions, types, and interfaces that live in files other than the one being edited.
The benchmark construction is rigorous: it takes real repositories, identifies completion points that demonstrably require cross-file context (by verifying that single-file context produces wrong answers), and evaluates completion quality via Exact Match (EM) and Edit Similarity (ES) against ground truth.
For TypeScript specifically, this is well-shaped for the structural question: can a code intelligence tool correctly surface the right cross-file context for a given completion task?
What it doesn't measure
Blast radius and structural safety: CrossCodeEval measures whether the completion is correct. It does not measure whether the completed code is structurally safe, whether it introduces unexpected dependencies, crosses community boundaries, or increases blast radius inappropriately.
Determinism across runs: evaluation is typically single-run per model. Real production systems care whether the tool gives the same answer when you run it twice, which probabilistic completion tools do not guarantee.
Codebase-scale graph properties: CrossCodeEval uses individual repository samples, not enterprise-scale codebases. The structural complexity of a 500k-line monorepo is not represented.
Non-completion tasks: the benchmark only evaluates completion. Code intelligence includes querying ("what depends on this?"), planning ("what's the blast radius of this refactor?"), and governance ("which files changed ownership this sprint?"), none of which CrossCodeEval touches.
Spiderbrain's position relative to CrossCodeEval
Spiderbrain is not primarily a completion engine, so CrossCodeEval is not its native benchmark. Its structural outputs, blast radius, community membership, spikescore, transitive closure, do not map to EM/ES metrics.
Where Spiderbrain is relevant to CrossCodeEval-style tasks is in context selection: given a completion point, which files should be in the model's context window? The hypothesis (currently being validated) is that a graph-selected context slice produces higher EM/ES scores than a vector-selected or heuristic-selected slice, because structural relevance is a better proxy for completion relevance than semantic similarity alone.
The right benchmark for Spiderbrain
The right evaluation for Spiderbrain is a deployment-safety benchmark, not a completion benchmark:
- Precision of blast radius: does the computed blast radius correctly predict which modules are affected by a change?
- Community detection stability: do community boundaries remain stable across semantically equivalent refactors?
- Context selection recall: does the graph-selected context slice include all the files a model needs to complete a task correctly?
These benchmarks do not currently exist as published standards. Building them is part of the research roadmap.
Why the benchmark gap matters
The code intelligence category is currently evaluated almost entirely on completion quality. This creates a benchmark monoculture: tools optimise for EM/ES on CrossCodeEval-style tasks, even if the tasks most engineers face daily are structural rather than completive.
The teams that feel the pain of the gap, the ones whose "AI passed the test but broke production" post-mortems pile up, are the ones who will push for structural benchmarks. That pressure is building. The tooling that can answer "was this change structurally safe?" will be evaluated on metrics that don't yet exist in the literature.
Spiderbrain is building toward those metrics. CrossCodeEval is the starting point, not the finish line.