The capability-recall study
The strongest quantitative result in the project, measured head-to-head against the tools auditors actually run. Capa runs against three real treatments on 25 hand-built Python / Capa pairs, scoring 48 per-function facts on two distinct questions. The decisive finding: under closed-world SBOM semantics, Capa never clears a function incorrectly, where the best dataflow tool clears ten.
Lower is better: a false-clearance silently tells a consumer a function is safe when its code can reach the authority.
A claim is only as useful as what its absence means
A Capa program ships a manifest of capability claims: for every function, which authorities it could possibly reach. An auditor verifies them by re-deriving the SBOM from source with the compiler itself, not a second analyser.
An SBOM is a closed list. What is not listed for a function is implicitly excluded. A consumer reading "this function reaches nothing sensitive" treats that silence as a clearance. So the dangerous failure mode is not a missing detection in the abstract; it is a function the record silently clears while the code can in fact exercise the authority. The study measures exactly that, against the tools that produce such records today.
Four treatments, 48 facts, two questions
The corpus is 25 hand-built Python / Capa pairs: 20 covering direct and via-helper authority, plus 5 covering via-dispatch and via-data indirection. The unit is one (python_function, capability) fact, 48 in total.
Package-granularity SBOM, the Syft / cdxgen style. No per-function granularity at all.
A good-faith pattern heuristic.
Good-faith dataflow analysis (python-all 7.1.2), the strongest real tool on the corpus. Direct-fact recall is 36/36 with zero over-attribution, so every dispatcher miss is the tool's limit, not a weak query.
The capability record the Capa compiler emits, with a sound provably-excluded channel proved in Agda.
Two distinct questions, reported separately, never collapsed into one number. Q1 is positive attribution: does the treatment credit capability C to function F? Q2 is false-clearance under closed-world semantics: does the treatment leave a true fact silently cleared?
False-clearance under closed-world semantics
A treatment commits a false-clearance for a true fact (F, C) when it gives the consumer no way to know F can exercise C. Absence equals exclusion, so a silently blank function reads as cleared.
CodeQL false-clears the ten via-dispatch / via-data dispatcher facts it cannot resolve. Capa's manifest gives each (F, C) three states: reachable, provably-excluded (sound, proved in Agda), or not-determined. The ten dispatcher facts land in not-determined, not excluded, so Capa clears nothing: 0 false-clearances by construction.
On attribution, Capa ties the best dataflow tool, exactly
Does the treatment attribute C to function F? Identical criterion for all: C appears in the treatment's output for F.
CodeQL and Capa both attribute 38/48: the 36 direct facts plus the 2 via-helper facts; neither attributes any of the 10 dispatcher facts. Capa does not see more than CodeQL on Q1. Its advantage is not attributing more: it is never clearing a function incorrectly, which is Q2.
The three regimes
The corpus separates three regimes by how the authority is reached. Each cell shows Q1 (attributes?) and Q2 (false-clears?).
The same record on two real enterprise programs
Reading the real SBOM the Capa build emits (capa --manifest main.capa) on two enterprise programs that exist only in Capa, to show the scale of the per-function record on real code.
capa_paymentguard
625 provably-excluded factscapa_claimdesk
2,295 provably-excluded factsBetween 88% and 94% of functions are provably pure, and no sensitive axis is held by more than ~4% of functions (paymentguard's Fs at 4.3%, three of seventy, is the worst case). These sound facts have no counterpart in any dependency SBOM: a dependency SBOM for capa_claimdesk lists six packages and cannot tell a consumer that exactly two functions reach the network, or that 187 functions are provably side-effect-free.
Every number is committed and reproducible
Each figure on this page is anchored to a versioned ground truth and a deterministic harness in the compiler repository. The CodeQL treatment reads pre-computed facts so the harness stays deterministic.
The study, in the repository →
The full corpus, ground truth, and both halves (breadth and depth).
The breadth summary →
The Q1 / Q2 tables, the three regimes, the per-pair results: the canonical source for this page.
run_study.py →
Regenerate the breadth tables end to end from the corpus and committed CodeQL facts.
scratch_codeql/REPRODUCE.md →
How the CodeQL 2.25.6 facts were produced with the good-faith reachability query.