The SAMATE project recognizes the value and importance of sound static code analyzers. During the 6th Static Analysis Tool Exposition (SATE VI), the NIST team evaluated static analyzers with respect to the SATE VI Ockham Sound Analysis Criteria.

In brief, these criteria are:
• The tool is claimed to be sound.
• For at least one weakness class and one test case, the tool produces findings for a minimum of 75% of appropriate sites.
• Even just one incorrect finding disqualifies a tool for SATE VI.

The definition of a finding includes reporting a buggy site as buggy, but also passing a correct code location without alarm. In other words, in order to satisfy the SATE VI Ockham Sound Analysis Criteria, all defects must be found, and the rate of false alarms must be low.

Astrée satisfies these criteria with excellent results: The tool was run on 28 sets of test cases from the Juliet 1.3 C test suite, including test cases for buffer overflows/underflows, invalid pointer dereferences, integer overflows/underflows, divisions by zero, use of uninitialized variables, dead code, infinite loops, double free, and use after free, running on a total of 18,954 buggy sites. All 18,954 were reported by Astrée. Additionally, Astrée discovered thousands of unintended defects in the Juliet 1.3 benchmark set.

The SAMATE report emphasizes Astrée’s outstanding precision, both with respect to the analysis results as well as the analysis model itself: “Alarms from Astrée led us to find and fix thousands of mistakes in what was intended as the Juliet known-bug list, manifest.xml. Because Astrée analyzes code very precisely and we checked meticulously, details of modeling that otherwise would be inconsequential showed up and had to be resolved.”

The full report was published in May 2020 and is available for free as PDF from
nvlpubs.nist.gov/nistpubs/ir/2020/NIST.IR.8304.pdf