Why Low-Severity Findings Say More About Your Audit Than Critical Bugs Many audit firms focus their sales pitch on number of Highs found as if this number isn't just noise without plugging in context: prior audits, peer review, test coverage levels, code complexity, line count and many other metrics. It's the lowest form of salesmanship, not different from comparing for example, USB drive quality by their length in millimeters. To show an alternative we first must assert the correctness of several supporting claims: - The probability of accidental bug injection has no bias towards higher impacts (developers are not more reckless in high-stakes code, usually the opposite). - The same comprehensive methodologies used to discover flaws of various severities would also discover high severity issues (opposite does not hold). - There are much higher requirements for a random bug to qualify as high severity (often it would be gated behind unreachable conditions, or touch non-critical functionality). - From basic statistics: higher sampling rate correlates with lower expected deviation/variance and thus a more accurate measurement. Let's define an audit report as the result of sampling a codebase's quality. We deduce that the expected true (no misses) number of Highs is much lower than Lows, and the expected deviation around it is much higher (due to smaller sample). In other words, the number of Highs tells us very little about the number of missed highs. So surprisingly, a 1 High, 10 Lows report is more reassuring than a 10 Highs, 1 Low report all else being equal. Although in fact the vast majority of salesmen would prefer to show the latter as an indication of quality. The point is that a high-frequency metric is a better tool to measure low-frequency outcomes. Web3 builders, next time firms wave you their Crit/High counts and X billions of $ secured line, you know where to focus to search for true signal. Web3 auditors, recognize there is no consistent secret formula for finding all the Highs without also searching for the Lows - every Low not fully investigated is a potential High - and give your best attention to every single line. Your client will thank you for it. Low severity is defined as concrete coding mistakes that don't result in higher level impacts. Doesn't include formatting, best practices and filler findings.
4,31K