What are the costs of false positives and false negatives?

False positives and false negatives are as dangerous in static analysis tools as they are in Covid-19 testing. False positives say you have a problem when you don’t. False negatives tell you everything is fine when it isn’t.

In medical testing and screening, it’s very clear why scientists do everything they can to limit the number of false positives and false negatives. A false positive Covid-19 test result, for example, would disrupt your life with quarantining, unwarranted anxiety and unnecessary treatments. A false negative is even worse, for both you and the friends and loved ones you meet.

It’s a similar, although less transparent and clear cut, situation with static analysis tools. Here, the danger with false negatives is that program defects go undetected. As a result, products get released with potentially disastrous results in the field.

Defective program components can also easily corrupt other program components (which may otherwise be perfect) in the same device. This happened in the Toyota sudden unintended acceleration case. Either way, havoc ensues.

Too many positives is not a good thing

The problem with false positives is that identifying them can involve a massive waste of resources. Engineers have to investigate all the positives to identify those that are true. If there are lots of positives and many of them are false, engineers might have no choice other than to only check a sample: as a result, an unknown number of true positives might be missed.

Another risk is that the engineer, not knowing the positive is false, makes unnecessary changes to the software to ‘fix’ the non-existent problem. This can introduce code defects or reduce code quality and cause problems further down the line. Again, they may release defective products with potentially damaging results.

The solution is a more accurate static analysis tool

Your tool should not only produce as few false negatives and false positives as possible, but also ‘confine’ or distinguish between definitely true positives (reported as violation messages) and possibly false positives (reported as caution messages). The tool also needs to document under what specific conditions the false negatives are possible.

Achieving this balance involves taking great care with the static analysis algorithms. Most importantly, it must ensure that any information loss occurs as late as possible in the process. We can illustrate this ‘information loss’ by going back to our Covid-19 test analogy.

There are many potential reasons why a Covid-19 test could produce a false positive or false negative result. The clinician may not observe all the proper hygiene rules, such as hand washing, for example. Maybe they make mistakes later in the process such as when they take the sample or with careless labeling procedures.

The garbage-in, garbage-out principle

Again, the situation is similar with static analysis tools. Low quality tools adopt imprecise parsing or fail to take into account the implementation-defined aspects of the C/C++ language. That means information is lost at the very beginning of the process that may result in false positives and false negatives. You can’t improve on this at later stages in the algorithm.

Washing your hands is the critical step

An inaccurate interpretation of the program text is the equivalent of a clinician not washing their hands or not wearing gloves before starting your Covid-19 test. The information loss has been made during the early stages and can’t be put right by more careful actions later. However, if the information remains until a more advanced stage, the analysis tool can be more precise as it has already ruled out many possible causes.

We discuss this issue of the implementation-defined aspects of the C/C++ language in more detail in this paper and in our webinar “MISRA C++: A Subset of C++ for the Development of High-Integrity Systems.” You can watch the recording here.

If you'd like to see our previous webinars, or share them with your colleagues, head over to our YouTube channel. The full schedule of webinars is available on our website. And of course, if you have any questions, please do get in touch.

Roberto Bagnara, PhD is CTO of BUGSENG, a leading provider of solutions and services for static code analysis. He is also a member of the ISO/IEC JTC1/SC22/WG14 - C Standardization Working Group and the MISRA C Working Group.