Measuring the complexity of processor bugs to improve testbench quality
Blog by Philippe Luc, April 4, 2022.
What defines the complexity of a processor bug and how to measure it?
Experience taught me that we can define the complexity of a bug by counting the number of independent events or conditions that are required to hit the bug.
What do we consider an event?
Let’s take a simple example. A typical bug is found in the caches, when a required hazard is missing. Data corruption can occur when:
- A cache line at address @A is Valid and Dirty in the cache.
- A load at address @B causes an eviction of line @A.
- Another load at address @A starts.
- The external write bus is slower than the read, so the load @A completes before the end of the eviction.
External memory returns the previous data because the most recent data from the eviction got lost, causing data corruption.
In this example, 4 events – or conditions – are required to hit the bug. These 4 events give the bug a score of 4, or in other words a complexity of 4.
Classifying processor bugs
To measure the complexity of a bug, we can come up with a classification that will be used by the entire processor verification team. In a previous blog post, we discussed 4 types of bugs and explained how we use these categories to improve the quality of our testbench and verification. Let’s go one step further and combine this method with bug complexity.
An easy bug can require between 1 and 3 events to be triggered. The first simple test fails. A corner case is going to need 4 or more events.
Going back to our example above, we have a bug with a score of 4. If one of the four conditions is not present, then the bug is not hit.
A constrained random testbench will need several features to be able to hit the example above. The sequence of addresses should be smart enough to reuse previous addresses from previous requests, delays on external buses should be sufficiently atypical to have fast Reads and slow-enough Writes.
A hidden case will need even more events. Perhaps a more subtle bug has the same conditions as our example, but it only happens when an ECC error is discovered on the cache, at the exact same time as an interrupt happens, and only when the core finishes an FPU operation that results in a divide-by-zero error. With typical random testbenches, the probability to have all these conditions together is extremely low, making it a “hidden” bug.
Making these hidden bugs more reachable in the testbench is improving the quality of verification. It consists in making hidden cases become corner cases.

Analyzing the complexity of a bug helps improve processor quality
Using bug complexity to improve the quality of a verification testbench
This classification and methodology is useful only if it is used from the moment verification starts and throughout the project development, for 2 reasons:
- Bugs must be fixed as they are discovered. Leaving a level 2 or 3 bug unfixed means that a lot of failures happen when launching large soak testing. Statistically, a similar bug (from the same squadron) that requires more events could be unnoticed.
- Bug complexity is used to improve and measure the quality of a testbench. As the level of complexity matches with the number of events required to trigger the bug, the higher the complexity score the more stressing the testbench is. Keeping track and analyzing the events that triggered a bug is very useful to understand how to tune random constraints or to create a new functional coverage point.
Finally, by combining this approach with our methodology that consists of hunting bugs flying in squadrons, we ensure high-level quality verification that helps us be confident that are going beyond verification sign-off criteria.
What's New in Codasip

06/28/22
Blog
by Roddy Urquhart
Embedded World 2022 – the RISC-V genie is out of the bottle

06/22/22
Press Release
by David Marsden
Codasip adds Veridify secure boot to RISC-V processors

06/21/22
Codasip Studio
by David Marsden
Codasip Studio Mac extends potential to design for differentiation with RISC-V

06/21/22
Press Release
by David Marsden
Codasip L31 customizable RISC-V core is an Embedded World Best in Show

06/14/22
Press Release
by David Marsden
Codasip appoints Mike Eftimakis as VP of Strategy and Ecosystem

05/23/22
Blog
by Keith Graham
Single unified toolchain empowering processor research

05/16/22
Blog
by Rupert Baines
Design for differentiation: architecture licenses in RISC‑V

05/10/22
Press Release
by David Marsden
Codasip appoints SH Lee to deliver RISC-V innovations to Korean OEMs

05/05/22
Blog
by Jamie Broome
Building the highway to automotive innovation

05/03/22
Press Release
by David Marsden
Codasip adopts Siemens’ OneSpin tools for formal verification

05/02/22
Blog
by Keith Graham
Processor architecture optimization is not a barrier for university researchers

04/29/22
Blog
by Philippe Luc
Building a Swiss cheese model approach for processor verification