Unit Testing: Is More Coverage Better?
Everyone thinks that more coverage is always better, but that's not always the case.
People want high code coverage, but to achieve something like 80% code coverage or higher, you often end up executing code mindlessly. It’s like asking a pianist to cover 100% of the piano keys rather than hit just the keys that make sense in the context of a given piece of music. When he plays the piece, he gets whatever amount of key coverage makes sense.
Another problem is that when code coverage exceeds a certain level (above about 70%), the test suite becomes increasingly difficult to maintain. You typically have very fine-granularity assertions, and these assertions are very sensitive to code changes. As a result, many assertion failures are reported and need to be dealt with each day—but these assertion failures typically don’t alert you to any significant changes.
Many people ask me if they can get 100% code coverage. The answer is yes… but don't worry about it unless you are actually required to demonstrate 100% code coverage (for instance, for regulatory compaliance).
Going back to the previous analogy, a piano concerto is complete when all the notes have been covered. A test suite is complete when every use case has been covered.
Each use case should have at least one test case. This might be a unit test, another test, or a combination of test types. When I have all of my use cases captured as test cases, then I can run my regression test suite nightly and it will tell me if the day’s changes broke or negatively impacted the functionality related to my use cases. This gives you a safety net that’s essential when you’re constantly evolving the application.
That being said, measuring code coverage does actually tell you some interesting things.
Say you have one test case for every use case, and this represents only 40% code coverage. This might mean that 60% of your code is unrelated to your use cases... which is a bit scary. Do you have a lot of useless code? Is the application being written without a clear understanding of what it’s supposed to do? Or do you have major omissions in your use case definitions? Maybe you’re missing some use cases for your requirements. Or maybe your requirements themselves are incomplete. This is definitely worth exploring.
On the other hand, what if your coverage is high even though you don’t have many test cases? This might be a sign that you have well-written tight code with a lot of reuse.