Why Regression Testing is a Development Testing Lifesaver
By Rich Newman, Principal Software Engineer at Parasoft
Lately, I've been working to implement more and better support in Insure++ (Parasoft's memory error detection tool) for the new C++11 standard -- not easy to do since every compiler out there has its own interpretation of the standard. The process actually started some time back when C++11 was called C++0x and it wasn't clear when it would be finalized and published. That was when Dustin, a member of my team, created our C++11 test suite using sample code and individual proposals from the C++ Working Group. At that time we only needed to support GCC 4.4, so Dustin locked down the test suite to execute only against GCC 4.4 and only run the C++11 tests that correspond to the things GCC 4.4 supports.
In addition to Dustin's test suite, our nightly process includes hundreds of thousands of test cases (accounting for variations in compiler and operating system), each of which represents a customer-reported issue, a Parasoft-discovered issue, or -- more commonly -- a feature that may or may not have been implemented yet.
After spending several days integrating the new parser, analyzer, and instrumentation engine, the code was compiling, and it was time to run tests. Thankfully, Dustin's hard work was about to pay off in the extreme. I was easily able to unlock the C++11 test suite so that it ran all C++11 feature tests against all GCC compilers 4.4 and greater (none of the GCC compilers prior to 4.4 have support for C++11). To my great disappointment, most of the test cases failed. Even the ones which had previously been failing were now failing differently. When I examined the test case logs, however, I was quite amused.
C++11 added, among other things, a new keyword: "constexpr". Any C++function which is declared "constexpr" must be fully resolvable at compile time. It must consist of only a return statement, and that return statement can only contain other "constexpr" entities. Instrumenting such a function is neither necessary, nor desirable. Nearly all of the C++11 test cases were failing because Insure++ hadn't yet been configured to handle "constexpr". It just saw a function and instrumented it normally. The resulting code is not compilable.
After looking closely at the changes introduced into the parse tree, I found the relevant flag which determines whether or not a particular function has been declared as "constexpr". I then configured Insure++ to handle such functions appropriately. After that, most of the C++11 test cases ran correctly, even those which were previously failing.
That suite of regression tests was both a lifesaver and a timesaver. I will never understand the development teams who claim they have no time to write and automated tests, but I will remain grateful that I work for a company that truly understands their benefit.
Want to read more about Parasoft's experiences with regression testing? Read our blog on Unit Testing: How We Eased the Burden of Our Regression Test Process.
Parasoft’s industry-leading automated software testing tools support the entire software development process, from when the developer writes the first line of code all the way through unit and functional testing, to performance and security testing, leveraging simulated test environments along the way.