Are Your Continuous Tests Too Fragile for Agile?

Posted on Jul 16, 2015

fragile_agileWith a fragile test suite, the Continuous Testing that's vital to Agile development just isn't feasible. If you truly want to automate the execution of a broad test suite—embracing unit, component, integration, functional, performance, and security testing—during continuous integration, you need to ensure that your test suite is up to the task. How do you achieve this?

Some best practices are to ensure that tests are: 

  • Logically-componentized: Tests need to be logically-componentized so you can assess the impact at change time. When tests fail and they're logically correlated to components, it is much easier to establish priority and associate tasks to the correct resource.

  • Incremental: Tests can be built upon each other, without impacting the integrity of the original or new test case.

  • Repeatable: Tests can be executed over and over again with each incremental build, integration, or release process.

  • Deterministic and meaningful: Tests must be clean and deterministic. Pass and fail have unambiguous meanings. Each test should do exactly what you want it to do—no more and no less. Tests should fail only when an actual problem you care about has been detected. Moreover, the failure should be obvious and clearly communicate what went wrong.

  • Maintainable within a process: A test that's out of sync with the code will either generate incorrect failures (false positives) or overlook real problems (false negatives). An automated process for evolving test artifacts is just as important as the construction of new tests.

  • Embedded within a prescriptive workflow based on results: When a test does fail, it should trigger a process-driven workflow that lets team members know what's expected and how to proceed. This typically includes a prioritized task list.

For more tips, see the Code Curmudeon's slide deck for his Are Your Continuous Tests Too Fragile for Agile? session from the recent Better Software Conference.