Non-Functional Testing: An Imperative for Shifting Left
In this guest blog, Frank Puranik from iTrinegy discusses how to test in the real-world networks that your applications will be deployed in, for realistic and early-stage non-functional testing.
If you’re anything like so many software testing departments I’ve spoken to, you're rolling your eyes already. “We haven’t even got our functional test fully sorted out," I can hear you saying. "How can we possibly start on non-functional testing??”
The answer is, “You have to."
If you don’t, it will cost you big in remedial costs, lost revenue, reputational damage, and star ratings.
Stay with me for a moment and I’ll explain why…
What happens when we ignore testing in the real-world network that the application is deployed in
You can do all the functional testing you like to ensure that every button works and every calculation is correct, but still, when your customer uses your application in a real-world network and they get something like:
...they’re going to think you didn’t test it at all. (That real example is from a released app by a multimillion company.)
It’s worse than that, though.
They might even think their account has been hacked (note the unauthorized 401 message), when actually what happened was that the network gave up at the wrong point in the transaction.
So here’s a “non-functional” problem that has created a critical application issue:
- The app has issued a very user unfriendly error message (401 - really?!)
- The app has misled the user about the nature of the problem (it looks like a security issue)
- The app provides no information about what to do now
How, then, can we justify completing all functional testing when there are gaping holes in the behavior of the application for non-functional (environmental, in our example) reasons?
The importance of risk analysis
Good practice says that during application design, development, and project management of the complete development lifecycle, we need to look at the risks of issues with our application and spend our time (and dollars) accordingly, not simply blindly focusing efforts on completing “all” the functional testing.
Our example above shows a relatively high risk event, given the propensity of mobile networks to deliver extremely variable connections, dependent on location, other users, weather conditions, and time of day. If you’re not developing for the mobile environment, you might say that won’t affect us, but there are other issues too, such as:
- The application may run unacceptably slowly in the real-world network
- The application may display interminable spinning circles of waiting, hour glasses, or just not respond in the real-world environment
- The application may exhibit timeouts and resultant poor error messages in the real-world environment
Neither of these issues, despite initial appearances, are performance related. We’re not talking about a little, or even a lot, slower than expected, we’re talking about issues that make the user give up completely.
These are, therefore, all risks that the project management of the application cannot ignore or rule as out-of-scope.
How do these risks get brought in?
It’s very simple really. Here are some examples of dev/test approaches that we commonly see:
- The designers may use network communication approaches that cannot work in real world networks at all
- Prototypes are developed and demonstrated in perfect LAN networks
- Developers will build and unit test their code in perfect LAN networks
- Most separate testing is functional and this will also use perfect LAN networks
So the first time most applications see real-world networks, with all their foibles, is:
- For internal applications: pre-production
- For external applications: beta testers or customers
Either way, by now, you’ve spent a lot of time and money developing your product down a particular direction that there may be no easy or cheap way out of. Its fundamental communication design may be incompatible with the network environment it must operate in.
In general terms then, our non-functional problem is an environmental one.
It’s akin to a car manufacturer building a car for off-road use, while, at all initial stages (prototyping etc), using only smooth paved roads, and somehow being surprised that it didn’t work well off-road. It wouldn’t happen -- in this process everybody understands the “off road risks” and factors them into the design accordingly.
The Solution: Shift Left
To make the case for shifting left, the first thing we need to understand is that there is a real and very tangible risk here. You can’t ignore the risk unless your application will never use the network at all, or only be used in a local (LAN) network.
We need to shift “Testing in the Network” to the left, starting with real-world network testing of the prototypes (so that we don’t go too far down a design blind alley) and continuing this throughout all development, unit testing, etc.
Some organizations (by which I mean the whole company when applications are being developed for external users, or the development department, if applications are being created for internal use) do recognize that, but still take the risk, leaving network testing very (or too) late because it is so hard to get a real-world network into the development and test environment.
There are answers to that. While it is true that doing development and testing in a real network is difficult, iTrinegy produces Virtual Test Network products that, delivered as hardware or virtual appliances, let you create any style and quality of network you need. And when coupled with service virtualization, to emulate the behavior of dependencies that are constrained or out of your control, development teams finally have FULL control over the complete test environment.
This makes it easy to have your test environment available at all application development stages and effectively makes this test network a part of every non-functional test you do. In this way, you can fully shift “Testing in the Network” to the left.
Your rewards include:
- Eliminating remedial costs due to poor customer networks
- Increasing customer satisfaction, including visible metrics like Star Ratings and customer recommendations
- Increasing your customers' revenues and therefore yours (as your end users are more productive)
Frank Puranik is a Senior Technical Specialist at iTrinegy, a world leader in Application Risk Management and is trusted by governments, military organizations and enterprises across the globe. Frank regularly provides expertise to companies across all industry sectors on how real-world network conditions can kill application performance. Customers are able to draw from Frank’s 20 years+ experience and enhance their Functional and Performance Test capabilities by introducing real-world network tests into their processes, thereby validating application performance prior to actual rollout or migration.