IoT Needs a Different Perspective on Testing
The scale of IoT is large — probably larger than most embedded manufactures have previously coped with. Moreover, individual devices are just part of a larger whole that consists of many varieties and types of devices. In my last post, I discussed how IoT’s promise is in the services it provides and the conglomeration of many "things" into something of value to the enterprise. It is important to deal with this scale in both (1) device ("things") requirements, and (2) the value it adds to the whole automation ("services") in the software development lifecycle, in order to meet cost, schedule, and time-to-market constraints. Testing, in particular, must scale to expected operational environment, plus provide support for critical non-functional requirements such as performance and security.
The Internet of Services needs automation
Automation in the software development process becomes critical as the scale of the IoT implementation rises. Security and non-functional requirements become more important as connectivity and scale increases. Service-based automated testing becomes critical during integration and system testing phases, while also enabling testing for security, stability, and performance.
Let’s take a look at Parasoft’s depth and breadth in test automation, as shown in Figure 1, as it applies to each phase of the SDLC. The key takeaway is how each solution complements the other and scales as the product grows. Unit testing is complemented by static and run-time analysis. During integration, unit testing progresses to API and service testing tools, which then progress to service virtualization.
Figure 1: Test automation solutions and where they are introduced into the software development lifecycle.
Most developers are familiar with some aspects of test automation; however, many are not thinking in terms of services or services-based testing and virtualization. As the paradigm shifts from “things” to “services,” the benefit of service-based testing becomes more apparent.
Risk and Cost Reduction with Service Based Testing in IoT
As a product progresses through the development process into integration, testing becomes more complicated and expensive. This is also the stage that many critical bugs are found, including security and performance issues not detectable at the unit level. A big risk is the availability of testing staff, labs, and hardware resources.
The solution here is automation. Automation provides a way to increase testing productivity, repeatability, and scope of system testing. An automated, services-based approach to testing is critical for the success of a newly-developed IoT device.
Benefits include the following:
- Increased Testing Productivity
Automation can assist in the generation of service-based tests, and testing loads that are repeatable and extensible. Runtime analysis tools are run alongside live tests to detect and trace errors. Defects are more easily caught and fixed than with manual testing. Once a fix is made, automation provides easy regression tests. Repeating tests as part of a continuous testing, agile, or iterative development process is supported.
- Removing hardware resource bottlenecksOne of the biggest issues in test scheduling is the availability of dependent components. These may be other devices on the network, or higher level functions in the IoT network. Simulating these dependencies removes testing bottleneck while making an entire virtual lab available on every tester and developer’s desktop.
- Scaling to current and future needsAs system integration progresses, the scope of testing increases to include more and more dependent components. Scalability is key in IoT systems, since the device under test is expected to perform in a highly-complex environment. During the device’s lifecycle environment, complexity will increase, and automated testing must scale with the product.
- Practical and realistic performance and security testingDevelopment teams do what they can to test performance and security during integration, but without automation, it’s often time-consuming and expensive to set up realistic scenarios in the lab. A services-oriented approach provides a framework for specifying and verifying performance and testing security.
Beyond Functional Test: Security and Performance
A pure functional test isn’t enough to bring a device to market. Non-functional requirements such as security and performance are critical, but are some of the hardest characteristics to test for. A device that has poor performance or poor security simply isn’t competitive, but meeting time-to-market constraints and properly testing performance and security is a serious challenge for IoT devices. Service orientation provides a common way to specify requirements (e.g. performance on a per-service bases) and for testing security (e.g. attacks exploiting exposed services and APIs).
Performance and Load Testing
Performance and security requirements are likely to be expressed in terms of a quality-of-service statement. For example, an HVAC system might be required to maintain building temperature to 75 degrees within two hours based on a 5-degree difference in outside temperature, reporting current temperature every 10 seconds. If designing a thermostat for this product, you can test the functionality at the individual level and perhaps the performance too. However, if a deployed system consists of hundreds of thermostats, performance of the device is just a small part of a complex network of other devices.
Security requirements are often system-level and vague. In the HVAC system example, a thermostat might be required not to fail under heavy network loads. A denial-of-service attack relies on flooding the target with heavy traffic, sometimes with malformed packets. Penetration and fuzz testing is used in these cases to make sure devices can withstand a hostile network environment. Scalability available in a virtual environment exceeds what’s possible with real hardware.
The next logical step for automated service-based testing is virtualization. A complete virtual environment is possible by simulating all dependent components plus a full suite of test stimuli. The benefit of service virtualization is not just realistic and repeatable test environments but the ability to duplicate and deploy at will thus creating a “virtual lab” — a lab that’s just as effective as the real thing but at a fraction of the cost.
Test automation, in general, is critical to meeting IoT product goals such as time-to-market and budget. A service-based approach increases testing productivity, and allows for highly-repeatable and scalable tests for performance and security, along with the ability to build virtual test environments. As the complexity of IoT devices and their deployment environment grows, the need for scalable, service-level testing becomes more compelling.