ALM Best Practices

The Acute Need for Widespread Change in Safety Critical and Embedded Development

The following is an excerpt from VDC Research's new report on embedded software testing tools and practices…

The growing volume, complexity, value and potential liabilities associated with software resonate with engineers across different market sectors and are already driving change within many organizations. In many instances, this change has manifested itself primarily through a growing recognition of the importance of more sophisticated automated tooling that offers potential engineering labor savings or improved software quality assurance, such as those targeted at software testing. However, while the piecemeal use of these products may offer incremental opportunities for improvement in development efficiency or quality, they cannot guarantee project success.

Going forward, we anticipate that it will become increasingly critical for organizations to embrace the corresponding need for change in the methods and processes surrounding these tools. Not only can the implementation of standardized development processes help improve quality by aligning development teams’ actions with established best practices,
but it can also aid engineering resource planning by improving the predictability of project schedule adherence. With no significant deceleration of software content growth in sight, we believe that the evaluation and adoption of automated test and verification tools will not only emerge as a critical means for quality assurance but also as an increasingly important agent to remain competitive in a rapidly changing marketplace.

Want to read more? Download the complete report—free of charge.

****

You can learn more about embedded testing at Parasoft's website, embedded testing resource center and safety-critical testing resource center.

Safety Critical Development – Process Standards Compliance

The following is an excerpt from VDC Research's new report on embedded software testing tools and practices…

Whereas the use of automated test tools can help engineers identify potential errors in their code bases, more organizations (and industries) are now taking additional steps earlier and across their entire development cycle to limit the introduction of software defects. For example, many of the industries in which a system’s failure could lead to injury or loss of life have established development process guidelines under the assumption that this guidance can foster end software quality and thus correct product functionality. Not surprisingly, engineers within those safety-critical industries reported the highest current rate of alignment to process standards.

 

However, many of the industries that have widely adopted process standards (such as the military/aerospace and automotive industries) are poised for change. Although many of these standards (such as DO-178B) are widely regarded as best practices, they have become somewhat antiquated in their ability to adapt to some of the needs and preferences of today’s engineers. However, some of the newer process standards (such as DO-178C for avionics and ISO 26262 for automotive) are being designed with a focus on modularity, formal verification methods, and model-based design – all of which can place a greater emphasis on the need for new tooling.

Although safety-critical markets have an established lineage and need for such development standards, the interest in their utility as best practices to control quality and schedule performance is now extending within consumer facing verticals as well. While device failure in these sectors does not present the same risk of tort liabilities that would be associated with loss of life, the potential financial impact of a product recall or missed market window can be just as debilitating. In fact, engineers from the mobile phone industry – who need to navigate especially tight project development cycles and high rates of software content growth – actually reported the highest expected rate of growth in process standard utilization. As a result, we anticipate that this acknowledgement of the potential value of more formalized and rigorous engineering processes will subsequently drive an increasing number of these organizations to reevaluate their incumbent tool suites.

Although most of the existing process standards were designed with a primary objective of improving the reliability of the end product being developed, our research has also shown that projects aligning to a particular process standard are less likely to be behind schedule than those that are not aligning to a standard. While there is a litany of variables that can influence project schedule (resources, tools, experience, etc.)  – and schedule adherence does not necessarily mean faster time to market or a reduction in complexity – these findings do suggest that process standards make schedules at least more predictable if not even more easily managed over their duration.

It should also be noted that engineers whose projects align to process standards also often report a higher frequency of use of automated development and test tools. Not only do many of these standard-adhering projects have safety-critical requirement concerns that can drive the use of more sophisticated tools, but automated tooling can also facilitate the management of, documentation for, and adherence to the standards themselves.

 

Want to read more? Download the complete report-Free of Charge!

Automated Defect Prevention for Embedded Software Quality

The following is an excerpt from VDC Research's new report on embedded software testing tools and practices…

 

A Market Conditioned to Failure

The world of embedded devices and systems has evolved rapidly over recent years as the functionality required for these systems grows more sophisticated and complex. In conjunction with this change, the software on these systems is rapidly emerging as a primary agent for device and system differentiation and resource investment for many organizations. Engineering organizations are now forced to balance expanding feature set requirements and this increasing demand for software content creation within often shrinking time-to-market windows. With 40% of the engineers recently responding to VDC’s Software & System Development Survey reporting projects that are already behind schedule, these business and engineering challenges place efficient management of software content creation and validation as an increasingly critical component of corporate profitability and a new potential point of organizational differentiation.

These market trends and pressures impact both consumer-facing device classes and safety-critical industries such as avionics, automotive or medical devices. In fact, these issues are often compounded in safety-critical markets due to the existing pressures and costs associated with system failures. As a result, many of these embedded system manufacturers have begun to look for ways to improve, if not overhaul, their existing development processes and tools in the interest of producing higher quality products with predictable and safe functionality.

As part of their efforts to achieve this process optimization, many organizations are adopting test and verification tools that automate an increasingly broad scope of complementary defect detection and prevention practices. Consistent application of such tools in the context of an overarching standardized process appears to be a promising path for not only improving software quality, but also delivering this software faster and more predictably.

 

Code Growth Testing the Limits of Traditional Methods

As previously discussed, functionality requirements on embedded devices has been fueling a significant amount of code growth within the embedded market, with respondents to our Software and System Development Survey expecting on average a 17% increase in lines of code for their next project. As one would expect, this rapid growth combined with the perpetual time to market pressures is placing an increasing amount of strain on engineering organizations’ ability to maintain software code quality and functionality.

The use of automated test tools to audit software quality and functionality is not a new or novel concept in the embedded market. However, the aforementioned growth in software content, complexity and importance is now driving a wider range of organizations to adopt the technology as the efficacy and feasibility of traditional manual testing techniques has continued to diminish. Whereas there are other metrics beyond growth in lines of code that can provide additional valuable assessments on increases in software complexity, the demonstrated correlation between the use of automated test tools and code base sizes clearly indicates that many engineering organizations already recognize the utility and value of automated test tools in assessing the quality of large code bases.

 

 

 

Want to read more? Download the complete report—free of charge.

Service Virtualization Webinar – Test More, More Effectively

Why testers run only 50% of their tests—and how service virtualization can help

 
Recent studies show that only 50% of test plans are actually executed. Access to the application under test's dependent applications (mainframes, 3rd-party services, etc.) is typically granted only in a tightly-scheduled timeframe—and so much time is required to configure the test environment that little time is left for actual testing.

Service virtualization provides testers unconstrained access to dependent applications via flexible test environments that are easily configured for distinct testing requirements. Service virtualization steps in where server/hardware virtualization leaves off, providing easy access to the behavior of dependent apps that aren't practical or cost-effective to fully "virtualize" for testing purposes.

Discover how service virtualization can empower your organization's QA teams to test more efficiently and effectively. Join voke principal analyst Theresa Lanowitz and Wayne Ariola, VP of Strategy at Parasoft, for a 45-minute webinar that:

• Discusses the challenges that impede thorough and accurate testing of today's composite apps.

• Explains the real-world impacts of such testing delays and limitations.

• Introduces new technologies that enable testers to access the exact test environments they need–any time and anywhere.

You'll learn how service virtualization not only provides 24/7 access to the environments needed to test, but also significantly reduces the CAPEX and OPEX associated with establishing, configuring, and maintaining test environments.

Reserve your spot today – spaces are limited

***

Missed our previous service virtualization webinar that introduced Parasoft Virtualize? View the recording now

You can also get more information on Service Virtualization at our Service Virtualization Resource Center.

Service Virtualization for Performance Testing Video

Service virtualization helps you test more effectively by emulating realistic behavior & performance of the dependent applications you need to test against—and letting you easily adjust that behavior & performance to meet different testing needs.

Watch our 2–minute video—Service Virtualization for Performance Testing—to learn how Parasoft's service virtualization can help you:

• Test vs. realistic transaction performance
• Test constrained dependent resources without scheduling restrictions
• Test extensively—without access and transaction fees
• Test vs. a broad array of functional & performance conditions—with minimal setup
• Get the exact test environment you need—on demand

 

***

You can learn more about service virtualization, service virtualization for performance testing and test environment management on Parasoft's website.

Video Highlights from the Design West Embedded Systems Conference

 

Quite a few people stopped by the Parasoft booth at the Embedded Systems Conference in San Jose. Engineers, QA testers and developers from safety-critical (medical device, automotive, mil/aero, etc.) and consumer electronics industries were looking for solutions to help them meet compliance requirements and overcome the distinct challenges associated with embedded software development and testing.

Embedded Testing: Automated Defect Prevention, Static Analysis

In this video, Parasoft Senior Sales Engineer Rich Newman shows one interested attendee how Parasoft C/C++test's static analysis automatically exposes potential coding defects so they can be fixed before they impact software safety and reliability.

 

 

For more extensive demos of Parasoft's embedded testing technologies, see our Embedded Testing Resource Center.

A Quick Look Around the Floor

If you've seen one show, you've seen them all. But that doesn't stop one from wondering who was there if you didn't get a chance to make it. Here is a quick look at a few of the neighbors cozied-up to the Parasoft booth.

 

 

Intel Showcases Atom Processor's Ability

The system in this video is built on Intel's Atom processor. The system is an assemblage of music-making machines surrounded by hoppers filled with golf ball-sized balls. The hoppers launch the balls at the machines, which plays a beat or note on collision. The mesmerizing demonstration was a nice way to appreciate how an embedded system can entertain, as well as provide safety-critical services.

 

***

You can learn more about embedded testing, safety-critical development and static analysis at Parasoft's web site.

Gain Access to Constrained Components - Service Virtualization Video

Service virtualization helps you break free from constraints when you need to test against dependent applications controlled by 3rd parties, other departments or partners.

Watch our 2- minute video—Service Virtualization – Gain Access to Constrained Components—to learn how Parasoft's service virtualization can help you:

• Test against constrained dependent resources without scheduling hassles
• Test extensively—without access and transaction fees
• Test vs. a broad array of conditions—with minimal setup
• Get the exact test environment you need—on demand

 

If you’d like an overview of how Parasoft's service virtualization provides easy access to the constrained components that impede development and testing, read our Service Virtualization eBook or visit the Parasoft Virtualize page.

Developers & Testers: Complete Our Test Environment Access Survey for a Chance to Win

 

At Parasoft, we truly value feedback from the marketplace. To help us better understand—and alleviate—your pains related to test environment access, would you please take 2-3 minutes to share your experiences via a simple 9-question survey?

This brief survey focuses on “Dev and QA Test Environment Access”. It covers questions like:

• How many dependent apps are associated with your AUT?
• What level of access you have to those dependent apps?
• What percentage of your dependent apps do you exercise during testing?
• How difficult is it to access test environments at convenient times?
• Do you need to schedule access to test environments?

To thank you for taking the time to complete this survey, we'll enter you in a drawing for one of three $100 Amazon gift cards and one of ten $10 Starbucks gift cards. Just provide your name and email address at the end of the survey if you would like to participate.

Complete the survey now

New Service Virtualization Videos

Service Virtualization Overview

With today's complex interdependent systems, multiple development and test teams are commonly vying for limited system access.  Performance testing, parallel / agile development, and functional testing efforts are complicated when the application under test or development interacts with dependent applications that are:

• Not yet completed
• Still evolving
• Controlled by a third-party or partner
• Available for testing only in limited capacity or at inconvenient times
• Difficult to provision or configure in a test environment
• Needed for simultaneous access by different teams with varied test data setup and other requirements
• Restricted or costly to use for load and performance testing

Watch our 2- minute Service Virtualization video to learn how Parasoft's service virtualization can help you:

• Gain easy access to constrained components that impede development & testing
• Cut scheduling constraints with ubiquitous access to an accurate test environment
• Eliminate process bottlenecks with rapid access to evolving or unavailable dependent apps

Service Virtualization & Agile Development

Deadlocks are endemic to parallel and Agile development environments, where different teams are simultaneously working on interconnected system components—and each team needs to access the others' components in order to complete its own tasks. But when a team ends up waiting for access to dependent functionality, agility is stifled.

Watch our 2–minute Service Virtualization & Agile Development video to learn how service virtualization can help your team break free from these constraints.

• Rapidly model virtual assets that reflect the anticipated behavior of still-evolving components
• Capture existing behavior from live systems, then adjust to represent expected future behavior
• Easily update virtual asset behavior as requirements change
• Rapidly deploy virtual assets for consistent team-wide access
• Eliminate the need for brittle stubs/mocks

Any Service Virtualization Video Requests?

What types of topics would you like us to cover in additional service virtualization videos?

 

***

Learn more about service virtualization visit our website.

Static Analysis and Dynamic Analysis

By Boguslaw Czwartkowski, Parasoft Professional Services Manager

Static analysis is a broad term used to describe several different types of analyses:

• Static code analysis (a.k.a. pattern-based static analysis): This type of static analysis looks for code patterns that violate defined coding rules. In addition to ensuring that code meets uniform expectations for regulatory compliance or internal initiatives, it also helps teams prevent defects such as resource leaks, performance and security issues, logical errors, and API misuse.
• Flow-based static analysis: This type of static analysis involves finding and analyzing the various paths that can be taken through the code, both by “control” (i.e. the order in which lines can be executed) and by data (i.e., the sequences in which a variable or similar entity can be created, changed, used and destroyed).  This can expose problems that lead to critical defects such as memory corruptions (buffer overwrites), memory access violations, null pointer dereferences, race conditions or deadlocks. It can also detect security issues by pointing out paths that bypass security-critical code (for example, code that performs authentication or encryption).
• Metrics analysis: This involves measuring and visualizing various aspects of the code. It can help detect existing defects, but more often it warns of potential difficulty in preventing and detecting future defects when code is maintained. This is done by finding complexity and unwieldiness such as overly large components, excessive nesting of loops, too-lengthy series of decisions and convoluted intercomponent dependencies.

Why Static Analysis is so Valuable

The thing all these types of static analysis have in common is that they involve scanning (i.e., having a program examine) source code. This is a very fast and easy way to expose critical defects. It achieves 100% coverage and its results are 100% objective. It's almost impossible to argue against doing it continually.

Beyond the Scope of Static Analysis

Static analysis provides information to help predict what may happen when code is integrated and executed. It detects defects based on what the tool considers a defect (typically, this can be customized to suit your preferences and priorities). However, it cannot tell you when the system under test or in production does NOT do what it is expected to do—or does something that nobody expected it would do.

The challenge here is observing the unexpected behavior. For example a transaction may appear to proceed correctly to a user (or tester or test execution tool) whereas a component has actually thrown an unhandled exception and failed to process it correctly. A control system may respond quickly and correctly under test for 3 days, yet be leaking memory and heading for a crash on its 4th day in production. Fixing all defects detected by static analysis gives no assurance against other defects that will cause failures like these.

That's why it's important to apply the definition of failure to internal as well as external behavior, even after integration. The internal failure must be detected before it manifests itself externally.

Dynamic Analysis - Runtime Error Detection

At Parasoft, we refer to this activity as runtime error detection, a form of dynamic analysis. Here, distinctions among testing types start to blur. Dynamic analysis is analytical testing in that the intention is to examine the test item rather than exercise it. It is white box testing in that we examine internal rather than external behavior. Yet the code under test must be executed, and that is done by running the same black box tests used for dynamic testing.

Runtime error detection detects and reports an internal failure at the instant it occurs, so it is easy for the tester to precisely correlate it with test actions for incident reporting. Like good static analysis, it provides full technical details to enable the developer to isolate and fix the underlying defect. Runtime error detection extends the capability of empirical testing at all levels, from unit to acceptance, by making it possible to detect an internal failure that indicates an otherwise-unobservable external failure has occurred or will occur after testing has stopped.

 

***

Photo Credit: Dexxus

You can learn more about runtime error detection and static analysis at Parasoft's web site.