Test software quality

USA: Automated test software has increased rapidly over the last decade due to the need for highly customizable, flexible, and capable measurement systems. Software-centric test solutions are the only viable approach for delivering complex technologies under aggressive timelines, limited resources, and constant product churn.

The trend toward increased product complexity and capability has made a direct impact on test software and the importance of test system reliability, performance, and accuracy. Test software has prompted increased focus on ensuring quality and reliability of test software through life-cycle management and development practices formerly reserved for embedded systems.

Some organizations are voluntarily applying these development practices to improve test software and build more feature-rich and defect-free test solutions, but a growing number of industries will be required to use similar practices to comply with regulatory standards.

Products developed for regulated industries such as automotive, aerospace, and medical must comply with rigorous development standards and certifications. Revisions to these standards place increased scrutiny on quality and accuracy of test tools—creating an increased burden of proof to demonstrate testers have been qualified for use. Though businesses outside these industries will not feel an immediate impact from these trends, they can benefit from detecting defects earlier in the life cycle and drive down product development cost.

For example, ISO 26262, an automotive standard for safety critical electrical and electronic systems, was  developed in response to growing reliance on embedded control systems. It strives to standardize development and test processes across vendors to decrease defect risk, find defects earlier, and ensure safety as a critical consideration. ISO 26262 is one of several standards that recognize the importance of software-based test solutions and enumerates specific requirements for test tool qualification.

These requirements include creation of a risk mitigation plan that assesses the impact and criticality of test tools, and documentation of the steps and processes used to address high-risk areas. The overall goal is to ensure that a testing tool can be used with confidence to accurately validate embedded software without introducing defects.

ISO 26262 is derived from IEC 61508, making it one of a family of standards that spans industries including nuclear, rail, and medical. While these standards vary based on industry, they are derived from the same philosophies regarding identification and mitigation of risk through validation processes.

As an industry familiar with the need for reliability and safety, avionics has long been a bellwether for other industries. As with automotive, increased complexity of embedded avionics control systems has given rise to new standards including DO-254 and revision C of DO-178. These standards illustrate increased focus on test systems because they enumerate requirements for test tool qualification.

In particular, DO-178C has a new section titled “Software Tools Qualification Considerations” that examines tool development life cycle and documentation artifacts. The rigor of the process and the granularity of the documentation are based on the tool qualification level, which is defined by the criticality of test tools.

As the complexity of the systems we deliver increases, so does our test  equipment; thus, a tight focus on ensuring quality and safety across all phases of development is paramount to both Lockheed  Martin and our customers. As a result, many of the latest regulatory standards for the defense and aerospace industry place a greater emphasis on employing rigorous and strict software development practices to ensure reliable and accurate test results, said Sean Donner, senior software engineer, Lockheed Martin.

While terminology and implementation of these standards are different, they prescribe recommended practices to document test requirements and prove requirements have been satisfied through iterative testing, reviews, and requirements traceability. Example guidelines and processes include:
* Test requirements.
* Coding standards and stylistic guidelines.
* Software documentation and traceability.
* Software unit tests.
* Build and deploy processes.

These processes and the relationship between them is depicted using the V-model, which illustrates going from high-level requirements to a deployed solution and the corresponding test and review phases required to verify those requirements have been satisfied. The V-model is used to visualize the development process for a system, of which component testing is a part, but a similar series of steps should be followed to comply with these standards and/or ensure quality and reliability of the test solution.

Recommended best practices for developing test software follow software engineering guidelines, which include steps to enforce regular code reviews and testing cycles. Configuration management best practices are paramount for the sake of identifying and tracking changes to source code and other artifacts throughout development.

Software specifications are used to define expected system behavior under conditions in the original risk assessment, and these are used to define unit tests integrated into continuous testing cycles for regression testing. Structured environments may go so far as to mandate that multiple unique individuals review and approve any new code incorporated into a project.

One of the primary goals of structured development approaches is to manage and mitigate the risk of introducing change. It’s important to anticipate and manage the risk(s) associated with introducing changes to a system. Continuous integration addresses this by encouraging integration and testing of code, which requires automated testing tools including static and dynamic analysis of code. The goal is to find software defects as early as possible and minimize the cost of fixing them.

There is a correlation between increased system complexity and the need for greater focus on test software quality. These standards set a high bar for process and quality, but software engineering best practices benefiting businesses ensures that test systems meet increasingly demanding feature and performance requirements.

- National Instruments.