Software Testing

Verification and Validation (V\&V) of software is critical to producing high-quality robust systems. As the performance and capabilities of embedded microprocessors and microcontrollers improves, they are finding applications in a wider range of systems from microwave ovens and VCRs to medical equipment (computer controlled X-ray machines), anti-lock brake systems for automobiles, and aircraft flight control systems (virtually all military aircraft, the Airbus 320 and the Boeing 777). The quality and reliability of the software that controls these systems can have very real consequences. For consumer products, the popularity of the product can suffer and repair costs can be high. For safety critical applications, failure of the software control can result in injury and even loss of life (there have been several documented cases of computer controlled X-ray equipment giving patients dangerously high doses of radiation because of faults in the controlling software and software errors are suspect in an Airbus A320 crash.) These increasingly demanding applications require that more sophisticated evaluation techniques be applied during software production.

Verification of software and firmware is extremely expensive and time consuming. Poor software verification techniques can delay the release of products and result in products that, once released, suffer from reliability and performance problems. Finding problems early is one of the benefits of a systematic verification process.

Systematic application of V\&V techniques throughout the design process are critical. Specifications must be checked for consistency and completeness, designs must be verified against specifications, implementations must be verified agains the designs and finally the implementation must be validated agains the specification. For software systems the implementations are code in a given language and verification and validation requires software testing. While all phases of verification are time consuming, software testing can be extremely time consuming and tedious for the test engineer. However, because software has a formal representation dictated by the syntax of the programming language used, many aspects of software verification can be automated with the use of software tools.

Testing can be classified into two broad categories: structural testing and functional testing. {\em Structural testing} takes into account the structure of the software in developing tests, while {\em functional testing} considers only the expected functionality of the software (the specification) and develops test cases accordingly. There are many distinctions between structural and functional testing. For example, functional testing is typically done by a testing team while structural testing is done by the software designers. Also, functional tests are created from the software specification while structural test are created directly form the code. As a result, functional tests are from the user's view of the software while structural tests are the designers view of the code. Therefore, functional testing is based on an independent view of the software while structural testing can suffer from the same misconceptions of the specification which resulted in the faults that are the target of the testing.

Functional testing is never ending because a piece of software has an infinite number of possible inputs. Structural testing, on the other hand, is based on the source code and the and paths through the software which (may be quite large but) are finite in number. Structural testing can also be extremely tedious and, therefore, prone to more ``human'' errors. In addition, functional tests will not change with modifications to the the software, for example if a section of code is re-written to improve performance. The structural tests, however would need to be completely rewritten. Hypothetically, if infinite testing were possible functional testing can find all bugs, but unit testing can not find all errors since errors in the control flow may imply paths are not covered or that the code does not meet the specification. Also, much of functional testing can be redundant since inputs to programs naturally fall into equivalence classes which are handled by given paths through the code. Therefore, a number of seemingly different inputs can result in testing of the same paths through the code. Structural testing, on the other hand, can eliminate useless test cases through explicit observation of the paths present. Finally, structural testing and functional testing can not be view in strict isolation. Most notably, layers of software blur the distinction between the testing techniques, functionality to one layer of the software is structure to higher layers. It is the goal of this research to use structural testing metrics as a measure of functional tests.

Go Back