Automated test generation from algebraic specifications

Abstract

This thesis is a contribution to work on the specification-based testing of computing systems. The development of computing systems is a challenging task. A great deal of research has been directed at support for analysis, design and implementation aspects, yielding a wide range of development techniques. However, the crucial area of system testing remains relatively under-explored. Because a project may spend a good part of its budget on testing, even modest improvements to the cost-effectiveness of testing represent substantial improvements in project budgets. Relatively little literature has been devoted to the entire testing process, including specification, generation, execution and validation. Most of the academic literature seems to assume a revolutionary change of the testing framework. On the contrary industry follows a more traditional approach consisting of trusted methods and based on personal experience. There is a need for testing methods that improve the effectiveness of testing but do so at reasonable cost and which do not require a revolutionary change in the development technology. The novel goal of the work described in this thesis is to "lift" traditional testing so that it takes advantage of system specifications. We provide a framework - hepTEsT- which is motivated by this goal. To that end, hepTEsT is a framework consisting of a specification language, a technology for generating tests in accordance with test strategies, a means of applying the tests to the implementations and support for validation of outcomes against the specification-based tests. We will first categorise different testing methodologies and then examine some of the past and present approaches to test data: we develop only the necessary theoretical foundations for hepSPEc and always consider the requirements of testing. The formalism hepSPEc for system description is based upon a well-defined algebraic approach. It utilises a novel approach allowing the description of finite domains in a way suitable for engineering purposes. The engineers' tasks are to provide an adequate description of the system in hepSPEC. The approach proposed in this thesis is grounded in the traditional approach to testing where test data is provided to the system under test and the outcome is compared to the expected outcome. To enhance the capabilities of the framework a general order on test inputs is proposed to be used in test strategies. Traditional testing strategies requiring an order on test inputs are introduced and their realisation in hepTEsT discussed as well as a proposal of new strategies which lend themselves to this particular approach. The manipulation of the specification yields abstract test cases which are then transformed into test cases suitable for the chosen implementation of the system. This transformation, called test reification, is necessary to bridge the "abstraction gap" between the abstract specification-derived tests and the concrete implementation on which the test must run. The transformation is necessary in order for the approach to be practical and is achieved through homomorphisms which are expressed in specially adapted grammars. This transformation is also applied to the generated test outcome and is aimed there at easing test result validation. The utility of the hepTEsT approach is illustrated by means of a simple example, a larger case study and one carried out within the aviation industry.