2. The aims and problems of acceptance testing software
The aim of any testing scheme is to ensure that the customer gets substantially the software that he ordered and it must provide the customer with convincing evidence that this is so. However, any attempt to test software comes up against several basic problems.
One of the main problems is that of specifying unambiguously the facilities and performance to be provided by an item of software. This must be done in such a way as to allow the manufacturer to introduce improvements as time goes by and yet be capable of being written into a contract. A problem closely associated with this is that of defining the acceptance criteria to be used. These must be capable of unambiguous application and leave the supplier in no doubt as to the performance required.
Software can only function through the action of the hardware; fundamentally, only the combined effects of the software and hardware can influence events outside the computer, software alone is useless. Any testing method can only test the validity of the whole software/hardware edifice. Further, any testing procedure must take account of the fact that software is often capable of operating in several different environments representing the many possible configurations and patterns of system usage.
A problem under failure conditions is to determine what has failed; is it the software, the hardware, or a subtle combination of both? As systems become more complex, this latter condition is more likely. Modern software is called upon to interact in real time with its environment. The external stimuli can occur almost at random. Any attempt to test exhaustively such a system is going to be very difficult. It is, perhaps, fair to say that it is going to be as intellectually challenging to test the complex software systems of the future as it will be to design them initially.
3. The requirements of a software testing procedure
A test procedure should fulfil the following requirements:
(a) Enable a user to check that he gets the product that he ordered. Any discrepancies should be detected and clearly defined so that meaningful discussions can take place between the user and the supplier.
(b) Be reasonably easy to apply and provide an effective measure of the acceptability of the software.
(c) Provide an incentive for the suppliers to improve their own Quality Assurance procedures for software.
(d) Check the ‘efficiency’ of the software.
(e) Must be applicable from the first deliveries of a system onwards. Before a machine is ordered, a customer should have carried out a detailed appraisal of the competing systems. The tests carried out prior to final acceptance are to confirm that the information on which the chosen system was selected and finally ordered were correct. Thus, the software testing procedure may be regarded as the culmination of this appraisal. However, the depth of any such appraisal will depend upon the circumstances under which a system was selected and the organization doing the selection; thus, any test procedure should not rely upon any previous appraisal for its success.
4. Present software testing procedures
Most customers accept the software without any form of test; indeed, their contracts with the suppliers would often not allow them to do so even if they wished to. Those attempts that are made are usually limited to providing demonstrations of the more important features and the user writing special programmes that he believes will test areas important to him.
The United Kingdom government’s standard computer contract allows for demonstrations of items of software to be called for during the acceptance trials. However, because of the limited time available, it is realized that these demonstrations do not represent a thorough test of the software: merely a check on the availability and a very limited test of certain critical items. When these demonstrations are based on a prior knowledge of the state of the software they can be arranged to highlight its failings and thus allow some action to be taken under the contract but without such knowledge these demonstrations are of little value. The United Kingdom government’s Computer Advisory Service