will, in general, have appraised the software of a computer before it is bought for use in the government service and will subsequently keep it under review. It is this fact that enables the present procedure to work reasonably well with the currently available software. However, it would be unrealistic to suppose that such a procedure will provide the user much protection against faults in the very complex systems that are likely to become available in the next few years.
The greatest benefits are obtained from a testing procedure in the early life of a new range of computers. Once a usable version of the required software is available, the basic needs of the user are satisfied. If subsequent issues are in error, it is inconvenient but not disastrous. However, this does not mean that testing standards can be relaxed for new issues of current packages, although it may make it less urgent. Seemingly, quite minor changes may have a profound effect upon the operation of a package. It is certainly necessary to check that a new issue of a package is compatible with its predecessor. It must not be expected that a test procedure will guarantee error-free software but it should tell the user what is the state of the software he is being offered so that he may decide whether it is capable of doing his work effectively.
5. Possible test strategies
There are, fundamentally, two different methods of determining whether a product meets its specification. One can analyse the product in great detail and from this determine if it is in accordance with its specification, or one can measure its performance experimentally and see if the results are in accord with the specification; the number and sophistication of the experiments can be varied to provide the degree of confidence required of the results.
Our current software appraisals are biased towards the analytic approach but the limited time and effort available and the often limited access to detailed information prevent the appraisal doing away with the need for an experimental approach to software testing.
6. A proposed test procedure
The proposed test procedure attempts to remove, as far as possible, software testing from the hardware acceptance trials. The aim of the procedure is to test software as soon as it becomes available and merely to check that each installation has a good copy of the required items in its library. It is intended for use by large organizations rather than by individual small users.
It is proposed that software should be rigorously tested outside the acceptance trials. It requires that items of software should be given ‘type’ approval on a number of prescribed configurations as soon as possible after company release. During an acceptance trial, it should merely be proved that a correct copy of the software exists at the installation for that particular configuration.
Various levels of type approval would be required but only the more important packages need be subjected to all levels of the testing envisaged. The suggested levels of approval are:
(a) Documentation. This would check that the user manuals are available, are in accordance with the specifications issued by the supplier, that they are of an acceptable standard of presentation and literacy and are self-consistent. Basically, the question that should be answered is ‘is it likely that the potential user will be able to use the documentation to write effective programs for his installations’. It would also be necessary to cheek that an acceptable up-dating system is provided so that a user can be sure that he has up-to-date manuals. The greater part of the work involved in this level of approval could be done as part of the normal appraisal of software carried out before a machine is ordered.
(b) Availability. This would check that an item of software had been released and was in general conformity with the documentation. This would not be a rigorous check but merely a cheek of a package’s existence in a working form on a number of prescribed configurations. This check should be carried out as soon as possible after the contractor makes the software available. These tests would be very similar to the current demonstrations of software during acceptance trials.
(e) Detailed verification of facilities. This stage would consist of testing the specified facilities of each package in detail and checking the package’s interaction with others with which it can be associated. Such tests should include the running of special test programs to check on facilities, plus ‘soak’ or random tests using a wide variety of typical user type programs from a wide variety of sources.