On the subject of pre-release checking, there is the question of how a manufacturer can ensure that a system he distributes will have been adequately checked for use in met environment. It seems to me that the only way to solve such problems is to have the manufacturer simulate my environment, or even use my environment directly, via communication lines.
Kolence: Users should expect to have to test software that is supplied to them, to ensure that it works in their environment. A large manufacturer cannot test out his software on all the environments in which it will operate, It is for this reason that manufacturers typically provide an on-site representative to help the user adapt a general system to his particular environment.
6.3 SYSTEM EVALUATION
The problem of system evaluation runs through all three areas of Design, Production and Service. This section is based on discussion of those aspects of the subject directly related to the system when in a user’s environment.
6.3.1 . ACCEPTANCE TESTING
By far the most extensive discussion of the acceptance testing of software systems was that given by Llewelyn and Wickens in the paper The Testing of Computer Software. For convenience this is reproduced in its entirety in Section 9, and just the conclusions are paraphrased below.
Llewelyn and Wickens: We are attempting to design a method by which a large organization could test the software supplied by computer manufacturers to its installations. In particular we wish to ensure that new installations can rapidly take on the work for which they were purchased by ensuring that their planning can be based on the best available information with regard to what software exists, and how well it performs. The present situation is that a customer has to purchase his software almost as an act of faith in the supplier — this surely cannot be allowed to continue.
Kolence: The manufacturers are always under pressure from the users to give them something that works even if it is not complete. The classic trade-off in software production is between features and schedule — not between working well and not working well. It is important therefore for users to receive an accurate set of specifications, ahead of time, of what is currently being produced, rather than of what is ultimately intended (as was the case, I believe, with published specifications for OS/360) .
Opler: (from Acceptance testing of large programing systems)
»The proper testing of large programming systems is virtually impossible; but with sufficient resources, enough testing can be performed to allow a good evaluation to be made. Most current systems operate under a wide variety of hardware configurations and with highly varied software component selection. A test plan must be developed considering all elements of the written specification (hardware, programming language, system facilities, documentation, performance, reliability, etc.) and describing steps to validate compliance of the final programming system.
For large systems, enormous resources of computing equipment which can configure all available components are required. Many months of computer testing are often required. A significant, sometimes neglected, area of testing is in checking external documentation (user and operator manuals) for accuracy and clarity and checking internal documentation (flow charts and listings) against the final distributed program.«
Dijkstra: Testing is a very inefficient way of convincing oneself of the correctness of a program.
Llewelyn: Testing is one of the foundations of all scientific enterprise. In fact it would be good to have independent tests of system function and performance published.
Galler: When the hardware doesn’t work, the user doesn’t pay — why should he have to pay for non-working software?
Babcock: We need software meters analogous to the present hardware meters, so that our rental costs can be adjusted to allow for time lost through software errors as well as hardware errors.