The likely effort required would be:
(a) For computers using pre-written benchmark programs, about one man-month per language.
(b) For operating systems, up to six man-months per system.
8. Criteria of acceptance
The supplier would have to have a criterion to aim at for each level or the approval procedure and it is here that the greatest difficulty arises. Software is a highly versatile product whose performance depends very much on the precise use made of it and thus useful generalised measures of software performance are not easily found. The value of a piece of software is still very much a matter of opinion but the testing procedure is aimed at finding facts and highlighting failures so that reasonable discussion should be possible between the parties.
Approval at Level 1 could initially be given as soon as the appropriate set of manuals become available for each package and had been used to write a few simple programs. Criticisms would be fed back to the supplier but it is not realistic to expect manuals to be re-cast at the ‘approving’ authority’s request, although a long-term influence might be expected. Of course, it could be expected that errors would be corrected as soon as possible. Approval at Level 1 might be interpreted as indicating that usable manuals are generally available to customers.
Approval of Level 2 could be given as soon as demonstrations of a package had been witnessed by the approving body. The demonstration programs would, of course, have to be acceptable to the approving body. If the supplier’s own validation procedures were acceptable, then documentary evidence of their completion might be acceptable instead of a demonstration.
Approval at Level 3 would be given when all the facilities offered or called for in the contract were shown to be working. However, we cannot expect perfection and it might be necessary to give partial approval in some cases. An alternative might be to divide the facilities into ‘required’ and ‘desired’ and call for 100 per cent implementation of the required facilities and, say, 70 per cent of the ‘desired’ ones, together with, say, three months’ grace to complete the implementation.
The requirements for approval at Level 4 can often be well defined; for example, ‘shall occupy not more than 8,000 words of core store’, ‘shall compile at 1,000 statements per minute for a program of 200 statements’, etc.
9, Effort and cost estimates for the approval scheme
The total cost of implementing the previously described Type Approval scheme can be expected to vary greatly from system to system. The following figures are for testing a typical currently available operating system in accordance with the above proposals.
Software Item Effort in Man-months Machine time in hours
Level 1 2 3 4 Level 1 2 3 4
Languages
Algol 2 1 2 1 0 1 2 1
Cobol 2 1 2 1 0 1 2 1
Fortran 2 1 2 1 0 1 2 1
R.P.G. 2 1 1 1 0 1 2 1
Assembly code 3 1 2 1 0 1 3 1
Operating system
Supervision 3 1 3 2 0 1 3 1
File Control packages 3 1 2 2 0 1 3 1
Peripheral handling routines 3 1 2 1 0 1 3 1
Program testing aids 3 1 2 1 0 1 3 1
Application packages
Sort routines 2 1 1 1 0 1 1 1
Miscellaneous 3 2 3 1 0 1 1 1
Totals: 28 12 22 13 0 11 25 11