fully designed test-cases to run successfully before the system can be released. The test progress consists of a number of discrete events, i.e. test-case successes, which tend to occur in some pattern, and which pattern can be predicted before-hand. If the pattern of successes followed a normal distribution, the test history curve would follow the Cumulative Normal Distribution curve, figure 10. Other distributions produce other cumulative curves, of which a selection is shown in figure 11. From studying test histories of previous developments, a judgement can be made about the nature of the curve applicable to a particular project. For most new projects curves around the 2.5 value are found to be applicable. For improvements to an existing product, successes tend to occur early on, and curve 1.0 can be used. In the case of an existing product being adopted for new environment, for example converting a language processor to run under a different operating system, successes are slow to start but rise quickly once under way, as typified by the higher valued curves.
As an example of the use of this technique, figure 12 shows the actual case history of a project, which was a major extension of a large component of OS/360.
The original test prediction, curve A, was made at the beginning of August. By mid-September it was clear that actual progress, shown by the irregular line, was lagging about 2 weeks behind the plan. The end-date was therefore postponed by four weeks, to allow for the cumulative effect of the lag, and a new prediction made, curve B. In October, it was again evident that the plan was not being met, but by working a substantial amount of overtime, this was corrected to reach the required objective.
There are several important conditions that must be observed if this technique is to be of any value. First, the system build plan must be such that simple test-cases can be run as early as possible, so that testing stretches over a maximum period. This allows the measurement of progress against prediction to give warnings of trouble at an early stage. Second, the set of test-cases must be constructed so that simple tests can be used in the early stages, building up progressively to more complex cases. Third, for the method to work smoothly, careful measures must be taken to avoid changes to the build system causing serious regression, i.e. the familiar event where a correction introduces further errors. With these provisos, test planning and status control can be a valuable tool in managing the later part of a development cycle.«
There was some discussion of the problems of estimating the costs of software projects:
Ercoli: I maintain that if proper management and accounting methods are used so that a software job can be considered as composed by well defined and normalized parts then the cost per instruction in implementing each part is roughly $5 per debugged instructions. This is also the average cost of software for known applications and for efforts below the limit of about 15 man-years. If the general cost of software production is considered then there are wide variations: as low as $1 per instruction using semiautomatic means of software production (such as compiler generators) or as high as $20 per instruction in cases of very large operating systems. I would have liked to have replaced these figures by a rough graph on semilogarithmic paper indicating cost per instruction against size of project but it was lost with my luggage.
Endres: I believe that cost/instruction can vary by a factor of 50.
Salle: It is very difficult to use cost per instruction in estimates because there is no general agreement on what the average cost per instruction is, and also because the ratio varies according to which phase of the production process your are in. The number of tables, number of interfaces, etc., are at least as important for costing as the number of instructions. If these figures are to be used for planning they should be correlated with a number of other important parameters such as range of hardware configurations, number of customers, research content, etc. In fact I think cost/instruction should be the last thing one uses for planning purposes.
Barton: I’ve just realized that the cost/instruction measure has to be meaningless. Now I don’t know the right measure but I suggest as a first approximation to one a weighting which would take into account the number of times the instruction had been used.
5.2.2. PERSONNEL FACTORS
Many of the problems of producing large software systems involve personnel factors:
David: (from Some thoughts about production of large computer systems (2))