Opler: (from Measurement and analysis of software in production)
»1. Who, When, Why?
a. Guidance Measurement by Production Group, starting at the earliest existence of code in measurable form, for purposes of analyzing: 1) Conformity to design requirements; 2) conformity to internal conventions; 3) identifications of erroneous or deficient areas; 4) Identification of areas subject to optimizing by tuning. As separate modules are combined, measurements are repeated.
b. Completion measurement by Production Group, immediately prior to delivery, for quality assurance of the final product.
c. Formal measurement by control group to determine if quality of final product is acceptable.
2. What is measured?
a. Performance: Space, speed, throughput, turn around.
b. Language: compliance with requirements, accuracy of object system. External Function: error isolation, configuration modularity/clear documentation, availability, installation ease, modification ease.
d. Internals: serviceability, reliability (freedom from mistakes), conformity to standards.
3. How are measurements made?
a. Gross (external) measurements: typical programs, mixes, streams; data files; special test programs for language conformity; for mathematical accuracy; for standards conformity.
b. Fine (internal) measurements: by special hardware monitors; by special software packages; by built-in measurement schemes.
c. By use of product operation: serviceability; configuration modularity; installation ease«
Opler: It is important to emphasize the necessity for providing sufficient resources during production of a software system to design and conduct performance measurements, and to feed the results back to those concerned, in both the design and the production groups. One warning is in order. It is fatally easy to concern oneself with only those quantities which are easy to measure, and to ignore other, possibly more important, quantities whose measurement is more difficult. Monitoring aids must be an integral part of the system, so that every attempt to use the system is potentially a source of useful data.
Kinslow: There is a whole class of what production management tends to think of as ‘auxiliary’ functions, such as system generation, system start-up, system shut-down, and monitoring. If the budget gets tight, monitoring is the first to go, because of a lack of appreciation for its importance.
Fraser: We found it useful to monitor a system, feed the results into a simulator, and then experiment with the simulator, as being much easier to fiddle with than the actual system.
Pinkerton provided a survey of the various techniques that have been used for monitoring systems. Because of its length this survey is reproduced in Section 9, rather than here.
Finally, a description was given by Gillette of a method of automating both the testing and performance monitoring of a system.
Gillette: (from Aids in the production of maintainable software)