(4) continuous software monitoring
Provided that care is taken to control the overhead and that hardware-level resolution is not required, the most desirable data collection procedure allows continuous monitoring of events under program control. Such a facility can be turned on and off during normal operation, easily changed to adjust the quantity and kinds of data, and it provides a complete description of the monitored phenomena in a given period. The author’s experience in implementing this kind of facility in the Michigan Terminal System (MTS) has shown that it can be done with surprisingly good resolution and low overhead, to produce invaluable data for the purposes mentioned above.
The following general observations may be useful in the implementation of a data collection facility (DCF);
a. A DCF should be included in the design of a system, for part of it will be embedded in the supervisor at a low level, and careful planning is necessary to ensure a clean implementation.
b. A DCF is very useful for the development of the system itself, and therefore should be produced early in that development.
c. For reasons of both effectiveness and generality, as much of the analysis of collected data as possible should be deferred for independent processing.
d. Both the collection and analysis phases of operation require a variety of data selection modes, e.g. by type of datum, by task, by recording time. Good selection capability reduces both the collection overhead and data reduction time.
II. Systems Evaluation
Once measures have been taken to provide accurate operating system data, it becomes reasonable to consider the problem of evaluating system performance. The performance criteria one uses clearly depend on one’s motives for evaluation. Within a single installation, one usually focusses on criteria of utilization of both hardware and software components, and on standards of the quality of service provided. Discussions of both these factors require a basis for characterizing the workload under which given levels of utilization and response have been obtained. Indices chosen to describe these three operational factors are biased by the types of performance data available and the way in which they are obtained.
Because distinct installations rely on different performance data and emphasize different criteria for evaluation, it is difficult to compare their performance. Thus there is an additional need to define standards for the descriptions of workload, utilization and response so that performance in different environments can be meaningfully compared. It would be particularly useful to provide such standards for general purpose multi-access systems (where the need is greatest!) in such a way that they also apply to simpler multi-programming and single-thread systems as special cases.