Getting Started

Documentation

Glish

Learn More

Programming

Contact Us

Version 1.9 Build 1556

News	FAQ
Search	Home

Performance benchmarks

This module defines the standard AIPS++ performance benchmarks. The primary purpose of the benchmark tool is to allow routine and automated performance monitoring of representative data reduction threads. This, in turn, allows performance problems due to unrelated code evolution to be identified and corrected early in development.

Performance benchmarks are invariably an imperfect measure of the performance of any software package. They are inherently difficult to define for a number of reasons, including:

Coverage

Benchmarks need to cover a set of operations which together are representative of the average use profile of the system. The heterogeneous nature of large radio astronomy reduction packages, including AIPS++, makes this very difficult to define in any unique or absolute sense. This is further compounded by the broad range of applications for the package, including near real-time use at telescopes and use in off-line reduction environments. The benchmarks defined here attempt to address this balance by focusing on the common scientific reduction threads most frequently in use.

Scientific relevance

Benchmarks need to be cast in terms of their scientific relevance if they are to be useful. In a pipeline environment, for example, the reduction time of any given thread is most sensibly expressed as the fraction of the observing time of the data under reduction. In an interactive environment, the relevant time-scale is the time it takes to set input parameters for the operation being performed. In either case, there is a point at which performance improvements no longer improve scientific throughput in any meaningful sense. Conversely, expressing benchmarks in terms of their scientific relevance isolates those cases which should be and can be profitably improved.

Dependence

Benchmarks typically depend on a large number of parameters, including uv-data set size, image size and deconvolution algorithm type, amongst many others. Scientific throughput is therefore a scalar function over many dimensions. In addition, the constituent components in the overall performance function are often non-linear in any individual parameter (e.g. Oⁿ(p)), so the overall performance function cannot be linearly extrapolated from one part of parameter space to another. Any performance benchmark is only sampling the overall performance function at one point in the parameter hyperspace. This reinforces the need for benchmarks to be representative of a broad range of common data reduction threads.

Environment

Benchmark measurements may be very sensitive to the immediate configuration of the local computing environment. This includes factors such as disk I/O protocols (SCSI versus IDE), background operating system I/O caching policies, operating system and architecture, networked (NFS) or non-networked disk access, available memory and memory utilization policies in the applications. All influence the measurement obtained for any given benchmark, and need to be carefully controlled for reliable benchmark measurements.

Inter-comparisons

Performance comparisons between different software systems are particularly difficult, primarily because it is seldom possible to inter-compare exactly the same operation in each package, due to differences in implementation and approach. In addition, great care needs to be taken in controlling the environment in making such comparisons. Packages which have small memory models are difficult to compare with packages which have large memory models, for example, given the substantial difference in access times for main memory and disk. The benchmarks defined here do however include scripts for comparable (or similar) operations in other packages in order to provide an approximate estimate of relative performance. However such numbers are invariably highly unreliable for the reasons stated here.