| Version 1.9 Build 1556
|
|
Next: Overview of benchmark tool functions:
Up: benchmark - Module
Previous: benchmark - Module
This module defines the standard AIPS++ performance benchmarks. The
primary purpose of the benchmark tool is to allow routine and
automated performance monitoring of representative data reduction
threads. This, in turn, allows performance problems due to unrelated
code evolution to be identified and corrected early in development.
Performance benchmarks are invariably an imperfect measure of the
performance of any software package. They are inherently difficult to
define for a number of reasons, including:
- Coverage
- Benchmarks need to cover a set of operations which
together are representative of the average use profile of the
system. The heterogeneous nature of large radio astronomy reduction
packages, including AIPS++, makes this very difficult to define in any
unique or absolute sense. This is further compounded by the broad
range of applications for the package, including near real-time use at
telescopes and use in off-line reduction environments. The benchmarks
defined here attempt to address this balance by focusing on the common
scientific reduction threads most frequently in use.
- Scientific relevance
- Benchmarks need to be cast in terms of
their scientific relevance if they are to be useful. In a pipeline
environment, for example, the reduction time of any given thread is
most sensibly expressed as the fraction of the observing time of the
data under reduction. In an interactive environment, the relevant
time-scale is the time it takes to set input parameters for the
operation being performed. In either case, there is a point at which
performance improvements no longer improve scientific throughput in
any meaningful sense. Conversely, expressing benchmarks in terms of
their scientific relevance isolates those cases which should be and can
be profitably improved.
- Dependence
- Benchmarks typically depend on a large number of
parameters, including uv-data set size, image size and deconvolution
algorithm type, amongst many others. Scientific throughput is
therefore a scalar function over many dimensions. In addition, the
constituent components in the overall performance function are often
non-linear in any individual parameter (e.g. On(p)), so the overall
performance function cannot be linearly extrapolated from one part of
parameter space to another. Any performance benchmark is only sampling
the overall performance function at one point in the parameter
hyperspace. This reinforces the need for benchmarks to be
representative of a broad range of common data reduction threads.
- Environment
- Benchmark measurements may be very sensitive to the
immediate configuration of the local computing environment. This
includes factors such as disk I/O protocols (SCSI versus IDE),
background operating system I/O caching policies, operating system and
architecture, networked (NFS) or non-networked disk access, available
memory and memory utilization policies in the applications. All
influence the measurement obtained for any given benchmark, and need
to be carefully controlled for reliable benchmark measurements.
- Inter-comparisons
- Performance comparisons between different
software systems are particularly difficult, primarily because it is
seldom possible to inter-compare exactly the same operation in each
package, due to differences in implementation and approach. In
addition, great care needs to be taken in controlling the environment
in making such comparisons. Packages which have small memory models
are difficult to compare with packages which have large memory models,
for example, given the substantial difference in access times for main
memory and disk. The benchmarks defined here do however include
scripts for comparable (or similar) operations in other packages in
order to provide an approximate estimate of relative
performance. However such numbers are invariably highly unreliable for
the reasons stated here.
Next: Overview of benchmark tool functions:
Up: benchmark - Module
Previous: benchmark - Module
  Contents
  Index
Please send questions or comments about AIPS++ to aips2-request@nrao.edu.
Copyright © 1995-2000 Associated Universities Inc.,
Washington, D.C.
Return to AIPS++ Home Page
2006-10-15