Infrastructure
articles:

Measurement
Set Version 2

The Data
Repository

In AIPS++, observed data from both interferometers and single-dish telescopes are stored in a common data format, referred to as a Measurement Set (MS). The data format is designed to allow data to be recorded in a generic format that is not instrument-specific, but extensible to include local data required for individual instruments. The MS format is an important counterpart to the generic calibration formalism, known as the Measurement Equation (Hamaker, Bregman and Sault 1996; A&AS 137, 117), which underpins calibration and imaging in AIPS++, and is similarly designed to be instrument-independent.

Measurement Sets in AIPS++ are implemented as AIPS++ Tables, and are fully visible to scientific users of the package using the table and tablebrowser tools. Additional tools such as ms and visplot allow higher-level access to the data stored in an MS.

For the past several years, AIPS++ has used MS version 1, as defined in AIPS++ Note 191. This format has served the project well, but over time needed to be extended to provide additional scientific features. The discussion of possible revisions to the MS v1 format took place within the AIPS++ project as a whole, and included significant contributions from throughout the project. The resulting MS v2 format was formally adopted by the project at the end of this process, and is defined in AIPS++ Note 229. This is the MS data format expected by AIPS++ applications from release v1.5 (November 2000) onwards. A conversion tool, ms1toms2 has been provided to allow conversion of existing MSv1 datasets. These can also be filled from FITS or other data formats into MSv2 directly.

The highlights of the new data format include:

• Closer integration of single-dish and interferometer data, including support for generalized back-end devices, loads and reference signals, and scheduling heuristics.
• Optional support for both lag-spectra and triple-product data in the main table.
• Expanded VLBI support, including correlator models and frames, pulsar gating, frequency groups and space VLBI.
• A more generalized flagging model, allowing both command- and row-based flags.
• Infrastructure to allow diverse observations and sub-arrays to be combined flexibly.
• Separation of antenna pointing, source position and field center, and polynomial representation of these positions for near-field or moving objects.
• Expanded representation of history data.
• Generalized representation of Doppler tracking, covering single-dish and interferometry Doppler tracking modes.
• Additions to the system calibration and weather tables.
• Expanded representation of polarization and frequency information.

Infrastructure
Articles:

Measurement
Set Version II

The Data
Repository

Release 1.4 will be the first release to contain the new AIPS++ data repository. This repository was developed to move the data elements out of the source tree. By moving the data which change slowly out of the source tree, the download time for each AIPS++ build is lessened, and larger data sets can be used and distributed.

CVSup was chosen for use with the data repository for several reasons:

1. it does rsync updates of large binary files, which allows only the modified tail-end of binary files to be shipped and then patched onto the existing older version of the files.
2. it has optimizations for dealing with RCS and CVS repositories which permit it to minimize traffic.
3. it is used by FreeBSD so it has a large body of users who maintain it and ensure that any security problems are handled.
4. once the specification file is created, it is easy for users to keep the data repository up to date and to monitor the updates as they happen (we plan to include CVSup binaries with the release).

The repository is divided into sections based on content:

• ephemerides
• geodetic
• catalogs
  • sources
  • lines
• demo
• individual institutions, e.g. NRAO, BIMA, etc.

Our goal is to continue to expand the repository and in time to allow users to choose which pieces they wish to keep locally and synchronized, and which pieces should be accessed on-demand. One of the next things we plan to add shortly is the Poynter and Pickett Submillimeter, Millimeter, and Microwave Spectral Line Catalogue. This and other catalogs should make the repository useful not only for internal AIPS++ applications, but also for direct use by end users.