Module MeasurementSets

Changes made in the current development cycle can be found in the changelog.

Description (classes)

Handle storage and access of telescope data

Prerequisite

Tables module
AIPS++ Note 191

Review Status

Reviewed By:: Bob Garwood
Date Reviewed:: 1997/02/01

Etymology

The MeasurementSet is where all data are ultimately to be found in AIPS++. Since, this is a collection of measurements (either actual or simulated), the term MeasurementSet seems appropriate. Often we use the abbreviation (and typedef) MS for MeasurementSet.

Synopsis

The MeasurementSets module handles storage of telescope data and access to it. The MeasurementSet is the class that gives access to the data.

A MeasurementSet (MS) is a Table with subtables stored as keywords. For each of these tables there is a separate class: the main table is MeasurementSet, the subtables are MSAntenna, MSArray, MSFeed, MSField, MSObsLog, MSObservation, MSSource, MSSpectralWindow, MSSysCal, MSWeather.

Class hierarchy and Table layout

Each table has a number of predefined columns and keywords, a subset of which is required to be present. The column and keyword layout of each table is described in AIPS++ Note 191 and in a separate class which contains two enum definitions. The enum classes, e.g., MSMainEnums and MSAntennaEnums, just contain a PredefinedColumn (PDC) enum and a PredefinedKeyword (PDK) enum. These enum definitions are used as template arguments for the generic class MSTable<PDC,PDK> from which MeasurementSet and all the subtable classes are derived. Thus, e.g., the class MSAntenna is derived from MSTable<MSAntennaEnums::PredefinedColumns, MSAntennaEnums::PredefinedKeywords>.

The MSTable class provides a large number of common column and keyword helper functions. They are useful when creating a Table following the MeasurementSet conventions from scratch and assist in following the agreed upon column and keyword naming conventions.

MSTable in turn is derived from Table, thus all the MS table classes are Tables. Many operations on a MeasurementSet are Table operations. See the Tables module for a list of those operations.

The reason for this class hierarchy is to provide each of the table classes with a separate namespace for its column and keyword enums, so that e.g., they can all have a column named TIME, while sharing as much code as possible. The MSTable class forwards any substantial work to the MSTableImpl class which does the actual work, it is a purely implementation class with static functions not of interest to the user.

Access to columns

To simplify the access of columns, and catch type errors in the column declarations for column access objects (TableColumns) at compile time, there is a helper class for each (sub)table (e.g., MSFieldColumns). The helper class for the MeasurementSet, MSColumns gives access to the main table columns and the helper objects for all subtables. A read-only version of these classes is also provided (e.g., ROMSFieldColumns).

At present these classes are separate from the Table classes, mainly to ensure that the member functions are only called on valid, completely constructed MeasurementSet Tables. They also represent a large amount of 'state' in the form of TableColumn objects of which only a couple may actually be used.

Units and Measures

Columns in the MeasurementSet and its subtables can have several keywords attached to describe the contents of the column. The UNIT keyword contains an ASCII string describing the unit of the values in the column. The unit member function (in MSTable) will return this unit string, it can be used to construct a Unit object for a particular column.

The MEASURE_TYPE keyword gives the AIPS++ Measure that applies to the column (if any). See the Measures module for a list of available Measures and their use.

The MEASURE_REFERENCE keyword gives (part of) the reference frame needed to interpret the values in a column. An example is J2000 for the POSITION column. A number of static functions in MeasurementSet are available to create a 'template' Measure for a column, which has the MEASURE_TYPE filled in. Currently the following functions exist: directionMeasure, positionMeasure, epochMeasure and frequencyMeasure. They return a Measure which can then be filled with a value from a particular row from the column to obtain, e.g., a proper MDirection Measure for the phase center.

Reference Tables

Each of the MS classes has a member function referenceCopy which takes the name of a new Table and a list (Block) of column names to create a Table which references all columns in the original table, except for those listed. The listed columns are new columns which can be written to without affecting the original Table. The main use of this is for the synthesis package where corrected and model visibilities are stored as new DATA columns in an MS which references the raw MS for the other columns. Except for these special cases, the use of this function will be rare.

DATA and FLOAT_DATA columns

To accommodate both synthesis and single dish data efficiently, it was decided that a MeasurementSet can have a Complex DATA column, a float FLOAT_DATA column or both. If it has only a FLOAT_DATA column, the corresponding DATA column can be created with the makeComplexData() member function. In special cases, both columns could be present but filled for different rows, with an extra index defined indicating in which column to look (e.g., multi-feed single dish with cross correlations). The details of this last scheme are yet to be worked out. The table consistency checks (isValid()) do not require the presence of either column.

Unset values in MeasurementSet Tables

For ID columns, the rule is that a value of -1 indicates that there is no corresponding subtable in which to look up details. An example is the PULSAR_ID column, which should be set to -1 if the optional PULSAR subtable does not exist.

The rules for non integer unset values in MS tables have not settled down yet. For Floating point and Complex values the recommended practice is to set the values to the FITS and IEEE value NaN, with a bit pattern of all 1's. In much of the present filler code unused values are filled with 0 instead.

Table destruction

Upon destruction, the table and all subtables are checked to see that the MeasurementSet remains valid, i.e., all required columns are present. An exception is thrown if not all required columns are present Nevertheless, the table will be flushed to disk if it is writable - preserving its state.

MS shorthand

While the class name, MeasurementSet, is descriptive, it is often too long for many common uses. The typedef MS is provided as a convenient shorthand for MeasurementSet. The example below uses this typedef.

Example

This example illustrates creation and filling of the MeasurementSet.

      // create the table descriptor
      TableDesc simpleDesc = MS::requiredTableDesc()
      // set up a new table
      SetupNewTable newTab("simpleTab", simpleDesc, Table::New);
      // create the MeasurementSet
      MeasurementSet simpleMS(newTab);
      // now we need to define all required subtables
      // the following call does this for us if we don't need to
      // specify details of Storage Managers or non-standard columns.
      simpleMS.createDummySubtables(Table::New);
      // fill MeasurementSet via its Table interface
      // For example, construct one of the column access objects.
      TableColumn feed(simpleMS, MS::columnName(MS::FEED1));
      uInt rownr = 0;
      // add a row
      simpleMS.addRow();
      // set the values in that row, e.g. the feed column
      feed.putScalar(rownr,1);
      // Access the position column in the ANTENNA subtable
      ArrayColumn<Double> antpos(simpleMS.antenna(),
                                 MSAntenna::columnName(MSAntenna::POSITION));
      // Add a row to it and fill in the position
      simpleMS.antenna().addRow();
      Array<Double> position(3); 
      position(0)=1.; position(1)=2.; position(2)=3.;
      antpos.put(0,position);
      // .
      // For standard columns the above can be done more easily using 
      // the MSColumns object.
      // Create the MSColumns
      MSColumns msc(simpleMS);
      // and fill in the position
      msc.antenna().position().put(0,position);

Example

This example illustrates read only access to an existing MeasurementSet and creation of an MDirection Measure for the phase center.

      // Create the MeasurementSet from an existing Table on disk
      MeasurementSet ms("myMS"); 
      // Create the RO column access objects for main table and subtables
      ROMSColumns msc(ms);
      // show data from row 5
      cout << msc.data()(5) << endl;
      // show phase center for row 3 in field table
      Vector<double> phaseCtr=msc.field().phaseCenter()(3);
      cout << phaseCtr<<endl;
      // now create a Measure for the phaseCenter
      MDirection phaseCenterMeasure =
         MS::directionMeasure(msc.field().phaseCenter());
      // put the value from row 3 in the Measure and print it
      phaseCenterMeasure.set(MVPosition(phaseCtr));
      cout <<"phase center:"<< phaseCenterMeasure<<endl;

Motivation

The attempt is to define a single, extensible, Table format that will be able to cope with all, or at least most, radio telescope data. Having a single MeasurementSet should make it easier to combine data from different instruments. The format of the MeasurementSet, table with subtables, was chosen to be able to cope with items varying at different rates more efficiently than a 'flat' Table layout would allow.

To Do

Incorporate the MSColumn classes in the MeasurementSet classes?
Variable (row to row) ReferenceFrame (e.g., J2000 mixed with galactic, different Frequency reference frames mixed in the same MS, etc.). This could be done with a column named "column_name"_MEASURE_REFERENCE for each column with varying Measure reference frames.

Classes

AlmaTI2MS -- AlmaTI2MS: Convert ALMA-TI data to MS format (full description)

CorrDataKeywords -- CorrDataKeywords: process CORRDATA-ALMATI FITS keywords (full description)

DataParKeywords -- DataParKeywords: process DATAPAR-ALMATI FITS keywords (full description)

FITSIDItoMS1 -- FITSIDItoMS converts a FITS-IDI binary table to an AIPS++ Table. (full description)

MS1ToMS2Converter -- Class to convert a MeasurementSet v1 to v2. (full description)

MSAntenna -- A Table intended to hold a MeasurementSet ANTENNA table. (full description)

MSAntennaColumns -- A class to provide easy read-write access to MSAntenna columns (full description)

MSAntennaEnums -- Enums for the MeasurementSet ANTENNA table (full description)

MSAntennaIndex -- Class to handle lookup or indexing into a MS ANTENNA subtable (full description)

MSColumns -- A class to provide easy read-write access to MeasurementSet columns (full description)

MSConcat -- A class with functions for concatenating MeasurementSets (full description)

MSDataDescColumns -- A class to provide easy read-write access to MSDataDescription columns (full description)

MSDataDescIndex -- Class to handle lookup or indexing into a MS DATA_DESC subtable (full description)

MSDataDescription -- A Table intended to hold a MeasurementSet DATADESCRIPTION table. (full description)

MSDataDescriptionEnums -- Enums for the MeasurementSet DATA_DESCRIPTION table (full description)

MSDerivedValues -- MSDerivedValues calculates values derived from a MS (full description)

MSDoppler -- A Table intended to hold a MeasurementSet DOPPLER table. (full description)

MSDopplerColumns -- A class to provide easy read-write access to MSDoppler columns (full description)

MSDopplerEnums -- Enums for the MeasurementSet DOPPLER table (full description)

MSDopplerIndex -- (full description)

MSDopplerUtil -- A utility class for MS Doppler tracking information (full description)

MSFeed -- A Table intended to hold a MeasurementSet FEED table. (full description)

MSFeedColumns -- A class to provide easy read-write access to MSFeed columns (full description)

MSFeedEnums -- Enums for the MeasurementSet FEED table (full description)

MSFeedIndex -- (full description)

MSField -- A Table intended to hold a MeasurementSet FIELD table. (full description)

MSFieldColumns -- A class to provide easy read-write access to MSField columns (full description)

MSFieldEnums -- Enums for the MeasurementSet FIELD table (full description)

MSFieldIndex -- Class to handle lookup or indexing into a MS FIELD subtable (full description)

MSFitsIDI -- MSFitsIDI: Convert FITS-IDI data to MS format (full description)

MSFitsInput -- UV FITS to MeasurementSet filler (full description)

MSFitsOutput -- Write a MeasurementSet to a random group uvfits file. (full description)

MSFlagCmd -- A Table intended to hold a MeasurementSet FLAG_CMD table. (full description)

MSFlagCmdColumns -- A class to provide easy read-write access to MSFlagCmd columns (full description)

MSFlagCmdEnums -- Enums for the MeasurementSet FLAG_CMD table (full description)

MSFlagger -- MSFlagger specifies selections on a MeasurementSet (full description)

MSFreqOffIndex -- (full description)

MSFreqOffset -- A Table intended to hold a MeasurementSet FREQ_OFFSET table. (full description)

MSFreqOffsetColumns -- A class to provide easy read-write access to MSFreqOffset columns (full description)

MSFreqOffsetEnums -- Enums for the MeasurementSet FREQ_OFFSET table (full description)

MSHistory -- A Table intended to hold a MeasurementSet OBSERVATIONLOG table. (full description)

MSHistoryColumns -- A class to provide easy read-write access to MSHistory columns (full description)

MSHistoryEnums -- Enums for the MeasurementSet HISTORY table (full description)

MSHistoryHandler -- A class to provide a simple interface to history writing (full description)

MSInterval -- Small helper class to specify an 'interval' comparison (full description)

MSIter -- An iterator class for MeasurementSets (full description)

MSLister -- List visibility records from a Measurement Set (full description)

MSMainColumns -- A class for easy read-write access to MeasurementSet main table columns (full description)

MSMainEnums -- Enums for the MeasurementSet main table (full description)

MSObservation -- A Table intended to hold a MeasurementSet OBSERVATION table. (full description)

MSObservationColumns -- A class to provide easy read-write access to MSObservation columns (full description)

MSObservationEnums -- Enums for the MeasurementSet OBSERVATION table (full description)

MSObservationIndex -- Class to handle lookup or indexing into an MS OBSERVATION subtable (full description)

MSPointing -- A Table intended to hold a MeasurementSet POINTING table. (full description)

MSPointingColumns -- A class to provide easy read-write access to MSPointing columns (full description)

MSPointingEnums -- Enums for the MeasurementSet POINTING table (full description)

MSPointingIndex -- (full description)

MSPolarization -- A Table intended to hold a MeasurementSet POLARIZATION table. (full description)

MSPolarizationColumns -- A class to provide easy read-write access to MSPolarization columns (full description)

MSPolarizationEnums -- Enums for the MeasurementSet POLARIZATION table (full description)

MSPolarizationIndex -- Class to handle lookup or indexing into an MS POLARIZATION subtable (full description)

MSPrimaryGroupHolder -- A helper class for MSFitsInput (full description)

MSProcessor -- A Table intended to hold a MeasurementSet PROCESSOR table. (full description)

MSProcessorColumns -- A class to provide easy read-write access to MSProcessor columns (full description)

MSProcessorEnums -- Enums for the MeasurementSet PROCESSER table (full description)

MSRange -- MSRange determines ranges of values in a MeasurementSet (full description)

MSReader -- Read from an MS, coordinating all of the subtables in the process (full description)

MSSelUtil -- Helper class for MSFlagger with templated static function (full description)

MSSelUtil2 -- Helper class for MSSelector/DOms with templated static functions (full description)

MSSelection -- MSSelection: Class to represent a selection on an MS (full description)

MSSelectionKeywords -- MSSelectionKeywords specifies selection keywords for the MeasurementSet (full description)

MSSelector -- MSSelector specifies selections on a MeasurementSet (full description)

MSSimFeedBuf -- a container for data destined for an MS FEED table (full description)

MSSimFeedRec -- a container for data destined for an MS FEED record (full description)

MSSimulator -- Create an empty MeasurementSet from observation and telescope descriptions. (full description)

MSSource -- A Table intended to hold a MeasurementSet SOURCE table. (full description)

MSSourceColumns -- A class to provide easy read-write access to MSSource columns (full description)

MSSourceEnums -- Enums for the MeasurementSet SOURCE table (full description)

MSSourceIndex -- (full description)

MSSpWindowColumns -- A class to provide easy read-write access to MSSpectralWindow columns (full description)

MSSpWindowIndex -- Class to handle lookup or indexing into a MS SPECTRAL_WINDOW subtable (full description)

MSSpectralWindow -- A Table intended to hold a MeasurementSet SPECTRAL_WINDOW table. (full description)

MSSpectralWindowEnums -- Enums for the MeasurementSet SPECTRAL_WINDOW table (full description)

MSState -- A Table intended to hold a MeasurementSet STATE table. (full description)

MSStateColumns -- A class to provide easy read-write access to MSState columns (full description)

MSStateEnums -- Enums for the MeasurementSet STATE table (full description)

MSSummary -- Provides and lists information about the header of an image (full description)

MSSysCal -- A Table intended to hold a MeasurementSet SYSCAL table. (full description)

MSSysCalColumns -- A class to provide easy read-write access to MSSysCal columns (full description)

MSSysCalEnums -- Enums for the MeasurementSet SYSCAL table (full description)

MSSysCalIndex -- (full description)

MSTable -- A Table intended to hold astronomical data (full description)

MSTableImpl -- An implementation class for the MeasurementSet to share code. (full description)

MSTableIndex -- (full description)

MSTileLayout -- An helper class for deciding on tile shapes in MeasurementSets (full description)

MSValidIds -- (full description)

MSWeather -- A Table intended to hold a MeasurementSet WEATHER table. (full description)

MSWeatherColumns -- A class to provide easy read-write access to MSWeather columns (full description)

MSWeatherEnums -- Enums for the MeasurementSet WEATHER table (full description)

MSWeatherIndex -- (full description)

MeasurementSet -- A Table intended to hold astronomical data (a set of Measurements). (full description)

NewMSSimulator -- Create an empty MeasurementSet from observation and telescope descriptions. (full description)

ROMSAntennaColumns -- A class to provide easy read-only access to MSAntenna columns (full description)

ROMSColumns -- A class to provide easy read-only access to MeasurementSet columns (full description)

ROMSDataDescColumns -- A class to provide easy read-only access to MSDataDesc columns (full description)

ROMSDopplerColumns -- A class to provide easy read-only access to MSDoppler columns (full description)

ROMSFeedColumns -- A class to provide easy read-only access to MSFeed columns (full description)

ROMSFieldColumns -- A class to provide easy access to MSField columns (full description)

ROMSFlagCmdColumns -- A class to provide easy read-only access to MSFlagCmd columns (full description)

ROMSFreqOffsetColumns -- A class to provide easy read-only access to MSFreqOffset columns (full description)

ROMSHistoryColumns -- A class to provide easy read-only access to MSHistory columns (full description)

ROMSMainColumns -- A class for easy read-only access to MeasurementSet main table columns (full description)

ROMSObservationColumns -- A class to provide easy read-only access to MSObservation columns (full description)

ROMSPointingColumns -- A class to provide easy read-only access to MSPointing columns (full description)

ROMSPolarizationColumns -- A class to provide easy read-only access to MSPolarization columns (full description)

ROMSProcessorColumns -- A class to provide easy read-only access to MSProcessor columns (full description)

ROMSSourceColumns -- A class to provide easy read-only access to MSSource columns (full description)

ROMSSpWindowColumns -- A class to provide easy read-only access to MSASpectralWindow columns (full description)

ROMSStateColumns -- A class to provide easy read-only access to MSState columns (full description)

ROMSSysCalColumns -- A class to provide easy read-only access to MSSysCal columns (full description)

ROMSWeatherColumns -- A class to provide easy read-only access to MSWeather columns (full description)

SDAntennaHandler -- (full description)

SDDataDescHandler -- (full description)

SDFITSDataIterator -- Steps through an SDFITS DATA cell, serving up chunks appropriate for an MS. (full description)

SDFITSHandler -- (full description)

SDFeedHandler -- (full description)

SDFieldHandler -- (full description)

SDHistoryHandler -- (full description)

SDMainHandler -- (full description)

SDObservationHandler -- (full description)

SDPointingHandler -- (full description)

SDPolarizationHandler -- (full description)

SDSourceHandler -- (full description)

SDSpWindowHandler -- (full description)

SDSysCalHandler -- (full description)

SDWeatherHandler -- (full description)

SubMS -- SubMS provides functionalities to make a subset of an existing MS (full description)

almati2ms -- Implementation of the almati2ms DO (full description)

fitsidi2ms -- Implementation of the fitsidi2ms DO (full description)

ms -- Implementation of the ms DO (full description)

ms2fromms1 -- Definition for ms1 to ms2 converter DO (full description)

sdfits2ms -- This is the SDFITS to MS converter distributed object (DO). (full description)