Summary

This note describes the format of calibration tables in CASA/AIPS++. The calibration table format is currently at revision v2.0.

Introduction

The calibration and imaging formalism used in AIPS++is based closely on the generic instrument model of Hamaker, Bregman and Sault (1996). This formalism, referred to as the Measurement Equation (ME), is instrument-independent and represents calibration effects in both the visibility and image plane as Jones matrices, with arbitrary parametrization and polarization basis. It's use in AIPS++is described by Cornwell (1995) and Noordam (1995), and references therein.

The calibration table format used in AIPS++was designed to support the fundamental principles on which the ME formalism is based. The definition of a standard calibration table format allows calibration components to be stored permanently for subsequent retrieval, application or modification. The current calibration table format revision level is v2.0.

The underlying principles on which the calibration table data format is based are described below.

Data representation: AIPS++calibration tables are stored as AIPS++Tables, in keeping with the general rule in AIPS++ for all data visible to end users. As such, there is full user access to all data stored in a calibration table. These data can be retrieved, inspected and modified in the general user interface. The implementation of the calibration tables also uses other common Tables infrastructure such as TableMeasures.
ME formalism: the calibration table format was chosen to support the key assumptions of the ME formalism, including support for: i) arbitrary polarization bases; ii) arbitrary Jones matrix parametrization; and, iii) generic, instrument-independent representation of calibration components.
Unified calibration: the calibration table format has been designed to support both single-dish and synthesis calibration, antenna- and baseline-based Jones matrices, as well as visibility-plane and image-plane calibration components in a common format, with specialization where appropriate. This was chosen to maximize the re-use of calibration information and allow it to be transferred between different observing contexts.
MS integration: the calibration table format is fully integrated with the MeasurementSet (MS) data format, supports all data representations possible in the MS, and re-uses all applicable column and keyword names for common underlying physical concepts. A calibration table can be attached to one or more MeasurementSets, but exists as a stand-alone table with a user-specified name. This supports both single-project and multi-project calibration schemes.
History tracking: the calibration table format supports full tracking of the calibration history attached to each calibration solution or step in the calibration process.

Calibration table format

This section describes and defines all fields in the calibration table format. At the highest level, the calibration table consists of a main table, CAL_MAIN, and two sub-tables stored as table keywords CAL_DESC and CAL_HISTORY. The CAL_DESC sub-table describes the shape and coordinates of the array columns in the main calibration table. CAL_HISTORY references the associated calibration history for each row in the main table. All tables are described separately below.

CAL_MAIN

The ME contains several different Jones matrix calibration components which are enumerated in the references listed above. In general, the Jones matrix calibration components fall into the following broad categories:

: Antenna-based, visibility-plane, non-solvable: i) P - parallactic angle; and, ii) C - polarization configuration.
: Antenna-based, visibility-plane, solvable: i) T - atmospheric correction; ii) G - electronic gain; iii) D - instrumental polarization; iv) B - bandpass; and v) F - ionosphere correction.
: Interferometer-based, visibility-plane, solvable: The only supported interferometer-based correction at present is a multiplicative, solvable term (M). An additive term is also allowed by the ME.
: Antenna-based, image-plane, non-solvable: i) P - parallactic angle.
: Antenna-based, image-plane, solvable: i) T - atmosphere correction; ii) D - instrumental polarization; iii) F - ionosphere correction; and iv) E - voltage pattern.

Note that the same Jones matrix can appear as both a visibility-plane and image-plane correction in the ME, as appropriate. In addition, an individual Jones matrix component may either be discretely sampled or modeled using arbitrary parameters.

Jones matrices of different types (time-variable, solvable, baseline-based, antenna-based and parametrized) share as many common sections in the CAL_MAIN table as appropriate, thus representing the inheritance and specialization inherent to the family of Jones matrix types. This inheritance relationship is illustrated by the VisJones and SkyJones class families in the AIPS++library. Each calibration table holds only Jones calibration matrices of one type (e.g. GJones, or PJones etc.).

The key sections in the CAL_MAIN table are as follows:

Primary MS indices: The first section contains the MS indices representing the MS data from which the calibration solution was derived or which otherwise label the calibration data. Fundamental amongst these are the time and interval of each row. Indices over which the MS data were averaged prior to the solution, or which are not applicable, are set to -1.
Secondary MS fields: The primary MS indices can be used to retrieve any secondary MS information by direct lookup in the associated MS. However, a limited subset of MS information is stored directly in the calibration table, to minimize the need for lookup in the MS for the most common secondary information. This breaks database normalization, but as is commonly the case, is warranted when dictated by efficiency. This also allows the calibration table to be used in many cases when the associated MS is not available. Where these column names are not unique within the MS as a whole, they are prefixed by the associated MS subtable name (e.g. SOURCE_CODE).
Gain values: The calibration gain factors, represented as sampled complex (1x1), (2x2) or (4x4) Jones matrices, in array format over optional axes of spectral window identifier, frequency channel and sky coordinates, are stored in a GAIN column. This column is optional for parametrized calibration components.
Reference frame: A section is provided to represent the reference antenna, feed, receptor, frequency and direction for calibration parameters, where appropriate.
Solution statistics: A section is defined to contain the statistical properties of each calibration solution. This includes boolean flags for the calibration gain arrays and the fit and fit weight, both per array element and for the solution interval as a whole.
Jones matrix parametrization: Customized columns required to represent parametrized Jones matrices are defined in a separate section. These are different for each parametrized type.
Sub-table pointers: Indices per CAL_MAIN row, into the CAL_DEC and CAL_HISTORY sub-tables, are defined separately.

CAL_DESC

The CAL_DESC sub-table defines the dimensions of the array-based columns in CAL_MAIN, and specifies the coordinates of the array axes. These coordinates include frequency, receptor polarization, and for image-plane components, directions and regions in the image-plane.

The frequency labeling of discretely sampled calibration gain values, as well as parametrized calibration models, may differ from the frequency labeling in the associated MS, if calibration is derived by aggregating or sub-dividing the MS spectral windows. Hence, the spectral windows and frequency channels along the calibration array axes, as described in CAL_DESC, may differ from the MS itself. Where there is a direct mapping to MS spectral windows and channel ranges, this is recorded.

The polarization axis is defined in terms of a list of receptor polarizations. As an enumerated coordinate, these overlap with those used in the associated MS.

For parametrized calibration components, the coordinate axes in CAL_DESC define the coordinate ranges over which the parameters are valid, and also the dimensions of the model parameters stored as arrays. In this case, the frequency and direction coordinates also define the units and frame for the parameters.

The image-plane direction array axis is defined by specifying an array or directions and associated regions about each direction in the image plane. This allows arbitrary discrete sampling of image-plane Jones calibration matrices, as well as arbitrary parameter domains for image-plane calibration models.

CAL_HISTORY

The CAL_HISTORY sub-table records the calibration context for the solutions recorded in each main calibration table row. This includes the parameters used by the calibration solver, the other calibration tables which may have been applied in the solution, any data selection, as well as an arbitrary set of notes the user may wish to attach to the calibration entry for future reference. All CAL_HISTORY fields are in string form, with a general keyword-value format.

CAL_MAIN: antenna-based, non-solvable, visibility-plane components

CAL_MAIN: antenna-based, non-solvable, visibility-plane components
Name	Format	Units	Measure	Comments
Columns
Primary MS indices
TIME	Double	s	EPOCH	Interval midpoint
TIME_EXTRA_PREC	Double	s		Extra TIME precision
INTERVAL	Double	s		Calibration interval
ANTENNA1	Int			Antenna no.
FEED1	Int			Feed on ANTENNA1
FIELD_ID	Int			Field id.
ARRAY_ID	Int			Subarray number
OBSERVATION_ID	Int			Observation id.
SCAN_NUMBER	Int			Scan number
PROCESSOR_ID	Int			Processor id.
PHASE_ID	Int			Phase id.
STATE_ID	Int			State id.
PULSAR_BIN	Int			Pulsar bin number
PULSAR_GATE_ID	Int			Pulsar gate id.
Secondary MS fields
FREQ_GROUP	Int			Freq. group
FREQ_GROUP_NAME	String			Freq. group name
FIELD_NAME	String			Field name
FIELD_CODE	String			Field code
SOURCE_NAME	String			Source name
SOURCE_CODE	String			Source code
CALIBRATION_GROUP	Int			Source cal. group
Gain values
GAIN	Complex(a)			Gain values
Reference
REF_ANT	Int(b)			Ref. antenna
REF_FEED	Int(b)			Ref. feed
REF_RECEPTOR	Int(b)			Ref. receptor
REF_FREQUENCY	Double(b)	Hz	FREQUENCY	Ref. freq.
MEAS_FREQ_REF	Int			Measures freq. ref.
REF_DIRECTION	Double(b)		DIRECTION	Ref. dir.
MEAS_DIR_REF	Int			Measures dir. ref.
Sub-table pointers
CAL_DESC_ID	Int			CAL_DESC index
CAL_HISTORY_ID	Int			CAL_HISTORY index

Notes:
: (i) The dimensions of the array-based columns, which can vary by row, and are defined in the CAL_DESC sub-table are as follows: (a) (N_jones, N_jones, N_spw, N_chan); and (b) (N_receptors, N_spw, N_chan).
: (ii) All MS primary indices point into the associated MS, as defined in the CAL_DESC sub-table as MS_NAME. See AIPS++note 229 for a full description of the MS data format.
TIME: Mid-point (not centroid) of calibration interval.
TIME_EXTRA_PREC: Extra TIME precision.
INTERVAL: Time interval for which this calibration solution is valid, or over which it was determined.
ANTENNA1: Antenna number ( $\geq$ 0), and a direct index into the ANTENNA sub-table rownr of the associated MS.
FEED1: Feed number ( $\geq$ 0).
FIELD_ID: Field identifier ( $\geq$ 0).
ARRAY_ID: Subarray identifier ( $\geq$ 0), which identifies data in separate subarrays.
OBSERVATION_ID: Observation identifier ( $\geq$ 0), which identifies data from separate observations.
SCAN_NUMBER: Arbitrary scan number to identify data taken in the same logical scan. Not required to be unique.
PROCESSOR_ID: Processor indentifier ( $\geq$ 0), and a direct index into the PROCESSOR sub-table rownr.
PHASE_ID: Switching phase identifier ( $\geq$ 0)
STATE_ID: State identifier ( $\geq$ 0), as defined in Section 3.1.5.
PULSAR_BIN: Pulsar bin number for the data record. Pulsar data may be measured for a limited number of pulse phase bins. The pulse phase bins are described in the PULSAR sub-table and indexed by this bin number.
PULSAR_GATE_ID: Pulsar gate identifier ( $\geq$ 0), and a direct index into the PULSAR_GATE sub-table rownr.
FREQ_GROUP: The frequency group to which the spectral window belongs. This is used to associate spectral windows for joint calibration purposes.
FREQ_GROUP_NAME: The frequency group name; user specified.
FIELD_NAME: Field name; user specified.
FIELD_CODE: Field code indicating special characteristics of the field; user specified.
SOURCE_NAME: Source name; user specified.
SOURCE_CODE: Source code, used to describe any special characteristics f the source, such as the nature of a calibrator. Reserved keyword, including ("BANDPASS CAL").
CALIBRATION_GROUP: Calibration group number to which this source belongs; user specified.
GAIN: Array of calibration gain values, expressed as Jones matrices in an array of dimension (b), which is defined together with the array coordinates in the CAL_DESC sub-table.
REF_ANT: Array of reference antenna numbers, of dimension (b).
REF_FEED: Array of reference feed numbers, of dimension (b).
REF_RECEPTOR: Array of reference receptor numbers, of dimension (b).
REF_FREQUENCY: Array of reference frequencies, of dimension (b).
MEAS_FREQ_REF: Array of reference frequency Measure references, of dimension (b).
REF_DIRECTION: Array of reference directions, of dimension (b).
MEAS_DIR_REF: Array of reference direction Measure references, of dimension (b).

CAL_MAIN: additions for antenna-based, solvable, visibility-plane components

CAL_MAIN: additions for antenna-based, solvable, visibility-plane components
Name	Format	Units	Measure	Comments
Columns
Solution statistics
TOTAL_SOLUTION_OK	Bool			Validity of total solution
TOTAL_FIT	Float			Total fit
TOTAL_FIT_WEIGHT	Float			Total fit weight
SOLUTION_OK	Bool(a)			Solution validity mask
FIT	Float(a)			Fit array
FIT_WEIGHT	Float(a)			Fit weight array

Notes:
: (i) The dimension (a) of the array-based columns is: (N_jones, N_jones, N_spw, N_chan). These dimensions are defined in the CAL_DESC sub-table, and can vary by row.
TOTAL_SOLUTION_OK: False if the current calibration solution interval is flagged, else True.
TOTAL_FIT: Total fit for the solution interval as a whole, as $\sqrt{{\chi^2}\over{\Sigma {\rm weight}}}$ .
TOTAL_FIT_WEIGHT: Total fit weight for the solution interval as a whole ( $\Sigma$ weight).
SOLUTION_OK: Validity mask for each element of the calibration solution array, of dimension (a).
FIT: Fit for each element of the calibration solution array, of dimension (a), as defined in the TOTAL_FIT_WEIGHT description above.
FIT_WEIGHT: Fit weight for each element of the calibration solution array, of dimension (a).

CAL_MAIN: additions for interferometer-based, solvable, visibility-plane components

CAL_MAIN: additions for interferometer-based, solvable, visibility-plane components
Name	Format	Units	Measure	Comments
Columns
MS primary indices
ANTENNA2	Int			Second antenna no.
FEED2	Int			Feed on ANTENNA2

Notes:
ANTENNA2: Second antenna number ( $\geq$ 0) of the interferometer pair, and a direct index into the ANTENNA sub-table rownr of the associated MS.
FEED2: Feed number ( $\geq$ 0) on ANTENNA2.

CAL_DESC: calibration array dimensions and coordinates

CAL_DESC: calibration array dimensions and coordinates
Name	Format	Units	Measure	Comments
Columns
NUM_SPW	Int			No. cal. spw ids
NUM_CHAN	Int			No. cal. freq. chan.
NUM_RECEPTORS	Int			No. cal. receptors
N_JONES	Int			Jones matrix dim.
SPECTRAL_WINDOW_ID	Int(N_spw)			Associated MS spw ids
CHAN_FREQ	Double( N_spw, N_chan)	Hz	FREQUENCY	Cal chan. freq.
MEAS_FREQ_REF	Int			Measures freq. ref.
CHAN_WIDTH	Double( N_spw, N_chan)	Hz		Cal. chan. width
CHAN_RANGE	Int( 2, N_spw, N_chan)			Associated MS chan. range
POLARIZATION_TYPE	String( N_receptors)			Receptor polarization
JONES_TYPE	String			Matrix type
MS_NAME	String			MS name

Notes:
NUM_SPW: Number of calibration spectral windows.
NUM_CHAN: Number of calibration frequency channels.
NUM_RECEPTORS: Number of calibration polarization receptors.
N_JONES: Dimension of the Jones matrix. For antenna-based corrections this is less than or equal to the number of polarization receptors (one or two).
SPECTRAL_WINDOW_ID: Associated MS spectral window id.'s. Set to -1 if the calibration frequency windows span multiple MS spectral windows.
CHAN_FREQ: Array of calibration array frequencies.
MEAS_FREQ_REF: Measures reference for the calibration frequencies.
CHAN_WIDTH: Array of calibration frequency channel widths.
CHAN_RANGE: Associated MS frequency channel range.
POLARIZATION_TYPE: Receptor polarization type (e.g. "R", "L", "X" etc.)
JONES_TYPE: The matrix type, as a reserved keyword ("DIAGONAL", "SCALAR", or "GENERAL").
MS_NAME: Name of the associated MeasurementSet.

CAL_DESC: additions for image-plane components

CAL_DESC: additions for image-plane components
Name	Format	Units	Measure	Comments
Columns
NUM_DIR	Int			No. of directions
DIRECTION	Doube(2, N_dir)		DIRECTION	Direction array
MEAS_DIR_REF	Int			Meas. dirn. ref.
REGION	Record(N_dir)			Associated regions.

Notes:
: (i) Note that the array dimensions in CAL_MAIN are increased by (N_dir) for the case of image-plane calibration components.
NUM_DIR: Number of image-plane calibration directions.
DIRECTION: Array or image-plane calibration directions.
MEAS_DIR_REF: Measures direction reference.
REGION: Associated region for each direction.

CAL_HISTORY: calibration table history

CAL_HISTORY: calibration table history
Name	Format	Units	Measure	Comments
Columns
CAL_PARMS	String			Solver parameters
CAL_TABLES	String			Associated cal. tables
CAL_SELECT	String			Calibration selection
CAL_NOTES	String			Calibration notes

Notes:
: (i) All CAL_HISTORY fields are in string form in a general keyword-value format.
CAL_PARMS: The parameters used for the solver in deriving the calibration solution.
CAL_TABLES: Associated calibration tables applied when solving for the current Jones component.
CAL_SELECT: Data selection applied when deriving the calibration solution.
CAL_NOTES: Associated user notes for the calibration solution.