Previous Up Next

Chapter 2  Visibility Data Import, Export, and Selection

To use CASA to process your data, you first will need to get it into a form that is understood by the package. These are “Measurement Sets” for synthesis (and single dish) data, and “image tables” for images.

There are a number of tasks used to fill telescope-specific data, to import/export standard formats, to list data contents, and to concatenate multiple datasets. These are:

In CASA, there is a standard syntax for selection of data that is employed by multiple tasks. This is described in § 2.3.

There are also tasks for the import and export of image data using FITS:

2.1  CASA Measurement Sets

Data is handled in CASA via the table system. In particular, visibility data are stored in a CASA table known as a Measurement Set (MS). Details of the physical and logical MS structure are given below, but for our purposes here an MS is just a construct that contains the data. An MS can also store single dish data (essentially a set of auto-correlations of a 1-element interferometer), though there are also data formats more suitable for single-dish spectra (see § 8).

A full description of the Measurement Set can be found at http://casa.nrao.edu/Memos/229.html.

Inside the Toolkit:

Measurement sets are handled in the ms tool. Import and export methods include ms.fromfits and ms.tofits.

Note that images are handled through special image tables, although standard FITS I/O is also supported. Images and image data are described in a separate chapter.

The headers of any FITS files can be displayed in the logger with the listfits task:

#  listfits :: List the HDU and typical data rows of a fits file:
fitsfile            =         ''        #  Name of input fits file

Unless your data was previously processed by CASA or software based upon its predecessor aips++, you will need to import it into CASA as an MS. Supported formats include some “standard” flavors of UVFITS, the VLA “Export” archive format, and most recently, the ALMA Science Data Model (ASDM) format. These are described below in § 2.2.

Once in Measurement Set form, your data can be accessed through various tools and tasks with a common interface. The most important of these is the data selection interface (§ 2.3) which allows you to specify the subset of the data on which the tasks and tools will operate.

2.1.1  Under the Hood: Structure of the Measurement Set

Inside the Toolkit:

Generic CASA tables are handled in the tb tool. You have direct access to keywords, rows and columns of the tables with the methods of this tool.

It is not necessary that a casual CASA user know the specific details on how the data in the MS is stored and the contents of all the sub-tables. However, we will occasionally refer to specific “columns” of the MS when describing the actions of various tasks, and thus we provide the following synopsis to familiarize the user with the necessary nomenclature. You may skip ahead to subsequent sections if you like!

All CASA data files, including Measurement Sets, are written into the current working directory by default, with each CASA table represented as a separate sub-directory. MS names therefore need only comply with UNIX file or directory naming conventions, and can be referred to from within CASA directly, or via full path names.

An MS consists of a MAIN table containing the visibility data. and associated sub-tables containing auxiliary or secondary information. The tables are logical constructs, with contents located in the physical table.* files on disk. The MAIN table consists of the table.* files in the main directory of the ms-file itself, and the other tables are in the respective subdirectories. The various MS tables and sub-tables can be seen by listing the contents of the MS directory itself (e.g. using Unix ls), or via the browsetable task (§ 3.6).

See Fig 2.1 for an example of the contents of a MS directory. Or, from the casa prompt,

CASA <1>: ls ngc5921.ms
IPython system call: ls -F ngc5921.ms
ANTENNA           POLARIZATION     table.f1        table.f3_TSM1  table.f8
DATA_DESCRIPTION  PROCESSOR        table.f10       table.f4       table.f8_TSM1
FEED              SORTED_TABLE     table.f10_TSM1  table.f5       table.f9
FIELD             SOURCE           table.f11       table.f5_TSM1  table.f9_TSM1
FLAG_CMD          SPECTRAL_WINDOW  table.f11_TSM1  table.f6       table.info
HISTORY           STATE            table.f2        table.f6_TSM0  table.lock
OBSERVATION       table.dat        table.f2_TSM1   table.f7
POINTING          table.f0         table.f3        table.f7_TSM1

Note that the MAIN table information is contained in the table.* files in this directory. Each of the sub-table sub-directories contain their own table.dat and other files, e.g.

CASA <2>: ls ngc5921.ms/SOURCE
IPython system call: ls -F ngc5921.ms/SOURCE
table.dat  table.f0  table.f0i  table.info  table.lock

Figure 2.1: The contents of a Measurement Set. These tables compose a Measurement Set named ngc5921.demo.ms on disk. This display is obtained by using the File:Open menu in browsetable and left double-clicking on the ngc5921.demo.ms directory.

Each “row” in a table contains entries for a number of specified “columns”. For example, in the MAIN table of the MS, the original visibility data is contained in the DATA column — each “cell” contains a matrix of observed complex visibilities for that row at a single time stamp, for a single baseline in a single spectral window. The shape of the data matrix is given by the number of channels and the number of correlations (voltage-products) formed by the correlator for an array.

Table 2.1 lists the non-data columns of the MAIN table that are most important during a typical data reduction session. Table 2.2 lists the key data columns of the MAIN table of an interferometer MS. The MS produced by fillers for specific instruments may insert special columns, such as ALMA_PHASE_CORR, ALMA_NO_PHAS_CORR and ALMA_PHAS_CORR_FLAG_ROW for ALMA data filled using the importasdm filler (§ 2.2.1). These columns are visible in browsetable and are accessible from the toolkit in the ms tool (e.g. the ms.getdata method) and from the tb “table” tool (e.g. using tb.getcol).

Note that when you examine table entries for IDs such as FIELD_ID or DATA_DESC_ID, you will see 0-based numbers.



Table 2.1: Common columns in the MAIN table of the MS.
ParameterContents
ANTENNA1First antenna in baseline
ANTENNA2Second antenna in baseline
FIELD_IDField (source no.) identification
DATA_DESC_IDSpectral window number, polarization identifier pair (IF no.)
ARRAY_IDSubarray number
OBSERVATION_IDObservation identification
POLARIZATION_IDPolarization identification
SCAN_NUMBERScan number
TIMEIntegration midpoint time
UVWUVW coordinates

The MS can contain a number of “scratch” columns, which are used to hold useful versions of other columns such as the data or weights for further processing. The most common scratch columns are:

The creation and use of the scratch columns is generally done behind the scenes, but you should be aware that they are there (and when they are used).



Table 2.2: Commonly accessed MAIN Table data-related columns. Note that the columns ALMA_PHASE_CORR, ALMA_NO_PHAS_CORR and ALMA_PHAS_CORR_FLAG_ROW are specific to ALMA data filled using the importasdm filler.
ColumnFormatContents
DATAComplex(Nc, Nf)complex visibility data matrix (= ALMA_PHASE_CORR by default)
FLAGBool(Nc, Nf)cumulative data flags
WEIGHTFloat(Nc)weight for a row
WEIGHT_SPECTRUMFloat(Nc, Nf)individual weights for a data matrix
ALMA_PHASE_CORRComplex(Nc, Nf)on-line phase corrected data (Not in VLA data)
ALMA_NO_PHAS_CORRBool(Nc, Nf)data that has not been phase corrected (Not in VLA data)
ALMA_PHAS_CORR_FLAG_ROWBool(Nc, Nf)flag to use phase-corrected data or not (not in VLA data)
MODEL_DATAComplex(Nc, Nf)Scratch: created by calibrater or imager tools
CORRECTED_DATAComplex(Nc, Nf)Scratch: created by calibrater or imager tools

Data flags can be set in the MS, too. Whenever a flag is set, the data will be ignored in all processing steps but not physically deleted from the MS. The flags are channel-based and stored in the MS FLAG subtable. Backups can be stored in the ’MS.flagversions’ file that can be accessed via the flagmanager (§ 3.2).

The most recent specification for the MS is Aips++ Measurement Set definition version 2.0 (http://casa.nrao.edu/Memos/229.html).

2.2  Data Import and Export

There are a number of tasks available to bring data in various forms into CASA as a Measurement Set:

2.2.1  ALMA: Filling of Science Data Model (ASDM) data

Under the Hood:

The importasdm task is just an interface to the stand-alone asdm2MS application. To find out the command-line arguments to this application, do asdm2MS --help .

The ALMA and JVLA projects have agreed upon a common archival science data model (ASDM) format, and have jointly developed the software to fill this data into CASA. In the ASDM format, the bulk of the data is contained in large binary data format (BDF) tables, with the meta-data and ancillary information in XML tables. This is structured as a directory, like the MS, and was designed to be similar to the MS to facilitate conversion.

The content of an ASDM can be listed with the task asdmsummary:

#  asdmsummary :: Summarized description of an ASDM dataset.
asdm                =         ''        #  Name of input ASDM directory

with an output that contains the list and positions of the antennas, followed by the parameters of each scan like observation time, source name, frequency and polarization setup:

Input ASDM dataset : TDEM0008.sb3373760.eb3580330.55661.22790537037
========================================================================================
ASDM dataset :TDEM0008.sb3373760.eb3580330.55661.22790537037
========================================================================================

Exec Block : ExecBlock_0
Telescope : JVLA
Configuration name : B
Observer name : Dr. Juergen Ott
The exec block started on 2011-04-10T05:28:13.200000000 and ended on 2011-04-10T10:27:12.300000256

27 antennas have been used in this exec block.
        Id     Name         Make Station    Diameter         X              Y             Z
     Antenna_0  ea01    UNDEFINED   W36        25    -1606841.96   -5042279.689    3551913.017
     Antenna_1  ea02    UNDEFINED   E20        25     -1599340.8   -5043150.965    3554065.219
     Antenna_2  ea03    UNDEFINED   E36        25   -1596127.728   -5045193.751    3552652.421
     Antenna_3  ea04    UNDEFINED   W28        25   -1604865.649    -5042190.04    3552962.365
     Antenna_4  ea05    UNDEFINED   W08        25   -1601614.091   -5042001.653    3554652.509
     Antenna_5  ea06    UNDEFINED   N24        25    -1600930.06   -5040316.397    3557330.397
     Antenna_6  ea07    UNDEFINED   E32        25   -1597053.116   -5044604.687    3553058.987
     Antenna_7  ea08    UNDEFINED   N28        25   -1600863.684   -5039885.318    3557965.319
     Antenna_8  ea09    UNDEFINED   E24        25    -1598663.09   -5043581.392    3553767.029
     Antenna_9  ea10    UNDEFINED   N32        25   -1600781.039   -5039347.456    3558761.542
     Antenna_10  ea11    UNDEFINED   E04        25    -1601068.79    -5042051.91    3554824.835
     Antenna_11  ea12    UNDEFINED   E08        25   -1600801.926   -5042219.366    3554706.448
     Antenna_12  ea14    UNDEFINED   W12        25   -1602044.903   -5042025.824    3554427.832
     Antenna_13  ea15    UNDEFINED   W24        25   -1604008.742   -5042135.828    3553403.707
     Antenna_14  ea16    UNDEFINED   N12        25   -1601110.052   -5041488.079    3555597.439
     Antenna_15  ea17    UNDEFINED   W32        25   -1605808.656   -5042230.082    3552459.202
     Antenna_16  ea18    UNDEFINED   N16        25   -1601061.961    -5041175.88    3556058.022
     Antenna_17  ea19    UNDEFINED   W04        25   -1601315.893    -5041985.32    3554808.305
     Antenna_18  ea20    UNDEFINED   N36        25   -1600690.606   -5038758.734    3559632.061
     Antenna_19  ea21    UNDEFINED   E12        25    -1600416.51    -5042462.45    3554536.041
     Antenna_20  ea22    UNDEFINED   N04        25   -1601173.979   -5041902.658    3554987.518
     Antenna_21  ea23    UNDEFINED   E16        25   -1599926.104   -5042772.967    3554319.789
     Antenna_22  ea24    UNDEFINED   W16        25   -1602592.854   -5042054.997      3554140.7
     Antenna_23  ea25    UNDEFINED   N20        25   -1601004.709   -5040802.809    3556610.133
     Antenna_24  ea26    UNDEFINED   W20        25   -1603249.685   -5042091.404    3553797.803
     Antenna_25  ea27    UNDEFINED   E28        25   -1597899.903   -5044068.676    3553432.445
     Antenna_26  ea28    UNDEFINED   N08        25    -1601147.94   -5041733.837    3555235.956

Number of scans in this exec Block : 234

scan #1 from 2011-04-10T05:28:13.200000000 to 2011-04-10T05:33:35.500000256
        Intents : OBSERVE_TARGET
        Sources : 1331+305=3C286
        Subscan #1 from 2011-04-10T05:28:13.200000000 to 2011-04-10T05:33:35.500000256
                Intent : UNSPECIFIED
                Number of integrations : 322

                 Binary data in uid:///evla/bdf/1302413292901
                 Number of integrations : 322
                 Time sampling : INTEGRATION
                 Correlation Mode : CROSS_AND_AUTO
                 Spectral resolution type : FULL_RESOLUTION
                 Atmospheric phase correction : AP_UNCORRECTED
                 SpectralWindow_0 : numChan = 256, frame = TOPO,
                 firstChan = 8484000000, chandWidth = 125000 x Polarization_0 : corr = RR,LL

scan #2 from 2011-04-10T05:33:35.500000256 to 2011-04-10T05:35:35.200000000
        Intents : OBSERVE_TARGET
        Sources : 1331+305=3C286
        Subscan #1 from 2011-04-10T05:33:35.500000256 to 2011-04-10T05:35:35.200000000
                Intent : UNSPECIFIED
                Number of integrations : 119

                 Binary data in uid:///evla/bdf/1302413293280
                 Number of integrations : 119
                 Time sampling : INTEGRATION
                 Correlation Mode : CROSS_AND_AUTO
                 Spectral resolution type : FULL_RESOLUTION
                 Atmospheric phase correction : AP_UNCORRECTED
                 SpectralWindow_0 : numChan = 256, frame = TOPO,
                 firstChan = 8484000000, chandWidth = 125000 x Polarization_0 : corr = RR,LL

scan #3 from 2011-04-10T05:35:35.200000000 to 2011-04-10T05:36:34.999999488
        Intents : OBSERVE_TARGET
        Sources : 1331+305=3C286
        Subscan #1 from 2011-04-10T05:35:35.200000000 to 2011-04-10T05:36:34.999999488
...

The importasdm task will fill SDM1.2 and SDM1.3 format data into a CASA visibility data set (MS). ALMA data was recorded in SDM1.2 format from October 2009 until May 2011. Since May 2011, ALMA is using the SDM 1.3 format. In particular all science data from cycle 0 will be in SDM1.3. The JVLA also started using SDM1.2 in October 2009 and continues to use this format as of October 2011. importasdm can read all of the above formats. The parameter useversion can be used to enable the options process_syspower, process_caldevice, and process_pointing.

The default inputs of importasdm are:

#  importasdm :: Convert an ALMA Science Data Model observation into a
CASA visibility file (MS) or single-dish data format (Scantable)
asdm                =         ''        #  Name of input asdm directory (on
                                        #   disk)
vis                 =         ''        #  Root name of the ms to be created.
                                        #   Note the .ms is NOT added
createmms           =      False        #  Create a multi-MS output
singledish          =      False        #  Set true to output single-dish data
                                        #   format
corr_mode           =      'all'        #  specifies the correlation mode to be
                                        #   considered on input. A quoted string
                                        #   containing a sequence of ao, co,
                                        #   ac,or all separated by whitespaces
                                        #   is expected
srt                 =      'all'        #  specifies the spectral resolution
                                        #   type to be considered on input. A
                                        #   quoted string containing a sequence
                                        #   of fr, ca, bw, or all separated by
                                        #   whitespaces is expected
time_sampling       =      'all'        #  specifies the time sampling
                                        #   (INTEGRATION and/or SUBINTEGRATION)
                                        #   to be considered on input. A quoted
                                        #   string containing a sequence of i,
                                        #   si, or all separated by whitespaces
                                        #   is expected
ocorr_mode          =       'ca'        #  output data for correlation mode
                                        #   AUTO_ONLY (ao) or CROSS_ONLY (co) or
                                        #   CROSS_AND_AUTO (ca)
compression         =      False        #  Flag for turning on data compression
lazy                =      False        #  Make the MS DATA column read the ASDM
                                        #   Binary data directly (faster import,
                                        #   smaller MS)
asis                =         ''        #  Creates verbatim copies of the
                                        #   ASDMtables in the ouput measurement
                                        #   set.  Value given must be a string
                                        #   of table names separated by spaces;
                                        #   A * wildcard is allowed.
wvr_corrected_data  =       'no'        #  Specifies which values are considerd
                                        #   in the SDM binary data to fill the
                                        #   DATA column in the MAIN table of the
                                        #   MS. Expected values for this option
                                        #   are: no, for uncorrected data
                                        #   (default), yes, for the corrected
                                        #   data, and both, for for corrected
                                        #   and uncorrected data. Note if both
                                        #   is selected two measurement sets are
                                        #   created, one with uncorrected data
                                        #   and the other with corrected data.
scans               =         ''        #  processes only the specified scans.
                                        #   This value is a semicolon separated
                                        #   list of scan specifications. A scan
                                        #   specification consists in an exec
                                        #   bock index followed by the :
                                        #   character;  followed by a comma
                                        #   separated list of scan indexes or
                                        #   scan index ranges. A scan index is
                                        #   relative to the exec block it
                                        #   belongs to. Scan indexes are 1-based
                                        #   while exec blocks are 0-based. "0:1"
                                        #   or "2:2~6" or
                                        #   "0:1,1:2~6,8;2:,3:24~30" "1,2" are
                                        #   valid values for the option. "3:"
                                        #   alone will be interpreted as, all
                                        #   the scans of the exec block#3.  An
                                        #   scan index or a scan index range not
                                        #   preceded by an exec block index will
                                        #   be interpreted as, all the scans
                                        #   with such indexes in all the exec
                                        #   blocks.  By default all the scans
                                        #   are considered.
ignore_time         =      False        #  All the rows of the tables Feed,
                                        #   History, Pointing, Source, SysCal,
                                        #   CalDevice, SysPower, and Weather are
                                        #   processed independently of the time
                                        #   range of the selected exec block /
                                        #   scan.
process_syspower    =       True        #   The SysPower table is processed if
                                        #   and only if this parameter is set to
                                        #   true.
process_caldevice   =       True        #  The CalDevice table is processed if
                                        #   and only if this parameter is set to
                                        #   true.
process_pointing    =       True        #  The Pointing table is processed if
                                        #   and only if this parameter is set to
                                        #   true. If set to False, the POINTING
                                        #   table is empty in the resulting MS
process_flags       =       True        #  Create online flags in the FLAG_CMD
                                        #   sub-table.
     tbuff          =        0.0        #   Time padding buffer (seconds)
     applyflags     =      False        #  Apply the flags to the MS.
     savecmds       =      False        #  Save flag commands to an ASCII file
     outfile        =         ''        #  Name of ASCII file to save flag
                                        #   commands

flagbackup          =       True        #  Back up flag column before applying
                                        #   flags.
verbose             =      False        #  Output lots of information while the
                                        #   filler is working
overwrite           =      False        #  Over write an existing MS(s)
showversion         =      False        #  Report the version of asdm2MS being
                                        #   used
useversion          =       'v3'        #  Version of asdm2MS to be used ('v3'
                                        #   (default, should work for all data))
bdfflags            =      False        #  Set the MS FLAG column according to
                                        #   the ASDM _binary_ flags
with_pointing_correction =      False   #   add (ASDM::Pointing::encoder -
                                        #   ASDM::Pointing::pointingDirection)
                                        #   to the value to be written in
                                        #   MS::Pointing::direction
remove_ref_undef    =      False        #  if set to True then apply
                                        #   fixspwbackport on the resulting
                                        #   MS(es).
convert_ephem2geo   =       True        #  if True, convert any attached
                                        #   ephemerides to the GEO reference
                                        #   frame (time-spacing not changed)

If scans is set, then importasdm processes only the scans specified in the option’s value. This value is a semicolon separated list of scan specifications. A scan specification consists in an exec bock index followed by the character ’:’ followed by a comma separated list of scan indexes or scan index ranges. A scan index is relative to the exec block it belongs to. Scan indexes are 1-based while exec blocks are 0-based. The expressions

         "0:1"
         "2:2~6"
         "0:1,1:2~6,8;2:,3:24~30"
         "1,2"
         "3:"

are all valid values for the selection. The "3:" selector will be interpreted as ’all the scans of the exec block 3’. An scan index or a scan index range not preceded by an exec block index will be interpreted as ’all the scans with such indexes in all the exec blocks’. By default all the scans are considered.

When process_flags=True the task will create online flags based on the Flag.xml, Antenna.xml and SpectralWindow.xml files and copy them to the FLAG_CMD sub-table of the MS. The flags will NOT be applied unless the parameter applyflags is set to True. Optionally, the flags can also be saved to an external ASCII file if savecmds is set to True. The flags can later be applied to the MS using task flagdata in list mode. See § 3.4.

When bdfflags=True the task will apply online flags contained in the ASDM BDF data by calling the executable bdflags2MS which the user can also do from the OS prompt. This is recommended for ALMA data.

If singledish=True, output data format is scantable (single-dish data format, see 8) instead of MS. In that case, you must specify name or id of the antenna that you want to obtain data. This can be done by using antenna parameter that is defined as a subparameter of singledish. For single-dish mode, only auto-correlation data are filled, i.e. ocorr_mode is forcibly set to ’ao’.

The option createmms prepares the output file for parallel processing and creates a multi-MS (see Sect. 10, and 10.2).

2.2.1.1  Import of ASDM data with option lazy=True

With release 4.3, the parameter ’lazy’ (default = False) is fully tested and operational. If the default value False is chosen, importasdm will (as in previous versions) fill the visibilities into a newly created DATA column of the MS converting them from their binary format in the ASDM to the CASA Table format.

If, however, lazy is set to True, the task will create the DATA column with an ALMA data-specific storage manager, the (asdmstman), which enables CASA to directly read the binary data from the ASDM with on-the-fly conversion. No redundant copy of the raw data is created.

This procedure has the advantage that it saves more than 60% disk space and at least in some cases makes the access to the DATA column 10% faster because the data I/O volume is decreased. For the same reason, it also accelerates the import itself by ca. a factor 2. The acceleration is particularly large in the applycal task and here particularly on standard SATA disks.

E.g., if your ASDM has a size of 36 GB, the import with default parameters will turn this into an MS of 73 GB size (total disk space consumption = 36 GB + 73 GB = 109 GB). With lazy=True, the imported MS has a size of only 2 GB (total disk space consumption = 36 GB + 2 GB = 38 GB). I.e. your total disk space savings are ca. 65%. Even when you compare to the case where you delete the ASDM after normal import, the solution with lazy import and keeping the ASDM will save you ca. 48% disk space (in the example above 38 GB compared to 73 GB).

The only caveats are the following:

  1. You must not delete your ASDM. You can, however, move it but you have to update the reference stored in the MS. Symbolic links will work. See below on how to use the tool method ms.asdmref() to manipulate the ASDM reference.
  2. The lazily imported DATA column is read-only. But in any normal data reduction, the DATA column (as opposed to CORRECTED_DATA) is treated as read-only anyway.

The lazily imported MS is numerically identical with the traditionally imported MS and so are all results derived from the MSs. The setting lazy=True might be made the default setting in future CASA releases.

An important additional tool to manipulate lazily imported MSs is the new method ms.asdmref() in the ms tool. If the MS is imported from an ASDM with option lazy=True, the DATA column of the MS is virtual and directly reads the visibilities from the ASDM. A reference to the original ASDM is stored with the MS. If the ASDM needs to be moved to a different path, the reference to it in the MS needs to be updated. This can be achieved with ms.asdmref().

The method takes one argument: abspath. When called with abspath equal to an empty string (default), the method just reports the currently set ASDM path or an empty string if the ASDM path was not set, i.e. the MS was not lazily imported.

If you want to move the referenced ASDM to a different path, you can set the new absolute path by providing it as the value of abspath to the method.

            ms.open('uid___A12345_X678_X910.ms',False)
            ms.asdmref('/home/alma/myanalysis/uid___A12345_X678_X910')
            ms.close()

will set the new location of the referenced ASDM to /home/alma/myanalysis/uid___A12345_X678_X910.

Note that the lazily imported MS can be moved without any restrictions independently from the referenced ASDM as long as the absolute path to the ASDM remains accessible, even across file systems.

2.2.2  Janksy VLA: Filling of Science Data Model (ASDM) data

Under the Hood:

The importevla task is a modified version of the importasdm task, that includes import of online flags from the Flag.xml table into the FLAG_CMD MS table, and a streamlined set of parameters.

The importevla task will fill SDM data from the Jansky VLA (or ALMA) into a MS, along with online flagging data contained in the Flag.xml SDM table. Otherwise, it behaves as importasdm but with a streamlined parameter set.

The default inputs are:

#  importevla :: Convert an Science Data Model observation into a CASA Measurement Set
asdm                =         ''        #  Name of input asdm directory (on disk)
vis                 =         ''        #  Root name of the ms to be created. Note the .ms
                                        #   is NOT added
ocorr_mode          =       'co'        #  Fill correlation mode AUTO_ONLY (ao),
                                        #   CROSS_ONLY (co) or CROSS_AND_AUTO (ca)
compression         =      False        #  Flag for turning on data compression
asis                =         ''        #  Create verbatim copies of these SDM tables in
                                        #   the MS.
scans               =         ''        #  List of scans to fill (default is all scans).
verbose             =      False        #  Output lots of information while the filler is
                                        #   working
overwrite           =      False        #  Over write an existing MS
online              =       True        #  Create online flags
     tbuff          =        0.0        #  Time padding buffer (in seconds)

flagzero            =       True        #  Create flag commands for zero points
     flagpol        =       True        #  Create flag commands for cross-hand
                                        #   correlations

shadow              =       True        #  Create flag commands for shadowed data
     tolerance      =        0.0        #  Amount of shadow allowed (in meters)
     addantenna     =         ''        #  File name or dictionary with additional antenna
                                        #   names, positions and diameters

applyflags          =      False        #  Apply flag commands to MS
savecmds            =      False        #  Save flag commands to an ASCII file
flagbackup          =       True        #  Back up flag column before applying flags

ALERT: If you want to use your JVLA online flags then you must use importevla rather than importasdm. The flagcmd task will process these flags. Also, if you have run importevla in CASA 3.3 or earlier, the flag syntax will be processed by the task oldflagcmd.

The default action of importevla is to construct the FLAG_CMD MS table based on the settings of online, flagzero, and shadow (and sub-parameters). If applyflags=True then these flags will be applied after filling. We recommend you use the flagcmd task after filling to examine these flags and then apply.

See importasdm (§ 2.2.1) for a description of the common parameters. Some differences:

Note that importevla automatically loads in VLA switched power information (unlike in previous versions).

The online parameter controls creation of online flags from the Flag.xml SDM table. The tbuff parameter adds a time “buffer” padding for these flags in both directions to deal with timing mis-matches. ALERT: For JVLA data taken before April 2011, you should set tbuff to a value (in seconds) equal to 1.5× the integration time.

The flagzero parameter controls creation of clipping commands to flag visibilities with amplitudes that are exact zeros. If flagpol=True then it will flag the cross-hands (e.g. RL and LR) as well, which might result in low but correct values of these correlations being thrown out (but can catch erroneous zeros also). ALERT: This facility is provided as the JVLA correlator, particularly in 2010, occasionally produces visibilities with zero or very small values that need to get flagged out.

The shadow parameter turns on creation of flag commands to remove antenna time ranges where they are shadowed by other antennas in the array. By default it will flag based on the antenna diameter, but if you want more lenient or conservative flagging then set the tolerance sub-parameter, where the shadowed antennas are flagged for all baselines that are shorter than radius1 + radius2tolerance (the radii are those for the antennas as listed in the ANTENNA subtable). addantenna can be a file that defines the positions of antennas that are on the ground but do not appear in the MS. They can still shadow antennas in the array.

savecmds will save all flagging commands in the flagdata and flagcmd syntax (§ 3.4 and 3.5) to a file to be applied later or for bookkeeping.

A flag backup can be performed using the flagbackup parameter. It saves all current flags to the ’*.flagversions’ file of the MS, before all new flags are applied.

2.2.3  VLA: Filling data from archive format (importvla)

VLA data in archive format (i.e., as downloaded from the VLA data archive) are read into CASA from disk using the importvla task. The inputs are:

#  importvla :: import VLA archive file(s) to a measurement set:

archivefiles  =         ''   #  Name of input VLA archive file(s)
vis           =         ''   #  Name of output visibility file
bandname      =         ''   #  VLA frequency band name:''=>obtain all bands in archive files
frequencytol  =   150000.0   #  Frequency shift to define a unique spectral window (Hz)
project       =         ''   #  Project name:  '' => all projects in file
starttime     =         ''   #  start time to search for data
stoptime      =         ''   #  end time to search for data
applytsys     =       True   #  apply nominal sensitivity scaling to data & weights
autocorr      =      False   #  import autocorrelations to ms, if set to True
antnamescheme =      'new'   #   'old' or 'new'; 'VA04' or '4' for ant 4
keepblanks    =      False   #  Fill scans with empty source names (e.g. tipping scans)?
evlabands     =      False   #  Use updated eVLA frequencies and bandwidths

The main parameters are archivefiles to specify the input VLA Archive format file names, and vis to specify the output MS name.

ALERT: The scaling of VLA data both before and after the June 2007 Modcomp-turnoff is fully supported, based on the value of applytsys.

The NRAO Archive is located at:

Note that archivefiles takes a string or list of strings, as there are often multiple files for a project in the archive.

For example:

   archivefiles = ['AP314_A950519.xp1','AP314_A950519.xp2']
   vis = 'NGC7538.ms'

The importvla task allows selection on the frequency band. Suppose that you have 1.3 cm line observations in K-band and you have copied the archive data files AP314_A95019.xp* to your working directory and started casa. Then,

   default('importvla')
   archivefiles = ['AP314_A950519.xp1','AP314_A950519.xp2','AP314_A950519.xp3']
   vis = 'ngc7538.ms' 
   bandname = 'K' 
   frequencytol = 10e6
   importvla()

If the data is located in a different directory on disk, then use the full path name to specify each archive file, e.g.:

archivefiles=['/home/rohir2/jmcmulli/ALMATST1/Data/N7538/AP314_A950519.xp1',\
     '/home/rohir2/jmcmulli/ALMATST1/Data/N7538/AP314_A950519.xp2',\
     '/home/rohir2/jmcmulli/ALMATST1/Data/N7538/AP314_A950519.xp3']

Important Note: importvla will import the on-line flags (from the VLA system) along with the data. Shadowed antennas will also be flagged. The flags will be put in the MAIN table and thus available to subsequent tasks and tools. If you wish to revert to unflagged data, use flagmanager (§ 3.2) to save the flags (if you wish), and then use flagdata (§ 3.4) with mode=’manualflag’ and unflag=True to toggle off the flags.

The other parameters are:

2.2.3.1  Parameter applytsys

The applytsys parameter controls whether the nominal sensitivity scaling (based on the measured TSYS, with the weights scaled accordingly using the integration time) is applied to the visibility amplitudes or not. If True, then it will be scaled so as to be the same as AIPS FILLM (i.e. approximately in deciJanskys). Note that post-Modcomp data is in raw correlation coefficient and will be scaled using the TSYS values, while Modcomp-era data had this applied online. In all cases importvla will do the correct thing to data and weights based on an internal flag in the VLA Archive file, either scaling it or unscaling based on your choice for applytsys.

If applytsys=True and you see strange behavior in data amplitudes, it may be due to erroneous TSYS values from the online system. You might want to then fill with applytsys=False and look at the correlation coefficients to see if the behavior is as expected.

2.2.3.2  Parameter bandname

The bandname indicates the VLA Frequency band(s) to load, using the traditional bandname codes. These are:

Note that as the transition from the VLA to JVLA progresses, the actual frequency ranges covered by the bands will expand, and additional bands will be added (namely ’S’ from 1-2 GHz and ’A’ from 26.4-40 GHz).

2.2.3.3  Parameter frequencytol

The frequencytol parameter specifies the frequency separation tolerated when assigning data to spectral windows. The default is frequencytol=150000 (Hz). For Doppler tracked data, where the sky frequency changes with time, a frequencytol < 10000 Hz may produce too many unnecessary spectral windows.

2.2.3.4  Parameter project

You can specify a specific project name to import from archive files. The default ’’ will import data from all projects in file(s) archivefiles.

For example for VLA Project AL519:

   project = 'AL519'    # this will work
   project = 'al519'    # this will also work

while project=’AL0519’ will NOT work (even though that is what queries to the VLA Archive will print it as - sorry!).

2.2.3.5  Parameters starttime and stoptime

You can specify start and stop times for the data, e.g.:

   starttime = '1970/1/31/00:00:00'
   stoptime = '2199/1/31/23:59:59'

Note that the blank defaults will load all data fitting other criteria.

2.2.3.6  Parameter autocorr

Note that autocorrelations are filled into the data set if autocorr=True. Generally for the VLA, autocorrelation data is not useful, and furthermore the imaging routine will try to image the autocorrelation data (it assumes it is single dish data) which will swamp any real signal. Thus, if you do fill the autocorrelations, you will have to flag them before imaging.

2.2.3.7  Parameter antnamescheme

The antnamescheme parameter controls whether importvla will try to use a naming scheme where JVLA antennas are prefixed with EA (e.g. ’EA16’) and old VLA antennas have names prefixed with VA (e.g. ’VA11’). Our method to detect whether an antenna is JVLA is not yet perfected, and thus unless you require this feature, simply use antnamescheme=’old’.

2.2.3.8  Parameter evlabands

The evlabands=True option is provided to allow users to access JVLA frequencies outside the standard VLA tunings (e.g. the extended C-band above 6 GHz). ALERT: use of this option for standard VLA data will cause unexpected associations, such as X-band data below 8 GHz being extracted to C-band (as the JVLA C-band is 4–8 GHz). Use with care.

2.2.4  Import ATCA and CARMA data

There are several ways to import data from ATCA and CARMA into CASA. The data from these arrays has historically been processed in MIRIAD. For simple cases (single source and frequency) exporting from MIRIAD to UVFITS format and importing using importuvfits (see § 2.2.7) often works ok, although some fixes to the resulting MeasurementSet may be needed.

The importmiriad task (§ 2.2.5) reads MIRIAD visibility data and can handle multiple frequencies and sources in the input. Since it does not apply any calibration, make sure to apply it beforehand in MIRIAD.

The importatca task (§ 2.2.6) reads the ATCA archive format (RPFITS) directly, avoiding the need to go through MIRIAD to load the data. It can handle ATCA data from both the old and new (CABB) correlator.

2.2.5  Import MIRIAD visibilities (importmiriad)

The task importmiriad allows one to import visibilities in the MIRIAD data format to be converted to a MS. The task has mainly be tested on data from the ATCA and CARMA telescopes and the imputs are:

#  importmiriad :: Convert a Miriad visibility file into a CASA MeasurementSet
mirfile             =         ''        #  Name of input Miriad visibility file
vis                 =         ''        #  Name of output MeasurementSet
tsys                =      False        #  Use the Tsys to set the visibility weights
spw                 =      'all'        #  Select spectral windows
vel                 =         ''        #  Select velocity reference (TOPO,LSRK,LSRD)
linecal             =      False        #  (CARMA) Apply line calibration
wide                =      'all'        #  (CARMA) Select wide window averages
debug               =          0        #  Display increasingly verbose debug messages

The mirfile parameter specifies a single MIRIAD visibility file which should have any calibration done in MIRIAD already applied to it.

Set the tsys parameter to True to change the visibility weights from the MIRIAD default (usually the integration time) to the inverse of the noise variance using the recorded system temperature.

The spw parameter can be used to select all or some of the simultaneous spectral windows from the input file. Use the default of ’all’ for all the data or use e.g., spw=’0,2’ to select the first and third window.

The vel parameter can be used to set the output velocity frame reference. For ATCA this defaults to ’TOPO’ and for CARMA it defaults to ’LSRK’. Only change this if your data comes out with the incorrect velocity.

The linecal parameter is only useful for CARMA data and can apply the line calibration if it is stored with the MIRIAD data.

The wide parameter is only useful for CARMA data and can select which of the wide-band channels should be loaded.

2.2.6  Import ATCA RPFITS data (importatca)

The data from the ATCA is available from the archive in files in the RPFITS format. These files can be imported into CASA with the importatca task.

#  importatca :: Import ATCA RPFITS file(s) to a measurement set
files               =['*.C1234']        #  Name of input ATCA RPFits file(s)
vis                 = 'c1234.ms'        #  Name of output visibility file
                                        #   (MeasurementSet)
options             =         ''        #  Processing options: birdie, reweight,
                                        #   noxycorr, fastmosaic, hires, noac
                                        #   (comma separated list)
spw                 =       [-1]        #  Specify the spectral windows to use,
                                        #   default=all
nscans              =     [0, 0]        #  Number of scans to skip followed by
                                        #   number of scans to read
lowfreq             =   '0.1GHz'        #  Lowest reference frequency to select
highfreq            =   '999GHz'        #  Highest reference frequency to select
fields              =       ['']        #  List of field names to select
edge                =          8        #  Percentage of edge channels to flag.
                                        #   For combined zooms, this specifies
                                        #   the percentage for a single zoom
                                        #   window

The files parameter can take a string or a list of strings as input and also allows the use of wildcards as shown in the example above.

For older ATCA continuum data (before the CABB correlator, April 2009) use options=’birdie,reweight’ to suppress internally generated RFI.

The options parameter:

The spw parameter takes a list of integers and can be used to select one or more of the simultaneous frequencies. With CABB there can be up to 34 spectra. The order of the frequency bands in the RPFITS file is: the two continuum bands (0 and 1), followed by the zoom bands for the first frequency and then the zoom bands for the second frequency. Note that this spw parameter does not take a string with wildcards. Use spw=-1 to get all the data.

The nscans parameter can be used to select part of a file, e.g., to retrieve a few test scans for a quick look.

The lowfreq and highfreq parameters select data based on the reference frequency.

The fields parameter selects data based on the field/source name.

The edge parameter specifies how many edge channels to discard as a percentage of the number of channels in each band. E.g., the default value of 8 will discard 82 channels from the top and bottom of a 2048 channel spectrum.

2.2.7  UVFITS Import and Export

The UVFITS format is not exactly a standard, but is a popular archive and transport format nonetheless. CASA supports UVFITS files written by the AIPS FITTP task, and others.

UVFITS is supported for both import and export.

2.2.7.1  Import using importuvfits

To import UVFITS format data into CASA, use the importuvfits task:

CASA <1>: inp(importuvfits)
fitsfile            =         ''  # Name of input UVFITS file
vis                 =         ''  # Name of output visibility file (MS)
antnamescheme       =      'old'  # For VLA only; 'new' or 'old'; 'VA04' or '04' for VLA ant 4

This is straightforward, since all it does is read in a UVFITS file and convert it as best it can into a MS.

For example:

   importuvfits(fitsfile='NGC5921.fits',vis='ngc5921.ms')

ALERT: CARMA data can be loaded into CASA. However,

tb.open("c0104I/ANTENNA",nomodify=False)
namelist=tb.getcol("NAME").tolist()
for i in range(len(namelist)):
 name = 'CA'+namelist[i]
 print ' Changing '+namelist[i]+' to '+name
 namelist[i]=name
 
tb.putcol("NAME",namelist)
tb.close()

2.2.7.2  Import using importfitsidi

Some uvfits data is written in the FITS-IDI standard. Those files can be imported into CASA with the importfitsidi task:

#  importfitsidi :: Convert a FITS-IDI file to a CASA visibility data set
fitsidifile         =       ['']        #  Name(s) of input FITS-IDI file(s)
vis                 = 'ngc5921.demo.ms' #  Name of output visibility file (MS)
constobsid          =      False        #  If True, give constant obs ID==0 to
                                        #   the data from all input fitsidi
                                        #   files (False = separate obs id for
                                        #   each file)
scanreindexgap_s    =        0.0        #  min time gap (seconds) between
                                        #   integrations to start a new scan

The constobs parameter can be used to give all visibilities the same observation id of 0. scanreindexgap_s controls the gap that defines different scans.

Example:

   importuvfits(fitsidifile='NGC1300.fits',vis='NGC1300.ms')

2.2.7.3  Export using exportuvfits

The exportuvfits task will take a MS and write it out in UVFITS format. The defaults are:

#  exportuvfits :: Convert a CASA visibility data set to a UVFITS file:
vis                 =         ''        #  Name of input visibility file
fitsfile            =         ''        #  Name of output UV FITS file
datacolumn          = 'corrected'       #  Visibility file data column
field               =         ''        #  Select field using field id(s) or field name(s)
spw                 =         ''        #  Select spectral window/channels
antenna             =         ''        #  Select data based on antenna/baseline
timerange           =         ''        #  Select data based on time range
avgchan             =          1        #  Channel averaging width (value > 1 indicates averaging)
writesyscal         =      False        #  Write GC and TY tables, (Not yet available)
multisource         =       True        #  Write in multi-source format
combinespw          =       True        #  Export the spectral windows as IFs
     padwithflags   =       True        #  Fill in missing data with flags to fit IFs

writestation        =       True        #  Write station name instead of antenna name
overwrite           =      False        #  Overwrite output file if it exists?

For example:

   exportuvfits(vis='ngc5921.split.ms',
                fitsfile='NGC5921.split.fits',
                multisource=False)

The MS selection parameters field, spw, antenna, and timerange follow the standard selection syntax described in § 2.3.

ALERT: The nchan, start, and width parameters will be superseded by channel selection in spw. Currently, there is a time parameter rather than timerange.

The datacolumn parameter chooses which data-containing column of the MS (see § 2.1.1) is to be written out to the UV FITS file. Choices are: ’data’, ’corrected’, and ’model’.

There are a number of special parameters that control what is written out. These are mostly here for compatibility with AIPS.

The writesyscal parameter toggles whether GC and TY extension tables are written. These are important for VLBA data, and for JVLA data. ALERT: Not yet available.

The multisource parameter determines whether the UV FITS file is a multi-source file or a single-source file, if you have a single-source MS or choose only a single source. Note: the difference between a single-source and multi-source UVFITS file here is whether it has a source (SU) table and the source ID in the random parameters. Some programs (i.e. difmap) only accept single-source files. If you select more than one source in fields, then the multisource parameter will be overridden to be True regardless.

The combinespw parameter allows, if some conditions are met, exporting all of spectral windows (SpW) as a set of "IF"s in a single "FREQID" setup instead of giving each SpW its own FREQID in the FITS file. In this context an IF (Intermediate Frequency) is a specialization of an SpW, where each IF in a UV FITS file must have the same number of channels and polarizations, each channel must have the same width, and each IF must be present (even if flagged) throughout the entire observation. If these conditions are not met the data must be exported using multiple FREQIDs, the UV FITS equivalent of a general SpW. This matters since many (sub)programs will work with multiple IFs, but not multiple FREQIDs. For example, a UV FITS file with multiple FREQIDs can be read by AIPS, but you may find that you have to separate the FREQIDs with SPLIT before you can do very much with them. Therefore combinespw=True should be True if possible. Typically MSes where each band was observed simultaneously can be exported with combinespw=True. MSes where the tuning changed with time, e.g. 10 minutes at 4.8 GHz followed by 15 minutes at 8.4 GHz, should be exported to multiple UV FITS files using spw to select one tuning (set of simultaneous SpWs) per file.

The multisource parameter determines whether the UV FITS file is a multi-source file or a single-source file, if you have a single-source MS or choose only a single source. Note: the difference between a single-source and multi-source UVFITS file here is whether it has a source (SU) table and the source ID in the random parameters. If you select more than one source in fields, then the multisource parameter will be overridden to be True regardless.

The combinespw parameter allows combination of all spectral windows at one time. If True, then all spectral windows must have the same shape. For AIPS to read an exported file, then set combinespw=True.

The writestation parameter toggles the writing of the station name instead of antenna name.

2.2.8  Handling Measurement Set metadata and data

There are tasks provided for basic listing and manipulation of Measurement Set data and metadata. These include:

2.2.9  Summarizing your MS (listobs)

Once you import your data into a CASA Measurement Set, you can get a summary of the MS contents with the listobs task.

The inputs are:



#  listobs :: List the summary of a data set in the logger or in a file
vis                 = 'day2_TDEM0003_10s_norx' #  Name of input visibility file (MS)
selectdata          =       True        #  Data selection parameters
     field          =         ''        #  Field names or field index
                                        #  numbers: '' ==>all, field='0~2,3C286'
     spw            =         ''        #  spectral-window/frequency/channel
     antenna        =         ''        #  antenna/baselines: ''==>all, antenna ='3,VA04'
     timerange      =         ''        #  time range: ''==>all,timerange='09:14:0~09:54:0'
     correlation    =         ''        #  Select data based on correlation
     scan           =         ''        #  scan numbers: ''==>all
     intent         =         ''        #  Select data based on observation intent: ''==>all
     feed           =         ''        #  multi-feed numbers: Not yet implemented
     array          =         ''        #  (sub)array numbers: ''==>all
     uvrange        =         ''        #  uv range: ''==>all; uvrange
                                        #  ='0~100klambda', default units=meters
     observation    =         ''        #  Select data based on observation ID: ''==>all

verbose             =       True        
listfile            =         ''        #  Name of disk file to write output: ''==>to terminal
listunfl            =      False        #  List unflagged row counts?
                                        #  If true, it can have significant negative performance impact.

The summary (of the selected data) will be written to the logger, to the casapy-YYYYMMDD-HHMMSS.log file, and optionally to a file specified in the listfile parameter. For example,

   listobs('n5921.ms')

results in the logger messages:

listobs(vis="day2_TDEM0003_10s_norx",selectdata=True,spw="",field="",
        antenna="",uvrange="",timerange="",correlation="",scan="",
        intent="",feed="",array="",observation="",verbose=True,
        listfile="",listunfl=False)
================================================================================
           MeasurementSet Name:  /Users/jott/casa/casatest/casa4.0/irc/day2_TDEM0003_10s_norx      MS Version 2
================================================================================
   Observer: Mark J. Mark Claussen     Project: T.B.D.  
Observation: EVLA
Data records: 290218       Total integration time = 10016 seconds
   Observed from   26-Apr-2010/03:21:56.0   to   26-Apr-2010/06:08:52.0 (UTC)
   
   ObservationID = 0         ArrayID = 0
  Date        Timerange (UTC)          Scan  FldId FieldName             nRows     SpwIds   Average Interval(s)    ScanIntent
  26-Apr-2010/03:21:51.0 - 03:23:21.0     5      2 J0954+1743                2720  [0, 1]  [10, 10] 
              03:23:39.0 - 03:28:25.0     6      3 IRC+10216                 9918  [0, 1]  [10, 10] 
              03:28:38.0 - 03:29:54.0     7      2 J0954+1743                2700  [0, 1]  [10, 10] 
              03:30:08.0 - 03:34:53.5     8      3 IRC+10216                 9918  [0, 1]  [10, 10] 
...
           (nRows = Total number of rows per scan) 
Fields: 4
  ID   Code Name                RA               Decl           Epoch   SrcId      nRows
  2    D    J0954+1743          09:54:56.823626 +17.43.31.22243 J2000   2          65326
  3    NONE IRC+10216           09:47:57.382000 +13.16.40.65999 J2000   3         208242
  5    F    J1229+0203          12:29:06.699729 +02.03.08.59820 J2000   5          10836
  7    E    J1331+3030          13:31:08.287984 +30.30.32.95886 J2000   7           5814
Spectral Windows:  (2 unique spectral windows and 1 unique polarization setups)
  SpwID  Name      #Chans   Frame   Ch1(MHz)  ChanWid(kHz)  TotBW(kHz)  Corrs          
  0      Subband:0     64   TOPO   36387.229       125.000      8000.0  RR  RL  LR  LL
  1      Subband:0     64   TOPO   36304.542       125.000      8000.0  RR  RL  LR  LL
Sources: 10
  ID   Name                SpwId RestFreq(MHz)  SysVel(km/s) 
  0    J1008+0730          0     0.03639232     -0.026       
  0    J1008+0730          1     0.03639232     -0.026       
  2    J0954+1743          0     0.03639232     -0.026       
  2    J0954+1743          1     0.03639232     -0.026       
  3    IRC+10216           0     0.03639232     -0.026       
  3    IRC+10216           1     0.03639232     -0.026       
  5    J1229+0203          0     0.03639232     -0.026       
  5    J1229+0203          1     0.03639232     -0.026       
  7    J1331+3030          0     0.03639232     -0.026       
  7    J1331+3030          1     0.03639232     -0.026       
Antennas: 19:
  ID   Name  Station   Diam.    Long.         Lat.                Offset from array center (m)                ITRF Geocentric coordinates (m)        
                                                                     East         North     Elevation               x               y               z
  0    ea01  W09       25.0 m   -107.37.25.2  +33.53.51.0       -521.9407     -332.7782       -1.1977 -1601710.017000 -5042006.928200  3554602.355600
  1    ea02  E02       25.0 m   -107.37.04.4  +33.54.01.1          9.8247      -20.4292       -2.7808 -1601150.059500 -5042000.619800  3554860.729400
  2    ea03  E09       25.0 m   -107.36.45.1  +33.53.53.6        506.0591     -251.8666       -3.5832 -1600715.948000 -5042273.187000  3554668.184500
  3    ea04  W01       25.0 m   -107.37.05.9  +33.54.00.5        -27.3562      -41.3030       -2.7418 -1601189.030140 -5042000.493300  3554843.425700
  4    ea05  W08       25.0 m   -107.37.21.6  +33.53.53.0       -432.1158     -272.1493       -1.5032 -1601614.091000 -5042001.655700  3554652.509300
  5    ea07  N06       25.0 m   -107.37.06.9  +33.54.10.3        -54.0667      263.8720       -4.2292 -1601162.593200 -5041829.000000  3555095.890500
  6    ea08  N01       25.0 m   -107.37.06.0  +33.54.01.8        -30.8810       -1.4664       -2.8597 -1601185.634945 -5041978.156586  3554876.424700
  7    ea09  E06       25.0 m   -107.36.55.6  +33.53.57.7        236.9058     -126.3369       -2.4443 -1600951.588000 -5042125.911000  3554773.012300
  8    ea12  E08       25.0 m   -107.36.48.9  +33.53.55.1        407.8394     -206.0057       -3.2252 -1600801.916000 -5042219.371000  3554706.449900
  9    ea15  W06       25.0 m   -107.37.15.6  +33.53.56.4       -275.8288     -166.7451       -2.0590 -1601447.198000 -5041992.502500  3554739.687600
  10   ea19  W04       25.0 m   -107.37.10.8  +33.53.59.1       -152.8599      -83.8054       -2.4614 -1601315.893000 -5041985.320170  3554808.304600
  11   ea20  N05       25.0 m   -107.37.06.7  +33.54.08.0        -47.8454      192.6015       -3.8723 -1601168.786100 -5041869.054000  3555036.936000
  12   ea21  E01       25.0 m   -107.37.05.7  +33.53.59.2        -23.8638      -81.1510       -2.5851 -1601192.467800 -5042022.856800  3554810.438800
  13   ea22  N04       25.0 m   -107.37.06.5  +33.54.06.1        -42.5986      132.8623       -3.5431 -1601173.953700 -5041902.660400  3554987.536500
  14   ea23  E07       25.0 m   -107.36.52.4  +33.53.56.5        318.0523     -164.1848       -2.6960 -1600880.570000 -5042170.388000  3554741.457400
  15   ea24  W05       25.0 m   -107.37.13.0  +33.53.57.8       -210.0944     -122.3885       -2.2581 -1601377.008000 -5041988.665500  3554776.393400
  16   ea25  N02       25.0 m   -107.37.06.2  +33.54.03.5        -35.6245       53.1806       -3.1345 -1601180.861480 -5041947.453400  3554921.628700
  17   ea27  E03       25.0 m   -107.37.02.8  +33.54.00.5         50.6647      -39.4832       -2.7249 -1601114.365500 -5042023.153700  3554844.945600
  18   ea28  N08       25.0 m   -107.37.07.5  +33.54.15.8        -68.9057      433.1889       -5.0602 -1601147.940400 -5041733.837000  3555235.956000


using the (default) verbose=True option. The most useful extra information that verbose=True gives is the list of the scans in the dataset.

2.2.10  MMS summary (listpartition)

Similar to listobs, listpartition shows the summary of a Multi-Measurement Set (MMS).

The inputs are:

#  listpartition :: List the summary of a multi-MS data set in the logger or in a file
vis                 =         ''        #  Name of multi-MS or normal MS.
createdict          =      False        #  Create and return a dictionary with
                                        #   sub-MS information
listfile            =         ''        #  Name of ASCII file to save output:
                                        #   ''==>to terminal

For example,

   listpartition('n5921.mms')

results in the logger messages:

This is a multi-MS with separation axis = scan,spw
Sub-MS               Scan  Spw    Nchan  Nrows   Size  
ngc5921.mms.0000.ms  2     [0]    [63]   1890    27M   
                     4     [0]    [63]   756           
                     5     [0]    [63]   1134          
                     6     [0]    [63]   6804          
ngc5921.mms.0001.ms  1     [0]    [63]   4509    28M   
                     3     [0]    [63]   6048          
                     7     [0]    [63]   1512          

2.2.11  Listing MS data (listvis)

The listvis task will print to the terminal (or file) listing of the data in your MS. The inputs are:

#  listvis :: List measurement set visibilities.
vis                 =         ''        #  Name of input visibility file
options             =       'ap'        #  List options: ap only
datacolumn          =     'data'        #  Column to list: data, float_data, corrected, model,
                                        #   residual
field               =         ''        #  Field names or index to be listed: ''==>all
spw                 =        '*'        #  Spectral window:channels: '\*'==>all, spw='1:5~57'
selectdata          =      False        #  Other data selection parameters
observation         =         ''        #  Select by observation ID(s)
average             =         ''        #  Averaging mode: ==>none (Not yet implemented)
showflags           =      False        #  Show flagged data (Not yet implemented)
pagerows            =         50        #  Rows per page
listfile            =         ''        #  Output file

For example:

Units of columns are: Date/Time(YYMMDD/HH:MM:SS UT), UVDist(wavelength), Phase(deg), UVW(m)
WEIGHT: 7
FIELD: 2
SPW: 0
Date/Time:                           RR:                 RL:                 LR:                 LL:                                             
2010/04/26/      Intrf UVDist  Chn    Amp     Phs  Wt F   Amp     Phs  Wt F   Amp     Phs  Wt F   Amp     Phs  Wt F         U         V         W
------------|---------|------|----|--------------------|-------------------|-------------------|-------------------|---------|---------|---------|
  03:21:56.0 ea01-ea02  72363    0: 0.005  -124.5   7   0.005    25.7   7   0.001   104.6   7   0.000    23.4   7     -501.93   -321.75    157.78
  03:21:56.0 ea01-ea02  72363    1: 0.001    -4.7   7   0.001  -135.1   7   0.004   -14.6   7   0.001    19.9   7     -501.93   -321.75    157.78
  03:21:56.0 ea01-ea02  72363    2: 0.002    17.8   7   0.002    34.3   7   0.005  -114.3   7   0.005  -149.7   7     -501.93   -321.75    157.78
  03:21:56.0 ea01-ea02  72363    3: 0.004   -19.4   7   0.003   -79.2   7   0.002   -89.0   7   0.004    31.3   7     -501.93   -321.75    157.78
  03:21:56.0 ea01-ea02  72363    4: 0.001   -16.8   7   0.004  -141.5   7   0.005   114.9   7   0.006   105.2   7     -501.93   -321.75    157.78
  03:21:56.0 ea01-ea02  72363    5: 0.001   -29.8   7   0.009   -96.4   7   0.002  -125.0   7   0.002   -64.5   7     -501.93   -321.75    157.78
...
Type Q to quit, A to toggle long/short list, or RETURN to continue [continue]: 

ALERT: We are working on improving the format of the listvis output.

2.2.12  Listing and manipulating MS metadata (vishead)

The vishead task is provided to access keyword information in the Measurement Set. The default inputs are:

#  vishead :: List, get, and put metadata in a measurement set
vis            =          ''   #  Name of input visibility file
mode           =      'list'   #  options: list, summary, get, put
   listitems   =          []   #  items to list ([] for all)

The mode = ’summary’ option just gives the same output as listobs.

For mode = ’list’ the options are: ’telescope’, ’observer’, ’project’, ’field’, ’freq_group_name’, ’spw_name’, ’schedule’, ’schedule_type’, ’release_date’.

CASA <29>: vishead('ngc5921.demo.ms',mode='list',listitems=[])
  Out[29]: 
{'cal_grp': (array([-1, -1, -1], dtype=int32), {}),
 'field': (array(['1331+30500002_0', '1445+09900002_0', 'N5921_2'], 
      dtype='|S16'),
           {}),
 'fld_code': (array(['C', 'A', ''], 
      dtype='|S2'), {}),
 'freq_group_name': (array(['none'], 
      dtype='|S5'), {}),
 'log': ({'r1': False}, {}),
 'observer': (array(['TEST'], 
      dtype='|S5'), {}),
 'project': (array([''], 
      dtype='|S1'), {}),
 'ptcs': ({'r1': array([[[-2.74392758]],

       [[ 0.53248521]]]),
           'r2': array([[[-2.42044692]],

       [[ 0.17412604]]]),
           'r3': array([[[-2.26020138]],

       [[ 0.08843002]]])},
          {'MEASINFO': {'Ref': 'J2000', 'type': 'direction'},
           'QuantumUnits': array(['rad', 'rad'], 
      dtype='|S4')}),
 'release_date': (array([  4.30444800e+09]),
                  {'MEASINFO': {'Ref': 'TAI', 'type': 'epoch'},
                   'QuantumUnits': array(['s'], 
      dtype='|S2')}),
 'schedule': ({'r1': False}, {}),
 'schedule_type': (array([''], 
      dtype='|S1'), {}),
 'source_name': (array(['1331+30500002_0', '1445+09900002_0', 'N5921_2'], 
      dtype='|S16'),
                 {}),
 'spw_name': (array(['none'], 
      dtype='|S5'), {}),
 'telescope': (array(['VLA'], 
      dtype='|S4'), {})}

You can use mode=’get’ to retrieve the values of specific keywords, and likewise mode=’put’ to change them. The inputs are:

mode           =      'get'    #  options: list, summary, get, put
   hdkey       =         ''    #  keyword to get/put
   hdindex     =         ''    #  keyword index to get/put, counting from zero. ==>all

and

#  vishead :: List, summary, get, and put metadata in a measurement set
mode           =      'put'    #  options: list, summary, get, put
   hdkey       =         ''    #  keyword to get/put
   hdindex     =         ''    #  keyword index to get/put, counting from zero. ==>all
   hdvalue     =         ''    #  value of hdkey

For example, a common operation is to change the Telescope name (e.g. if it is unrecognized), e.g.

CASA <36>: vishead('ngc5921.demo.ms',mode='get',hdkey='telescope')
  Out[36]: 
(array(['VLA'], 
      dtype='|S4'), {})

CASA <37>: vishead('ngc5921.demo.ms',mode='put',hdkey='telescope',hdvalue='JVLA')

CASA <38>: vishead('ngc5921.demo.ms',mode='get',hdkey='telescope')
  Out[38]: 
(array(['JVLA'], 
      dtype='|S5'), {})

2.2.13  MS statistics (visstat)

ALERT: This is still a prototype task.

The visstat task is provided to obtain simple statistics for a Measurement Set, useful in regression tests.

The inputs are:

#  visstat :: Displays statistical information from a measurement set
vis              =      ''   #  Name of input visibility file
axis             =   'amp'   #  Which values to use
   datacolumn    =  'data'   #  Which data column to use (data, corrected, model)

useflags         =    True   #  Take flagging into account?
spw              =      ''   #  spectral-window/frequency/channel
field            =      ''   #  Field names or field index numbers: ''==>all, field='0~2,3C286'
selectdata       =    True   #  More data selection parameters (antenna, timerange etc)
   antenna       =      ''   #  antenna/baselines: ''==>all, antenna = '3,VA04'
   timerange     =      ''   #  time range: ''==>all, timerange='09:14:0~09:54:0'
   correlation   =      ''   #  Select data based on correlation
   scan          =      ''   #  scan numbers: ''==>all
   array         =      ''   #  (sub)array numbers: ''==>all
   uvrange       =      ''   #  uv range: ''==>all; uvrange = '0~100klambda', default units=meters

Running this task returns a record (Python dictionary) with the statistics, which can be captured in a Python variable. For example,

CASA <42>: mystat = visstat('ngc5921.demo.ms',axis='amp',datacolumn='corrected',field='0')

CASA <43>: mystat
  Out[43]: 
{'CORRECTED': {'max': 51.938671112060547,
               'mean': 14.796444141750133,
               'medabsdevmed': 0.28020858764648438,
               'median': 14.764373779296875,
               'min': 0.81362706422805786,
               'npts': 514916.0,
               'quartile': 0.56053066253662109,
               'rms': 14.829294204711914,
               'stddev': 0.98650836609147285,
               'sum': 7618925.8316934109,
               'sumsq': 113234125.12642419,
               'var': 0.97319875636846753}}

CASA <44>: print mystat['CORRECTED']['stddev']
0.986508366091

The options for axis are:

    axis='amplitude'      #    or ('amp')
    axis='phase'
    axis='imag'
    axis='scan_number'
    axis='flag'

The phase of a complex number is in radians with range (−π, π).

2.2.14  Concatenating multiple datasets (concat)

Once you have your data in the form of CASA Measurement Sets, you can go ahead and process your data using the editing, calibration, and imaging tasks. In some cases, you will most efficiently operate on single MS for a particular session (such as calibration). Other tasks will (eventually) take multiple Measurement Sets as input. For others, it is easiest to combine your multiple data files into one.

If you need to combine multiple datasets, you can use the concat task. The default inputs are:

#  concat :: Concatenate several visibility data sets.
vis                 =         ''        #  Name of input visibility files to be
                                        #   concatenated
concatvis           =         ''        #  Name of output visibility file
freqtol             =         ''        #  Frequency shift tolerance for considering data
                                        #   as the same spwid
dirtol              =         ''        #  Direction shift tolerance for considering data
                                        #   as the same field
respectname         =      False        #  If true, fields with a different name are not
                                        #   merged even if their direction agrees
timesort            =      False        #  If true, sort by TIME in ascending order
copypointing        =       True        #  Copy all rows of the POINTING table.
visweightscale      =         []        #  List of the weight scaling factors to be
                                        #   applied to the individual MSs

The vis parameter will take a list of one or more MS. Usually, this will contain all the MS to combine. concat will presort the visibilities in time.

With visweightscale, a list of weights can be manually specified for the respective input data sets. They will be applied at the time of the combination. To determine the appropriate weights for this procedure, one can inspect the weights (Wt and WtSp axis parameters) of the input datasets in plotms.

The concatvis parameter contains the name of the output MS. If this points to an existing file on disk, then the MS in vis will appended to it, otherwise a new MS file is created to contain the concatenated data. Be careful here!

The timesort parameter can be used to make sure the output MS is in time order (e.g. if your input MS have concurrent times). This can possibly speed up some subsequent calibration operations.

Furthermore, the parameter copypointing can be used to control whether the POINTING table will be carried along in the concatenation process or if the output MS should not contain a POINTING table. This table is quite large for some data (e.g. ALMA) and is mainly needed for mosaic imaging. If you are certain that you will not need it, you can save time and diskspace by setting copypointing to False.

importasdm will fill the correct coordinates from ephemerides data into the SOURCE table. And, as stated in the ephemeris handling section 4.7.11.3, concat will correctly merge fields which use the same ephemeris.

The parameters freqtol and dirtol control how close together in frequency and angle on the sky spectral windows or field locations need to be before calling them the same.

ALERT: Note that if multiple frequencies or pointings are combined using freqtol or dirtol, then the data are not changed (i.e. not rephased to the single phase center). Use of these parameters is intended to be tolerant of small offsets (e.g. planets tracked which move slightly in J2000 over the course of observations, or combining epochs observed with slightly different positions).

For example:

   default('concat')
   vis = ['n4826_16apr.split.ms','n4826_22apr.split.ms']
   concatvis = 'n4826_tboth.ms'
   freqtol = '50MHz'
   visweightscale=['1',2']
   concat()

combines the two days in ’n4826_16apr.split.ms’ and ’n4826_22apr.split.ms’ into a new output MS called ’n4826_tboth.ms’, and the second MS is weighted twice the first one.

ALERT: Note that if you are concatenating MSs which use antennas which were moved between observations, the MS definition does only foresee a unique antenna ID, but not a unique name(!). The moved antenna will appear twice in the antenna list under the same name but on different stations and with two different IDs. The pair (’NAME@STATION’) will be the unique identifier.

If you would like to only concatenate the subtables of several MSs, not the bulk visibility data, you can use the task testconcat instead of concat to save time and diskspace. testconcat has the same parameters as concat. It produces an output MS with the concatenated subtables and an empty Main table.

Furthermore, the task virtualconcat permits to concatenate MSs into a multi-MS (MMS, see chapter 10) which is much faster as the data is moved into the MMS rather than copied and only some reindexing is done. The bulk data is not rewritten. If you want to keep a copy of the original MSs, set the parameter keepcopy of virtualconcat to True. The creation of that copy will of course consume some of the time you saved by doing a virtual concatenation. Otherwise virtualconcat offers the same functionality as concat.

2.3  Data Selection

Once in MS form, subsets of the data can be operated on using the tasks and tools. In CASA, there are three common data selection parameters used in the various tasks: field, spw, and selectdata. In addition, the selectdata parameter, if set to True, will open up a number of other sub-parameters for selection. The selection operation is unified across all the tasks. The available selectdata parameters may not be the same in all tasks. But if present, the same parameters mean the same thing and behave in the same manner when used in any task.

For example:

field               =         ''        #   field names or index of calibrators ''==>all
spw                 =         ''        #   spectral window:channels: ''==>all
selectdata          =      False        #   Other data selection parameters

versus

field               =         ''        #   field names or index of calibrators ''==>all
spw                 =         ''        #   spectral window:channels: ''==>all
selectdata          =       True        #   Other data selection parameters
     timerange      =         ''        #   time range: ''==>all 
     uvrange        =         ''        #   uv range''=all
     antenna        =         ''        #   antenna/baselines: ''==>all
     scan           =         ''        #   scan numbers: Not yet implemented
     msselect       =         ''        #   Optional data selection (Specialized. but see help)

The following are the general syntax rules and descriptions of the individual selection parameters of particular interest for the tasks:

2.3.1  General selection syntax

Most of the selections are effected through the use of selection strings. This sub-section describes the general rules used in constructing and parsing these strings. Note that some selections are done through the use of numbers or lists. There are also parameter-specific rules that are described under each parameter.

All lists of basic selection specification-units are comma separated lists and can be of any length. White-spaces before and after the commas (e.g. ’3C286, 3C48, 3C84’) are ignored, while white-space within sub-strings is treated as part of the sub-string (e.g. ’3C286, VIRGO A, 3C84’). In some cases, spaces need to be quoted, e.g. "’spw 1’" (note the double quote around the single quotes).

All integers can be of any length (in terms of characters) composed of the characters 0–9. Floating point numbers can be in the standard format (DIGIT.DIGIT, DIGIT., or .DIGIT) or in the mantissa-exponent format (e.g. 1.4e9). Places where only integers make sense (e.g. IDs), if a floating point number is given, only the integer part is used (it is truncated).

Range of numbers (integers or real numbers) can be given in the format 'N0~N1'. For integer ranges, it is expanded into a list of integers starting from N0 (inclusive) to N1 (inclusive). For real numbers, it is used to select all values present for the appropriate parameter in the Measurement Set between N0 and N1 (including the boundaries). Note that the '~' character is used rather than the more obvious ’-’ in order to accommodate hyphens in strings and minus signs in numbers.

Wherever appropriate, units can be specified. The units are used to convert the values given to the units used in the Measurement Set. For ranges, the unit is specified only once (at the end) and applies to both the range boundaries.

2.3.1.1  String Matching

String matching can be done in three ways. Any component of a comma separated list that cannot be parsed as a number, a number range, or a physical quantity is treated as a regular expression or a literal string. If the string does not contain the characters ’*’, ’{’, ’}’ or ’?’, it is treated as a literal string and used for exact matching. If any of the above mentioned characters are part of the string, they are used as a regular expression. As a result, for most cases, the user does not need to supply any special delimiters for literal strings and/or regular expressions. For example:

  field = '3'     # match field ID 3 and not select field named "3C286".

  field = '3*'    # used as a pattern and matched against field names. If
                  # names like "3C84", "3C286", "3020+2207" are found,
                  # all will match. Field ID 3 will not be selected
                  # (unless of course one of the above mentioned field
                  # names also correspond to field ID 3!). 

  field = '30*'   # will match only with "3020+2207" in above set. 

However if it is required that the string be matched exclusively as a regular expression, it can be supplied within a pair of ’/’ as delimiters (e.g. ’/.+BAND.+/’). A string enclosed within double quotes (’"’) is used exclusively for pattern matching (patterns are a simplified form of regular expressions - used in most UNIX commands for string matching). Patterns are internally converted to equivalent regular expressions before matching. See the Unix command "info regex", or visit http://www.regular-expressions.info, for details of regular expressions and patterns.

Strings can include any character except the following:

     ','   ';'   '"'  '/'   NEWLINE

(since these are part of the selection syntax). Strings that do not contain any of the characters used to construct regular expressions or patterns are used for exact matches. Although it is highly discouraged to have name in the MS containing the above mentioned reserved characters, if one does choose to include the reserved characters as parts of names etc., those names can only be matched against quoted strings (since regular expression and patterns are a super-set of literal strings – i.e., a literal string is also a valid regular expression).

This leaves ’"’, ’*’, ’{’, ’}’ or ’?’ as the list of printable character that cannot be part of a name (i.e., a name containing this character can never be matched in a MSSelection expression). These will be treated as pattern-matching even inside double double quotes (’" "’). There is currently no escape mechanism (e.g. via a backslash).

Some examples of strings, regular expressions, and patterns:

2.3.2  The field Parameter

The field parameter is a string that specifies which field names or ids will be processed in the task or tool. The field selection expression consists of comma separated list of field specifications inside the string.

Field specifications can be literal field names, regular expressions or patterns (see § 2.3.1.1). Those fields for which the entry in the NAME column of the FIELD MS sub-table match the literal field name/regular expression/pattern are selected. If a field name/regular expression/pattern fails to match any field name, the given name/regular expression/pattern are matched against the field code. If still no field is selected, an exception is thrown.

Field specifications can also be given by their integer IDs. IDs can be a single or a range of IDs. Field ID selection can also be done as a boolean expression. For a field specification of the form ’>ID’, all field IDs greater than ID are selected. Similarly for ’<ID’ all field IDs less than the ID are selected.

For example, if the MS has the following observations:

MS summary:
==========
FIELDID   SPWID   NChan       Pol        NRows     Source Name
---------------------------------------------------------------
 0          0      127         RR        10260     0530+135
 1          0      127         RR        779139    05582+16320
 2          0      127         RR        296190    05309+13319
 3          0      127         RR        58266     0319+415
 4          0      127         RR        32994     1331+305
 5          1       1      RR,RL,LL,RR   23166     KTIP

one might select

  field = '0~2,KTIP'          # FIELDID 0,1,2 and field name KTIP
  field = '0530+135'          # field 0530+135
  field = '05*'               # fields 0530+135,05582+16320,05309+13319

2.3.3  The spw Parameter

The spw parameter is a string that indicates the specific spectral windows and the channels within them to be used in subsequent processing. Spectral window selection (’SPWSEL’) can be given as a spectral window integer ID, a list of integer IDs, a spectral window name specified as a literal string (for exact match) or a regular expression or pattern.

The specification can be via frequency ranges or by indexes. A range of frequencies are used to select all spectral windows which contain channels within the given range. Frequencies can be specified with an optional unit — the default unit being Hz. Other common choices for radio and mm/sub-mm data are kHz, MHz, and GHz. You will get the entire spectral windows, not just the channels in the specified range. You will need to do channel selection (see below) to do that.

The spw can also be selected via comparison for integer IDs. For example, ’>ID’ will select all spectral windows with ID greater than the specified value, while ’<ID’ will select those with ID lesser than the specified value.

Spectral window selection using strings follows the standard rules:

  spw = '1'                   # SPWID 1
  spw = '1,3,5'               # SPWID 1,3,5
  spw = '0~3'                 # SPWID 0,1,2,3
  spw = '0~3,5'               # SPWID 0,1,2,3 and 5
  spw = '<3,5'                # SPWID 0,1,2,3 and 5
  spw = '*'                   # All spectral windows
  spw = '1412~1415MHz'        # Spectral windows containing 1412-1415MHz

In some cases, the spectral windows may allow specification by name. For example,

  spw = '3mmUSB, 3mmLSB'      # choose by names (if available)

might be meaningful for the dataset in question.

Note that the order in which multiple spws are given may be important for other parameters. For example, the mode = ’channel’ in clean uses the first spw as the origin for the channelization of the resulting image cube.

2.3.3.1  Channel selection in the spw parameter

Channel selection can be included in the spw string in the form ’SPWSEL:CHANSEL’ where CHANSEL is the channel selector. In the end, the spectral selection within a given spectral window comes down to the selection of specific channels. We provide a number of shorthand selection options for this. These CHANSEL options include:

The most common selection is via channel ranges 'START~STOP' or frequency ranges 'FSTART~FSTOP':

  spw = '0:13~53'             # spw 0, channels 13-53, inclusive
  spw = '0:1413~1414MHz'      # spw 0, 1413-1414MHz section only

All ranges are inclusive, with the channel given by, or containing the frequency given by, START and STOP plus all channels between included in the selection. You can also select the spectral window via frequency ranges 'FSTART~FSTOP', as described above:

  spw = '1413~1414MHz:1413~1414MHz'    # channels falling within 1413~1414MHz
  spw = '*:1413~1414MHz'               # does the same thing

You can also specify multiple spectral window or channel ranges, e.g.

  spw = '2:16, 3:32~34'       # spw 2, channel 16 plus spw 3 channels 32-34
  spw = '2:1~3;57~63'         # spw 2, channels 1-3 and 57-63
  spw = '1~3:10~20'           # spw 1-3, channels 10-20
  spw = '*:4~56'              # all spw, channels 4-56

Note the use of the wildcard in the last example.

A step can be also be included using '^STEP' as a postfix:

  spw = '0:10~100^2'          # chans 10,12,14,...,100 of spw 0
  spw = ':^4'                 # chans 0,4,8,... of all spw
  spw = ':100~150GHz^10GHz'   # closest chans to 100,110,...,150GHz

A step in frequency will pick the channel in which that frequency falls, or the nearest channel.

2.3.4  The selectdata Parameters

The selectdata parameter, if set to True (default), will expand the inputs to include a number of sub-parameters, given below and in the individual task descriptions (if different). If selectdata = False, then the sub-parameters are treated as blank for selection by the task.

The common selectdata expanded sub-parameters are:

2.3.4.1  The antenna Parameter

The antenna selection string is a semi-colon (’;’) separated list of baseline specifications. A baseline specification is of the form:

The selectors ANT1 and ANT2 are comma-separated lists of antenna integer-IDs or literal antenna names, patterns, or regular expressions. The ANT strings are parsed and converted to a list of antenna integer-IDs or IDs of antennas whose name match the given names/pattern/regular expression. Baselines corresponding to all combinations of the elements in lists on either side of ampersand are selected.

Integer IDs can be specified as single values or a range of integers. When items of the list are parsed as literal strings or regular expressions or patterns (see § 2.3.1 for more details on strings). All antenna names that match the given string (exact match)/regular expression/pattern are selected.

ALERT: Just for antenna selection, a user supplied integer (or integer list) is converted to a string and matched against the antenna name. If that fails, the normal logic of using an integer as an integer and matching it with antenna index is done. Note that currently there is no method for specifying a pure index (e.g. a number that will not first be checked against the name).

The comma is used only as a separator for the list of antenna specifications. The list of baselines specifications is a semi-colon separated list, e.g.

   antenna = '1~3 & 4~6 ; 10&11'

will select baselines between antennas 1,2,3 and 4,5,6 (’1&4’, ’1&5’, …, ’3&6’) plus baseline ’10&11’.

The wildcard operator (’*’) will be the most often used pattern. To make it easy to use, the wildcard (and only this operator) can be used without enclosing it in quotes. For example, the selection

   antenna = 'VA*'

will match all antenna names which have ’VA’ as the first 2 characters in the name (irrespective of what follows after these characters).

There is also a negation operator “!” that can be used to de-select antennas or baselines.

Some examples:

   antenna=''          # shows blank autocorr pages
   antenna='*&*'       # does not show the autocorrs
   antenna='*&&*'      # show both auto and cross-cor (default)
   antenna='*&&&'      # shows only autocorrs

   antenna='5&*'       # shows non-auto baselines with AN 5

   antenna='5,6&&&'    # AN 5 and 6 autocor
   antenna='5&&&;6&*'  # AN 5 autocor plus cross-cors to AN 6

   antenna='!5'        # baselines not involving AN 5

Antenna numbers as names: Needless to say, naming antennas such that the names can also be parsed as a valid token of the syntax is a bad idea. Nevertheless, antenna names that contain any of the reserved characters and/or can be parsed as integers or integer ranges can still be used by enclosing the antenna names in double quotes (' "ANT" '). E.g. the string

   antenna = '10~15,21,VA22'

will expand into an antenna ID list 10,11,12,13,14,15,21,22 (assuming the index of the antenna named ’VA22’ is 22). If, however, the antenna with ID index 50 is named ’21’, then the string

   antenna = '10~15,21,VA22'

will expand into an antenna ID list of 10,11,12,13,14,15,50,22. Keep in mind that numbers are FIRST matched against names, and only against indices if that matching fails. There is currently no way to force a selection to use the index, and if there an antenna with that name it will select that.

Read elsewhere (e.g. info regex under Unix) for details of regular expression and patterns.
Antenna stations Instead of antenna names, the antenna station names are also accepted by the selection syntax., e.g. ’N15’ for the JVLA.

ANT@STATION sections syntax Sometimes, data from multiple array configurations are stored in a single MS. But some antennas may have been moved during reconfiguration and the ’ANT@STATION’ syntax can distinguish between them. ’ANT’ is the antenna name or index and ’STATION’ is the antenna station name, e.g., ’EA12@W03’ selects antenna EA012 but only at times when it is positioned on station W03. Wildcards are accepted, e.g. ’EA12@*’ selects all visibilities from antenna EA12, and ’*@W03’ would select all antennas that are located on station ’W03’ during any observations included in the MS.

2.3.4.2  The scan Parameter

The scan parameter selects the scan ID numbers of the data. There is currently no naming convention for scans. The scan ID is filled into the MS depending on how the data was obtained, so use this with care.

Examples:

  scan = '3'                      # scan number 3.
  scan = '1~8'                    # scan numbers 1 through 8, inclusive
  scan = '1,2,4,6'                # scans 1,2,4,6
  scan = '<9'                     # scans <9 (1-8)

NOTE: ALMA and VLA/JVLA number scans starting with 1 and not 0. You can see what the numbering is in your MS using the listobs task with verbose=True (see § 2.2.9).

2.3.4.3  The timerange Parameter

The time strings in the following (T0, T1 and dT) can be specified as YYYY/MM/DD/HH:MM:SS.FF. The time fields (i.e., YYYY, MM, DD, HH, MM, SS and FF), starting from left to right, may be omitted and they will be replaced by context sensitive defaults as explained below.

Some examples:

  1. timerange='T0~T1': Select all time stamps from T0 to T1. For example:
      timerange = '2007/10/09/00:40:00 ~ 2007/10/09/03:30:00'
    

    Note that fields missing in T0 are replaced by the fields in the time stamp of the first valid row in the MS. For example,

      timerange = '09/00:40:00 ~ 09/03:30:00'
    

    where the YY/MM/ part of the selection has been defaulted to the start of the MS.

    Fields missing in T1, such as the date part of the string, are replaced by the corresponding fields of T0 (after its defaults are set). For example:

      timerange = '2007/10/09/22:40:00 ~ 03:30:00'
    

    does the same thing as above.

  2. timerange=’T0’: Select all time stamps that are within an integration time of T0. For example,
      timerange = '2007/10/09/23:41:00'
    

    Integration time is determined from the first valid row (more rigorously, an average integration time should be computed). Default settings for the missing fields of T0 are as in (1).

  3. timerange=’T0+dT’: Select all time stamps starting from T0 and ending with time stamp T0+dT. For example,
      timerange = '23:41:00+01:00:00'
    
    picks an hour-long chunk of time.

    Defaults of T0 are set as usual. Defaults for dT are set from the time corresponding to MJD=0. Thus, dT is a specification of length of time from the assumed nominal "start of time".

  4. timerange=’>T0’: Select all times greater than T0. For example,
      timerange = '>2007/10/09/23:41:00'
      timerange = '>23:41:00'                # Same thing without day specification
    
    Default settings for T0 are as above.
  5. timerange=’<T1’: Select all times less than T1. For example,
      timerange = '<2007/10/09/23:41:00'
    
    Default settings for T1 are as above.

An ultra-conservative selection might be:

  timerange = '1960/01/01/00:00:00~2020/12/31/23:59:59'

which would choose all possible data!

2.3.4.4  The uvrange Parameter

Rows in the MS can also be selected based on the uv-distance or physical baseline length that the visibilities in each row correspond to. This uvrange can be specified in various formats.

The basic building block of uv-distance specification is a valid number with optional units in the format N[UNIT] (the unit in square brackets is optional). We refer to this basic building block as UVDIST. The default unit is meter. Units of length (such as ’m’ and ’km’) select physical baseline distances (independent of wavelength). The other allowed units are in wavelengths (such as ’lambda’, ’klambda’ and ’Mlambda’ and are true uv-plane radii

uv = 
u2+v2
.     (2.1)

If only a single UVDIST is specified, all rows, the uv-distance of which exactly matches the given UVDIST, are selected.

UVDIST can be specified as a range in the format 'N0~N1[UNIT]' (where N0 and N1 are valid numbers). All rows corresponding to uv-distance between N0 and N1 (inclusive) when converted the specified units are selected.

UVDIST can also be selected via comparison operators. When specified in the format ’>UVDIST’, all visibilities with uv-distances greater than the given UVDIST are selected. Likewise, when specified in the format ’<UVDIST’, all rows with uv-distances less than the given UVDIST are selected.

Any number of above mentioned uv-distance specifications can be given as a comma-separated list.

Examples:

  uvrange = '100~200km'                     # an annulus in physical baseline length
  uvrange = '24~35Mlambda, 40~45Mlambda'    # two annuli in units of mega-wavelengths
  uvrange = '< 45klambda'                   # less than 45 kilolambda 
  uvrange = '> 0lambda'                     # greater than zero length (no auto-corrs)
  uvrange = '100km'                         # baselines of length 100km
  uvrange = '100klambda'                    # uv-radius 100 kilolambda

2.3.4.5  The correlation Parameter

The correlation parameter will select between different correlation products. They can be either the correlation ID or values such as ’XX’, ’YY’, ’XY’, ’YX’, ’RR’, ’LL’, ’RL’, ’LR’.

2.3.4.6  The intent Parameter

intent is the scan intent that was specified when the observations were set up. They typically describe what was intended with a specific scan, i.e. a flux or phase calibration, a bandpass, a pointing, an observation of your target, or something else or a combination. The format for the scan intents of your observations are listed in the logger when you run listobs. Minimum matching with wildcards will work, like ’*BANDPASS*’. This is especially useful when multiple intents are attached to scans.

2.3.4.7  The observation Parameter

The observation parameter can select between different observation IDs. They will be assigned to parts of a combined data set during a run of concat. Each input MS will receive its own observation id in the process.

2.3.4.8  The feed Parameter

The feed parameter can select between different feeds, e.g. for different feeds in a single dish multibeam array.

2.3.4.9  The msselect Parameter

More complicated selections within the MS structure are possible using the Table Query Language (TaQL). This is accessed through the msselect parameter.

Note that the TaQL syntax does not follow the rules given in § 2.3.1 for our other selection strings. TaQL is explained in more detail in Aips++ NOTE 199 — Table Query Language (http://aips2.nrao.edu/docs/notes/199/199.html). This will eventually become a CASA document. The specific columns of the MS are given in the most recent MS specification document: Aips++ NOTE 229 — Measurement‘ Set definition version 2.0 (http://aips2.nrao.edu/docs/notes/229/229.html). This documentation will eventually be updated to the CASA document system.

Most selection can be carried out using the other selection parameters. However, these are merely shortcuts to the underlying TaQL selection. For example, field and spectral window selection can be done using msselect rather than through field or spw:

  msselect='FIELD_ID == 0'                   # Field id 0 only
  msselect='FIELD_ID <= 1'                   # Field id 0 and 1
  msselect='FIELD_ID IN [1,2]'               # Field id 1 and 2
  msselect='FIELD_ID==0 && DATA_DESC_ID==3'  # Field id 0 in spw id 3 only

ALERT: The msselect style parameters will be phased out of the tasks. TaQL selection will still be available in the Toolkit.


Previous Up Next