UV Data Import

There are a number of tasks available to bring data in various forms into CASA as a MeasurementSet:

• ALMA and VLA Science Data Model format via importasdm and importevla
• historic VLA Archive format data via importvla
• ATCA Data via importatca
• MIRIAD Data from various telescopes via importmiriad
• GMRT Data via importgmrt
• UVFITS format can be imported into and exported from CASA (importuvfits, importfitsidi, and exportuvfits)

 

ALMA and VLA Filling of Science Data Model (ASDM) data

The ALMA and JVLA projects have agreed upon a common archival science data model (ASDM; sometimes also called SDM) format, and have jointly developed the software to fill this data into CASA. In the ASDM format, the bulk of the data is contained in large binary data format (BDF) tables, with the meta-data and ancillary information in XML tables. This is structured as a directory, like the MS, and was designed to be similar to the MS to facilitate conversion.

The content of an ASDM can be listed with the task asdmsummary:

with an output that contains the list and positions of the antennas, followed by the metadata of each scan like observation time, source name, frequency and polarization setup:

The importasdm task will fill SDM1.2 and SDM1.3 format data into a CASA visibility data set (MS). ALMA data was recorded in SDM1.2 format from October 2009 until May 2011. Since May 2011, ALMA is using the SDM 1.3 format. In particular all science data from cycle 0 onwards is in SDM1.3. The JVLA also started using SDM1.2 in October 2009 and continues to use this format. importasdm can read all of the above formats. The parameter useversion can be used to enable the options process_syspower, process_caldevice, and process_pointing.

The default inputs of importasdm are:

If scans is set, then importasdm processes only the scans specified in the option’s value. This value is a semicolon separated list of scan specifications. A scan specification consists of an exec block index followed by the character ’:’ followed by a comma separated list of scan indexes or scan index ranges. A scan index is relative to the exec block it belongs to. Scan indexes are 1-based while exec blocks are 0-based. The expressions

are all valid values for the selection. The "3:" selector will be interpreted as ’all the scans of the exec block 3’. A scan index or a scan index range not preceded by an exec block index will be interpreted as ’all the scans with such indexes in all the exec blocks’. By default all the scans are considered.

When process_flags=True the task will create online flags based on the Flag.xml, Antenna.xml and SpectralWindow.xml files and copy them to the FLAG_CMD sub-table of the MS. The flags will NOT be applied unless the parameter applyflags is set to True. Optionally, the flags can also be saved to an external ASCII file if savecmds is set to True. The flags can later be applied to the MS using task flagdata in list mode.

When bdfflags=True the task will apply online flags contained in the ASDM BDF data by calling the executable bdflags2MS which the user can also do from the OS prompt. This is recommended for ALMA data.

If singledish=True, output data format is scantable (single-dish data format) instead of MS. In that case, you must specify name or id of the antenna that you want to obtain data. This can be done by using antenna parameter that is defined as a subparameter of singledish. For single-dish mode, only auto-correlation data are filled, i.e. ocorr_mode is forcibly set to ’ao’.

The option createmms prepares the output file for parallel processing and creates a multi-MS.

 

Import of ASDM data with option lazy=True

With lazy=False, importasdm will fill the visibilities into a newly created DATA column of the MS converting them from their binary format in the ASDM to the CASA Table format.

If, however, lazy is set to True, the task will create the DATA column with an ALMA data-specific storage manager, the asdmstman, which enables CASA to directly read the binary data from the ASDM with on-the-fly conversion. No redundant copy of the raw data is created.

This procedure has the advantage that it saves more than 60% disk space and at least in some cases makes the access to the DATA column ≥ 10% faster because the data I/O volume is decreased. For the same reason, it also accelerates the import itself by ca. a factor 2. The acceleration is particularly large in the applycal task and here particularly on standard SATA disks.

E.g., if your ASDM has a size of 36 GB, the import with default parameters will turn this into an MS of 73 GB size (total disk space consumption = 36 GB + 73 GB = 109 GB). With lazy=True, the imported MS has a size of only 2 GB (total disk space consumption = 36 GB + 2 GB = 38 GB). I.e. your total disk space savings are ca. 65%. Even when you compare to the case where you delete the ASDM after normal import, the solution with lazy import and keeping the ASDM will save you ca. 48% disk space (in the example above 38 GB compared to 73 GB).

The only caveats are the following:

1. You must not delete your ASDM. You can, however, move it but you have to update the reference stored in the MS. Symbolic links will work. See below on how to use the tool method ms.asdmref() to manipulate the ASDM reference.
2. The lazily imported DATA column is read-only. But in any normal data reduction, the DATA column (as opposed to CORRECTED_DATA) is treated as read-only anyway.

The lazily imported MS is numerically identical with the traditionally imported MS and so are all results derived from the MSs.

An important additional tool to manipulate lazily imported MSs is the new method ms.asdmref() in the ms tool. If the MS is imported from an ASDM with option lazy=True, the DATA column of the MS is virtual and directly reads the visibilities from the ASDM. A reference to the original ASDM is stored with the MS. If the ASDM needs to be moved to a different path, the reference to it in the MS needs to be updated. This can be achieved with ms.asdmref().

The method takes one argument: abspath. When called with abspath equal to an empty string (default), the method just reports the currently set ASDM path or an empty string if the ASDM path was not set, i.e. the MS was not lazily imported.

If you want to move the referenced ASDM to a different path, you can set the new absolute path by providing it as the value of abspath to the method.

will set the new location of the referenced ASDM to /home/alma/myanalysis/uid___A12345_X678_X910. Contrary to what one would expect from the parameter name, you can also provide a relative path as abspath. This path will be interpreted as relative to the location of the MS.

 

Janksy VLA: Filling of Science Data Model (ASDM) data with importevla

The importevla task will fill SDM data from the Jansky VLA (or ALMA) into a MS, along with online flagging data contained in the Flag.xml SDM table. Otherwise, it behaves as importasdm but with a streamlined parameter set.

The default inputs are:

The default action of importevla is to construct the FLAG_CMD MS table based on the settings of online, flagzero, and shadow (and sub-parameters). If applyflags=True then these flags will be applied after filling. We recommend you use the flagcmd task after filling to examine these flags and then apply.

See importasdm for a description of the common parameters. Some differences:

• Note that importevla automatically loads in VLA switched power information (unlike in previous versions).
• The online parameter controls creation of online flags from the Flag.xml SDM table. The tbuff parameter adds a time “buffer” padding for these flags in both directions to deal with timing mis-matches.
  

  ALERT: For JVLA data taken before April 2011, you should set tbuff to a value (in seconds) equal to 1.5× the integration time.

• The flagzero parameter controls creation of clipping commands to flag visibilities with amplitudes that are exact zeros. If flagpol=True then it will flag the cross-hands (e.g. RL and LR) as well, which might result in low but correct values of these correlations being thrown out (but can catch erroneous zeros also).

  Info: This facility is provided as the JVLA correlator denotes missing or corrupted data with a value of zero.  These data points should be flagged.

• The shadow parameter turns on creation of flag commands to remove antenna time ranges where they are shadowed by other antennas in the array. By default it will flag based on the antenna diameter, but if you want more lenient or conservative flagging then set the tolerance sub-parameter, where the shadowed antennas are flagged for all baselines that are shorter than radius1 + radius2 − tolerance (the radii are those for the antennas as listed in the ANTENNA subtable). addantenna can be a file that defines the positions of antennas that are on the ground but do not appear in the MS. They can still shadow antennas in the array.
• savecmds will save all flagging commands in the flagdata and flagcmd syntax to a file to be applied later or for bookkeeping.
• A flag backup can be performed using the flagbackup parameter. It saves all current flags to the ’*.flagversions’ file of the MS, before all new flags are applied.

 

VLA: Filling data from archive format (importvla)

 

VLA data in archive format (i.e., as downloaded from the historic VLA data archive) are read into CASA from disk using the importvla task. The inputs are:

The main parameters are archivefiles to specify the input VLA Archive format file names, and vis to specify the output MS name.

Note that archivefiles takes a string or list of strings, as there are often multiple files for a project in the archive.

For example:

The importvla task allows selection on the frequency band. Suppose that you have 1.3 cm line observations in K-band and you have copied the archive data files AP314_A95019.xp* to your working directory and started casa. Then,

If the data is located in a different directory on disk, then use the full path name to specify each archive file, e.g.:

 

The other parameters are:

Parameter applytsys

The applytsys parameter controls whether the nominal sensitivity scaling (based on the measured TSYS, with the weights scaled accordingly using the integration time) is applied to the visibility amplitudes or not. If True, then it will be scaled so as to be the same as AIPS FILLM (i.e. approximately in deciJanskys). Note that post-Modcomp data is in raw correlation coefficient and will be scaled using the TSYS values, while Modcomp-era data had this applied online. In all cases importvla will do the correct thing to data and weights based on an internal flag in the VLA Archive file, either scaling it or unscaling based on your choice for applytsys.

If applytsys=True and you see strange behavior in data amplitudes, it may be due to erroneous TSYS values from the online system. You might want to then fill with applytsys=False and look at the correlation coefficients to see if the behavior is as expected.

Parameter bandname

The bandname indicates the VLA Frequency band(s) to load, using the traditional bandname codes. These are:

• ’4’ = 48-96 MHz
• ’P’ = 298-345 MHz
• ’L’ = 1.15-1.75 GHz
• ’C’ = 4.2-5.1 GHz
• ’X’ = 6.8-9.6 GHz
• ’U’ = 13.5-16.3 GHz
• ’K’ = 20.8-25.8 GHz
• ’Q’ = 38-51 GHz
• ’’ = all bands (default)

Note that as the transition from the VLA to JVLA progressed, the actual frequency ranges covered by the bands expanded, and additional bands were added (namely ’S’ from 2-4 GHz and ’A’ from 26.4-40 GHz).

Parameter frequencytol

The frequencytol parameter specifies the frequency separation tolerated when assigning data to spectral windows. The default is frequencytol=150000 (Hz). For Doppler tracked data, where the sky frequency changes with time, a frequencytol < 10000 Hz may produce too many unnecessary spectral windows.

Parameter project

You can specify a specific project name to import from archive files. The default ’’ will import data from all projects in file(s) archivefiles.

For example for VLA Project AL519:

while project=’AL0519’ will NOT work (even though that is what queries to the VLA Archive will print it as - sorry!).

Parameters starttime and stoptime

You can specify start and stop times for the data, e.g.:

Note that the blank defaults will load all data fitting other criteria.

Parameter autocorr

Note that autocorrelations are filled into the data set if autocorr=True. Generally for the VLA, autocorrelation data is not useful, and furthermore the imaging routine will try to image the autocorrelation data (it assumes it is single dish data) which will swamp any real signal. Thus, if you do fill the autocorrelations, you will have to flag them before imaging.

Parameter antnamescheme

The antnamescheme parameter controls whether importvla will try to use a naming scheme where JVLA antennas are prefixed with EA (e.g. ’EA16’) and old VLA antennas have names prefixed with VA (e.g. ’VA11’). Our method to detect whether an antenna is JVLA is not yet perfected, and thus unless you require this feature, simply use antnamescheme=’old’.

Parameter evlabands

The evlabands=True option is provided to allow users to access JVLA frequencies outside the standard VLA tunings (e.g. the extended C-band above 6 GHz).

 

Import ATCA and CARMA data

There are several ways to import data from ATCA and CARMA into CASA. The data from these arrays has historically been processed in MIRIAD. For simple cases (single source and frequency) exporting from MIRIAD to UVFITS format and importing using importuvfits often works ok, although some fixes to the resulting MeasurementSet may be needed.

The importmiriad task reads MIRIAD visibility data and can handle multiple frequencies and sources in the input. Since it does not apply any calibration, make sure to apply it beforehand in MIRIAD.

The importatca task reads the ATCA archive format (RPFITS) directly, avoiding the need to go through MIRIAD to load the data. It can handle ATCA data from both the old and new (CABB) correlator.

 

Import MIRIAD visibilities (importmiriad)

The task importmiriad allows one to import visibilities in the MIRIAD data format to be converted to a MS. The task has mainly been tested on data from the ATCA and CARMA telescopes and the inputs are:

The mirfile parameter specifies a single MIRIAD visibility file which should have any calibration done in MIRIAD already applied to it.

Set the tsys parameter to True to change the visibility weights from the MIRIAD default (usually the integration time) to the inverse of the noise variance using the recorded system temperature.

The spw parameter can be used to select all or some of the simultaneous spectral windows from the input file. Use the default of ’all’ for all the data or use e.g., spw=’0,2’ to select the first and third window.

The vel parameter can be used to set the output velocity frame reference. For ATCA this defaults to ’TOPO’ and for CARMA it defaults to ’LSRK’. Only change this if your data comes out with the incorrect velocity.

The linecal parameter is only useful for CARMA data and can apply the line calibration if it is stored with the MIRIAD data.

The wide parameter is only useful for CARMA data and can select which of the wide-band channels should be loaded.

 

Import ATCA RPFITS data (importatca)

The data from the ATCA is available from the archive in files in the RPFITS format. These files can be imported into CASA with the importatca task.

The files parameter can take a string or a list of strings as input and also allows the use of wildcards as shown in the example above.

For older ATCA continuum data (before the CABB correlator, April 2009) use options=’birdie,reweight’ to suppress internally generated RFI.

The options parameter:

• birdie - (pre-CABB data only) Discard edge channels and channels affected by internal RFI.
• reweight - (pre-CABB data only) Suppress ringing of RFI spikes by reweighting of the lag spectrum
• noxycorr - do not apply the xy phase correction as derived from the switched noise calibration, by default this is applied during loading of the data.
• fastmosaic - use this option if you are loading mosaic data with many pointings and only one or two integrations per pointing. This option changes the tiling of the data to avoid excessive I/O.
• hires - use this option if you have data in time binning mode (as used for pulsars) but you want to make it look like data with very short integration time (no bins).
• noac - discard the auto-correlation data

The spw parameter takes a list of integers and can be used to select one or more of the simultaneous frequencies. With CABB there can be up to 34 spectra. The order of the frequency bands in the RPFITS file is: the two continuum bands (0 and 1), followed by the zoom bands for the first frequency and then the zoom bands for the second frequency. Note that this spw parameter does not take a string with wildcards. Use spw=-1 to get all the data.

The nscans parameter can be used to select part of a file, e.g., to retrieve a few test scans for a quick look.

The lowfreq and highfreq parameters select data based on the reference frequency.

The fields parameter selects data based on the field/source name.

The edge parameter specifies how many edge channels to discard as a percentage of the number of channels in each band. E.g., the default value of 8 will discard 8 channels from the top and bottom of a 2048 channel spectrum.

 

UVFITS Import

 

The UVFITS format is not exactly a standard, but is a popular archive and transport format nonetheless. CASA supports UVFITS files written by the AIPS FITTP task, and others.

UVFITS is supported for both import and export.

Import using importuvfits

To import UVFITS format data into CASA, use the importuvfits task:

This is straightforward, since all it does is read in a UVFITS file and convert it as best it can into a MS.

For example:

 

Import using importfitsidi

Some uvfits data is written in the FITS-IDI standard. Those files can be imported into CASA with the importfitsidi task:

The constobs parameter can be used to give all visibilities the same observation id of 0. scanreindexgap_s controls the gap that defines different scans.

Example: