Previous Up Next

Chapter 3  Data Examination and Editing

3.1  Plotting and Flagging Visibility Data in CASA

The tasks available for plotting and flagging of data are:

The following sections describe the use of these tasks. Information on other related operations can be found in:

3.2  Managing flag versions with flagmanager

The flagmanager task will allow you to manage different versions of flags in your data. These are stored inside a CASA flagversions table, under the name of the MS <msname>.flagversions. For example, for the MS jupiter6cm.usecase.ms, there will need to be jupiter6cm.usecase.ms.flagversions on disk. This is created on import (by importvla or importuvfits) or when flagging is first done on an MS without a .flagversions (e.g. with plotxy).

By default, when the .flagversions is created, this directory will contain a flags.Original in it containing a copy of the original flags in the MAIN table of the MS so that you have a backup. It will also contain a file called FLAG_VERSION_LIST that has the information on the various flag versions there. The flagversions are cumulative, i.e. a specific version number contains all the flags from the lower version numbers, too.

The inputs for flagmanager are:


vis                 =         ''        #   Name of input visibility file (MS)
mode                =     'list'        #   Flag management operation (list,save,restore,delete)

The mode=’list’ option will list the available flagversions from the <msname>.flagversions file. For example:


CASA <102>: default('flagmanager')
CASA <103>: vis = 'jupiter6cm.usecase.ms'
CASA <104>: mode = 'list'
CASA <105>: flagmanager()
MS : /home/imager-b/smyers/Oct07/jupiter6cm.usecase.ms

main : working copy in main table
Original : Original flags at import into CASA
flagautocorr : flagged autocorr
xyflags : Plotxy flags

The mode parameter expands the options. For example, if you wish to save the current flagging state of vis=<msname>,


mode                =     'save'        #   Flag management operation (list,save,restore,delete)
     versionname    =         ''        #   Name of flag version (no spaces)
     comment        =         ''        #   Short description of flag version
     merge          =  'replace'        #   Merge option (replace, and, or)

with the output version name specified by versionname. For example, the above xyflags version was written using:


   default('flagmanager')
   vis = 'jupiter6cm.usecase.ms'
   mode = 'save'
   versionname = 'xyflags'
   comment = 'Plotxy flags'
   flagmanager()

and you can see that there is now a sub-table in the flagversions directory


CASA <106>: ls jupiter6cm.usecase.ms.flagversions/
  IPython system call: ls -F jupiter6cm.usecase.ms.flagversions/
  flags.flagautocorr  flags.Original  flags.xyflags  FLAG_VERSION_LIST

It is recommended that you use this facility regularly to save versions during flagging.

Note that if a flagversion already exists under a name, the task will give a warning and add a suffix ’.old.timestamp’ tp the previous version.

You can restore a previously saved set of flags using the mode=’restore’ option:


mode                =  'restore'        #   Flag management operation (list,save,restore,delete)
     versionname    =         ''        #   Name of flag version (no spaces)
     merge          =  'replace'        #   Merge option (replace, and, or)

The merge sub-parameter will control how the flags are restored. For merge=’replace’, the flags in versionname will replace those in the MAIN table of the MS. For merge=’and’, only data that is flagged in BOTH the current MAIN table and in versionname will be flagged. For merge=’or’, data flagged in EITHER the MAIN or in versionname will be flagged.

The mode=’delete’ option can be used to remove versionname from the flagversions:


mode                =   'delete'        #   Flag management operation (list,save,restore,delete)
     versionname     =         ''       #   Name of flag version (no spaces)

3.3  X-Y Plotting and Editing of the Data

There are three main X-Y plotting tasks in CASA:

3.3.1  MS Plotting and Editing using plotms

The principal way to get X-Y plots of visibility data and calibration tables is the plotms task. This task also provides editing capability. Plotms is a GUI-style plotter, based on Qt. It can either be started as a task within CASA (plotms) or from outside CASA (type casaplotms on the command line).

The current inputs to the plotms task are:


#  plotms :: A plotter/interactive flagger for visibility data.
vis                 =         ''        #  input MS (or CalTable) (blank for none)
gridrows            =          1        #  Number of subplot rows (default 1).
gridcols            =          1        #  Number of subplot columns (default 1).
rowindex            =          0        #  Row location of the plot (0-based, default 0)
colindex            =          0        #  Column location of the plot (0-based, default 0)
plotindex           =          0        #  Index to address a subplot (0-based, default 0)
xaxis               =         ''        #  plot x-axis (blank for default/current)
yaxis               =         ''        #  plot y-axis (blank for default/current)
selectdata          =      False        #  data selection parameters
averagedata         =       True        #  data averaging parameters
     avgchannel     =         ''        #  average over channel?  (blank = False, otherwise value in channels)
     avgtime        =         ''        #  average over time? (blank = False, other value in seconds)
     avgscan        =      False        #  only valid if time averaging is turned on.  average over scans?
     avgfield       =      False        #  only valid if time averaging is turned on.  average over fields?
     avgbaseline    =      False        #  average over all baselines?  (mutually exclusive with avgantenna)
     avgantenna     =      False        #  average by per-antenna?  (mutually exclusive with avgbaseline)
     avgspw         =      False        #  average over all spectral windows?
     scalar         =      False        #  Do scalar averaging?

transform           =      False        #  transform data in various ways?
extendflag          =      False        #  have flagging extend to other data points?
iteraxis            =         ''        #  the axis over which to iterate
customsymbol        =      False        #  set a custom symbol(s) for unflagged points
coloraxis           =         ''        #  selects which data to use for colorizing
customflaggedsymbol =      False        #  set a custom plot symbol for flagged points
plotrange           =         []        #  plot axes ranges: [xmin,xmax,ymin,ymax]
title               =         ''        #  Title written along top of plot
xlabel              =         ''        #  Text for horizontal axis. Blank for default.
ylabel              =         ''        #  Text for vertical axis. Blank for default.
showmajorgrid       =      False        #  Show major grid lines (horiz and vert.)
showminorgrid       =      False        #  Show minor grid lines (horiz and vert.)
showlegend          =      False        #  Show a legend on the plot.
plotfile            =         ''        #  Name of plot file to save automatically.
showgui             =       True        #  Show GUI
     clearplots     =       True        #  Remove any existing plots so new ones can replace them.

callib              =       ['']        #  Calibration library string or filename for on-the-fly calibration.

Almost all of these parameters can also be set or modified from inside the plotms window. Note that, if the vis parameter is set to the name of a measurement or calibration table set here, when you start up plotms, the entire measurement set will be plotted, which can be time consuming. It is probably best to leave all parameters blank for now, setting them as needed inside the plotms GUI.


Figure 3.1: A freshly-started plotms GUI window. Note that the Plots > Data tab is selected, which is discussed in § 3.3.1.1, 3.3.1.7, and 3.3.1.9.

3.3.1.1  Loading and Selecting Data

When plotms is first started, a window will appear as in Figure 3.1. It will, by default, display the Plots tab (as chosen from the tabs at the top of the plotms window—e.g., Plots, Flag, Tools...) and the Plots > Data tab (as chosen from the tabs on the far left side of the plotms window—e.g., Data, Calibration, Axes, Pages, Transform...). First, a measurement set should be loaded by clicking on Browse near the top of the Plots > Data tab, and selecting a .ms directory (just select the directory itself; do not descend into the .ms directory). A plot can now be made of the measurement set by clicking on Plot—but beware, this would plot the entire measurement set, and could take quite some time! It is probably better to select a subset of the measurement set using the Selection windows in the Plots > Data tab before clicking Plot.

The options for data selection are:

These are described in § 2.3. Note that, unlike when setting data selection parameters from the CASA command line, no quotation marks are needed around strings.

Once you have selected the desired subset of data, if you click Plot, plotms will by default plot amplitude versus time. See the next section for information about other possible axes.

For a given data selection, plotms will only load the data once. This speeds up plotting considerably when changing plot parameters such as different axes, colors etc. Sometimes, however, the data changes on disk, e.g., when other data processing tasks were applied. To force plotms to reload the data, checkmark the little force reload box left to the Plot’ button or press the SHIFT key while clicking the Plot button.

3.3.1.2  A Brief Note Regarding plotms Memory Usage

In order to provide a wide range of flexible interactive plotting options while minimizing the I/O burden, plotms caches the data values for the plot (along with a subset of relevant meta-info) in as efficient a manner as possible. For plots of large numbers of points, the total memory requirement can be quite large. plotms attempts to predict the memory it will require (typically 5 or 6 bytes per plotted point when only one axis is a data axis, depending upon the data shapes involved), and will complain if it believes there is insufficient memory to support the requested plot. For most practical interactive purposes (plots that load and draw in less than a few or a few 10s of minutes), there is usually not a problem on typical modern workstations (attempts to plot large datasets on small laptops might be more likely to encounter problems here).

The absolute upper limit on the number of simultaneously plotted points is currently set by the ability to index the points in the cache. For modern 64 bit machines, this is about 4.29 billion points (requiring around 25GB of memory). (Such plots are not especially useful interactively, since the I/O and draw become prohibitive.)

In general, it is usually most efficient to plot data in modest chunks of not more than a few hundred million points or less, either using selection or averaging. Note that all iterations are (currently) cached simultaneously for iterated plots, so iteration is not a way to manage memory use. A few hundred million points tends to be the practical limit of interactive plotms use w.r.t. information content and utility in the resulting plots, especially when you consider the number of available pixels on your screen.

In scripts, or for very large data sets, it can be desirable to use plotms in a non-interactive mode. This can be done by setting showgui=False and to directly plot into a png image specified by plotfile.


Figure 3.2: The Plots > Axes tab in the plotms GUI window, used to make a plot of Amp versus Channel.

3.3.1.3  Plot Axes

The X and Y axes of a plot are selected by clicking on the Plots > Axes tab on the left side of the plotms window, and choosing an entry from the drop-down menus below X Axis and Y Axis (see Figure 3.2). Possible axes are:

If the data axis selected from the drop-down menu is already stored in the cache (therefore implying that plotting will proceed relatively quickly), an “X” will appear in the checkbox next to In Cache?. If the data shall be reloaded from disk, the “force reload” checkmark should be set at the bottom of this display.

For relevant data axes like Amp and Phase, the user will be presented with the option to plot raw data or calibrated data. This can be selected via a drop-down menu called Data Column, located directly under the drop-down menu for X or Y Axis selection (see the Y axis in Figure 3.2). To plot raw data, select “data”; to plot calibrated data, select “corrected”. Note that this choice will only have an impact on a plot if a calibration table has been applied to the measurement set (see applycal, Sect. 4.6.1).

If a data model has been applied to the measurement set (e.g., with setjy, Sect. 4.3.5) it can be plotted by selecting “model” from the Data Column menu. Residuals can be plotted via “corrected-model”, “data-model”, “data/model”, and “corrected/model”.

3.3.1.4  Calibration Library

The keyword ’callib’ in the plotms task parameters can be used to provide a calibration library file (see Appendix G). This file allows one to apply calibration tables to the uncalibrated data on the fly, i.e. without a run of applycal beforehand. The tab Calibration on the right hand side contains a field to specify the calibration library or to specify the calibration library commands directly. There’s also a switch to either apply the calibration library to produce the “corrected” data (“Calibration On”) or to show an existing “corrected” data column (“Calibration Off”).

3.3.1.5  Tools

Various tools—selectable as icon buttons at the bottom of the plotms window—can be used to zoom, edit, and locate data. The icon buttons can be seen at the bottom of Figures 3.1 and 3.2, and are, from left to right:

Under the Options tab near the top of the plotms window one can select the layout of the page. For multiple plots per page, one can select the grid layout, ie the number of rows and columns that determine the number of sub-plots. The when changing plot axes, clear any existing regions or annotations checkbox determines when regions and annotation are deleted from the plot. The Tool Button Style drop-drop down menu determines if icons and/or text represent the buttons at the bottom of the plotms window.

It is possible to hide these icons by going to the View > Toolbars menu at the top of the plotms window and un-depressing the Tools option (except for Hide Drawing, which is hidden by clicking on View > Toolbars > Display). In addition, the above tools can also be accessed by clicking on the Tools tab near the top of the plotms window (just below the View menu).

The Tools tab also enables one additional tool, the Tracker. To use Tracker, click on the Hover and/or

Display checkbox, and place your mouse over the plot. Tracker will output the X and Y position of your mouse, either as text superimposed on the plot near your mouse (if Hover is selected) or in the blank window in the Tools tab (if Display is selected). Pressing the SPACE bar will copy the lines into the larger white box below to the right. This can be repeated many times and a log of positions and values will be created. The content in the box can then be easily copied and pasted into any other application that is used for data analysis. The Clear button wipes out the content of the box for a fresh start into new scientific adventures.

3.3.1.6  Interactive Flagging in plotms

Interactive flagging, on the principle of “see it — flag it”, is possible on the X-Y display of the data plotted by plotms. The user can use the cursor to mark one or more regions, and then flag, unflag, or list the data that falls in these zones of the display.

Using the row of icons buttons at the bottom of the plotms window (§ 3.3.1.5), click on the Mark Regions button (which will appear to depress), then mark a region by left-clicking and dragging the mouse (each click and drag will mark an additional region). You can get rid of all your regions by clicking on the Clear Regions. Once regions are marked, you can then click on one of the other buttons to take action:

  1. Flag — flag the points in the region(s),
  2. Unflag — unflag flagged points in the region(s),
  3. Locate — spew out a list of the points in the region(s) to the command line (Warning: this could be a long list!).

Figure 3.3 shows an example of marking regions and then clicking the Flag button. Whenever you click on a button, that action occurs without requiring an explicit disk-write. If you quit plotms and re-enter, you will see your previous edits.


Figure 3.3: Plot of amplitude versus time, before (left) and after (right) flagging two marked regions. To unflag these regions, mark the two same regions and click the Unflag button.

plotms does not automatically create flag backups in the <msname>.flagversions file. It is thus recommended to save the initial flags with the flagmanager task (§ 3.2) before starting plotms interactive flagging. Surely, important intermediate flagging stages may also be saved during plotms flagging in the same fashion.

Flags can also be extended with options in the Flagging tab, found near the top of the plotms window. Flag extension enables the user to plot a subset of the data and extend the flagging to a wider set. In this release, the only functional extensions are over channel and correlation.

By checking the boxes next to Extend Flags and Channel, flagging will be extended to other channels in the same spw as the displayed point. For example, if spw=’0:0’ and channel 0 is displayed, then flagging will extend to all channels in spw 0.

By checking the boxes next to Extend Flags and Correlation, flags will be extended beyond the correlations displayed. Currently the only option is to extend to All correlations, implying that all correlations will be flagged, e.g. with RR displayed, the correlations RR, RL, LR, and LL will all be flagged.

WARNING: use of flag extensions may lead to deletion of much more data than desired. Be careful!

3.3.1.7  Averaging Data

The Plots > Data tab enables averaging of the data in order to increase signal-to-noise of the plotted points or to increase plotting speed. The options for Averaging are:

The box next to a given Averaging mode needs to be checked for that averaging to take effect.

For example, to average n channels together, the user would click on the box next to Channels so that an “X” appears in it, and then type the number n in the empty box. When the user next clicks on Plot, every n channels will then be averaged together and plotted against the average channel numbers. The total number of channels plotted will be decreased by a factor of n.

Time averaging is a little trickier, as it is controlled by three fields. If the checkbox next to Time under Averaging is clicked on, a blank box with units of seconds will become active, along with two additional checkboxes: Scan and Field. If averaging is desired over a relatively short interval (say, 30 seconds, shorter than the scan length), a number can simply be entered into the blank box and, when the data are replotted, the data will be time averaged. Clicking on the Scan or Field checkbox in this case will have no impact on the time averaging.

These checkboxes become relevant if averaging over a relatively long time—say the entire observation, which consists of multiple scans—is desired. Regardless of how large a number is typed into the Time averaging blank box, only data within individual scans will be averaged together. In order to average data across scan boundaries, the Scan checkbox must be clicked on and the data replotted. Finally, clicking on the Field checkbox enables the averaging of multiple fields together in time.

Clicking on the All Baselines checkbox will average all baselines in the array together. Alternatively, the Per Antenna box may be checked, which will average all baselines for a given antenna together. In this case, all baselines are represented twice; baseline 3-24 will contribute to the averages for both antenna 3 and antenna 24. This can produce some rather strange-looking plots if the user also selects on antenna—say, if the user requests to plot only antenna 0 and then averages Per Antenna, In this case, an average of all baselines including antenna 0 will be plotted, but each individual baseline including antenna 0 will also be plotted (because the presence of baselines 0-1, 0-2, 0-3, etc. trigger Per Antenna averaging to try and compute averages for antennae 1, 2, 3, etc. Therefore, baseline 0-1 will contribute to the average for antenna 0, but it will also singlehandedly be the average for antenna 1.)

Spectral windows can be averaged together by checking the box next to All Spectral Windows. This will result in, for a given channel n, all channels n from the individual spectral windows being averaged together.

Finally, the default mode is vector averaging, where the complex average is formed by averaging the real and imaginary parts of the relevant visibilities. If Scalar is chosen, then the amplitude of the average is formed by a scalar average of the individual visibility amplitudes.

When averaging, plotms will prefer unflagged data. I.e., if an averaging bin contains any unflagged data at all, only the average of the unflagged will be shown. For averaging bins that contain only unflagged data, the average of that unflagged data will be shown. When flagging on a plot of averaged data, the flags will be applied to the unaveraged data in the MS.

3.3.1.8  Plot Symbols

Plot symbols are selected in the Plots > Display tab. Most fundamentally, the user can choose to plot unflagged data and/or flagged data. By default, unflagged data is plotted (the circle next to Default is checked under Unflagged Points Symbol), and flagged data is not plotted (the circle next to None is checked under Flagged Points Symbol. We note here that plotting flagged data on an averaged plot is undertaken at the user’s own risk, as the distinction between flagged points and unflagged points becomes blurred if data are averaged over a dimension that is partially flagged. Take, for example, a plot of amplitude versus. time where all channels are averaged together, but some channels have been flagged due to RFI spikes. In creating the average, plotms will skip over the flagged channels and only use the unflagged ones. The averaged points will be considered unflagged, and the flagged data will not appear on the plot at all.

A selection of None produces no data points, Default results in data points which are small circles (blue for unflagged data and red for flagged data), and Custom allows the user to define a plot symbol. If Custom plot symbols are chosen, the user can determine the symbol size by typing a number in the blank box next to px or by clicking on the adjacent up or down arrows. Symbol shape can be chosen from the drop-down menu to be either “circle”, “square”, “diamond”, or “pixel” (note than “pixel” only has one possible size). “autoscaling” attempts to adjust the size of the points from dots to circles of different sizes, depending on how many points are plotted. Symbol color can be chosen by typing a hex color code in the blank box next to Fill: (e.g., “ff00ff”), or by clicking on the ... button and selecting a color from the pop-up GUI. The adjacent drop-down menu provides options for how heavily the plot symbol is shaded with this color, from heaviest to lightest: “fill”, “mesh1”, “mesh2”, “mesh3”, and “no fill”. Finally, the plot symbol can be outlined in black (if Outline: Default is checked) or not (if Outline: None is checked). Note that if “no fill” and Outline: None are selected, the plot symbols will be invisible.

Finally, unflagged data points can be given informative symbol colors using the Colorize parameter. By checking the box next to Colorize and selecting a data dimension from the drop-down menu, the data will be plotted with colors that vary along that dimension. For example, if “corr” is chosen from the Colorize menu, “RR”, “LL”, “RL”, and “LR” data will each be plotted with a different color. Note that, currently, colorize and plotting flagged data appear to be incompatible; a plot can only include one of these special features at a time.

3.3.1.9  Summarizing Data

Information about the measurement set can be obtained from within plotms by clicking on the Summary button, found at the top menu bar. If “All” is chosen from the pull-down menu next to Summary, listobs-style output about scans, correlator configurations, and antennae will be written to the command line from which plotms was started. For more detail, click on the Verbose checkbox. For a specific subset of the data, choose a selection from the pull-down menu like “Antenna” or “Field”.

3.3.1.10  Defining Frequency and Velocity

If the user plans to plot Frequency, the reference frame must be defined. By default, the plotted frequency is simply that observed at the telescope. However, transformations can be made by choosing a Frame from the drop-down menu in the Plots > Transform tab. Frequency reference frames can be chosen to be:

Velocity is affected by the user’s choice of Frame, but it is also impacted by the choice of velocity definition and spectral line rest frequency. The velocity definition is chosen from the Velocity Defn drop-down menu in the Plots > Trans tab, offering selections of Radio, True, or Optical.

For more information on frequency frames and spectral coordinate systems, see the paper by Greisen et al. (A&A, 446, 747, 2006) 1.

Finally, the spectral line’s rest frequency in units of MHz should be typed into the blank box next to Rest Freq in the Plots > Trans tab. You can use the me.spectralline tool method to turn transition names into frequencies


CASA <16>: me.spectralline('HI')
  Out[17]: 
{'m0': {'unit': 'Hz', 'value': 1420405751.786},
 'refer': 'REST',
 'type': 'frequency'}

For a list of known lines in the CASA measures system, use the toolkit command me.linelist(). For example:


CASA <21>: me.linelist()
  Out[21]: 'HI H186A H185A H184A H183A H182A H181A H180A H179A H178A H177A H176A H175A 
H174A H173A H172A H171A H170A H169A H168A H167A H166A H165A H164A H163A H162A H161A H160A... 
He182A He181A He180A He179A He178A He177A He176A He175A He174A He173A He172A He171A He170A 
He169A He168A He167A He166A He165A He164A He163A He162A He161A He160A He159A He158A He157A...
C186A C185A C184A C183A C182A C181A C180A C179A C178A C177A C176A C175A C174A C173A C172A 
C171A C170A C169A C168A C167A C166A C165A C164A C163A C162A C161A C160A C159A C158A C157A... 
NH3_11 NH3_22 NH3_33 NH3_44 NH3_55 NH3_66 NH3_77 NH3_88 NH3_99 NH3_1010 NH3_1111 NH3_1212 
OH1612 OH1665 OH1667 OH1720 OH4660 OH4750 OH4765 OH5523 OH6016 OH6030 OH6035 OH6049 OH13433 
OH13434 OH13441 OH13442 OH23817 OH23826 CH3OH6.7 CH3OH44 H2O22 H2CO4.8 CO_1_0 CO_2_1 CO_3_2 
CO_4_3 CO_5_4 CO_6_5 CO_7_6 CO_8_7 13CO_1_0 13CO_2_1 13CO_3_2 13CO_4_3 13CO_5_4 13CO_6_5 
13CO_7_6 13CO_8_7 13CO_9_8 C18O_1_0 C18O_2_1 C18O_3_2 C18O_4_3 C18O_5_4 C18O_6_5 C18O_7_6 
C18O_8_7 C18O_9_8 CS_1_0 CS_2_1 CS_3_2 CS_4_3 CS_5_4 CS_6_5 CS_7_6 CS_8_7 CS_9_8 CS_10_9 
CS_11_10 CS_12_11 CS_13_12 CS_14_13 CS_15_14 CS_16_15 CS_17_16 CS_18_17 CS_19_18 CS_12_19 
SiO_1_0 SiO_2_1 SiO_3_2 SiO_4_3 SiO_5_4 SiO_6_5 SiO_7_6 SiO_8_7 SiO_9_8 SiO_10_9 SiO_11_10 
SiO_12_11 SiO_13_12 SiO_14_13 SiO_15_14 SiO_16_15 SiO_17_16 SiO_18_17 SiO_19_18 SiO_20_19 
SiO_21_20 SiO_22_21 SiO_23_22'

3.3.1.11  Shifting the Phase Center

The plot’s phase center can be shifted in the Plots > Trans tab. Enter the X and Y shifts in units of arcseconds in the blank boxes under Phase center shift.

3.3.1.12  Plot Ranges

The X and Y ranges of the plot can be set in the Plots > Axes tab. By default, the circle next to Automatic will be checked, and the ranges will be auto-scaled. To define the range, click on the circle below Automatic and enter a minimum and maximum value in the blank boxes (as for the X Axis in Figure 3.2. Note that if identical values are placed in the blank boxes (xmin=xmax and/or ymin=ymax), then the values will be ignored and a best guess will be made to auto-range that axis.

3.3.1.13  Plot Labels

The plot and axes labels which are displayed in the plot window are set in the Plots > Canvas tab. To change the plot title, under Canvas Title, click on the circle next to the blank box and enter the desired text. To change the X- and Y-axis labels, similarly click on the circles next to the blank boxes under Show X Axis and Show Y Axis and type the desired text in the blank box. To display these new labels, simply click the Plot button.

The user can determine the locations of axis labels in the Plots > Axes tab. The X-axis label switches from the bottom to the top of the plot depending on what is selected for Attach to:. Similarly for the Y-Axis, the user can choose to attach axis labels and tick marks to the Top or Bottom (note that the axis labels have been attached to the Bottom and Right in Figure 3.2.

Finally, axis labels can be removed all together by unchecking the boxes next to Show X Axis and Show Y Axis on the Plots > Canvas tab.

3.3.1.14  Grid Lines

A grid of lines can be superimposed on the plot using Grid Lines in the Plots > Canvas tab. “Major” grid lines are drawn at the locations of major tick marks, while “minor” grid lines are drawn at minor tick marks.

Grid line colors, thicknesses, and styles are selected independently for the “major” and “minor” grid lines. Desired line thickness should be typed into the blank boxes just to the right of the Major and Minor labels. Colors are set by clicking on the ... buttons. The blank boxes to the left of the ... buttons will then contain the hex codes for the selected colors (e.g., “808080”). Line styles can also be selected from the drop-down menus to the right of ... buttons.

3.3.1.15  Legend

A plot symbol legend can be added to the plot by clicking on the checkbox next to Legend in the Plots > Canvas tab. However, given the current functionalities of plotms, a symbol legend is of very limited use. This option is usefule when overplotting data.

3.3.1.16  The Options Tab

A few miscellaneous options are available in the Options tab, the last tab at the top of the plotms window. The Tool Button Style drop-drop down menu determines if icons and/or text represent the buttons in the toolbar near the bottom of the plotms window.

The Log Events drop down menu determines how verbose plotms is in documenting its actions on the command line.

There is a checkbox that determines the persistence of regions and annotations on new plots, labelled When changing plot axes, clear any existing regions and annotations.

A useful option is the fixed size for cached image checkbox. It determines how large the dots in the panel are with respect to the screen resolution. The values influence how the data is redrawn on the panel. When the Screen resolution is selected, the plotms window can be resized without redrawing on the canvas – a considerable speedup for large data sets. The penalty is that the dots of the data points are the size of a pixel on the screen, which may be very small for high resolution monitors.

Finally, the File chooser history limit determines the number of remembered directories in the file loading pop-up of the Browse selection of the Data tab.

3.3.1.17  Iteration

In many cases, it is desirable to iterate through the data that were selected in the Data tab. A typical example is to display a single baseline in a time vs. amplitude plot and then proceed to the next baselines step by step. This can be done via the Page tab on the left hand side of plotms. A drop-down menu allows you to select the parameter to be iterated on, such as baseline or spw (press plot after changing your selection). The plot titles in the main panel in plotms show which data slice is currently displayed. To proceed to the next plot use the green buttons below the main panel. The different button symbols let you to proceed panel by panel or to jump to the first or last panel directly. The number of plots per page can be selected under Options-> Grid, the last of the top row of tabs (corresponding to the gridrows and gridcols in the command line interface).

There are two scaling options for the axes: Global and Self. Global will use a common axis range based on data loaded with the selection criteria specified in the Data tab. Self readjusts the axes scaling to the data for each individual panel of the iteration.

Below, one can invoke multiple panels per display by selecting the number of rows and columns to be displayed on the canvas.

3.3.1.18  Overplotting

Different values of the same dataset can be shown at the same time. E.g. to add a second y-axis, press the “Add Y Axis Data” button under the “Axes” tab. Then select the parameters for the newly created axis by selecting from the now available “Y Axis Data” drop-down menu. If the two y-axes have the same units, they can be displayed both on the same axis. If they are different, e.g. Amplitude and Elevation (both versus time; see Fig. 3.4), one axis should be attached to the left and the other to the right hand side of the plot. Using more than a single y-axis data is also reflected in the “Display” tab where a drop-down menu appears in order to select multiple y-axis options.


Figure 3.4: Overplotting in plotms: Two different y-axes have been chosen for this plot, amplitude and elevation.

In the plotms input interface, you can overplot by invoking plotms more than once with clearplot=F. Each run of plotms corresponds to a plot to go on top of previous ones.

3.3.1.19  Plotting Multiple Data Sets

plotms can also plot more than a single dataset in separate panels. To do so, press “Add Plot” next to the “Plot” button. This will bring up a new data window where the plot parameters are defined. Right-click options are used to “Minimize”, “Maximize”, or “Close” these panels which helps to keep a better overview on the individual datasets. If Options-> Grid is selected to have more than a single panel, the different datasets will be shown side by side.

When plotms is run from the command line, the location of the plots can be defined as follows. gridcols and gridrows define the number of plots on the screen. To define the location where a subplot is to appear on this grid, use colindex and rowindex. If one uses a plotindex, this will be used as a label to address the plot. Each call of plotms with the same plotindex will overplot on the subplot where plotindex was defined the first time. Here is an example on multiple plotms calls:


#Plot in the second column, first row of a 2x2 grid and define this plot as plotindex=0
plotms(vis='vis1.ms', gridrows=2, gridcols=2, colindex=1, rowindex=0)

#Overplot in the same panel using a different axis and symbol for the second plot.
plotms(vis='vis2.ms', clearplots=False, plotindex=1, rowindex=0,
colindex=1, gridrows=2, gridcols=2, yaxislocation='right', symbolshape='circle')

#Define a second plot and give it a label plotindex=2, in the lower right corner of the grid.
plotms(vis='vis1.ms', clearplots=False, plotindex=2, rowindex=1,colindex=1, gridrows=2, gridcols=2)

#Move the plot with the overplot one panel to the left. This requires clearing
#the plots and rerunning the script specifications with the new plot locations.
plotms(vis='vis1.ms, gridrows=2, gridcols=2, colindex=0, rowindex=0, symbolshape='diamond')
plotms(vis='vis2.ms', clearplots=False, plotindex=1, rowindex=0, colindex=0,
                          gridrows=2, gridcols=2,yaxislocation='right',
                          symbolshape='circle')
plotms(vis=self.ms, clearplots=False, plotindex=2, rowindex=1,colindex=1,gridrows=2, gridcols=2)

3.3.1.20  Saving your plot

You can save a copy of a plot to file in the Plots > Export tab. Click the Browse button for a GUI-based selection of the directory and file name to which the plot will be saved. The file format can also be determined in this GUI by the suffix given to the filename: .png (PNG), .jpg (JPG), .ps (PS), .pdf (PDF), and txt (TEXT). Alternatively, the file format can be selected from the Format drop-down menu located just below the Browse button. In this case, plotms will add a suffix to the file name depending on the format chosen.

ALERT: The plot files produced by the PS and PDF options can be large and time-consuming to export. The JPG is the smallest.

The exported plot resolution can be manipulated using the High Resolution, DPI, and Size options. The size can be specified using height and/or width within or outside the GUI. The default (unset) value is -1 and the maximum allowed value is 65500 for height/width.

Click on Export to create the file, you may select to either plot only the current page or all pages (filenames will be automatically incremented).

The TEXT format will not save an image but the data points themselves. This allows one to dump the current plot into a file that is used in other programs for further processing. The reported data is the same as when using the locate button in plotms and the format looks like:


# x y chan scan field ant1 ant2 ant1name ant2name time freq spw corr offset currchunk irel
# Real Imag None None None None None None None MJD(seconds) GHz None None None None None
0.282938 0.0387583 31 5 2 1 12 ea02@E02 ea21@E01 4778968956.000 36.308479452 1 RR 26 0 26
0.263241 -0.00806698 31 7 2 1 12 ea02@E02 ea21@E01 4778969356.000 36.308479452 1 RR 29 1 28
0.258207 0.0301206 31 9 2 1 12 ea02@E02 ea21@E01 4778969745.000 36.308479452 1 RR 30 2 28
0.311155 -0.0180511 31 11 2 1 12 ea02@E02 ea21@E01 4778970133.250 36.308479452 1 RR 31 3 28
0.284589 -0.0628808 31 13 2 1 12 ea02@E02 ea21@E01 4778970522.250 36.308479452 1 RR 32 4 28

where x and y are the two plotted axes and the other columns contain additional information such as the baselines or frequencies. The three last columns offset, corrchunk, and irel are internal data management items for plotms and you most likely will never use them.

3.3.1.21  Exiting plotms

To exit the plotms GUI, select Quit from the File menu at the top of the plotms window. You can also dismiss the window by killing it with the “X” on the frame.

Alternatively, you can just leave it alone, and plotms will keep running in the background. If the data file changes in the background, you can force reloading the data via the ’force reload’ checkbox next to the ’Plot’ button. Alternatively, press SHIFT while clicking on ’Plot’ for the same purpose.

3.3.2  Plotting and Editing using plotxy

Inside the Toolkit: Access to matplotlib is also provided through the pl tool. See below for a description of the pl tool functions. ALERT: The plotxy code is fragile and slow, and is being replaced by the plotms (§ 3.3.1). We retain plotxy in this release as not all functionality is available yet in plotms.

Plotxy is a tool for visualizing and editing visibility data. Unlike plotms, it is useful in scripting, as it can non-interactively produce a hardcopy plot (see § 3.3.2.13). It also has multi-plot (§ 3.3.2.8), iteration (§ 3.3.2.3), and overplotting (§ 3.3.2.4) functionality—unlike plotms in the current release. Plotxy uses the matplotlib plotting library to display its plots. You can find information on matplotlib at http://matplotlib.sourceforge.net/.


Figure 3.5: The plotxy plotter, showing the Jupiter data versus uv-distance. You can see bad data in this plot. The bottom set of buttons on the lower left are: 1,2,3) Home, Back, and Forward. Click to navigate between previously defined views (akin to web navigation). 4) Pan. Click and drag to pan to a new position. 5) Zoom. Click to define a rectangular region for zooming. 6) Subplot Configuration. Click to configure the parameters of the subplot and spaces for the figures. 7) Save. Click to launch a file save dialog box. The upper set of buttons in the lower left are: 1) Mark Region. Press this to begin marking regions (rather than zooming or panning). 2,3,4) Flag, Unflag, Locate. Click on these to flag, unflag, or list the data within the marked regions. 5) Next. Click to move to the next in a series of iterated plots. Finally, the cursor readout is on the bottom right.

To bring up this plotter use the plotxy task. The inputs are:


#  plotxy :: X-Y plotter/interactive flagger for visibility data

vis              =         ''   #  Name of input visibility
xaxis            =     'time'   #  X-axis: def = 'time': see help for options
yaxis            =      'amp'   #  Y-axis: def = 'amp': see help for options
     datacolumn  =     'data'   #  data (raw), corrected, model, residual (corrected - model)

selectdata       =      False   #  Other data selection parameters
spw              =         ''   #  spectral window:channels: ''==>all, spw='1:5~57'
field            =         ''   #  field names or index of calibrators: ''==>all
averagemode      =         ''   #  Select averaging type: 'vector', 'scalar'
restfreq         =         ''   #  a frequency quanta or transition name. see help for options
extendflag       =      False   #  Have flagging extend to other data points?
subplot          =        111   #  Panel number on display screen (yxn)
plotsymbol       =        '.'   #  Options include . : , o ^ v > < s + x D d 2 3 4 h H | _
plotcolor        =  'darkcyn'   #  Plot color
plotrange        = [-1, -1, -1, -1]  #  The range of data to be plotted (see help)
multicolor       =     'corr'   #  Plot in different colors: Options: none, both, chan, corr
selectplot       =      False   #  Select additional plotting options (e.g, fontsize, title,etc)
overplot         =      False   #  Overplot on current plot (if possible)
showflags        =      False   #  Show flagged data?
interactive      =       True   #  Show plot on gui?
figfile          =         ''   #  ''= no plot hardcopy, otherwise supply name

ALERT: The plotxy task expects all of the scratch columns to be present in the MS, even if it is not asked to plot the contents. If you get an error to the effect "Invalid Table operation: Table: cannot add a column" then use clearcal() to force these columns to be made in the MS. Note that this will clear anything in all scratch columns (in case some were actually there and being used).

Setting selectdata=True opens up the selection sub-parameters:


selectdata       =       True   #   Other data selection parameters
     antenna     =         ''   #   antenna/baselines: ''==>all, antenna = '3,VA04' 
     timerange   =         ''   #   time range: ''==>all 
     correlation =         ''   #   correlations: default = '' 
     scan        =         ''   #   scan numbers: Not yet implemented
     feed        =         ''   #   multi-feed numbers: Not yet implemented
     array       =         ''   #   array numbers: Not yet implemented
     uvrange     =         ''   #   uv range''==>all; uvrange = '0~100kl' (default unit=meters)

These are described in § 2.3.

Averaging is controlled with the set of parameters


averagemode      =   'vector'   #  Select averaging type: vector, scalar
     timebin     =        '0'   #  Length of time-interval in seconds to average
     crossscans  =      False   #  Have time averaging cross scan boundaries?
     crossbls    =      False   #  have averaging cross over baselines?
     crossarrays =      False   #  have averaging cross over arrays?
     stackspw    =      False   #  stack multiple spw on top of each other?
     width       =        '1'   #  Number of channels to average

See § 3.3.2.9 below for more on averaging.

You can extend the flagging beyond the data cell plotted:


extendflag          =   True    #  Have flagging extend to other data points?
     extendcorr     =     ''    #  flagging correlation extension type
     extendchan     =     ''    #  flagging channel extension type
     extendspw      =     ''    #  flagging spectral window extension type
     extendant      =     ''    #  flagging antenna extension type
     extendtime     =     ''    #  flagging time extension type

See § 3.3.2.11 below for more on flag extension.

The restfreq parameter can be set to a transition or frequency:


restfreq            =    'HI'   #  a frequency quanta or transition name. see help for options
     frame          =  'LSRK'   #  frequency frame for spectral axis. see help for options
     doppler        = 'RADIO'   #  doppler mode. see help for options

See § 3.3.2.12 below for more on setting rest frequencies and frames.

Setting selectplot=True will open up a set of plotting control sub-parameters. These are described in § 3.3.2.2 below.

The interactive and figfile parameters allow non-interactive production of hardcopy plots. See § 3.3.2.13 for more details on saving plots to disk.

The iteration, overplot, plotrange, plotsymbol, showflags and subplot parameters deserve extra explanation, and are described below.

For example:


plotxy(vis='jupiter6cm.ms',                # jupiter 6cm dataset
       xaxis='uvdist',                     # plot uv-distance on x-axis
       yaxis='amp',                        # plot amplitude on y-axis
       field='JUPITER',                    # plot only JUPITER
       selectdata=True,                    # open data selection
       correlation='RR,LL',                #   plot RR and LL correlations
       selectplot=True,                    # open plot controls
       title = 'Jupiter 6cm uncalibrated') #   give it a title 

The plotter resulting from these settings is shown in figure 3.5.

ALERT: The plotxy task still has a number of issues. The averaging has been greatly speeded up in this release, but there are cases where the plots will be made incorrectly. In particular, there are problems plotting multiple spw at the same time. There are sometimes also cases where data that you have flagged in plotxy from averaged data is done so incorrectly. This task is under active development for the next cycle to fix these remaining problems, so users should be aware of this.

ALERT: Another know problem with (plotxy) is that it fails if the path to your working directory contains spaces in its name, e.g. /users/smyers/MyTest/ is fine, but /users/smyers/My Test/ is not!

3.3.2.1  GUI Plot Control

You can use the various buttons on the plotxy GUI to control its operation – in particular, to determine flagging and unflagging behaviors.

There is a standard row of buttons at the bottom. These include (left to right):

In a row above these, there are a set of other buttons (left to right):

These buttons are shared with the plotcal tool.

3.3.2.2  The selectplot Parameters

These parameters work in concert with the native matplotlib functionality to enable flexible representations of data displays.

Setting selectplot=True will open up a set of plotting control sub-parameters:


selectplot       =       True   #  Select additional plotting options (e.g, fontsize, title,etc)
     markersize  =        5.0   #  Size of plotted marks
     linewidth   =        1.0   #  Width of plotted lines
     skipnrows   =          1   #  Plot every nth point
     newplot     =      False   #  Replace the last plot or not when overplotting
     clearpanel  =     'Auto'   #  Specify if old plots are cleared or not
     title       =         ''   #  Plot title (above plot)
     xlabels     =         ''   #  Label for x-axis
     ylabels     =         ''   #  Label for y-axis
     fontsize    =       10.0   #  Font size for labels
     windowsize  =        5.0   #  Window size: not yet implemented

Inside the Toolkit:

For even more functionality, you can access the pl tool directly using Pylab functions that allow one to annotate, alter, or add to any plot displayed in the matplotlib plotter (e.g. plotxy).

The markersize parameter will change the size of the plot symbols. Increasing it will help legibility when doing screen shots. Decreasing it can help in congested plots. The linewidth parameter will do similar things to the lines.

The skipnrows parameter, if set to an integer n greater than 1, will allow only every nth point to be plotted. It does this, as the name suggests, by skipping over whole rows of the MS, so beware (channels are all within the same row for a given spw). Be careful flagging on data where you have skipped points! Note that you can also reduce the number of points plotted via averaging (§ 3.3.2.9) or channel striding in the spw specification (§ 2.3.3).

The newplot toggle lets you choose whether or not the last layer plotted is replaced when overplot=True, or whether a new layer is added.

The clearpanel parameter turns on/off the clearing of plot panels that lie under the current panel layer being plotted. The options are: ’none’ (clear nothing), ’auto’ (automatically clear the plotting area), ’current’ (clear the current plot area only), and ’all’ (clear the whole plot panel).

The title, xlabels, and ylabels parameters can be used to change the plot title and axes labels.

The fontsize parameter is useful in order to enlarge the label fonts so as to be visible when making plots for screen capture, or just to improve legibility. Shrinking can help if you have lots of panels on the plot also.

The windowsize parameter is supposed to allow adjustments on the window size. ALERT: This currently does nothing, unless you set it below 1.0, in which case it will produce an error.

3.3.2.3   The iteration parameter

Under Page one can select plot iterations parameters. There are currently four iteration options available: ’field’, ’antenna’, and ’baseline’. If one of these options is chosen, the data will be split into separate plot displays for each value of the iteration axis (e.g., for the VLA, the ’antenna’ option will get you 27 displays, one for each antenna). An example use of iteration:


  # choose channel averaging, every 5 channels
  plotxy('n5921.ms','channel',subplot=221,iteration='antenna',width='5')

The results of this are shown in Figure 3.6. Note that this example combines the use of width, iteration and subplot.


Figure 3.6: The plotxy iteration plot. The first set of plots from the example in § 3.3.2.3 with iteration=’antenna’. Each time you press the Next button, you get the next series of plots.

NOTE: If you use iteration=’antenna’ or ’baseline’, be aware if you have set antenna selection. You can also control whether you see auto-correlations or not using the appropriate syntax, e.g. antenna=’*&&*’ or antenna=’*&&&’ (§ 2.3.4.1).

3.3.2.4   The overplot parameter

The overplot parameter toggles whether the current plot will be overlaid on the previous plot or subpanel (via the subplot setting, § section:edit.plot.plotxy.subplot) or will overwrite it. The default is False and the new plot will replace the old.

The overplot parameter interacts with the newplot sub-parameter (see § 3.3.2.2).

See § 3.3.2.7 for an example using overplot.

3.3.2.5   The plotrange parameter

The plotrange parameter can be used to specify the size of the plot. The format is [xmin, xmax, ymin, ymax]. The units are those on the plot. For example,


   plotrange = [-20,100,15,30]

Note that if xmin=xmax and/or ymin=ymax, then the values will be ignored and a best guess will be made to auto-range that axis.

Unfortunately, the units for the time axis must be in Julian seconds. This is somewhat inconvenient as the usual time parameter is given in Julian days. To calculate the Julian seconds the me.epoch tool can be used. An example: For 02:00 UT on 2012/05/22, the value of MJD seconds can be calculated via


86400*(me.epoch('utc','2012/05/22')['m0']['value']+2/24.) 

which results in 4844368800.0.

3.3.2.6   The plotsymbol parameter

The plotsymbol parameter defines both the line or symbol for the data being drawn as well as the color; from the matplotlib online documentation (e.g., type pl.plot? for help):


    The following line styles are supported:
        -     : solid line
        --    : dashed line
        -.    : dash-dot line
        :     : dotted line
        .     : points
        ,     : pixels
        o     : circle symbols
        ^     : triangle up symbols
        v     : triangle down symbols
        <     : triangle left symbols
        >     : triangle right symbols
        s     : square symbols
        +     : plus symbols
        x     : cross symbols
        D     : diamond symbols
        d     : thin diamond symbols
        1     : tripod down symbols
        2     : tripod up symbols
        3     : tripod left symbols
        4     : tripod right symbols
        h     : hexagon symbols
        H     : rotated hexagon symbols
        p     : pentagon symbols
        |     : vertical line symbols
        _     : horizontal line symbols
        steps : use gnuplot style 'steps' # kwarg only
    The following color abbreviations are supported
        b  : blue
        g  : green
        r  : red
        c  : cyan
        m  : magenta
        y  : yellow
        k  : black
        w  : white
    In addition, you can specify colors in many weird and
    wonderful ways, including full names 'green', hex strings
    '#008000', RGB or RGBA tuples (0,1,0,1) or grayscale
    intensities as a string '0.8'.
    Line styles and colors are combined in a single format string, as in
    'bo' for blue circles.

3.3.2.7   The showflags parameter

The showflags parameter determines whether only unflagged data (showflags=False) or flagged (showflags=True) data is plotted by this execution. The default is False and will show only unflagged “good” data.

Note that if you want to plot both unflagged and flagged data, in different colors, then you need to run plotxy twice using overplot (see § 3.3.2.4) the second time, e.g.


> plotxy(vis="myfile", xaxis='uvdist', yaxis='amp' )
> plotxy(vis="myfile", xaxis='uvdist', yaxis='amp', overplot=True, showflags=True )

3.3.2.8   The subplot parameter

The subplot parameter takes three numbers. The first is the number of y panels (stacking vertically), the second is the number of xpanels (stacking horizontally) and the third is the number of the panel you want to draw into. For example, subplot=212 would draw into the lower of two panels stacked vertically in the figure.

An example use of subplot capability is shown in Fig 3.7. These were drawn with the commands (for the top, bottom left, and bottom right panels respectively):


plotxy('n5921.ms','channel',     # plot channels for the n5921.ms data set
       field='0',                  # plot only first field
       datacolumn='corrected',     # plot corrected data
       plotcolor='',               # over-ride default plot color
       plotsymbol='go',            # use green circles
       subplot=211)                # plot to the top of two panels

plotxy('n5921.ms','x',           # plot antennas for n5921.ms data set
       field='0',                  # plot only first field
       datacolumn='corrected',     # plot corrected data
       subplot=223,                # plot to 3rd panel (lower left) in 2x2 grid
       plotcolor='',               # over-ride default plot color
       plotsymbol='r.')            # red dots

plotxy('n5921.ms','u','v',       # plot uv-coverage for n5921.ms data set
       field='0',                  # plot only first field
       datacolumn='corrected',     # plot corrected data
       subplot=224,                # plot to the lower right in a 2x2 grid
       plotcolor='',               # over-ride default plot color
       plotsymbol='b,')            # blue, somewhat larger dots
                                   # NOTE: You can change the gridding
                                   # and panel size by manipulating
                                   # the ny x nx grid.


Figure 3.7: Multi-panel display of visibility versus channel (top), antenna array configuration (bottom left) and the resulting uv coverage (bottom right). The commands to make these three panels respectively are: 1) plotxy(’ngc5921.ms’, xaxis=’channel’, datacolumn=’data’, field=’0’, subplot=211, plotcolor=”, plotsymbol=’go’) 2) plotxy(’ngc5921.ms’, xaxis=’x’, field=’0’, subplot=223, plotsymbol=’r.’), 3) plotxy(’ngc5921.ms’, xaxis=’u’, yaxis=’v’, field=’0’, subplot=224, plotsymbol=’b,’,figfile=’ngc5921_multiplot.png’).

See also § 3.3.2.3 above, and Figure 3.6 for an example of channel averaging using iteration and subplot.

3.3.2.9  Averaging in plotxy

The averaging parameters and sub-parameters are:


averagemode      = 'vector' #  Select averaging type: vector, scalar
     timebin     =      '0' #  length of time in seconds to average, default='0', or: 'all'
     crossscans  =    False #  have time averaging cross over scans?
     crossbls    =    False #  have averaging cross over baselines?
     crossarrays =    False #  have averaging cross over arrays?
     stackspw    =    False #  stack multiple spw on top of each other?
     width       =      '1' #  number of channels to average, default: '1', or: 'all', 'allspw'

The choice of averagemode controls how the amplitudes are calculated in the average. The default mode is ’vector’, where the complex average is formed by averaging the real and imaginary parts of the relevant visibilities. If ’scalar’ is chosen, then the amplitude of the average is formed by a scalar average of the individual visibility amplitudes.

Time averaging is effected by setting the timebin parameter to a value larger than the integration time. Currently, timebin takes a string containing the averaging time in seconds, e.g.


   timebin = '60.0'

to plot one-minute averages.

Channel averaging is invoked by setting width to a value greater than 1. Currently, the averaging width is given as a number of channels.

By default, the averaging will not cross scan boundaries (as set in the import process). However, if crossscans=True, then averaging will cross scans.

Note that data taken in different sub-arrays are never averaged together. Likewise, there is no way to plot data averaged over field.

3.3.2.10  Interactive Flagging in plotxy

Hint!

In the plotting environments such as plotxy, the ESC key can be used to remove the last region box drawn.

Interactive flagging, on the principle of “see it — flag it”, is possible on the X-Y display of the data plotted by plotxy. The user can use the cursor to mark one or more regions, and then flag, unflag, or list the data that falls in these zones of the display.

There is a row of buttons below the plot in the window. You can punch the Mark Region button (which will appear to depress), then mark a region by left-clicking and dragging the mouse (each click and drag will mark an additional region). You can get rid of all your regions by clicking again on the Mark Region button (which will appear to un-depress), or you can use the ESC key to remove the last box you drew. Once regions are marked, you can then click on one of the other buttons to take action:

  1. Flag — flag the points in the region(s),
  2. Unflag — unflag flagged points in the region(s),
  3. Locate — spew out a list of the points in the region(s) to the logger (Warning: this could be a long list!).

Whenever you click on a button, that action occurs without forcing a disk-write (unlike previous versions). If you quit plotxy and re-enter, you will see your previous edits.


Figure 3.8: Plot of amplitude versus uv distance, before (left) and after (right) flagging two marked regions. The call was: plotxy(vis=’ngc5921.ms’, xaxis=’uvdist’, field=’1445*’).

A table with the name <msname>.flagversions (where vis=<msname>) will be created in the same directory if it does not exist already.

It is recommended that you save important flagging stages using the flagmanager task (§ 3.2).

3.3.2.11  Flag extension in plotxy

Flag extension is controlled using extendflag=T and its sub-parameters:


extendflag          =   True    #  Have flagging extend to other data points?
     extendcorr     =     ''    #  flagging correlation extension type
     extendchan     =     ''    #  flagging channel extension type
     extendspw      =     ''    #  flagging spectral window extension type
     extendant      =     ''    #  flagging antenna extension type
     extendtime     =     ''    #  flagging time extension type

The use of extendflag enables the user to plot a subset of the data and extend the flagging to a wider set.

ALERT: Using the extendflag options will greatly slow down the flagging in plotxy. You will see a long delay after hitting the Flag button, with lots of logger messages as it goes through each flag. Fixing this requires a refactoring of plotxy which is underway starting in Patch 4 development.

Setting extendchan=’all’ will extend the flagging to other channels in the same spw as the displayed point. For example, if spw=’0:0’ and channel 0 is displayed, then flagging will extend to all channels in spw 0.

The extendcorr sub-parameter will extend the flagging beyond the correlations displayed. If extendcorr=’all’, then all correlations will be flagged, e.g. with RR displayed RR,RL,LR,LL will be flagged. If extendcorr=’half’, then the extension will be to those correlations in common with that show, e.g. with RR displayed then RR,RL,LR will be flagged.

Setting extendspw=’all’ will extend the flagging to all other spw for the selection. Using the same example as above, with spw=’0:0’ displayed, then channel 0 in ALL spw will be flagged. Note that use of extendspw could result in unintended behavior if the spw have different numbers of channels, or if it is used in conjunction with extendchan.

WARNING: use of the following options, particularly in conjunction with other flag extensions, may lead to deletion of much more data than desired. Be careful!

Setting extendant=’all’ will extend the flagging to all baselines that have antennas in common with those displayed and marked. For example, if antenna=’1&2’ is shown, then ALL baselines to BOTH antennas 1 and 2 will be flagged. Currently, there is no option to extend the flag to ONLY baselines to the first (or second) antenna in a displayed pair.

Setting extendtime=’all’ will extend the flagging to all times matching the other selection or extension for the data in the marked region.

3.3.2.12  Setting rest frequencies in plotxy

The restfreq parameter can be set to a transition or frequency and expands to allow setting of frame information. For example,


restfreq            =    'HI'   #  a frequency quanta or transition name. see help for options
     frame          =  'LSRK'   #  frequency frame for spectral axis. see help for options
     doppler        = 'RADIO'   #  doppler mode. see help for options

Examples of transitions include:


   restfreq='1420405751.786Hz'  #  21cm HI frequency
   restfreq='HI'                #  21cm HI transition name
   restfreq='115.2712GHz'       #  CO 1-0 line frequency

For a list of known lines in the CASA measures system, use the toolkit command me.linelist(). For example:


CASA <14>: me.linelist()
  Out[14]: 'C109A CI CII166A DI H107A H110A H138B H166A H240A H272A
            H2CO HE110A HE138B HI OH1612 OH1665 OH1667 OH1720'

ALERT: The list of known lines in CASA is currently very restricted, and will be increased in upcoming releases (to include lines in ALMA bands for example).

You can use the me.spectralline tool method to turn transition names into frequencies


CASA <16>: me.spectralline('HI')
  Out[17]: 
{'m0': {'unit': 'Hz', 'value': 1420405751.786},
 'refer': 'REST',
 'type': 'frequency'}

(not necessary for this task, but possibly useful).

The frame sub-parameter sets the frequency frame. The allowed options can be listed using the me.listcodes method on the me.frequency() method, e.g.


CASA <17>: me.listcodes(me.frequency())
  Out[17]: 
{'extra': array([], 
      dtype='|S1'),
 'normal': array(['REST', 'LSRK', 'LSRD', 'BARY', 'GEO', 'TOPO', 'GALACTO', 'LGROUP',
       'CMB'], 
      dtype='|S8')}

The doppler sub-parameter likewise sets the Doppler system. The allowed codes can be listed using the me.listcodes method on the me.doppler() method,


CASA <18>: me.listcodes(me.doppler())
  Out[18]: 
{'extra': array([], 
      dtype='|S1'),
 'normal': array(['RADIO', 'Z', 'RATIO', 'BETA', 'GAMMA', 'OPTICAL', 'TRUE',
       'RELATIVISTIC'], 
      dtype='|S13')}

For most cases the ’RADIO” Doppler system is appropriate, but be aware of differences.

For more information on frequency frames and spectral coordinate systems, see the paper by Greisen et al. (A&A, 446, 747, 2006) 2.

3.3.2.13  Printing from plotxy

There are two ways to get hardcopy plots in plotxy.

The first is to use the “disk save” icon from the interactive plot GUI to print the current plot. This will bring up a sub-menu GUI that will allow you to choose the filename and format. The allowed formats are .png (PNG), .eps (EPS), and svg (SVG). If you give the filename with a suffix (.png, .eps, or svg) it will make a plot of that type. Otherwise it will put a suffix on depending on the format chosen from the menu.

ALERT: The plot files produced by the EPS option can be large, and the SVG files can be very large. The PNG is the smallest.

The second is to specify a figfile. You probably want to disable the GUI using interactive=False in this case. The type of plot file that is made will depend upon the filename suffix. The allowed choices are .png (PNG), .eps (EPS), and svg (SVG).

This latter option is most useful from scripts. For example,


   default('plotxy')
   vis = 'ngc5921.ms'
   field = '2'
   spw = ''
   xaxis = 'uvdist'
   yaxis = 'amp'
   interactive=False
   figfile = 'ngc5921.uvplot.amp.png'
   plotxy()

will plot amplitude versus uv-distance in PNG format. No plotxy GUI will appear.

ALERT: if you use this option to print to figfile with an iteration set, you will only get the first plot.

3.3.2.14  Exiting plotxy

You can use the Quit button to clear the plot from the window and detach from the MS. You can also dismiss the window by killing it with the X on the frame, which will also detach the MS.

You can also just leave it alone. The plotter pretty much keeps running in the background even when it looks like it’s done! You can keep doing stuff in the plotter window, which is where the overplot parameter comes in. Note that the plotcal task (§ 4.5.1) will use the same window, and can also overplot on the same panel.

If you leave plotxy running, beware of (for instance) deleting or writing over the MS without stopping. It may work from a memory version of the MS or crash.

3.3.2.15  Example session using plotxy

The following is an example of interactive plotting and flagging using plotxy on the Jupiter 6cm continuum VLA dataset. This is extracted from the script jupiter6cm_usecase.py available in the script area.

This assumes that the MS jupiter6cm.usecase.ms is on disk with flagautocorr already run.


default('plotxy')

vis = 'jupiter6cm.usecase.ms'

# The fields we are interested in: 1331+305,JUPITER,0137+331
selectdata = True

# First we do the primary calibrator
field = '1331+305'

# Plot only the RR and LL for now
correlation = 'RR LL'

# Plot amplitude vs. uvdist
xaxis = 'uvdist'
yaxis = 'amp'
multicolor = 'both'

# The easiest thing is to iterate over antennas
iteration = 'antenna'

plotxy()

# You'll see lots of low points as you step through RR LL RL LR
# A basic clip at 0.75 for RR LL and 0.055 for RL LR will work
# If you want to do this interactively, set
iteration = ''

plotxy()

# You can also use flagdata to do this non-interactively
# (see below)

# Now look at the cross-polar products
correlation = 'RL LR'

plotxy()

#---------------------------------------------------------------------
# Now do calibrator 0137+331
field = '0137+331'
correlation = 'RR LL'
xaxis = 'uvdist'
spw = ''
iteration = ''
antenna = ''

plotxy()

# You'll see a bunch of bad data along the bottom near zero amp
# Draw a box around some of it and use Locate
# Looks like much of it is Antenna 9 (ID=8) in spw=1

xaxis = 'time'
spw = '1'
correlation = ''

# Note that the strings like antenna='9' first try to match the 
# NAME which we see in listobs was the number '9' for ID=8.
# So be careful here (why naming antennas as numbers is bad).
antenna = '9'

plotxy()

# YES! the last 4 scans are bad.  Box 'em and flag.

# Go back and clean up
xaxis = 'uvdist'
spw = ''
antenna = ''
correlation = 'RR LL'

plotxy()

# Box up the bad low points (basically a clip below 0.52) and flag

# Note that RL,LR are too weak to clip on.

#---------------------------------------------------------------------
# Finally, do JUPITER
field = 'JUPITER'
correlation = ''
iteration = ''
xaxis = 'time'

plotxy()

# Here you will see that the final scan at 22:00:00 UT is bad
# Draw a box around it and flag it!

# Now look at what's left
correlation = 'RR LL'
xaxis = 'uvdist'
spw = '1'
antenna = ''
iteration = 'antenna'

plotxy()

# As you step through, you will see that Antenna 9 (ID=8) is often 
# bad in this spw. If you box and do Locate (or remember from
# 0137+331) it's probably a bad time.

# The easiest way to kill it:

antenna = '9'
iteration = ''
xaxis = 'time'
correlation = ''

plotxy()

# Draw a box around all points in the last bad scans and flag 'em!

# Now clean up the rest
xaxis = 'uvdist'
correlation = 'RR LL'
antenna = ''
spw = ''

# You will be drawing many tiny boxes, so remember you can
# use the ESC key to get rid of the most recent box if you
# make a mistake.

plotxy()

# Note that the end result is we've flagged lots of points
# in RR and LL.  We will rely upon imager to ignore the
# RL LR for points with RR LL flagged!

3.3.3  Plotting antenna positions using plotants

This task is a simple plotting interface (to the plotxy functionality) to produce plots of the antenna positions (taken from the ANTENNA sub-table of the MS).

The inputs to plotants are:


#  plotants :: Plot the antenna distribution in the local reference frame:
vis       =         ''   #  Name of input visibility file (MS)
figfile   =         ''   #  Save the plotted figure to this file

3.3.4  Plotting uv-coverages plotuv

A simple way to plot uv-coverages is offered by the task plotuv:


#  plotuv :: Plot the baseline distribution
vis                 =         ''        #  Name of input visibility file (MS)
field               =         ''        #  Select field using ID(s) or name(s)
antenna             =         ''        #  Select data based on antenna/baseline
spw                 =         ''        #  Select spectral window/channels
observation         =         ''        #  Select by observation ID(s)
array               =         ''        #  Select (sub)array(s) by array ID number
maxnpts             =     100000        #  Maximum number of points per plot.
colors              = ['r', 'y', 'g', 'b'] #  a list of matplotlib color codes
symb                =        ','        #  A matplotlib plot symbol code
ncycles             =          1        #  How many times to cycle through colors per
                                        #   plot.
figfile             =         ''        #  Save the plotted figure(s) using this name

plotuv provides basic selection of data as well as plotting style options. The difference to plotms is that plotuv is also plotting the Hermitian conjugates of the visibilities which produces the familiar symmetric plots. This is a remedy to the restriction in plotms to allow flagging of data. This is achieved via a unambiguous link from a displayed data point to a visibility. Plotting Hermitian conjugates would break this rule in plotms and plotuv is used instead to plot Hermitian conjugates.

3.4  Data Flagging using flagdata

flagdata can flag measurement sets and calibration tables with an elaborate selection syntax. It also contains auto-flagging routines.

For a full description of flagdata please visit:

http://www.aoc.nrao.edu/\~rurvashi/FlaggerDocs/FlaggerDocs.html

The inputs to flagdata are:


#  flagdata :: All-purpose flagging task based on data-selections and flagging modes/algorithms.
vis                 =         ''        #  Name of MS file or calibration table to flag
mode                =   'manual'        #  Flagging mode
     field          =         ''        #  Field names or field index
                                        #  numbers: '' ==> all, field='0~2,3C286'
     spw            =         ''        #  Spectral-window/frequency/channel: '' ==> all, spw='0:17~19'
     antenna        =         ''        #  Antenna/baselines: '' ==> all, antenna ='3,VA04'
     timerange      =         ''        #  Time range: '' ==> all,timerange='09:14:0~09:54:0'
     correlation    =         ''        #  Correlation: '' ==> all, correlation='XX,YY'
     scan           =         ''        #  Scan numbers: '' ==> all
     intent         =         ''        #  Observation intent: '' ==> all, intent='CAL*POINT*'
     array          =         ''        #  (Sub)array numbers: '' ==> all
     uvrange        =         ''        #  UV range: '' ==> all;
                                        #  uvrange ='0~100klambda', default units=meters
     observation    =         ''        #  Observation ID: '' ==> all
     feed           =         ''        #  Multi-feed numbers: Not yet implemented
     autocorr       =      False        #  Flag auto-correlations

action              =    'apply'        #  Action to perform in MS
                                        #  and/or in inpfile (none/apply/calculate)
     display        =         ''        #  Display data and/or
                                        #  end-of-MS reports at runtime (data/report/both).
     flagbackup     =       True        #  Back up the state of flags before the run

savepars            =      False        #  Save the current parameters
                                        #  to the FLAG_CMD table or to a file

vis can take a measurement set or calibration table. Data selection for calibration tables is limited to field, scan, time, antenna, spw, and observation. Since calibration tables do not have a FLAG_CMD table, parameter settings, if requested, can only be saved in external files.

The mode parameter (§3.4.2) selects the flagging algorithm and the following are available:


list        = list of flagging commands to apply to MS
manual      = flagging based on specific selection parameters
clip        = clip data according to values
quack       = remove/keep specific time range at scan beginning/end
shadow      = remove antenna-shadowed data
elevation   = remove data below/above given elevations
tfcrop      = automatic identification of outliers on the time-freq plane
rflag       = automatic detection of outliers based on sliding-window RMS filters
extend      = extend and/or grow flags beyond what the basic algorithms detect
summary     = report the amount of flagged data
unflag      = unflag the specified data

Flagging will only be applied to the data selection that is performed with the usual selection parameters (§ 2.3). The dataset is iterated-through in chunks (small pieces of data) consisting of one field, one spw, and a user-defined timerange (default is one scan). In addition to the typical antenna, spw, timerange, etc. selections, we would like to point out some addition of the correlation syntax for modes clip, tfcrop, and rflag. One can combine correlation products with simple mathematical expressions


'ABS', 'ARG', 'RE', 'IM', 'NORM' 

followed by the polarization products (using an underscore in between “_” )


'ALL', 'I', 'XX', 'YY', 'RR', 'LL', 'WVR' 

’WVR’ refers to the water vapour radiometer of ALMA data. Note that the operators ABS,ARG,RE, etc. are written only once as the first value. if more than one correlation is given, the operator will be applied to all of them. An example would be


correlation='RE_XX,XY'

which would select all real XX and XY polarization for flagging.

3.4.1  The action parameter

The keyword action controls whether the actual flagging commands will be applied or not and the options are the empty string , ’apply’ and ’calculate’.

apply is likely the most popular one as it applies the flags to the MS:


action              =    'apply'        #  Action to perform in MS and/or in inpfile
                                        #   (none/apply/calculate)
     display        =         ''        #  Display data and/or end-of-MS reports at runtime
                                        #   (data/report/both).
     flagbackup     =       True        #  Back up the state of flags before the run

flagbackup specifies if a backup of the current flags should be saved in the “*.flagversions” file. display can be ”, ’data’, ’report’, ’both’ where the empty string will report no individual flagging statistics, whereas ’data’ launches an interactive GUI to display data and flags for each chunk to browse through. The plots are time-frequency planes and both old and new flags are being overlaid for all correlations per baseline. In the GUI, one can step though all chunks for inspection and if the flagging is unsatisfactory, one can exit without applying the flags. If the flagging is acceptable, it is also possible to continue flagging without viewing all chunks (the number of chunks can be very large for typical JVLA and ALMA data sets. display=’report’ lists the flagging statistics at the end of the procedure on the screen and both starts the GUI and reports all statistics at the end.

action=’calculate’ calculates the flags but does not write them to the MS or calibration table. This is useful if one would like to inspect the computed flags in the GUI without a straight application:


action              = 'calculate'       #  Action to perform in MS and/or in inpfile
                                        #   (none/apply/calculate)
     display        =         ''        #  Display data and/or end-of-MS reports at runtime
                                        #   (data/report/both).

The empty string action=” will do nothing and is useful when the commands themselves shall only be written to the FLAG_CMD sub-table or to an external file using the savepars parameter to specify the filename.

savepars will save the flagging commands to a file that can be later used for input in flagdata via mode=’list’. It also shares the flagcmd syntax and can be used there. The file name is specified by outfile and, if empty, the FLAG_CMD table in the MS will be populated. A REASON can be given by the reason keyword which may be useful for bookkeeping as well as for unflagging data that are marked by specific REASON keywords. The overwrite parameter will control overwriting an existing file when saving the flag commands.

3.4.2  Flagging Modes

3.4.2.1  Manual Flag/Unflag


mode                =   'manual'        #  Flagging mode (list/manual/clip/shadow/quack/el
                                        #   evation/tfcrop/rflag/extend/unflag/summary)
     field          =         ''        #  Field names or field index numbers: '' ==> all,
                                        #   field='0~2,3C286'
     spw            =         ''        #  Spectral-window/frequency/channel: '' ==> all,
                                        #   spw='0:17~19'
     antenna        =         ''        #  Antenna/baselines: '' ==> all, antenna
                                        #   ='3,VA04'
     timerange      =         ''        #  Time range: '' ==>
                                        #   all,timerange='09:14:0~09:54:0'
     correlation    =         ''        #  Correlation: '' ==> all, correlation='XX,YY'
     scan           =         ''        #  Scan numbers: '' ==> all
     intent         =         ''        #  Observation intent: '' ==> all,
                                        #   intent='CAL*POINT*'
     array          =         ''        #  (Sub)array numbers: '' ==> all
     uvrange        =         ''        #  UV range: '' ==> all; uvrange ='0~100klambda',
                                        #   default units=meters
     observation    =         ''        #  Observation ID: '' ==> all
     feed           =         ''        #  Multi-feed numbers: Not yet implemented
     autocorr       =      False        #  Flag auto-correlations

The ’manual’ mode is the most straight-forward of all modes. All visibilities that are selected by the various data selection parameters will be flagged or unflagged, depending on the action parameter. autocorr is a shorthand for antenna=’*&&&’ to flag all auto correlations in the data.

3.4.2.2  List


mode                =     'list'        #  Flagging mode (list/manual/clip/shadow/quack/el
                                        #   evation/tfcrop/rflag/extend/unflag/summary)
     inpfile        =         ''        #  Input ASCII file, list of
                                        #  files or Python list of strings with 

                                        #   flag commands.
     reason         =      'any'        #  Select by REASON types

A list of flag commands can be provided through a file or a list of files, specified by the inpfile parameter. Each input line may contain a flagging mode with data selection parameters as well as parameters that are specific to that mode. All parameters that are not set will be reset to their default values (default mode is ’manual’). Each line of this file or list of strings will be taken as a command to the flagdata task. This mode=’list’ is similar to the task flagcmd with the inpmode=’list’ option.

An example for such a file would be:


mode='shadow'
mode='clip' clipminmax=[0,5] correlation='ABS_ALL'
mode='quack' quackmode='end' quackinterval=1.0
antenna='ea01' timerange='00:00:00~01:00:00'
antenna='ea11' timerange='00:00:00~03:00:00' spw='0~4'

Alternatively, this can be issued in the task directly like:


flagdata(vis='vis',mode='list',
inpfile=["mode='shadow'",
         "mode='clip' clipminmax=[0,5] correlation='ABS_ALL'",
         "mode='quack' quackmode='end' quackinterval=1.0"'
         "antenna='ea01' timerange='00:00:00~01:00:00'",
         "antenna='ea11' timerange='00:00:00~03:00:00' spw='0~4'"])

or via a variable


cmds=["mode='shadow',
           "mode='clip' clipminmax=[0,5] correlation='ABS_ALL'",
           "mode='quack' quackmode='end' quackinterval=1.0",
           "antenna='ea01' timerange='00:00:00~01:00:00'",
           "antenna='ea11' timerange='00:00:00~03:00:00' spw='0~4'"]


flagdata(vis='vis',mode='list', inpfile=cmds)

The syntax needs to be written with quotes e.g. mode=’manual’ antenna=’ea10’. There should be no space between key=value. Spaces are used to separate pairs of parameters, not commas.

3.4.2.3  Clip


mode                =     'clip'        #  Flagging mode (list/manual/clip/shadow/quack/
                                        #  elevation/tfcrop/rflag/extend/unflag/summary)
...
     datacolumn     =     'DATA'        #  Data column on which to operate
                                        #   (data,corrected,model,residual)
     clipminmax     =         []        #  Range to use for clipping
     clipoutside    =       True        #  Clip outside the range, or within it
     channelavg     =      False        #  Average over channels (scalar average)
     timeavg        =      False        #  Average over time ranges
     timebin        =         ''        #  Bin width for time averaging.
     clipzeros      =      False        #  Clip zero-value data

in addition to the regular selection parameters, mode=’clip’ also has an option to select between a number of scratch columns in datacolumn. This includes the usual DATA, CORRECTED, etc., and also clipping based on data weights WEIGHT, WEIGHT_SPECTRUM as well as other MS columns. clipminmax selects the range of values to be clipped – usually this is combined with clipoutside=True to clip everything but the values covered in clipminmax. The data can also be averaged over the selected spw channel ranges by setting channelavg=True, or time averages via timeavg=True and setting of timebin. clip will also flag ’NaN’, ’inf’, and ’-inf’ values by default and can flag exact zero values (these are sometimes produced by the JVLA correlator) using the clipzeros parameter.

Note : For modes clip, tfcrop and rflag, channel-ranges can be excluded from flagging by selecting ranges such as spw=’0:05;1063’. This is a way to protect known spectral-lines from being flagged by the autoflag algorithms.

3.4.2.4  Shadow


mode                =   'shadow'        #  Flagging mode (list/manual/clip/shadow/quack/
                                        #  elevation/tfcrop/rflag/extend/unflag/summary)
...
     tolerance      =        0.0        #  Amount of shadow allowed (in meters)
     addantenna     =         ''        #  File name or dictionary with additional antenna names,
                                        #   positions and diameters

This option flags shadowed antennas, i.e. when one antenna blocks part of the aperture of a second antenna that is behind the first one. Shadowing can be gradual and the criterion for a shadow flag is when a baseline is shorter than radius1 + radius2tolerance (where the radii of the antennae are taken from the MS antenna subtable); see Fig. 3.9. addantenna may be used to account for shadowing when antennas are not listed in the MS but are physically present. Please read the flagdata inline help for the syntax of this option.


Figure 3.9: This figure shows the geometry used to compute shadowed antennas.

3.4.2.5  Quack


mode                =    'quack'        #  Flagging mode (list/manual/clip/shadow/quack/
                                        #   elevation/tfcrop/rflag/extend/unflag/summary)
...
     quackinterval  =      0.0          #  Quack n seconds from scan beginning or end
     quackmode      =     'beg'         #  flag an interval at the beginning of scan
                          'endb'        #  flag an interval at the end of scan
                          'tail'        #  flag all but an interval at the beginning of 
                                           scan
                          'end'         #  flag all but an interval at end of scan

     quackincrement =      False        #  Flag incrementally in time?

quack is used to remove data at scan boundaries. quackinterval specifies the time in seconds to be flagged, and quackmode can be ’beg’ to flag the quackinterval at the beginning of each selected scan, ’endb’ at the end of scan. ’tail’ flags all but the beginning of scan and ’end’ all but the end of scan. The quackincrement is either True or False, depending if one wishes to flag the quackinterval from the first unflagged data in the scan, or from the scan boundaries independent of data being already flagged or not.



      Visual representation of quack mode when flagging a scan 
      with 1s duration. The following diagram shows what is flagged 
      for each quack mode when quackinterval is set to 0.25s. 
      The flagged part is represented by crosses (+++++++++)

                     scan with 1s duration
          --------------------------------------------
          beg
          +++++++++++---------------------------------
                                           endb
          ---------------------------------+++++++++++
                     tail
          -----------+++++++++++++++++++++++++++++++++
          end
          +++++++++++++++++++++++++++++++++-----------

3.4.2.6  Elevation


mode                = 'elevation'       #  Flagging mode (list/manual/clip/shadow/quack/
                                        #   elevation/tfcrop/rflag/extend/unflag/summary)
...
     lowerlimit     =        0.0        #  Lower limiting elevation (in degrees)
     upperlimit     =       90.0        #  Upper limiting elevation (in degrees)

Flagging based on the elevation of the antennae. This may be useful to avoid data taken at very low elevations or close to transit and the lowerlimit and upperlimit parameters specify the range of good elevations.

3.4.2.7  Tfcrop


mode                =   'tfcrop'        #  Flagging mode (list/manual/clip/shadow/quack/
                                        #   elevation/tfcrop/rflag/extend/unflag/summary
                                        #   )
...
     ntime          =     'scan'        #  Time-range to use for each chunk (in seconds
                                        #   or minutes)
     combinescans   =      False        #  Accumulate data across scans.
     datacolumn     =     'DATA'        #  Data column on which to operate
                                        #   (data,corrected,model,residual)
     timecutoff     =        4.0        #  Flagging thresholds in units of deviation
                                        #   from the fit
     freqcutoff     =        3.0        #  Flagging thresholds in units of deviation
                                        #   from the fit
     timefit        =     'line'        #  Fitting function for the time direction
                                        #   (poly/line)
     freqfit        =     'poly'        #  Fitting function for the frequency direction
                                        #   (poly/line)
     maxnpieces     =          7        #  Number of pieces in the polynomial-fits (for
                                        #   'freqfit' or 'timefit' = 'poly')
     flagdimension  = 'freqtime'        #  Dimensions along which to calculate fits
                                        #   (freq/time/freqtime/timefreq)
     usewindowstats =     'none'        #  Calculate additional flags using sliding
                                        #   window statistics (none,sum,std,both)
     halfwin        =          1        #  Half-width of sliding window to use with
                                        #   'usewindowstats' (1,2,3).
     extendflags    =       True        #  Extend flags along time,
                                        #  frequency and correlation.
     channelavg     =       False       #  Pre-average data across channels before 
                                        #  analyzing visibilities for flagging.                                        
     chanbin        =       False       #  Bin width for channel average in
                                        #  number of input channels.
     timeavg        =       False       #  Pre-average data across time before 
                                        #  analyzing visibilities for flagging
     timebin        =       False       #  Bin width for time average in seconds
                                         

TFCrop is an autoflag algorithm that detects outliers on the 2D time-frequency plane, and can operate on un-calibrated data (non bandpass-corrected). The original implementation of this algorithm is described in NCRA Technical Report 202 (Oct 2003).

The algorithm iterates through the data in chunks of time. For each chunk, the result of user-specified visibility-expressions are organized as 2D time-frequency planes, one for each baseline and correlation-expression result, and the following steps are performed.

As of CASA 4.6 the data can also be pre-averaged over the selected spw channel ranges by setting channelavg=True and chanbin to the desired bin (in number of channels), or time averaged over the selected time ranges by setting timeavg=True and timebin to the desired time range (in seconds). This averaging is independent from the tfcrop time/channel average, and allows to specify custom time/channel average bins, instead of averaging all data across time and/or channel direction.

  1. Calculate a bandshape template : Average the data across time, to construct an average bandpass. Construct an estimate of a clean bandpass (without RFI) via a robust piece-wise polynomial fit to the average bandpass shape.

    Note : A robust fit is computed in up to 5 iterations. It begins with a straight line fit across the full range, and gradually increases to ’maxnpieces’ number of pieces with third-order polynomials in each piece. At each iteration, the stddev between the data and the fit is computed, values beyond N-stddev are flagged, and the fit and stddev are re-calculated with the remaining points. This stddev calculation is adaptive, and converges to a value that reflects only the data and no RFI. At each iteration, the same relative threshold is applied to detect flags, and this results in a varying set of flagging thresholds, that allows deep flagging only when the fit represents the true data best. Iterations stop when the stddev changes by less than 10%, or when 5 iterations are completed.

    The resulting clean bandpass is a fit across the base of RFI spikes.

  2. Divide out this clean bandpass function from all timesteps in the current chunk. Now, any data points that deviate from a mean of 1 can be considered RFI. This step helps to separate narrow-band RFI spikes from a smooth but varying bandpass, in situations where a simple range-based clipping will flag good sections of the bandpass.
  3. Perform iterative flagging (robust flagging) of points deviating from a value of 1.

    Flagging is done in up to 5 iterations. In each iteration, for every timestep, calculate the stddev of the bandpass-flattened data, flag all points further than N times stddev from the fit, and recalculate the stddev. At each iteration, the same relative threshold is applied to detect flags. Optionally, use sliding-window based statistics to calculate additional flags.

  4. Repeat steps 1 and 3, but in the other direction (i.e. average the data across frequency, calculate a piece-wise polynomial fit to the average time-series, and find flags based on deviations w.r.to this fit.)

The default parameters of the tfcrop implementation are optimized for strong narrow-band RFI (see, e.g. Fig. 3.10). With broad-band RFI, the piece-wise polynomial can sometimes model it as part of the band-shape, and therefore not detect it as RFI. In this case, reducing the maximum number of pieces in the polynomial can help. This algorithm usually has trouble with noisy RFI that is also extended in time of frequency, and additional statistics-based flagging is recommended (via the ’usewindowstats’ parameter). It is often required to set up parameters separately for each spectral-window.

If frequency ranges of known astronomical spectral lines are known a-priori , they can be protected from automatic flagging by de-selecting those frequency-ranges via the ’spw’ data-selection parameter.

The extendflag parameter will clean up small portions of data between flagged data points along time and/or frequency when more than 50% of all timeranges or 80% of all channels are already flagged. It will also extend the flags to the other polarizations. Alternatively, mode=’extend’ can be used (Fig. 3.12).


Figure 3.10: This screenshot represents a run where ’tfcrop’ was run on a spw=’9’ with mainly narrow-band RFI. RIGHT : An example of protecting a spectral line (in this case, demonstrated on an RFI spike) by setting the spw-selection to spw=’0:0 45;53 63’. In both figures, the top row indicates the data before flagging, and the bottom row after flagging.

3.4.2.8  Rflag



mode                =    'rflag'        #  Flagging mode (list/manual/clip/shadow/quack/
                                        #   elevation/tfcrop/rflag/extend/unflag/summary
                                        #   )
...
     ntime          =     'scan'        #  Time-range to use for each chunk (in seconds
                                        #   or minutes)
     combinescans   =      False        #  Accumulate data across scans.
     datacolumn     =     'DATA'        #  Data column on which to operate
                                        #   (data,corrected,model,residual)
     winsize        =          3        #  Number of timesteps in the sliding time
                                        #   window [aips:fparm(1)]
     timedev        =         ''        #  Time-series noise estimate [aips:noise]
     freqdev        =         ''        #  Spectral noise estimate [aips:scutoff]
     timedevscale   =        5.0        #  Threshold scaling for timedev [aips:fparm(9)]
     freqdevscale   =        5.0        #  Threshold scaling for freqdev
                                        #   [aips:fparm(10)]
     spectralmax    =  1000000.0        #  Flag whole spectrum if freqdev is greater
                                        #   than spectralmax [aips:fparm(6)]
     spectralmin    =        0.0        #  Flag whole spectrum if freqdev is less than
                                        #   spectralmin [aips:fparm(5)]
     extendflags    =       True        #  Extend flags along time, frequency and correlation.
     channelavg     =       False       #  Pre-average data across channels before 
                                        #  analyzing visibilities for flagging.                                        
     chanbin        =       False       #  Bin width for channel average in
                                        #  number of input channels.
     timeavg        =       False       #  Pre-average data across time before 
                                        #  analyzing visibilities for flagging
     timebin        =       False       #  Bin width for time average in seconds     
                                       

RFlag is an autoflag algorithm based on a sliding window statistical filter. The RFlag algorithm was originally developed by Eric Greisen in AIPS (31DEC11). AIPS documentation : Subsection E.5 of the AIPS cookbook (Appendix E : Special Considerations for JVLA data calibration and imaging in AIPS3)

In RFlag, the data is iterated-through in chunks of time, statistics are accumulated across time-chunks, thresholds are calculated at the end, and applied during a second pass through the dataset.

The CASA implementation also optionally allows a single-pass operation where statistics and thresholds are computed and also used for flagging, within each time-chunk (defined by ’ntime’ and ’combinescans’).

For each chunk, calculate local statistics, and apply flags based on user supplied (or auto-calculated) thresholds.

As of CASA 4.6 the data can also be pre-averaged over the selected spw channel ranges by setting channelavg=True and chanbin to the desired bin (in number of channels), or time averaged over the selected time ranges by setting timeavg=True and timebin to the desired time range (in seconds). This averaging is independent from the rflag time/channel sliding window, as it performs not only an average but also a binning operation (so there is no data overlap between adjacent bins), and allows to specify independent time/channel bins.

  1. Time analysis (for each channel)
    1. Calculate local rms of real and imag visibilities, within a sliding time window
    2. Calculate the median rms across time windows, deviations of local rms from this median, and the median deviation
    3. Flag if local rms is larger than timedevscale x (medianRMS + medianDev)
  2. Spectral analysis (for each time)
    1. Calculate avg of real and imag visibilities and their rms across channels
    2. Calculate the deviation of each channel from this avg, and the median-deviation
    3. Flag if deviation is larger than freqdevscale x medianDev

The extendflag parameter will clean up small portions of data between flagged data points along time and/or frequency when more than 50% of all timeranges or 80% of all channels are already flagged. It will also extend the flags to the other polarizations. Alternatively, mode=’extend’ can be used.

Some examples (also see Fig. 3.11):

  1. Calculate thresholds automatically per scan, and use them to find flags. Specify scale-factor for time-analysis thresholds, use default for frequency.
    
    flagdata('my.ms', mode='rflag',spw='9',timedevscale=4.0,writeflags=True)
    
  2. Supply noise-estimates to be used with default scale-factors.
    
    flagdata(vis='my.ms', mode='rflag', spw='9', timedev=0.1, freqdev=0.5, writeflags=True);
    

    Two-passes. This replicates the usage pattern in AIPS.


Figure 3.11: Example of rflag on narrow-band RFI

3.4.2.9  Extend


mode                =   'extend'        #  Flagging mode (list/manual/clip/shadow/quack/el
                                        #   evation/tfcrop/rflag/extend/unflag/summary)
     field          =         ''        #  Field names or field index numbers: '' ==> all,
                                        #   field='0~2,3C286'
     spw            =         ''        #  Spectral-window/frequency/channel: '' ==> all,
                                        #   spw='0:17~19'
     antenna        =         ''        #  Antenna/baselines: '' ==> all, antenna
                                        #   ='3,VA04'
     timerange      =         ''        #  Time range: '' ==>
                                        #   all,timerange='09:14:0~09:54:0'
     correlation    =         ''        #  Correlation: '' ==> all, correlation='XX,YY'
     scan           =         ''        #  Scan numbers: '' ==> all
     intent         =         ''        #  Observation intent: '' ==> all,
                                        #   intent='CAL*POINT*'
     array          =         ''        #  (Sub)array numbers: '' ==> all
     uvrange        =         ''        #  UV range: '' ==> all; uvrange ='0~100klambda',
                                        #   default units=meters
     observation    =         ''        #  Observation ID: '' ==> all
     feed           =         ''        #  Multi-feed numbers: Not yet implemented
     ntime          =     'scan'        #  Time-range to use for each chunk (in seconds or
                                        #   minutes)
     combinescans   =      False        #  Accumulate data across scans.
     extendpols     =       True        #  If any correlation is flagged, flag all
                                        #   correlations
     growtime       =       50.0        #  Flag all 'ntime' integrations if more than X%
                                        #   of the timerange is flagged (0-100)
     growfreq       =       50.0        #  Flag all selected channels if more than X% of
                                        #   the frequency range is flagged(0-100)
     growaround     =      False        #  Flag data based on surrounding flags
     flagneartime   =      False        #  Flag one timestep before and after a flagged
                                        #   one (True/False)
     flagnearfreq   =      False        #  Flag one channel before and after a flagged one
                                        #   (True/False)

Although the modes tfcrop and rflag already have extendflag parameters, some autoflagging algorithms may still leave small islands of unflagged data behind, data that are surrounded by flagged visibilities in the time-frequency space. Although the algorithm may deem these visibilities as good ones, they are frequently affected by low-level RFI that spills from the adjacent, flagged points and one may wish to clean those up.

ntime specifies the time ranges over which to clean up, e.g. ’1.5min’ or ’scan’ which checks on all data within a scan. To span time ranges larger than scans, one can set combinescans to True.

extendpols=True would extend all flags to all polarization products when at least one of them is flagged.

growtime flags the entire time range for a flagged channel, when a certain fraction of flagged time intervals is exceeded.

growfreq is similar but extends the flags in frequency when a given fraction of channels is already flagged.

growaround checks for flagged data points in the time-frequency domain that neighbor a datum. The threshold is four data points. If more surrounding points are flagged, the central datum will be flagged, too.

flagneartime flags adjacent data points along the time axis, around a flagged datum

flagnearfreq flags neighboring channels.

For an example, see Fig. 3.12.


Figure 3.12: This screenshot represents a run where ’tfcrop’ was run only on ’ABS_RR’ (top row) and followed by an extension along time and correlations (bottom row).

3.4.2.10  Unflag


mode                =   'unflag'        #  Flagging mode (list/manual/clip/shadow/quack/
                                        #   elevation/tfcrop/rflag/extend/unflag/summary
                                        #   )
     field          =         ''        #  Field names or field index numbers: ''==>all,
                                        #   field='0~2,3C286'
     spw            =         ''        #  spectral-window/frequency/channel
     antenna        =     'ea01'        #  antenna/baselines: ''==>all, antenna
                                        #   ='3,VA04'
     timerange      =         ''        #  time range:
                                        #   ''==>all,timerange='09:14:0~09:54:0'
     correlation    =         ''        #  Select data based on correlation
     scan           =         ''        #  scan numbers: ''==>all
     intent         =         ''        #  Select data based on observation intent:
                                        #   ''==>all
     feed           =         ''        #  multi-feed numbers: Not yet implemented
     array          =         ''        #  (sub)array numbers: ''==>all
     uvrange        =         ''        #  uv range: ''==>all; uvrange ='0~100klambda',
                                        #   default units=meters
     observation    =         ''        #  Select data based on observation ID: ''==>all

The selection data will be unflagged.

3.4.2.11  Summary


mode                =  'summary'        #  Flagging mode (list/manual/clip/shadow/quack/
                                        #   elevation/tfcrop/rflag/extend/unflag/summary
                                        #   )
...
     minrel         =        0.0        #  minimum number of flags (relative)
     maxrel         =        1.0        #  maximum number of flags (relative)
     minabs         =          0        #  minimum number of flags (absolute)
     maxabs         =         -1        #  maximum number of flags (absolute). Use a
                                        #   negative value to indicate infinity.
     spwchan        =      False        #  Print summary of channels per spw
     spwcorr        =      False        #  Print summary of correlation per spw
     basecnt        =      False        #  Print summary counts per baseline

This mode reports the number of rows and data points that are flagged. The selection of reported points can be restricted (see inline help for details).

mode=’summary’ can also report back a dictionary if the task is run as


s = flagdata(..., mode='summary')

with a variable assigned, here ’s’.

3.5  Command-based flagging using flagcmd

The task flagcmd will flag the visibility data set or calibration table based on a specified set of flagging commands using a flagging syntax (see § 3.5.3). These commands can be input from the FLAG_CMD MS table, from a Flag.xml SDM table, from an ascii file, or from input python strings. Facilities for manipulation, listing, or plotting of these flags are also provided.

The inputs to flagcmd are:


#  flagcmd :: Flagging task based on batches of flag-commands
vis                 =         ''        #  Name of MS file or calibration table to flag
inpmode             =    'table'        #  Input mode for flag commands(table/list/xml)
     inpfile        =         ''        #  Source of flag commands
     tablerows      =         []        #  Rows of inpfile to read
     reason         =      'any'        #  Select by REASON types
     useapplied     =      False        #  Select commands whose rows
                                        #  have APPLIED column set to True

action              =    'apply'        #  Action to perform in MS and/or in inpfile
                                        #   (apply/unapply/list/plot/clear/extract)
     flagbackup     =       True        #  Automatically backup the
                                        #   FLAG column before execution

savepars            =      False        #  Save flag commands to the MS or to a file

The default input mode is inpmode=’table’ which directs the task to input flag commands from the FLAG_CMD internal MS table. See § 3.5.1 for more options.

The default operation mode is action=’apply’ directing the task to apply relevant flagging commands to the MS data main table. See § 3.5.2 for more options.

See § 3.5.3 for a description of the flagging command syntax.

It is possible to flag calibration tables using flagcmd, although we recommend using the flagdata task for this.

When using flagcmd to flag calibration tables, only the apply and list actions are supported. Because calibration tables do not have a FLAG_CMD sub-table, the default inpmode=’table’ can only be used if an MS is given in the inpfile parameter so that flags from the MS are applied to the calibration table directly. Otherwise, the flag commands must be given using inpmode=’list’, either from a file or from a list of strings.

3.5.1  Input modes inpmode

The inpmode parameter selects options for the input mode for the flagging commands.

Available inpmode options are:

3.5.1.1  Input mode ’table’

The default input mode is inpmode=’table’ which directs the task to input flag commands from a FLAG_CMD MS table. This has the sub-parameters:


inpmode             =    'table'        #  Input mode for flag commands(table/list/xml)
     inpfile        =         ''        #  Source of flag commands
     tablerows      =         []        #  Rows of inpfile to read
     reason         =      'any'        #  Select by REASON types
     useapplied     =      False        #  Select commands whose rows
                                        #  have APPLIED column set to
                                        #   True

If inpfile = ” then it will look for the FLAG_CMD table in the MS given by vis. You can use this sub-parameter to direct the task to look directly at another table.

The tablerows sub-parameter is a simple Python list of the row numbers of the table to consider in processing flags. The default is all rows.

The useapplied sub-parameter toggles whether only flag commands marked as not having been applied are considered (the default), or to allow (re)processing using all commands.

The reason sub-parameter selects the REASON type to process. The default ’any’ means all commands, note that reason=” would only select flags who have a blank REASON column entry.

One use case is to read the flag commands from the FLAG_CMD of an MS and apply them to a calibration table given in the parameter vis. Example:


     flagcmd(vis='cal-X54.B1', inpmode='table', 
             inpfile='uid___A002_X2a5c2f_X54.ms', action='apply')

3.5.1.2  Input flag mode ’list’

This mode allows one to insert a list of strings with flagging commands, the name of a file or a list of filenames that contains these commands equivalent to the mode=’list’ in flagdata (§ 3.4.2.2). E.g. a file flags.txt that contains


scan='1~3' mode='manual'
mode='clip' clipminmax=[0,2] correlation='ABS_XX' clipoutside=False
spw='9' mode='tfcrop' correlation='ABS_YY' ntime=51.0
mode='extend' extendpols=True

can be called via


flagcmd(vis,inpmode='list',inpfile='flags.txt')

Alternatively, the individual flagging commands can be directly provided in the call itself like


inpfile=["scan='1~3' mode='manual'", 
         "mode='clip' clipminmax=[0,2] correlation='ABS_XX' clipoutside=False",
         "spw='9' mode='tfcrop' correlation='ABS_YY' ntime=51.0",
         "mode='extend' extendpols=True"]

3.5.1.3  Input flag mode ’xml’

The input mode inpmode=’xml’ directs the task to input flag commands from a XML SDM online flagging Flag.xml file. When set this opens the sub-parameters:


inpmode             =      'xml'        #  Input mode for flag commands(table/list/xml)
     tbuff          =        0.0        #  Time buffer (sec) to pad flags
     ants           =         ''        #  Allowed flag antenna names to select by
     reason         =      'any'        #  Select by REASON types

This mode will look for a file called Flag.xml inside the MS directory specified under vis. Note that if the data was filled from the SDM using importevla (§ 2.2.2) then the relevant XML file will have been copied to the MS already.

The tbuff sub-parameter sets a padding buffer (in seconds) to the begin and end times of the online flags in the XML file. As in importevla, the online flag time buffer tbuff is specified in seconds, but in fact should be keyed to the intrinsic online integration time to allow for events (like slewing) that occur within an integration period. This is particularly true for JVLA data, where a tbuff value of 0.5× to 1.5× the integration time is needed. For example, if data were taken with 1-second integrations, then at least tbuff=0.5 should be used, likewise tbuff=5 for 10-second integrations. Note: For JVLA data you should use 1.5× (e.g. tbuff=15 for 10-second integrations) for data taken in early 2011 or before due to a timing error. We do not yet know what ALMA data will need for padding (if any).

The ants sub-parameter selects the antennas from which online flags will be selected (default is all antennas). For example, ants=’ea01’ is a valid choice for JVLA data.

The reason sub-parameter selects by the REASON field in the Flag.xml file. The default ’any’ means all commands. Note that reason=” would only select flags who have a blank REASON field entry.

3.5.2  Operation types action

The action selects options for operating on the selected flags and possibly the data.

Available action options are:

3.5.2.1  Apply flags — optype option ’apply’

The default operation mode is action=’apply’ directing the task to apply relevant flagging commands to the vis data main table.


action              =    'apply'        #  Action to perform in MS and/or in inpfile
                                        #   (apply/unapply/list/plot/clear/extract)
     flagbackup     =       True        #  Automatically backup the
                                        # FLAG column before execution

The flagbackup toggle sets whether a new copy of the MS main table FLAG column is written to the .flagversions backup directory for that MS before the requested flagging operation.

3.5.2.2  Unapply flags — action option ’unapply’

The unapply option allows unflagging of data based on the selected flag commands. This choice opens the sub-parameters:


action              =  'unapply'        #  Action to perform in MS and/or in inpfile
                                        #   (apply/unapply/list/plot/clear/extract)
     flagbackup     =       True        #  Automatically backup the
                                        #   FLAG column before execution

As in action=’apply’, it is possible to make a backup to the *.flagversions file by using flagbackup=True.

In order to guarantee that only the data selected in the command is unapplied, the framework will first unapply the selected rows and then re-apply the overlapping data that got unapplied in the first pass. This is a true unapply action, but it will take longer to process because it will re-apply all the remaining commands that have APPLIED = True!

3.5.2.3  List flags — action=’list’

The ’list’ option will give a listing of the flagging commands. This choice opens the sub-parameters:


action              =     'list'        #  Action to perform in MS and/or in inpfile
                                        # (apply/unapply/list/plot/clear/extract)
savepars            =       True        #  Save flag commands to the MS or to a file
     outfile        =         ''        #  Name of output file to save commands

This action lists the commands on the screen without applying them. One can save the flagging script to an file specified in the outfile parameter when savepars=True. If outfile is empty, it will save the commands to the MS given in vis.

The format of the listing output depends on the source of the flagging commands. A set of flagging commands specified through inpmode=’list’ will be listed directly. The flagging commands extracted through inpmode=’table’ will reflect the columns in the table:


        'Row', 'Timerange', 'Reason', 'Type', 'Applied', 'Lev', 'Sev', 'Command'

while commands from inpmode=’xml’ will be shown with the SDM XML table fields:


        'Key', 'FlagID', 'Antenna', 'Reason', 'Timerange'

3.5.2.4  Plot flags — action=’plot’

The ’plot’ option will produce a graphical plot of flags of time versus antenna. This choice opens the sub-parameters:


action              =     'plot'        #  Action to perform in MS and/or in inpfile
                                        #   (apply/unapply/list/plot/clear/extract)
     plotfile       =         ''        #  Name of output file to save plot

This is only useful for online flags or general flag commands that are specified by antenna plus timerange using the standard REASON codes that are known SDM Flag.xml enumerations.

If the plotfile sub-parameter is non-blank, then a plotfile will be made with that name instead of appearing in a matplotlib plotter window on the users workstation. There are additional parameters that control the shape of the output file, such as dimensions, and resolution.

ALERT: The plotted enumerations are currently only those known to be allowed JVLA online flags as of 15 April 2011, and include:


        'FOCUS', 'SUBREFLECTOR', 'OFF SOURCE', 'NOT IN SUBARRAY'

with all others being plotted as ’Other’.

3.5.2.5  Clear flags — action=’clear’

The ’clear’ action will delete selected rows from the FLAG_CMD MS table. This choice opens the sub-parameters:


action              =    'clear'        #  Action to perform in MS and/or in inpfile
                                        #   (apply/unapply/list/plot/clear/extract)
     clearall       =      False        #  Delete all rows from FLAG_CMD
     rowlist        =         []        #  FLAG_CMD rows to clear

The rowlist sub-parameter is a simple Python list of the row numbers of the table to consider in processing flags. The default is a blank list which indicates the desire to clear all rows.

In either case, if clearall=False then nothing will happen by default as a safeguard. If clearall=True, then a blank list will direct the deletion of the selected rows from the table.

ALERT: Use this option with care. You can easily mess up the FLAG_CMD table.

3.5.2.6  Extract Flag Commands— action=’extract’

The ’extract’ option will return the internal flagging dictionary to python:


action              =  'extract'        #  Action to perform in MS and/or in inpfile
                                        #   (apply/unapply/list/plot/clear/extract)

The value can be returned to a variable like:


myflagd = flagcmd(vis=msfile,useapplied=True,action='extract')

3.5.3  Flagging command syntax

A flagging command syntax has been devised to populate the COMMAND column of the FLAG_CMD table and to direct the operation of the flagcmd task.

The syntax is similar to flagdata, so please check help flagdata for more info.

You can also use help flagcmd inside casapy for this syntax guide also.

Commands are a string (which may contain internal "strings") consisting of KEY=VALUE pairs separated by whitespace (see examples below).

NOTE: There should be no whitespace between KEY=VALUE or within each KEY or VALUE, since the simple parser first breaks command lines on whitespace, then on "=".

Each key should only appear once on a given command line/string

There is an implicit "mode" for each command, with the default being ’manual’ if not given.

Comment lines can start with ’#’ and will be ignored.

  1. Data selection parameters (used by all flagging modes, see also § 2.3)
            
    timerange='' 
    antenna='' 
    spw='' 
    correlation='' 
    field='' 
    scan='' 
    feed=''
    array='' 
    uvrange='' 
    intent='' 
    observation=''
    

    Note: a command consisting only of selection key-value pairs is a basic "manual" operation, i.e. flag the data meeting the selection

  2. Modes specific parameters with default values (for further details, refer to the task flagdata, § 3.4.2).
    1. Mode manual
      
                    autocorr=False
      
    2. Mode clip
    3. Mode manual
      
                    datacolumn='DATA'
                    clipminmax=[]  
                    clipoutside=True
                    channelavg=False  
                    clipzeros=False
                    timeavg=False
                    timebin=''
      
      
    4. Mode shadow
      
                    tolerance=0.0
                    addantenna=''
      
    5. Mode quack
      
                    quackinterval=0.0     
                    quackmode='beg' 
                    quackincrement=False
      
    6. Mode elevation
      
                    lowerlimit=0.0
                    upperlimit=90.0
      
    7. Mode tfcrop
      
                    ntime='scan'
                    combinescans=False
                    datacolumn='DATA' 
                    timecutoff=4.0 
                    freqcutoff=3.0
                    timefit='line'
                    freqfit='poly'
                    maxnpieces=7 
                    flagdimension='freqtime' 
                    usewindowstats='none'
                    halfwin=1 
      
    8. Mode extend
      
                    ntime='scan'
                    combinescans=False
                    extendpols=True
                    growtime=50.0
                    growfreq=50.0
                    growaround=False
                    flagneartime=False
                    flagnearfreq=False
      
    9. Mode rflag
      
                    ntime='scan'
                    combinescans=False
                    datacolumn='DATA'
                    winsize=3
                    timedev=''
                    freqdev=''
                    timedevscale=5.0
                    freqdevscale=5.0
                    spectralmax=1000000.0
                    spectralmin=0.0
      
    10. Mode unflag
  3. Basic elaboration options for online and interface use
    
      id=''              # flag ID tag (not necessary)
      reason=''          # reason string for flag
      flagtime=''        # a timestamp for when this flag was generated (for 
                           user history use)
    

    NOTE: there is no flagtime column in FLAG_CMD at this time, but we will propose to add this as an optional column

    NOTE: These are currently ignored and not used

  4. Extended elaboration options for online and interface use Note: these are FLAG_CMD columns, but their use is not clear but included here for compatibility and future expansion
    
      level=N            # flagging "level" for flags with same reason
      severity=N         # Severity code for the flag, on a scale of 0-10 in order 
                           of increasing severity; user specified
    

3.6  Browse the Data

The browsetable task is available for viewing data directly (and handles all CASA tables, including Measurement Sets, calibration tables, and images). This task brings up the CASA Qt casabrowser, which is a separate program. You can launch this from outside casapy.

The default inputs are:


#  browsetable :: Browse a table (MS, calibration table, image)

tablename         =         ''  #   Name of input table

Currently, its single input is the tablename, so an example would be:


   browsetable('ngc5921.ms')

For an MS such as this, it will come up with a browser of the MAIN table (see Fig 3.13). If you want to look at sub-tables, use the tab table keywords along the left side to bring up a panel with the sub-tables listed (Fig 3.14), then choose (left-click) a table and View:Details to bring it up (Fig 3.15). You can left-click on a cell in a table to view the contents.


Figure 3.13: browsetable: The browser displays the main table within a frame. You can scroll through the data (x=columns of the MAIN table, and y=the rows) or select a specific page or row as desired. By default, 1000 rows of the table are loaded at a time, but you can step through the MS in batches.


Figure 3.14: browsetable: You can use the tab for Table Keywords to look at other tables within an MS. You can then double-click on a table to view its contents.


Figure 3.15: browsetable: Viewing the SOURCE table of the MS.

Note that one useful feature is that you can Edit the table and its contents. Use the Edit table choice from the Edit menu, or click on the Edit button. Be careful with this, and make a backup copy of the table before editing!

Use the Close Tables and Exit option from the Files menu to quit the casabrowser.

There are a lot of features in the casabrowser that are not fully documented here. Feel free to explore the capabilities such as plotting and sorting!

ALERT: You are likely to find that the casabrowser needs to get a table lock before proceeding. Use the clearstat command to clear the lock status in this case.


1
2
3

Previous Up Next