vlafiller - Tool

Postscript file available

A tool for converting VLA archive format data to a measurement set

include "vlafiller.g"

Description
A vlafiller is a tool that contains functions that allow you to convert data that has been stored in the VLA archive format to a measurement set. Note that VLA archive format versions earlier than version 23 (earlier than 1990) are not supported by the vlafiller.

The VLA archive format is radically different from FITS and is the native data format written by the online computers at the VLA. The VLA data that is archived at the AOC is stored, on tape, in this format. It is described more fully in the VLA Computer Memorandum 188.

Running the vlafiller tool is a multi-step process. These steps can be grouped into the following sequence.

1.: Create a vlafiller tool
2.: Specify the data source and sink
3.: Select some data filters (optional)
4.: Start the copying process
5.: Remove the tool

The advantage of a multi-step approach is that it gives the user fine grained, flexible control of the filling process. For many users this is unnecessary and the global functions vlafillerfromtape, vlafillerfromdisk bundle the vlafiller functions into a simpler one step approach.

Creating a vlafiller tool

The only way to create a vlafiller tool is to use the vlafiller constructor. Calling this function will start a precompiled executable, called vlafiller, usually running on your machine. This executable executes all the vlafiller functions. See the documentation for the vlafiller constructor for a more complete description of this function.

Specifying the data source and sink

The data source can either be a tape drive, disk file or the online computers. If it is a tape drive then it is assumed that a VLA archive format tape is already in the drive. The vlafiller knows about the layout and blocking structure of files on a VLA archive format tape and is able to rewind and skip to the any file on the tape. The files on a tape are numbered beginning at one. Users of tapes from the VLA archive need to be aware that the first file on these tapes is a tape label and hence the first file containing astronomical data will be the file numbered two.

An alternative to using the tapeinput function the VLA archive data may be copied from tape to disk using commands like:

 
    dd if=/dev/tape bs=26624 of=file1.vla

and you can use commands like

 
    mt -f /dev/tape fsf 2

to manually skip, in this example, over two tape files.

Then the diskinput function can be used to read this file. The advantage of this is that reading from disk is much faster and a lot less troublesome than reading from tape. It is also advantageous to do this if you need to read the data multiple times as shown in the examples below. The disadvantage is that you need about 160MBytes of spare disk space, as this is typically the size of a VLA archive file.

The vlafiller can also read data directly from the online computers at the VLA. This option is only available when the filler is run on a computer at the array operations center in Socorro. See the final example below for an illustration of how to do this.

The data sink is a measurement set. The vlafiller can append data to an existing measurement set^1.1 or create a new one. Only one measurement set can be written for each pass through the input data. If you need to write multiple measurement sets each containing a different subset of the input archive you should consider copying the data to disk as described above. See the documentation for the output function for a more complete description of how to specify the data sink.

Selecting data filters

One measurement set is able to contain all the data in any number of VLA archive format files. Such a measurement set will usually be quite large, potentially affecting data access times, and may contain unwanted data. The vlafiller has a number of functions that allow you to discard data prior to it being copied into the output measurement set. These functions all have names that begin with the word select, a typical one being selectproject.

It is possible to use any combination of select functions as each one filters the data on a different parameter^1.2. For example using the selectproject and selectfrequency functions will then only copy data from the specified project and in the specified frequency range. Other data in the same project or other projects that use the same frequency will not have their data copied to the measurement set.

Using a specific select function more than once will overwrite the previous selection. For example using the selectsource function with sourceA and then sourceB as arguments will discard any selection of sourceA (assuming you have not run the fill function between the two function calls).

Currently it is only possible to select data based on a limited number of parameters. Other selections must be done after the data is converted to a measurement set.

Starting the copying process

Once a vlafiller tool has been setup with input and output files the fill function is used to copy the data.

This function, unlike all the other ones in the vlafiller tool, can take some time. Because of this it can be run in the background ie., asynchronously. If run synchronously and from the command line you will not be able to type any further commands to glish until the command completes. If run asynchronously from the command line the function will return with a small positive integer that indicates the job number. This number can be used to query whether the function is still running and a message will appear in the logger when the function has finished. In either case the vlafiller tool will not execute any other functions until the fill function has completed. You need to start a new vlafiller server process, by calling the constructor with the forcenewserver argument set to true, to run two fill functions simultaneously.

You can signal for the filler to stop filling as soon as possible using the stop function. This function will shut the filling function down after it has completed processing the current record.

When the fill function terminates it closes the input source, but leaves the output measurement set open. This way you can open another input source and append data to the existing measurement set.

Removing the tool

When you have finished with the vlafiller you should shut down the process, using the done function, even though it is not consuming much processor time. This is to conserve virtual memory. Once a tool is shut down it can no longer be used. It also closes the output measurement set.

Example
The following example shows the minimum necessary to run the vlafiller.

The first line makes all the definitions for this tool available to the user. The second line calls the constructor. The third tells the filler that the input is to come from a file, in the current directory, called file1.vla. The fourth states that the output is to go to a measurement set called file1.ms. The fifth line does all the copying and the sixth shuts down the tool.

    include 'vlafiller.g'     #1
    v := vlafiller()          #2
    v.diskinput('file1.vla')  #3
    v.output('file1.ms')      #4
    v.fill()                  #5
    v.done()                  #5

This example could also be done much more simply using the vlafillerfromdisk global function as shown below.

    vlafillerfromdisk('file1.ms', 'file1.vla')

All the other examples in this section are too complicated to be done using the vlafillerfromdisk or vlafillerfromtape global functions.

Example
This example illustrates the way the vlafiller can be used to read all the calibrators from three files.

The first three lines are identical to the first example. The fourth line sets the output measurement set to file1.ms. As this file may already exist, if you have just done example shown above the overwrite flag is set to force deletion of any existing file prior to copying any data. The fifth line selects all the calibrators, irrespective of their calibration code. The sixth line copies all the calibrators from the file to the measurement set.

At this point the input VLA archive file is closed but the output measurement set is still open. Line seven opens a new VLA archive file, called file2.vla, and line eight copies all the calibrators from that file to the measurement set. Note that the selection done in line five is still current and that the second fill command appends to the existing measurement set rather than overwriting it.

This process continues for the third file before the process is shut down.

    include 'vlafiller.g'               #1
    v := vlafiller()                    #2
    v.diskinput('file1.vla')            #3
    v.output('file1.ms', overwrite=T)   #4
    v.selectcalibrator('*')             #5
    v.fill()                            #6
    v.diskinput('file2.vla')            #7
    v.fill()                            #8
    v.diskinput('file3.vla')            #9
    v.fill()                            #10
    v.done()                            #11

Example
This example illustrates how to make multiple passes through a file to do more complex selections. The selectsource function can only select one source. If you want to create a measurement set with exactly two sources you need to read the file twice as shown in the example below.

The first four lines are identical to the above example. The fifth line selects the first source 'ngc1234' and the sixth line copies this to the measurement set. As described above after calling the fill function the input data source is closed. So we need to re-specify it as shown in line seven. Line eight then sets the selection to read the other source and line nine appends it to the measurement set.

    include 'vlafiller.g'               #1
    v := vlafiller()                    #2
    v.diskinput('file1.vla')            #3
    v.output('file1.ms', overwrite=T)   #4
    v.selectsource('ngc1234')           #5
    v.fill()                            #6
    v.diskinput('file1.vla')            #7
    v.selectsource('cen-a')             #8
    v.fill()                            #9
    v.done()                            #10

Example
This example illustrates how to use the vlafiller to read from a tape on a remote machine and fill the data asynchronously (as tapes take a long time).

The first line is identical to previous examples. The second line starts up the vlafiller process on the computer called 'tapehost' (thats the one with the tape drive). The third line tells the vlafiller it is to read the first three files from the tape in the drive with a Unix device name of '/dev/nst0'. The fourth line tells the vlafiller to create the measurement set in the /tmp directory. This is the /tmp directory on the tapehost machine which is usually not the same as the /tmp directory on your local machine. To see the measurement set on your local machine you need to write to a disk that shares files (usually via NFS). The fifth line selects only the data from the specified project. The sixth line tells the filler to start copying the data asynchronously.

At this point glish returns before all the data has been copied. When all the data has been copied a message appears in the logger window. This may be missed if you are doing other things so in this example I set up a whenever that detects the event signalling that the job has finished and pops up a window to attract the users attention. Finally the tool is shut down.

    include 'vlafiller.g'                       #1
    v := vlafiller(host='tapehost')             #2
    v.tapeinput('/dev/nst0', files=[1,2,3])     #3
    v.output('/tmp/file1.ms')                   #4
    v.selectproject('ab123')                    #5
    jobid := v.fill(async=T, verbose=F)         #6
    whenever defaultservers.alerter() -> [as_string(jobid)] do {
      deactivate 
      messagebox("VLA filler has finished") 
    }
    ....
    v.done()

Example
The final example illustrates how to use the vlafiller to read data from the online computers.

The first two lines are identical to previous examples. The third line tells the vlafiller it is to read data from the online computers. The fourth line tells the vlafiller to create the measurement set in the current directory. The fifth line selects only the data from the specified project. The sixth line tells the filler to start copying the data. Records are processed as they are made available by the online computers. To stop the filling process the stop function is used. This cleanly terminates the filling function and the tool is shutdown using the done function.

    include 'vlafiller.g'                       #1
    v := vlafiller()                            #2
    v.onlineinput()                             #3
    v.output('realtime.ms')                     #4
    v.selectproject('ab123')                    #5
    v.fill(async=T)                             #6
    ....
    v.stop()                                    #7
    v.done()                                    #8

Scaling & weighting of VLA data

The vlafiller scales both the visibility data and computes the weights in a manner that is compatible with the 31DEC01 version of the AIPS task FILLM. The details of both the scaling and weight computation are described below.

The visibility data that is stored in the VLA archive is related to the the raw correlator coefficients by a scaling factor. This factor, which is different for each correlator, can be decomposed into antenna based terms called the nominal sensitivity (s_i). The raw correlation coeffecient ( $\rho_{ij}^{}$ ) is then related to the value stored in the VLA archive (v_ij) by the formula:

v_ij = $\displaystyle \rho_{ij}^{}$ ^. $\displaystyle \sqrt{s_i \cdot s_j}$

The nominal sensitivity is derived during the course of an observation from calibrated noise diodes on each antenna.

The visibility data in VLA archive format is represented using a 16-bit signed integer for each of the real and imaginary components. To increase the dynamic range that can be represented each group of visibilities is scaled by a signed exponent (e). A group of visibilities is, in continuum mode, all four-correlations ie., RR, LL, RL, LR, for one baseline. In spectral line mode it's all channels corresponding to one correlation.

In order to convert these integers to a complex number (d_r, d_i) the vlafiller uses the formula (the subscripts r & i denote the real and imaginary components):

(d_r, d_i) = (v_r, v_i)/2^{- (e + 8)}

All floating point operations are done in single precision. The factor of 8 added to the exponent results in scaling the complex data by 256. It is probably used to further scale the data so that it has nominal units of deka-Jansky's ie., 1.0 means 10Jy.

The vlafiller assigns weights to the data that are directly proportional to both the integration time and the observed bandwidth and indirectly related to the system temperature at each of the antennas. The mathematical description and justification of this is described in AIPS memo 103, ``Weighting data in AIPS'' by Ketan Desai. In essence it states that the weights should be given by:

w_ij = $\displaystyle {\frac{\Delta t \Delta \nu}{s_i s_j}}$


where: t is the integration time (typically 10 seconds)

		 

is the bandwidth (often 50MHz) 

		 s_i is the nominal sensitivity of antenna i (typically around .2)

AIPS modifies these weights by scaling them by a factor that has been empirically determined so that the rms determined from the weights ( $\sigma_{ij}^{}$ = ${\frac{1}{\sqrt{w_{ij}}}}$ ) corresponds to the measured rms in the visiblities when a calibrator is being observed. This factor is .12/10000^1.3. In order to maintain consistency with AIPS, AIPS++ also scales the weights by this factor.

Prior to version 1.5 of AIPS++ the vlafiller used a simpler algorithm for calculating weights that matched what was the default in older versions of AIPS. The old weights were simply the integration time, in seconds, divided by 10. The vlafiller contains a global function called vlafilleroldweights that allows you to change the weights in a measurement set so that the older scheme is used. You cannot change the weights in a measurement from the old scheme to the new scheme. Instead you need to reload the data.

Constructors

vlafiller

Construct an vlafiller tool.

Functions

diskinput	Specify the input data file.
done	Delete the vlafiller tool
fill	Copy the data
onlineinput	Read input data from the online computers.
output	Specify the output measurement set
selectband	Select which frequency-band to convert
selectcalibrator	Select which calibrators to convert
selectfrequency	Select which frequency-range to convert
selectproject	Select which project to convert
selectsource	Select which source to convert
selectsubarray	Select which sub-array to convert
selecttime	Select which time-range to convert
state	Show the state of the tool
stop	Stop the filling process
tapeinput	Specify the input tape device.

Next: vlafiller - Constructor Up: vla - Module Previous: vla - Module Contents Index

	Package	nrao
	Module	vla