 News |
FAQ
|
| Search
|
Home
|
| Getting Started | Documentation | Glish | Learn More | Programming | Contact Us |
|
| Version 1.9 Build 1367 |
|
The SDMeasurementSet is our initial attempt at a MeasurementSet for single dish data. The MeasurementSet is any collection of data produced by a telescope or a model. It includes astronomical data, instrument data, and monitor data.
The AIPS++ table system is clearly more than adequate as a container of data. Early designs of the MeasurementSet utilized the flexibility of the table system to it's fullest. These early designs consisted of multiple tables. Multiple tables are useful as an organizational aid: similar quantities can be grouped together. They may also be useful as a storage mechanism. Quantities that vary slowly or are constant can be grouped together to reduce the size of the data set. On the other hand, astronomical data has been traditionally viewed as coming in chunks with associated header words. This view is analogous to a row of a table, with each row being an atomic piece of the data.
The telescope-dependent nature of the MeasurementSet makes it difficult to define a general multiple table structure with enough shared parts to be useful from an object-oriented approach. Groupings of quantities are likely to be telescope dependent. Values that are constant or slowly varying at one telescope may vary rapidly at another telescope. Classes which use the MeasurementSet as well as application programs should be as telescope independent as possible. An application programmer should not need to know in which sub-table a quantity is found. Users should also not need to know in which sub-table a quantity is found.
The conflict between the desire to use the full flexibility of the table system for organizational needs and data storage efficiency on the one hand and the need to present the data in single ``atomic'' units or rows was a large factor in the slow development of a working MeasurementSet. This conflict was resolved in part with the realization that a suitable storage manager could reduce data set bloat. The current table system allows for each column to have it's own storage manager. Through careful choice of an appropriate storage manager, data set bloat can be avoided as needed for each column (there is no longer any need to group values into tables to accomplish this). Organizational needs can be dealt with through the use of reference tables. If a class or application finds it convenient to group columns X,Y, and Z together, that can be done without any need to impact the actual MeasurementSet. The MeasurementSet that we have implemented consists of a single table.
This section describes the specific layout of the table that is used in the initial SDMeasurementSet. All column names are unique (they only occur once in any given MeasurementSet) and have a fixed definition. The layout described here is more homogeneous that the ultimate MeasurementSet will likely require, primarily due to the lack of a range of suitable storage managers. The planned storage managers and use of virtual column engines will allow future MeasurementSets to be extremely heterogeneous.
All values in any column will have the same units. The specific unit for a given column is indicated by the value of an associated string keyword named ``UNIT''. Some columns are defined to have a specific unit (e.g.,integration time is always in seconds). For those columns, no ``UNIT'' keyword is required (and if present, is ignored). This convention will likely change when a complete units class is available.
The coordinate system used for celestial coordinate columns is indicated by associated keywords as described below. All values in any given column are in the same coordinate system although coordinate systems may differ between columns. Units are degrees for all angular columns (this is consistent with the FITS usage). This leads to a more homogeneous MeasurementSet than is probably necessary (one would certainly like to mix Galactic with Equatorial observations, etc). This decision was made for simplicity with the recognition that once a complete coordinate class was available this convention would likely change.
The following keywords, when associated with a column, indicate that that column is a celestial coordinate of the indicated type.
| CTYPE | a String keyword containing the first 4 characters of a valid FITS World Coordinate System ( e.g.,``RA-'' or ``DEC-'' for equatorial coordinates, ``GLON'' or ``GLAT'' for galactic coordinates, ``ELON'' and ``ELAT'' for ecliptic coordinates, etc). |
| EQUINOX | Ecliptic and equatorial coordinates require that the equinox of the coordinate system be specified ( e.g.,1950, 2000, etc). This is stored as double precision value. |
In all cases in this initial MeasurementSet, columns of Strings (source name, observing mode, integration state, instrument name) are likely to represent choices from a small set of possible values (for example, there are a limited number of observing modes available). These values may change frequently throughout an observing session (for example, an OTF data set consists of ``OTF'' observing modes - the mode where the actual on-the-fly data is taken - interspersed with ``TPMF'' observing modes - the total power mode used when taking the calibration scans at the ``OFF'' positions). For storage efficiency, these columns are stored as short unsigned integers with an associated keyword containing the array of possible string values. The integer actually stored in each row of the table points at the element of the associated string array where the actual string value is found. This will ultimately be implemented more transparently using virtual columns.
The following keywords apply to the entire MeasurementSet.
| Keyword | Data Type | Description |
|---|---|---|
| ReferenceDate | Double | The reference date, used to construct the full date from the UTime column. The units of this value are modified Julian date: ( JD - 2400000.5). Required. |
| Telescope | String | The name of the telescope. Optional. |
| Observer | String | The name of the observer. Optional. |
| Project | String | A string identifying the project. Optional. |
The optional keywords above are likely to be columns in the future so that a MeasurementSet can contain data from multiple telescopes, observers and projects.
The following columns are required to exist for the table to be considered a valid SDMeasurementSet. They are the primary means of sorting and selecting data.
| Column Name | Data Type | Description |
|---|---|---|
| TimeStamp | Int | A number guaranteed to be unique for each unique time in the MeasurementSet. Since time is a floating point value, by guaranteeing a unique integer for each time, comparisons are simpler. Traditionally, data from a specific time is assigned a number (the scan number). While this number is usually unique during a particular observing session, it is not unique over several sessions. This column is intended to take the place of atomic identification that scan number has traditionally provided during an observing session. Valid time stamps are positive. |
| ScanNumber | Int | As long as on-line control systems stamp the data with a scan number, observers will continue to use such numbers. These values are likely to be unique only over single days or possibly observing sessions. |
| SpectralWindow | Int | An integer specifying the spectral window of the data (previously this has been known as ``IF''). The current working definition is to use the ``sub-scan number'' used at the 140' and 12-m, but those conventions are somewhat arbitrary and telescope dependent. SpectralWindow needs a more complete definition. At a minimum, the characteristics of each SpectralWindow need to be available (possibly as associated keywords). |
| PolarizeType | Int | The polarization type appropriate for this row. The value of this integer is from the StokesTypes enumeration found in the Stokes class. Available enumerations appropriate for single dish data are RCircular, LCircular, and Linear. |
| Feed | Int | The feed number for multi-feed systems. Currently this is always 1 as there are no NRAO multi-feed systems presently taking data. |
| IntegrationState | String | A description of the integration state (known at NRAO single dish telescopes as ``phase''). For example, for a simple On/Off sequence, this might be ``ON-POS'' or ``OFF-POS''. This is the last indexing value to be used when all of the above do not yield a unique row. Most on-line systems (or fillers of an SDMeasurementSet) will average all IntegrationStates appropriately and make this column unnecessary. This will be used under those circumstances where it is desirable to edit individual parts of an integration (known to UniPOPS users as ``record editing''). |
| Flag | Bool | A flag. ``True'' indicates that the data is ``ok''. ``False'' indicates that the data is ``bad''. |
| Data | Double | This is a vector containing the actual data. All Data vectors must be the same length in a single MeasurementSet. Data of different lengths (even from the same observing session) must reside in different MeasurementSets. This restriction will be lifted in the future, although specific fillers may wish follow this rule. |
Most of the information in an SDMeasurementSet is considered optional. The specific SDMeasurementSet filler (which will be telescope dependent) will determine which additional columns are present. It is therefore extremely important that column names be registered so that applications can rely on standard definitions. The following have been tentatively identified as useful optional columns.
| Column Name | Data Type | Description |
|---|---|---|
| UTime | Double | The universal time in number of seconds since the ReferenceDate keyword. This time corresponds to the mid-point of the integration. |
| Object | String | A descriptive name. |
| ObservingMode | String | A string describing the observing mode. This will be telescope dependent although some attempt should be made to identify common observing modes ( e.g.,frequency switched, position switched, etc). |
| PolarizeAngle | Float | The polarization angle, meaningful for Linear PolarizeType only. |
| UserTag | Int | A tag set by the user to identify groups of rows. |
| IntegrationTime | Float | The total integration time (seconds). This is the real time spent actually collecting all of the data in this row. |
| EffectiveIntTime | Float | The effective integration time (seconds). This is the time that appears in the radiometer equation ( i.e.,use this time when calculating expected noise levels). The exact relationship of this quantity to IntegrationTime depends on the ObservingMode and is likely to be telescope dependent. |
| HSource | Double | |
| VSource | Double | ``Horizontal'' and ``Vertical'' source position in the coordinate system described by the associated keywords defined above. This is where the telescope was pointed (for position switched data, this is the ``Source'' or ``On'' position). |
| HReference | Double | |
| VReference | Double | ``Horizontal'' and ``Vertical'' reference position in the coordinate system described by the associated keywords defined above. This is a reference positions (for position switched data, this is the ``Off'' position). |
| Instrument | String | A descriptive string identifying the instrument. This will likely be split into constituent parts ( e.g.,receiver, feed, correlator, etc). |
| ReferenceChannel | Int | The reference channel for the Data array. This may be renamed to ReferencePixel to be more generic. |
| ReferenceValue | Double | The x-axis value at the ReferenceChannel. |
| DeltaValue | Double | The increment along the x-axis. Positive values increase with increasing channel (pixel) number. |
| RestFrequency | Double | The rest frequency corresponding to ReferenceChannel. |
| VelocityDef | String | The velocity definition (``Radio'', ``Optical'', ``Relativistic''). In the future, this may be more appropriately an Int with the values corresponding to an enumeration in much the same way that PolarizeType corresponds to enumerations found in the Stokes class. |
| VelocitySystem | String | The velocity reference frame (``LSR'', ``HELO'', ``EART'', ``BARI''). As with VelocityDef, in the future this may be an Int corresponding to an enumeration. |
| VelWRTReference | Double | The velocity (in the VelocityDef for that row) of the ReferenceChannel with respect to VelocitySystem. |
| SignalFreqOffset | Double | The frequency offset for frequency switched data for the ``signal''. |
| ReferenceFreqOffset | Double | The frequency offset for frequency switched data for the ``reference''. Note that this and SignalFreqOffset will not be adequate to describe more complex frequency switching observing modes. |
| SourceTSystem | Float | The system temperature (K) at the source position. |
| ReferenceTSystem | Float | The system temperature (K) at the reference position. |
| TReceiver | Float | The receiver temperature (K). |
| TCalibration | Float | The value of the noise tube diode or other calibration temperature (K) used to calibrate the data. |
| BeamFWHM | Float | The full telescope main beam width at half maximum at the observing frequency. |
| OffTimeStamp | Int | The TimeStamp corresponding to the designated ``Off'' or calibration data. If OffTimeStamp is equal to TimeStamp for that row, then that row is ``Off'' data. If OffTimeStamp is zero, there is no designated ``Off'' data for that row. |
| GainTimeStamp | Int | The TimeStamp corresponding to the designated per channel gain values (currently 12-m only). If GainTimeStamp is equal to TimeStamp for that row, then that row is Gain data. If GainTimeStamp is zero, there is no designated ``Gain'' data for that row. |
Often a region of the sky is scanned in some regular pattern (a ``map'' is made) through one command of the operator or observer. This will generate several rows in the MeasurementSet. The original map description may be available from the on-line data. The following columns are intended to hold such information when available. These can assist in imaging the data or they can be ignored. Maps always have two axes (in these names, then, n=1 or 2).
| MapNAXISn | Int | The number of map ``pixels'' on the nth axis. |
| MapCRVALn | Double | Coordinate value at the reference pixel on the nth axis. |
| MapCRPIXn | Double | The reference pixel on the nth axis. |
| MapCDELTn | Double | The coordinate increment on the nth axis. |
| MapCTYPEn | String | The WCS coordinate type for the map (applies to any coordinate expressed in these columns) for this particular row. |
| MapEQUINOX | Double | The equinox of the coordinate system. |
The following describe various flags and weight options (in addition to Flag).
| Sigma | Float | The theoretical RMS in the same units as the data. This value is usually provided by the operating system. It is used with weight when averaging or combining data. |
| Weight | Float | This is a unit-less number that is initially 1. The total effective sigma is weight*sigma. This allows the user to modify the effective sigma used when combining data without discarding the value provided by the operating system. |
| ChannelSigma | Float | Per-channel version of Sigma |
| ChannelWeight | Float | Per-channel version of Weight |
| ChannelFlag | Bool | Per-channel version of Flag |
All of the above are vectors that must be the same length as the data vector. There must be only one of Sigma and Weight or ChannelSigma and ChannelWeight in a given MeasurementSet. ChannelFlag can exist in conjunction with Flag. Flag turns ``on'' or ``off'' the entire row while ChannelFlag turns ``on'' or ``off'' individual values in the data vector.
Additional columns will be registered and added as needed.
SDMeasurementSet has been implemented as having a Table rather than derived from Table. The primary motivation for this was the design of Table at the time of this implementation. Table then consisted of ROTable (a read-only, or constant, table) and Table (a read/write table), which was derived from ROTable. In order to implement SDMeasurementSet as a Table following the above would require an ROSDMeasurementSet for read-only data which would be derived from ROTable. SDMeasurementSet would then need to be derived from ROSDMeasurementSet as well as from Table. This inheritance, with ROTable being in the inheritance tree twice, was felt to be a potential maintenance headache. It was decided to avoid the problem entirely by implementing SDMeasurementSet with a private Table data member. The most recent version of the Tables module eliminates ROTable. Since a MeasurementSet seems to clearly be a Table, we will use this opportunity to implement MeasurementSet as a Table in the near future.
An SDMeasurementSet is constructed from an existing table. During construction, the table is checked to see that the required keywords and columns as outlined above are present. A static member function, miminumDescriptor() provides the minimum table descriptor (i.e.,one that consists only of the required keywords and columns). This can be used to construct an empty table which would then be used to construct a valid SDMeasurementSet.
Normal table access to the columns would require that the correct spelling of the column name be used. For example, to construct an ArrayColumn of Float from a column named ``Data'' in an existing Table named tab, one would use:
ArrayColumn<Float> data(tab, "Data");
Mistakes in the string containing the column name would only be caught at run time (and then only if a column of the miss-typed name did not exist with the specific type).
In this implementation, we have provided a member function that returns either a reference or a pointer to the appropriate ArrayColumn or ScalarColumn object for each of the required and optional column names. Pointers are returned for the optional columns while references are returned for the required columns. Here are some examples from the SDMeasurementSet header file:
// Columns that are GUARANTEED to be in all SDMeasurementSets ScalarColumn<Int> &scanNumber(); ScalarColumn<Int> &timeStamp(); ArrayColumn<Float> &data(); // Optional columns ScalarColumn<Double> *uTime(); ScalarColumn<Float> *integrationTime();
The advantage of this scheme is that mistakes in the column names are caught at compile time rather than run time. We feel that this is an important advantage that should be retained in any future MeasurementSet. One clear disadvantage is that some columns (the required columns) are returned as references while the optional columns are returned as pointers. The current thoughts on the future MeasurementSet (to be shared by all types of data) are outlined later in this document and summarized here. We intend to provide an enumeration, with each registered column name having a corresponding entry in the enumeration. Member functions will provide a translations between the enumerated symbolic names and the registered column names. This will eliminated the cumbersome need to maintain a member function for each registered column name.
There are a few additional functions that SDMeasurementSet provides. The first 3 of these would be unnecessary if SDMeasurementSet was a Table, the last 2 are more typical of the types of convenience functions likely to be necessary in a MeasurementSet.
void attachNewTable(Table &attachToMe);
This allows one to attach a SDMeasurementSet to a different table from the one that was used when the class was constructed (thereby allowing the same SDMeasurementSet object to be used with several tables, although not at the same time).
Table &asTable(); This returns a reference to the currently attached Table. This allows the user to do Table operations (e.g.,sort, select, etc.). The results from such operations might then be used to construct a new SDMeasurementSet.
uInt nrow();
A convenience function that returns the number of rows in the attached Table.
static Bool isSDMeasurementSet(const TableDesc &desc);
This allows a user to test if a table descriptor is a valid for use by a SDMeasurementSet (this make sure that the required columns and keywords all exist).
Double referenceDate() const;
A convenience function that returns the value of the ReferenceDate keyword.
There is no substantial difference between SDMeasurementSet and ROSDMeasurementSet other than that const or RO objects are used.
A MeasurementSet must be filled. At the present time, the only access to astronomical data is through FITS and the current set of AIPS++ FITS classes. For single dish data from NRAO telescopes, UniPOPS provides a FITS writer that writes Single Dish FITS files. Single Dish FITS is standard binary table FITS with some conventions for describing single dish data (standard column names). A FITS binary table (and hence single dish FITS data) is easily placed into an AIPS++ table. Eventually AIPS++ will be able to fill a MeasurementSet from a telescope specific data format.
There are two facts about using FITS as the input data that impact the filler. First, the FITS classes only read sequentially from the input file. Second, single dish FITS binary tables require that the length of each data array in each row of the FITS table be the same length. For NRAO 12-m on-the-fly data, this means that the OTF rows are found in a separate FITS table from the calibration data. Additionally, the 12-m gain calibration data is contained in a separate file from the OTF and OFF data, which must also be written to a separate FITS binary table. So, in order to fill a single SDMeasurementSet with all of the data necessary to calibrate and image an OTF observing session, three input single dish FITS binary tables are required (the OTF data, the OFF source data and the gain calibration data).
In order to facilitate the merging of these three input FITS files into a single MeasurementSet, a SDMSSource class was developed. This class can also be used to fill data from a single FITS file. It is planned that this will ultimately be a base class from which specific SDMSSource classes will be derived to handle input from sources other than FITS (i.e.,telescope specific data formats). Filling a MeasurementSet from multiple data streams will be a useful option for other fillers.
The following background information is required to understand the excerpts from the SDMSSource header file which follows.
BinaryTable is the AIPS++ FITS class that converts a FITS binary table to an AIPS++ Table. A BinaryTable object is used to construct a SDMSSource.
Several SDMSSource objects are attached to a single SDMeasurementSet. They can fill the attached SDMeasurementSet until the universal time (UTime) of the data in the BinaryTable object changes or they can fill until they reach the end of theBinaryTable object.
A number of member functions are provided which are used to help an application coordinate several SDMSSource objects so that the single SDMeasurementSet is filled in the desired order.
class SDMSSource {
public:
// Construct one from a BinaryTable, the second argument
// is used to indicate whether or not this object should keep an
// index of the mapping of scan number to time stamp. Since
// The OFF and GAIN calibration information is referenced by
// scan number in the input FITS data and this information is
// reference by TimeStamp in an SDMeasurementSet, this index is
// available to the filling program so that the correct reference
// information can be filled into the SDMeasurementSet. This is
// important when filling from separate FITS files where the OFF
// and GAIN TimeStamps are not available to each individual
// SDMSSource. The default is to NOT keep an index.
SDMSSource(BinaryTable& bintab, Bool indexed = False);
~SDMSsource();
// provide a TableDesc appropriate for a SDMS. The default
// values for each column and keyword are derived from the first
// row of the attached BinaryTable. This can be used to construct
// the initial empty SDMS.
TableDesc &makeDesc();
// test to see if a SDMS is attached (it must be attached after
// the object is constructed).
Bool isAttached();
// attach the SDMSSource to an SDMeasurementSet
Bool attachSDMS(SDMeasurementSet& attachToMe);
// return a reference to the currently attached SDMeasurementSet
SDMeasurementSet& getSDMS();
// fill the SDMS until the time in the BinaryTable changes from
// its current value. Each time change generates a new TimeStamp.
// Start filling at the indicated starting row of the attached
// SDMS. The returned value is the next available row of the
// attached SDMS that could be filled. If the returned value is
// equal to the input value, nothing was filled to the SDMS
// (likely due to the attached BinaryTable being at the end of the
// FITS table).
uInt fill(uInt startRow);
// fill the SDMS until all the remaining data in the
// BinaryTable has been used. Start filling at the indicated
// starting row of the attached SDMS. The returned value is the
// next available row of the SDMS. If the returned value is
// equal to the input value, nothing could be filled to the SDMS.
uInt fillAll(uInt startRow);
// return the time associated with the next row of the
// BinaryTable. The value returned is UT seconds since the
// ReferenceDate of the SDMS. Returns 0 if not attached to an SDMS.
Double nextUTime();
// For an indexed SDMSSource object, return the timeStamp
// associated with a given scan number. Returns 0 if that scan
// number is not found in the index or the SDMSSource is not indexed.
uInt timeStamp(uInt scanNumber);
// The shape of the data column of the BinaryTable.
IPosition dataShape();
// The maximum number of rows this could potentially fill.
// For most FITS binary tables this is simply the number of rows
// in the binary table. For 12-m OTF binary tables, each row is
// really several individual spectra, each one of which will be
// a single row in the SDMS.
uInt nrows();
// Other information valid for the full SD-FITS binary table.
// The ReferenceDate, may not be the ReferenceDate of the SDMS.
Double refDate();
// The Telescope
String telescope();
// The observer
String observer();
// The project
String project();
// Is the BinaryTable 12-m OTF data
Bool isOTF(); private:
// this is not shown in this sample, an implementation detail.
};
The SDMS filler then has the following form:
The execution time to convert the trial data set described above from the on-line data files to FITS is roughly 5 minutes. The time to fill a SDMeasurementSet from these FITS files is approximately 10 minutes.
Once filled, then next step before the data is imaged is calibration. In AIPS++ , calibration is accomplished through a TelescopeModel. A TelescopeModell in general will consist of multiple TelescopeComponent objects. Each TelescopeComponent in turn consists of the information and methods required to perform a specific calibration step. A TelescopeComponent has an apply() method that actually calibrates the data and a solve() method that uses a SourceModel and the data to solve for some calibration parameters (that might then be used to calibrate the data through apply()). There is also a corrupt() method that is the inverse of apply(). The TelescopeModel then coordinates the methods of the various TelescopeComponents contained within it.
Our goal of imaging 12-m OTF data only requires one TelescopeComponent. We use this directly without any TelescopeModel. On-the-fly data at the 12-m is position switched data. This is an observing technique that is common at single dish telescopes. It is described in more detail earlier in this document.
A PositionSwitchedTC (position switched telescope component) is used to apply ``on minus off'' style calibration (and decalibration, ultimately, through corrupt()). There are two styles of such calibration within NRAO single dish telescopes (ON is the ``on source'' data - this is really what needs to be calibrated, OFF is the ``off source'' data).
= ((
- 
)/) 
= ((
- )/) 
The OFF and GAIN data are determined by the values of the OffTimeStamp and GainTimeStamp columns for the row to be calibrated. Currently, this information is always taken directly from the SDMS being calibrated. Eventually, a user will be able to construct a PositionSwitchedTC that will maintain separate tables of where the OFFs and GAINs are. This will allow the user to alter those values without altering the underlying data. These tables will initially consist of the values found in the MeasurementSet. This is referred to as a ``sucking'' TelescopeComponent because the internal calibration information is ``sucked'' from the original data.
At the moment, apply() creates a new table. In the future, when virtual tables are available, it will perform on-demand (also called on-the-fly calibration, not to be confused with on-the-fly data). On-demand calibration simply means that apply() doesn't actually make a new copy of the data (as the current implementation does). The use of virtual tables means that the actual calibration may not occur until the user requests a specific value - the calibration is done ``on-demand'' as needed by the user.
We have not yet implemented corrupt(). solve() does nothing for this TelescopeComponent since there is nothing to solve for.
There are also a few utility functions that allow a user to set the type of normalization, query the type of normalization, and verify that the SDMS contains the required columns for calibration.
Since the current implementation of PositionSwitchedTC creates a new table when the apply() function is used, it is simplest to have a separate application perform the calibration. In the future, applications will be able to calibrate ``on-demand'' and it will be possible to go directly from a SDMS through apply() to an imager (or other object requiring calibrated data) within a single application.
Our calibrator is extremely simple. Here is the full calibrator:
#include <iostream.h>
#include <aips/Tables.h>
#include <aips/Error.h>
#include <aips/Input.h>
#include <dish/SDMeasurementSet/SDMeasurementSet.h>
#include <dish/SDTelModel/PosSwitchTC.h>
main(int argc, char **argv) {
try {
Input inputs(1);
inputs.Create("raw","","Input: raw, uncalibrated data");
inputs.Create("calibrated","calibrated.data","Output:calibrated.data");
inputs.ReadArguments(argc, argv);
String rawFile(inputs.GetString("raw");
String calibratedFile(inputs.GetString("calibrated");
ROTable rawTable(rawFile);
ROSDMeasurementSet raw(rawTable);
PositionSwitchedTC fixer(PositionSwitchedTC::VECTOR);
SDMeasurementSet calibrated = fixer.apply(raw, calibratedFile);
} catch (AipsError x) {
cerr << "Error - " << x.getMesg() << endl;
} end_try;
return 0;
}
The PositionSwitchedTC is constructed so that it attempts to use vector normalization. As currently implement, the calibrator attempts to calibrate the entire table. Applications will eventually need to do any selection options on the table so that only those portions that need to be calibrated are calibrated.
The time to calibrate the test data set is approximately 3 minutes.
The final step in our initial single dish implementation is imaging. Eventually this will be handled by a MeasurementModel. A MeasurementModel consists of an invert() and a predict() method. invert() ``inverts'' the data to obtain a model of the sky. Formally this can be modeled as an actual matrix inversion although in practice for typical single dish MeasurementModels, there will be no need to do any matrix inversion. In fact, it should be possible to implement a number of common single dish MeasurementModels in glish. predict() takes a model of the sky and ``predicts'' what the data would look like given the assumptions of the MeasurementModel.
We have not attempted to build a MeasurementModel. We have simply built an application that grids the irregularly spaced and over sampled 12-m OTF data onto a regular, appropriately sampled grid. This application will become the invert() method of a specific MeasurementModel.
The basic gridding functions required for the OTF imaging problem are the same as those required for gridding UV data from an interferometer. In the single dish case, we are gridding directly to the image plane and no fft of the gridded data is required. The gridding is really a convolution at each of the observed data points on the sky and is independent of frequency. The GridTool class provides the functionality that we require.
Much of our gridder is devoted to determining the parameters appropriate to the input calibrated SDMeasurementSet so that all of the values will be gridded. There is currently no way to select a portion of an SDMeasurementSet, although such an operation is trivial to implement. The user can indicate through command line inputs the gridding parameters they wish to use (the size of the cube, the center position of the cube on the sky, the dimensions of each pixel on the sky, etc), if they need to override the default values or the values that the gridder will choose based on the MeasurementSet.
The DummyImage class is used to hold the gridded values. Eventually the new Image class will be used, but this was not available when the gridder was written. This should be a rather smooth transition. GridTool is used to do the actual gridding or convolution onto the cube. The resulting DummyImage is written out as a FITS file. This last step (conversion to FITS) will not be necessary once the Image class is used (unless requested by the user to transport their data to another analysis package or for long term storage).
Our first implementation of gridder ran extremely slowly. This single step in the process of converting 12-m OTF data into an image took longer than the time spent taking the data and this first image was only from 1 polarization and 1 spectral window (1/4 of the total data set). This is clearly unacceptable.
The bootle neck is in the GridTool class. Indexing into the cube holding the gridded data and the cube holding the sum of the weights at each data point is through virtual function calls. These calls are extremely expensive. We have replaced the inner-most loop (where the values are multiplied by the appropriate weights and placed into the two cubes) with a simple 10 line fortran routine (not counting comments and variable declarations). This step has increased the execution time so that the gridding time is now roughly comparable to the equivalent step in classic AIPS.
The time to grid the calibrated test data after implementing these enhancements is just under 4 minutes for 1 polarization of the 128 channel spectral window.