Rationale

My opinions are strongly influenced by three things. First modifying the Single Source Fits Filler to calculate Parallactic angle. Second, prodded by Bob's original proposal on the structure of the MS, I've also spent some time re-reading the Diamond et al document on "FITS format for Interferometry Data Exchange". I have also benefited from a very useful discussion with Athol Kemball. D et al. describe the interface between the VLBA correlator and the AIPS system in the form of a (largish) number of Binary tables. It is interesting as a description of the information that must be passed on from Correlator to Post-Processing system and also what the Post-Processing system needs to store when making the data set persistent after some processing. It maps onto the AIPS (not AIPS++) format reasonably closely, I think. My precis of the format is simplified deliberately. The data file can contain time-independent tables at the beginning:

• Array geometry
• Frequency
• Source

Then the actual visibility data comes in quanta containing

• Flagging information
• Calibration
• Antenna
• UV
• (the actual complex data)

There are in addition general tables that can be present:

• Gain Curve
• Interferometer Model
• Phase cal
• Weather
• Bandpass
• Baseline correction

Then there are some VLBA specific tables

• Calc model
• Emphemeris
• Sampler Statistics

While one can argue with the details of what is stored in each table, I think that one cannot easily argue with these as categories of information. Nearly all of this information must show up in the MS in some form. It appears to me to be a very straightforward principle that "publishing" the columns in each one of these tables in the MS is a very bad idea. I suspect that we would end up with about 200 columns of information. Thus the current form of the One Big Table for the MS should be modified. We have identified things that belong in a Frequency table, a Source table, an Observation table, etc. Thus we need these tables, plus keys that enable access into each one, and the original data. That's what a MS should be. The change in philosophy is actually not that big since all I've proposed is that some columns in One Big Table should be moved off to separate tables.

In thinking about the details of the columns and subtables, I (and others) have become convinced that we need to unite Synthesis and SD MeasurementSets. A storage manager can handle the disk space problem. Hence the description given below does unify the two. The reasons for this unification are:

• It is a fundamental goal of AIPS++ to unify SD and synthesis processing where appropriate.
• Many of the concepts from one domain are beginning to show up in the other. For example, modern single dish spectrometers can and will perform cross correlation of different polarizations. Synthesis arrays keep careful track of antenna characteristics, single dish processing would benefit from this in some applications.
• The proposed model telescope is fundamentally a correlation device. The correlation is only purely real in special circumstances: ANTENNA1=ANTENNA2 and FEED1=FEED2 and the same RECEPTOR used on both FEEDS. In other words, we are rapidly approaching the point when single dish and real data will no longer be synonymous. For this reason, I've resisted replacing ANTENNA1, ANTENNA2 by Vector<Int> ANTENNA. Similarly for FEED.

The disadvantages to a merging of SD and synthesis are:

• More memory shuffling, half of the data being empty.
• More processing, half of the multiplications being fruitless.
• Unnecessary complication of something that is mercifully simpler than synthesis.

If necessary we could define a FLOAT_DATA engine to counter some of these problems.

One other major point that should be made is that the MS is fundamentally a repository of information, both from the telescope and from subsequent operations such as calibration. The MS itself should therefore have few member functions that impose any policy such as, for example, how values are interpolated. I've assumed that such an access layer exists but is independent of the MS. The MeasurementEquation relies on one such interface, VisSet and VisIter, that may serve as a starting point for a more general interface.

In checking the proposal, I've tried to think about many different types of observing: single dish, simple interferometry, VLBI, mosaicing observations, multi-beam, phased arrays, phased feeds. We have a lot of bases to cover so I wouldn't be surprised if I'd missed something.