 News |
FAQ
|
| Search
|
Home
|
| Getting Started | Documentation | Glish | Learn More | Programming | Contact Us |
|
| Version 1.9 Build 1367 |
|
J.E.Noordam
August 1995, version 2.0
File: /aips++/nfra/nfrarequ.latex
This memo contains the NFRA requirements with respect to AIPS++. It is an update of User Specification Memo nr 112 (Dutch Requirements for AIPS++), which was an input document for AIPS++ User Specification Memo nr 115 (AIPS++ Consortium User Specifications) of 7 April 1992. Although the essence of the requirements has not changed significantly, an update was deemed useful in the light of the developments in the AIPS++ project since 1992.
Version 1.0 of this document was discussed with the potential Dutch AIPS++ user community at a workshop in Dwingeloo, on 14 June 1995, and with the AIPS++ Project Manager. It was then updated with the received comments and suggestions, clarified where necessary, and submitted as an AIPS++ User Specification Memo.
The NFRA policy towards AIPS++ has the following main pillars:
NFRA has always played a very active role in the Consortium (and has probably kept it alive by doing so). It has made the largest contribution to the Centre after NRAO (Van Diepen, Olnon, the `UVCI efforts' in Dwingeloo in 1994/95, development of the matrix-based Measurement Equation), and intends to continue to do this. NFRA has also earmarked some extra funds (about 200 kfl over 2 years) to stimulate the project at key points when necessary. Part of this money has already been used to strengthen the new Socorro Team by helping ATNF to send Mark Wieringa there, and to make it possible to bring forward the hiring of a new programmer.
The local AIPS++ group in Dwingeloo is now preparing to follow close on the heels of the AIPS++ Centre in Socorro. The group will concentrate on uv-calibration and imaging (UVCI), with emphasis on the special NFRA interests listed in section 1.2 below. The group will work within the framework of the overall AIPS++ project as much as possible, according to the priorities set by the AIPS++ Project Manager, unless urgent local concerns dictate otherwise.
The interests of NFRA are parallel but not identical to those of other Consortium members. In particular, NFRA seems to be the only member that has expressed the wish to change over from its current WSRT uv-data reduction package (NEWSTAR) to AIPS++ as soon as possible (starting in 1996), because it cannot afford to work on two packages at the same time. This has prompted NFRA to insist that the implementation of `everyday' calibration by the Centre should get a higher priority than was originally planned. Moreover, there are some specific areas, usually connected to WSRT, EVN and Square Km Array (SKAI), to which NFRA attaches particular importance. The following list is not in any specific order:
NFRA would like to start changing over from its current data reduction package (NEWSTAR) to AIPS++ during 1996. This time-scale is driven particularly by the needs of the new WSRT Telescope Management System (TMS), and other aspects of the WSRT upgrade.
Jan 1996 Decision whether to use NEWSTAR intermediately for TMS
Aug 1996 First production release of TMS
Aug 1996 Prototype of new WSRT backend arrives (TMS needed)
Sep 1996 New WSRT frontends arrive (TMS needed)
Jan 1997 Start of simulations for the Square Km Array (SKAI)
Jul 1997 Final possibility for TMS to adopt AIPS++
Nov 1997 Second production release of TMS
Jan 1998 WSRT WENSS survey finished: NEWSTAR pensioned off.
Apr 1998 Final production release of TMS
Dec 1999 WSRT WHISP survey finished.
1999 Choice of SKAI concept
2005 Integration of SKAI prototype element(s) in WSRT
2010 SKAI fully operational
The basic assumption is that the Centre will indulge in an intensive UVCI design acitivity during the remainder of 1995, and produce a complete UVCI system for the ATCA in the beginning of 1996. The NFRA AIPS++ group will then add WSRT-specific features where necessary, while becoming AIPS++ application programmers in the process. After that, the group will be in a position to contribute new functionality.
In the meantime, it is assumed that the AIPS++ infrastructure will develop in step with the applications, and that the interests of other Consortium members will drive the development of other (non-UVCI) AIPS++ functionality like Image Analysis.
It is highly desirable for NFRA to be able to use AIPS++ for the TMS for its first production release in August 1996: the fall-back option of using NEWSTAR for 1-2 years is certainly possible, but time-consuming and confusing. A definite decision about this will be taken in January 1996.
The following General Requirements are either fundamental, or not easy to place in a specific category. They are not necessarily in order of importance:
Explicit design: Long overdue. Essential for a proper management of the project. The absence of an explicit design has made it very difficult for anyone but the `inner circle' to contribute to AIPS++: the potentially large pool of person-power is used ineffectively. The reliance on an implicit design has also increased the danger of `single-point failures'.
TMS interface: An interface (software bus) that allows other systems like the WSRT TMS to use AIPS++ functions, and that allows AIPS++ to communicate with external packages.
Version management: The TMS will rely on certain AIPS++ functionality for its critical on-line operations. It will therefore only adopt a new AIPS++ release after a period of extensive testing the TMS-critical bits. This means that the TMS (and perhaps other on-line systems?) may work with `frozen' versions of AIPS++ at times.
HP9000 platform: Most NFRA (and JIVE) workstations are of this make (although the local AIPS++ group has a SunSparc20 for development). AIPS++ should be `ported' to HP's by the time of the first release in early 1996.
Programmers support: Documentation. Examples. Template applications. Graphical tools to pipeline AIPS++ modules. Schools?
User support: Documentation. Quality control. User feedback.
Global Services like Time and Earth motion should be provided centrally by AIPS++. This includes collection of data from Bureaus of Standards etc by the Centre. NB: A similar software structure is needed to deal with local ionospheric data (see section 6.4). It would be a nice example of code reuse if elements of the software structure that deals with global Time could be copied for that purpose.
Future tiger teams: One of the potential advantages of AIPS++ is expected to be that isolated experts scattered around the globe will be able to collaborate more effectively in tackling difficult problems. Once AIPS++ is operational (in 1996/7), international `tiger teams' should be set up to study subjects like Map Reliability, interference detection and rejection, or refinement of the Measurement Equation.
While it is true that too much generality causes delays and may lead to an unworkable result, it is also true that unfortunate design decisions in existing packages have sometimes made the implementation of new functionality unnecesarily difficult and ugly. (I deliberately refrain from giving examples, but we can all think of some good ones). The design of AIPS++ should not exclude any of the known observing, calibration and imaging modes, and somehow create a `better than usual' environment for the natural inclusion of new ones. The following principles could be adopted:
Simulation of corrupted uv-data is extremely important for testing the correctness of calibration procedures etc. It should be available at the very beginning, and not be an after-thought, like in most existing packages. In a somewhat later stage, simulation will also be an important tool in the study of the limitations of calibration and imaging. And finally, simulation is an important tool in the development of new instruments like the Square Km Array.
The following extra functionality is all that is needed:
Tools for specification: The user has to be able to specify Sky Models, Measurement Equations, and the actual values of all kinds of instrumental effects, in uv-plane and image-plane.
Tools for assessing the results: Convenient visualisation and quantification of the difference between simulated input and the resulting output.
MS-filler from WSRT archive data: Needed soon. The route via NEWSTAR and UVFITS is cumbersome and possibly restrictive, and will eventually disappear. NFRA (FMO) will split the design into a part that is generic, and can be used for other telescopes, and a part that is WSRT-specific.
Reading corrections from NEWSTAR SCN-files: May be needed while some NEWSTAR calibration functions are not yet in AIPS++. One way of dealing with this is to transfer corrected data from NEWSTAR to AIPS++ via UVFITS. If that proves impractical for some reason, it may be desirable to transfer NEWSTAR corrections to AIPS++, which requires much less work than transferring uv-data. The latter can always be read from the WSRT archive format.
Editing means removing unwanted uv-data, for whatever reason. This can be done by separating selected data into a secondary MS, or by setting (reversible) flags. Editing is usually done interactively, for instance by pointing to the screen, and can be very time-consuming. An alternative way to identify `bad' data is by filtering (see below).
Filtering means applying some kind of software filter to the data, with the object of flagging or subtracting unwanted parts. Examples are the subtraction of continuum flux from line data, removing interference, or removing the effects of a strong source outside the primary beam (like the Sun or CasA). In view of the increasing data volume, and the rising level of interference, filtering will be of increasing importance. The development of sophisticated and reliable algorithms should get a high priority.
Implementation suggestion: For multi-dimensional filtering operations it might be efficient and/or convenient to transfer the uv-data temporarily from the MS to a 3D cube, with axes time, frequency channel, and interferometer nr. The user defines the dimensions of the cube, which might represent a subset of the data, or a sliding time window. Such `fit-cubes' might be implemented as a separate class, with generalised filter functions as methods.
NEWSTAR has 8 flags per uv-data value, so that they can be flagged for different reasons. This makes it possible to undo the result of some flagging operations, without having to reset all flags (which may represent many hours of work).
After uv-data has been filtered, it will often be desirable to collect the subtracted part into a secondary Measurement Set, for instance to make a separate map to check whether anything valuable has been thrown away by the filtering proces. Another reason for the use of secondary MS's is `data-decimation' for quick-look purposes, possibly after smoothing. In short, there are many reasons to integrate the concept of secondary MS's as receptacles for subsets of physically modified uv-data into AIPS++ in a natural fashion.
In the latest thinking, the Measurement Model enshrines all the information about the instrument, in the form of a Measurement Equation and actual values for its parameters. (This differs from the original Green Bank Model). A Solver estimates (better) values for instrumental parameters. In this section, a list is given of the instrumental effects for which NFRA requires Solvers.
Manual input: Sometimes the user already has an alternative source for parameter values, for instance data about the inonosphere. In that case, there must be tools to use these data.
Using redundant spacings is important for calibration. Traditionally, spacings are defined as redundant when they occupy the same cell in uv-space, which is usually also a function of polarisation, frequency, and time. Redundancy is `stronger' if the cell is smaller. The concept can be generalised to stating that visibilities are redundant if a nominal Measurement Equation (defined by the user) produces the same values for them for a user-defined trial Sky Model, to an accuracy specified by the user. Redundancy equations represent a `higher truth' than Selfcal equations because they do not depend upon a detailed Sky Model, which may be incomplete. Therefore, Solvers must be equipped with the extra book-keeping capability to make use of redundant spacings.
IF-based, i.e for each IF-channel either amplitude and phase, or a complex number.
Phase screen over the array: This can be treated as a parametrised expression that links the individual phases for each IF.
X/Y Phase-Zero Difference (PZD): Tricky and important, because
it will translate Stokes U into Stokes V (for the WSRT). For very high
accuracy measurements of V, the PZD has to be calibrated to a degree.
There are various ways to estimate it:
- Standard WSRT procedure: An unpolarised point source is
observed with `crossed' dipoles. This will no longer be possible
with the new frontends.
- Standard NEWSTAR procedure: first correct for leakage,
using an unpolarised source, then look at a source with strong
U-polarisation.
- Mosaicing: create artificial U, by moving the
telescope so that a strong source is in the U-lobe of the off-axis
instrumental polarisation. The great advantage is that this method can
be used on an arbitrary source, during the observations.
- Minimise the V-polarisation of the short baselines
when observing very extended sources like large-scale galactic
foreground. Works very well, but only for special cases.
- Inject a noise signal simultaneously into X and Y
channels. Experience at the ATCA indicates that the overall PZD can be
kept to a few degrees, even though the dipoles and the atmosphere are
excluded. The new WSRT frontends will have this capability too.
IF Delay offsets: They cause phase gradients across the frequency bands, and thus cause decorrelation. They cannot be properly corrected afterwards, so they have to be set right before the observation. NB: Delay offsets seem to play a role also in the relatively high closure errors (decorrelation!) that the WSRT tied array causes in VLBI.
IF bandpass shape: Phase and amplitude.
Undoing Tsys: This is a correction that has been physically applied on-line at the WSRT. It is not entirely correct for certain fields with lots of 90cm flux in the galactic plane. It must be possible to undo the on-line correction, and replace it with a new one.
Antenna positions: These are measured by looking at the time-behaviour of Selfcal phase of a strong calibrator. For greater accuracy, multiple calibrators in different parts of the sky are used.
Should probably be solved semi-simultaneously (i.e. alternating iteratively) with the G-matrix. For linearly polarised receptors, the real part of the leakage terms is to first order a dipole position angle error. Therefore, it is important to account for Faraday rotation at the same time.
Beamshape/Off-axis polarisation: Measured by stepping through a grid of pointing positioins around a strong calibrator source with known polarisation. If a satisfactory analytical expression can be found, one may solve for its parameters. Otherwise, the results must be smoothed by a suitable function to reduce the measurement noise, and stored in an `lmf-cube' (position(l,m) and frequency(f)). NFRA will start measuring them this fall, using NEWSTAR.
Pointing errors: Time-dependent, different per antenna. A procedure is needed in AIPS++ to replace the existing program for measuring the WSRT pointing model. TMS requirement.
Mirror surface: Holography. Measuring strategy: One reference antenna is pointed at a strong source, while another steps through a grid of pointing positions around it. The output is Fourier Transformed to give the deviations of the antenna mirror surface. This strategy can also be used to detremine the beam-shape.
Non-isoplanaticity: Time-dependent, different per antenna. May be important for the SKAI (and GMRT), and in some cases for the WSRT. It has been suggested that this effect should be modelled by a (relatively thin) phase screen at an altitude of several km above the telescope. Alternatively, one might tackle it via time-dependent position parameters for the Sky Model components.
Correction for Faraday rotation is vital for the NFRA search for linearly polarised sources at 90/50 cm (WENSS), which are candidates for msec pulsars. The ionospheric Faraday rotation can vary up to 15 degr per hour at these frequencies, especially in the daytime. This can completely `wash out' the linear polarisation in a 12-hr observation.
External: Estimates of the Faraday rotation can be derived to a certain extent from external measurements of the ionosphere (ionosonde, chirpsounder, GPS). Especially 2-frequency GPS is promising, because it measures the effect through the ionosphere. All methods measure the Total Electron Content (TEC) of the ionosphere. They all require a model of the TEC distribution to estimate the actual Faraday rotation along the direction of the observed source.
Faraday Selfcal: Determines the Faraday rotation variations with the help of a strong polarised source during the observations. In general, the angle separation with the target source will be so large (> 1 degr) that an ionospheric TEC model will be needed to calculate the Faraday rotation for the target source.
What is needed:
- a local mechanism to store and retrieve the ionosphere data as
a function of time.
- an AIPS++ mechanism to plug in and use TEC models to calculate
the Faraday rotation in a certain direction at a certain time.
NB: If a way could be found to make a map of the total amount of
linear polarisation by Fourier Transforming
Pvis = 
, all this Faraday stuff
would be less urgent. See also `exotic maps' in section 8.
Additive (A-matrix): The increasingly popular WSRT 327 MHz broadband (80 MHz) system has additive ifr-based errors, which show up as constant non-zero values in the selfcal residuals (at least when the dominating source is a point source in the centre of the field). Until they are understood, they must be removed in software.
Multiplicative (M-matrix): ...
In the most recent thinking, a Sky Model (SM) is the projection on the sky of a 3D astrophysical Source Model (SOMOD?). The latter is not foreseen in the early AIPS++, and will not be discussed further here.
A Sky Model must be instrument-independent by definition. Otherwise it cannot be used to combine the data from observations with different instrumental settings (e.g. mosaicing), or even from entirely different instruments. But a Sky Model should perhaps know about its general limitations, i.e. the parts of Phase Space about which it has no information.
NFRA has always championed an `able' Sky Model like the one used in NEWSTAR and VLBI. The NEWSTAR Sky Model contains a mixture of (fully deconvolved) Parametrised Source Components (PSC) and traditional CLEAN Components (CC). The parameters of PSCs are: Flux (I,Q,U,V), position (l,m, not tied to a grid), shape (elliptic gaussian only), Faraday rotation measure, spectral index. In AIPS++, one could add other spectral parameters, and time-dependence of all parameters. NB: Braun has suggested that it might be possible to model some sources of EM interference as a special kind of Sky Model component.
Since the Measurement Equation (ME) of a radio telescope does not have a proper inverse, the best way to generate and improve a Sky Model is by fiddling its parameters (and the parameters of the Measurement Model) until the predicted uv-model is consistent with the measured uv-data. This will require solvers of the same type as those for instrumental parameters. The problem with CCs is that they are only partially deconvolved: a point source will be represented by a collection of CCs at neighbouring grid-points. Also, the effects of bandwidth smearing etc will cause a point source to be interpreted as an extended source. For the Sky Model to be truly instrument-independent, CCs have to be converted into PSCs by means of some kind of `residual deconvolution' before the Sky Model can be `updated'.
One way to make a polarised Sky Model is to CLEAN all four I,Q,U,V maps simultaneoulsy, convert the CCs into PSCs, and fiddle them further using the actual uv-data.
It is clear that the thinking about Sky Models is still immature within the Consortium. Ideas and experience will have to be pulled together into an integrated set of concepts. This should be a high priority, before too many of the old concepts get entrenched in prototypes etc.
East-West arrays have advantages for wide-field mapping, especially at low frequencies (300 MHz), because they can handle their geometry exactly.
Robust weighting, as developed by Briggs [], automatically finds the optimum on a sliding scale between `uniform' and `natural' weighting, depending on the available uv-data. This might be the answer to the problem of adding all the redundant WSRT spacings into a map while retaining the low sidelobe level of the regular WSRT beamshape is retained. Such a weighting scheme is needed in any case to combine the data from an arbitrary collection of telescopes into a single map.
Polarisation calibration using mosaicing: Bregman has suggested that one might use the (known) off-axis instrumental polarisation to calibrate other polarisation parameters during the observation of an arbitrary source. Especially the X/Y Phase Zero Difference, using the artificial Stokes U. This strengthens the requirement in section 3 that mosaicing should be the default mode of observing.
Tied Array: Apart from their use for VLBI, tied arrays can also be formed into local interferometers. For instance by splitting the WSRT array into two or more tied arrays of one or more antennas each, whose outputs are then correlated. The purpose is to get high sensitivity and a very long spectrum (16.000 channels). This mode requires modelling and subtraction of background sources. The residuals are processed with single-dish type software.
Exotic maps: Example: A map of
P2vis = Q2vis + U2vis,
to detect sources with high linear polarisation, without worrying
about Faraday rotation. Since the FT of a square is an
autocorrellation, this will cause a very high noise peak in the centre
of the map. Pogrebenko has suggested that this can be avoided by
taking
P2(t) = Q(t).Q(t + 
t) + U(t).U(t + t),
i.e. multiplying the corresponding uv-data of successive integration
intervals.
Drift-Scanning: The antenna's do not track the sky, but the fringe-stopping does. This is a form of mosaicing in which the mechanical gears are not worn out so quickly, and no observing time is lost by changing the pointing centre. It may be a can of worms, but the possibility should be considered.
NFRA places a high importance to this, but assumes that the requirements of other Consortium partners will drive the development of Image Analysis (not to be confused with Visualisation!). No specific needs were identified at the AIPS++ workshop in Dwingeloo on 14 June 1995. The Groningen group will rely on GIPSY for the immediate future, and is not contemplating any direct involvement in AIPS++ in 1995/6.
Assumed to be Glish for the time being. Must be consistent, complete, self-evident, and able to manipulate AIPS++ objects.
Walter Jaffe has emphasised that packages like AIPS and IRAF could be described as `schedulers of large operations'. These packages have few facilities for the intuitive manipulation of (small) chunks of data. These two modes should be reconciled in AIPS++. NB: This should remove an important psychological obstacle in the unification of synthesis and single dish observing.
Jay Chengalur has submitted a list of requirements to Dick Crutcher, who answered that all of them will be met in the next version of AipsView. Other people can check it out for themselves by signing up as AipsView "beta tester" (Email: aipsview@ncsa.uiuc.edu).
We need powerful, efficient and convenient visualisation of at least images and uv-data. 3D visualisation should be available on common machines.
The needs of the TMS are a prime driver for the NFRA requirements for AIPS++ in terms of timescale. In this sense, the TMS plays the same role for NFRA as the GBT does for NRAO. The first production version of the TMS has to be operational in August 1996, and it would be highly desirable if it could use AIPS++ from the start. The alternative of using NEWSTAR as an interim solution is viable but unattractive. Therefore, a separate checklist is given here for the minimum required TMS functionality. If the prospects for the items on this list look good in January 1996, the TMS project team will be able to lean on AIPS++ from the start. For a more detailed treatment of TMS needs, see [3].
The SKAI has officially been launched as the next large project of NFRA. It consists of a radio telescope with a total collecting area of one square km, distributed over an array of 30-100 elements, each with a diameter of 100-200 m. Most elements will be within an ellipse of 30-50 km, but a few `outriggers' will be up to 200 km away. Wavelength coverage is 150-1500 MHz. The current plan is to select the concept in 1999, build 1-3 prototype elements and integrate them with the WSRT before 2005, and build the entire instrument before 2010.
We assume that AIPS++ will be flexible enough to grow in response to new developments for 15-20 years (!), so that it will still be around when the SKAI comes on-line. But in any case, AIPS++ will be needed for simulations during the development phase, starting in 1997.
If the SKAI elements would be conventional parabolic dishes, it would have the same processing requirements as the GMRT (except for multiple simultaneous beams). However, it seems more likely that the indivudual elements will be some form of phased dipole array, preferably without moving parts. Such an element may have unconventional properties:
Moreover, prototypes of these unusual antennas have to be integrated in the WSRT. Fortunately, all this will not present a problem as long as SKAI antennas meet the requirement that can be fully described by means of an antenna-based `Jones' matrix in the Measurement Equation. This is the case for all concepts that have been considered to date. So, ias long as AIPS++ is driven by a Measurement Equation, it will be able to meet the requirements of the SKAI with virtually no extra effort (amazing, isn't it?)
Finally, since the correlator will have at least one million channels, the data volume will be huge. Distributed and Parallel Processing will be required.
