AIPS++ note 177 Copyright (C) 1994,1995 Associated Universities, Inc. Washington DC, USA. Permission is granted to make and distribute verbatim copies of this document provided the copyright notice and this permission notice are preserved on all copies. Permission is granted to copy and distribute modified versions of this document under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one. Permission is granted to copy and distribute translations of this document into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the AIPS++ Consortium. The AIPS++ consortium may be reached by email at aips2-request@nrao.edu. The postal address is: AIPS++ Consortium, c/o NRAO, 520 Edgemont Rd., Charlottesville, VA. 22903-2475 USA. =============================================================================== Development Plan for the Interferometry (UV-plane) Calibration and Imaging (Aperture Synthesis) Application Development Stream of AIPS++ J.E. Noordam 1 December 1994 1. Introduction =============== AIPS++ applications are sub-divided into groups, which can be developed semi-independently in worldwide "Application Development Streams" (see AIPS++ note 177 "AIPS++ Development Plan", Simon & Noordam, 1994). The management and coordination of these streams is still a bit unclear, but will be shaped by experience. This document provides details about the plans and organization of the Stream that deals with Interferometry (UV-plane) Calibration and Imaging, or Aperture Synthesis. The Aperture Synthesis set of applications could be be considered the heart of AIPS++, because it deals with the UV-data that are generated by the various Consortium interferometers. One of the difficulties in this area is the development of a design that is rich enough to do full justice, now and in the future, to a range of instruments that have been optimized for rather different observational "niches". But at the same time, the fundamental similarities of the interferometers must be reconciled, as much as possible, in a single generic formalism. It is expected that it will take several cycles of the development process before the Aperture Synthesis design might possibly reach the elegance of something like the "A-matrix" formalism (see AIPS++ memo 147 "Recommendations for the AIPS++ Imaging Model", Cornwell, 1992 and AIPS++ note 148 "Recommendations for the AIPS++ Telescope Model", Cornwell, 1992). The trick is to to let AIPS++ grow and renew itself in a natural manner, without alienating the users too much. 2. Tentative phases of Aperture Synthesis development ===================================================== The following phases can be tentatively identified for the Aperture Synthesis Application Development Stream. It should be noted that the later phases are increasingly more hazy, and serve as indications of things that should be done at some point. ** Phase 0: Preparation ----------------------- The early development of AIPS++ has not been easy, since no-one really knew how to tackle such a project. Among the more notable false starts were trying to implement prototype applications without a proper design and a sufficiently mature infrastructure, and working in isolated and sub-critical groups. Nevertheless, many good things were produced: - Period: Jan 1992 - Dec 1994 - Infrastructure development (1992-4) Mainly at the AIPS++ Center in Charlottesville, with important contributions from NFRA and ATNF - The "Green Bank Model" (Feb 1992) - The "A-matrix" formalism (1992, Cornwell) - Socorro period (1993-4), produced various designs - The "Prototyping Bible" (May 1994, Dwingeloo/Jodrell) - Various Aperture Synthesis prototype appl. (Holdaway, Bhatnagar, Wieringa, Willis) - Single Dish applications (1994, Garwood, Glendenning) - AIPS++ Review (Dec 1994) - Alpha Release (version 0.5, Feb 1995) for application programmers, The infrastructure is now deemed to be sufficiently mature to start writing applications in earnest. A concise and up-to-date description of the Aperture Synthesis (and Single Dish) design is given in "The Design of AIPS++" (AIPS++ memo 111, Glendenning, 1994). The firmness of the Aperture Synthesis design owes much to the practical work in the Single Dish stream, which is a few months ahead of Aperture Synthesis. ** Phase 1: Basic Aperture Synthesis applications ----------------------------------- - Period: Jan 1995 - May 1995 - Concentrated effort in one place (Dwingeloo) - Final design framework for basic Aperture Synthesis applications - Look forward to advanced Aperture Synthesis applications (break the design?): + Multi-pointing Aperture Synthesis (mosaicking) + Multi-waveband Aperture Synthesis (wide-band imaging) + Multi-telescope Aperture Synthesis (e.g., MERLIN and VLA combined) - Implementation of generic VS, SM, MM - Implementation of specific TM's, MS's for most of the Consortium interferometers - Simulation of corrupted UV-data (first priority!) includes mechanisms for specifying instrumental effects - Basic reading, calibration, imaging and deconvolution. At the end of a four-month period of concentrated effort, it should be possible to read, calibrate and image (and simulate!) UV-data sets for most of the various Consortium interferometers. Moreover, each site will have a trained AIPS++ Aperture Synthesis programmer, who can build on what is there, and train other locals. Hopefully, they will also have learned to think generically about synthesis imaging. See appendix A for more details of the "Dwingeloo effort". ** Phase 2: Preparation for Beta release (version 0.9) ------------------------------------------------------ - Period: Mar 1995 - Aug 1995 Overlaps with phase 1 (TM builders continue at home) - Distributed over Consortium sites A real commitment from the local managers is needed! - Polishing of specific TM's and MS's - Integration with the rest of AIPS++ e.g., image analysis and visualization - Thorough testing (simulations, automatic procedures) - Code policing - Documentation - Beta Release (version 0.9), Aug 1995 - AIPS++ Summer School, Aug 1995? This is where the AIPS++ Operations group really gets its act together. ** Phase 3: Advanced Aperture Synthesis applications ----------------------------------------------------- - Period: Sep 1995 - Dec 1995 - How is this organized? A 2-3 week workshop, and then distributed? Who coordinates the activity? - Use feedback from astronomical users of the Beta release - Is an update of the Aperture Synthesis Framework needed? - Implementation of: + Multi-pointing Aperture Synthesis (mosaicking) + Multi-waveband Aperture Synthesis (wide-band imaging) + Multi-telescope Aperture Synthesis (e.g., MERLIN and VLA) - More mechanisms for Source Model generation and analysis - More detailed Telescope Models (secondary effects) - Image-plane effects (e.g., non-isoplanaticity) - Polarization VLBI - etc ** Phase 4: Preparation for full release (version 1.0) ------------------------------------------------------ - Period: Nov 1995 - Jan 1996 - Integration of all Aperture Synthesis applications into a coherent whole - Integration with the rest of AIPS++ - Thorough testing (simulations, automatic procedures) - Code policing - Documentation - First full release of AIPS++ (version 1.0), Jan 1996 ** Phase 5: The next cycle ... ------------------------------ - Period: Jan 1996 - May 1996: - How is this organized? A 2-3 week workshop, and then distributed? Who coordinates the activity? - Real-time imaging? - Interface with optical interferometry? - Towards A-matrix elegance? NB: At this point, users all over the world should start writing "unofficial" applications in the AIPS++ "Freedom Layer". The mechanisms for dealing with this in a helpful way must be developed. 3. The "A-matrix" formalism =========================== This formalism was proposed by Tim Cornwell (op. cit.) in 1992. Because it could be an important goal of the AIPS++ Aperture Synthesis development, a short summary of the basic idea will be given here. The A-matrix formalism is an abstraction of how Telescopes convert Sky Brightness into measured data, and vice versa. Many instruments are linear, and their effect can therefore be represented by a matrix (A) operating on an input sky vector (I), to produce a data vector (D). The "Measurement Model" is the matrix equation AI=D. Cornwell points out that the matrix A can represent a Fourier Transform (interferometer), convolution, single dish imaging, beam-switched total power imaging, various kinds of mosaicking etc. He then demonstrates that many of our usual data reduction can be described in terms of matrix operations: various ways of imaging, deconvolution, mosaicking etc. At this moment, all of the Consortium telescopes conform to this abstraction in principle (i.e., if instrumental imperfections are ignored). It follows that any algorithm that is implemented in this formalism can be applied generically to all Consortium telescopes, as long as the telescope-specific A-matrix is known. It is important to note that it is NOT suggested that the actual implementation of the formalism should be in terms of standard linear algebra operations. Since the matrices can be (very) large and sparse, this would be very inefficient in terms of memory use and processing speed. Instead, some of the shortcuts described in the literature should be used. But these implementation details would be hidden from the user, who would think in terms of general matrix operations. At this moment (end 1994), the step to a full A-matrix implementation seems a little too large. Also, it is not yet clear whether this concept will present any special implementation problems, or whether some instrumental effects might violate the assumption of linearity. Therefore, the first Aperture Synthesis applications will probably be implemented along more conventional lines, albeit with some important innovations. However, the Aperture Synthesis designers are aware of the A-matrix concept, and the present design does not preclude it in any way. =========================================================================== Appendix: Details of the "Dwingeloo effort" =========================================== In early 1995, the Aperture Synthesis Application Development Stream will be launched with a four-month concentrated effort, in which the right people will work together in one place, with a minimum of other distractions, in a suitable environment, for a sufficiently long time. This effort will be hosted by NFRA in Dwingeloo, and the following people will participate: 1995 Jan Feb Mar Apr May Jun week xxxx xxxx xxxx xxxx xxxx xxxx FF GG R II II Central Design Team (F,G,I): - Mark Wieringa xx xxxx xxxx xxxx xx F,G,R,I ATNF/WSRT - Dave Shone xx x xx ?? F,G,R,I EVN/MERLIN - Brian Glendenning xx x F System Architect Telescope Model Builders: - Tony Willis xxxx ?? DRAO TM/MS, (SM) - Peter Teuben xx xx? BIMA TM/MS - Phil Diamond xxx xx VLBA/EVN/VLA TM/MS - Sanjay Bhatnagar xxxx GMRT TM/MS - Friso Olnon xxxx xxxx xxxx xxxx xxxx xx WSRT MS / documentation - Jan Noordam xxxx xxxx xxxx xxxx xxxx xx WSRT TM / chief whip Available: - Ger van Diepen xxxx xxxx xxxx xxxx xxxx xx Infrastructure - Bob Sault xx xxxx xxxx xxxx xxxx Helpful comments, ATNF Visiting: - Richard Simon x Project Manager Explanation: - F: Design of Aperture Synthesis Application Framework. - G: Generic versions of TM,MS,VS,MM,SM produced. - R: Mid-term Review (internal) - I: Integration period (the last month). - The basic time unit is a week: - Crosses (x) indicate a definite commitment. - Exclamation mark (!) indicates an urgent suggestion - Question mark (?) indicates a possibility - SM: Source Model (generic) - VS: Visibility Set (generic) - MM: Measurement Model (generic) - TM: Telescope Model (specific) - MS: Measurement Set (specific) The most important group will be the Central Design Team, which will have considerable freedom to design the Aperture Synthesis Applications Framework and to design and implement the generic Aperture Synthesis Classes. At the same time, selected representatives from most (all?) Consortium interferometers will be invited to visit Dwingeloo for a few weeks each, to implement their specific versions of the Telescope Model and Measurement Set. Their feedback will be noted by the Design Team, but they will be prevented from getting under the Designers' feet. The organization is based on the assumption that the "basic" Aperture Synthesis design framework is already fairly stable before January, and that there already are working prototypes of the Measurement Set (MS), UVFITS filler of the MS, VisSet (VS) and Telescope Model (TM). Then the TM/MS builders can get to work, perhaps even before they arrive in Dwingeloo. The Design Team can then concentrate its wisdom on the conceptually somewhat more difficult Source Model and Measurement Model, and on trying to break the basic design by looking forward to advanced Aperture Synthesis applications. The Dwingeloo Aperture Synthesis effort will be actively supported by the AIPS++ Center, which will provide a programming environment in the form of the "alpha" release of AIPS++. The Center will also send its System Architect (Glendenning) to Dwingeloo for 3 weeks, at the beginning of the period. And, faithful l to the Fountain Model of OO software development, the Center will endeavor to be responsive to the emerging needs of the application builders.