|
|
|
T. Cornwell, Project Manager NRAO-AOC AIPS++, the Astronomical Information Processing System, is an ambitious project being developed by an international consortium of four observatories. About 25 people are actively involved in designing and writing the system, and over 1 million lines of C++ and Glish code have been written so far. We plan to publicly release the system in mid 1998. The pre-release system is in active use at various consortium sites for scientific observations; (the HIPASS survey using a 21cm multibeam receiver system on the Parkes telescope), support of a real-time system (the Telescope Management System at WSRT), and engineering work (development of the Green Bank Telescope). In addition, a number of testers have been working with two internal pre-releases of the system. Details of many of these initiatives are given in this newsletter.
The project is ambitious in a number of
ways: first as an international collaboration
between observatories with different goals
and cultures; second, adopting new software
methods to cut development
times and costs; third, as an
attempt to share not only libraries of
software but conceptual models of
radio-telescopes; fourth, as a
sociologically interesting challenge to
perform technically difficult
development using people spread all
over the globe; and finally, and most
importantly, as an attempt to aid
astronomers in doing science
with radio-telescopes by providing new
tools and applications for
their work.
|
Parkes Multibeam Project
David Barnes,University of Melbourne and Lister Staveley-Smith, ATNF-Epping
Background
The
Parkes telescope
has devoted about 25% of its observing time
to HI multibeam surveys of the
southern sky (HIPASS) and the
Zone of Avoidance since March 1997. Early on
in the project, three main requirements
were set for the data reduction
software:
No existing software package was able to provide us with the required functionality and satisfy the above requirements. So, in late 1995, we made the decision to use the AIPS++ library. We predicted then that this would make it harder to develop our software, but would give us access to all the AIPS++ applications as they came along. This prediction proved largely correct in retrospect. At times it has been hard to develop and maintain a system using a library that was shifting under our feet. However, recent tools such as the images module have proved to be extremely useful, allowing us to use the power of the AIPS++ image format for operations such as Hanning smoothing and moment analysis. It was also a surprise to us how useful Glish (and its graphical interface) was in pseudo-realtime applications, and Glish is now popular for the control of instrumentation at the Parkes telescope.
(continued on page 3)
| |
| February 1998 | AIPS++ Newsletter | Page 2 |
|
Project News Summary
(continued from page 1) | ||
All of these ambitious goals have already been realized in one way or another. I think that the collaboration has worked well at both the observatory and the personal level. We have been able to develop and share complicated libraries (notable examples being the Table system and the Glish system), and conceptual models (the MeasurementEquation formalism using our synthesis processing code), which are now being | turned into astronomical applications taking full advantage of the object-oriented methods that we adopted. Now that AIPS++ has been demonstrated in the specialized areas mentioned above, our next challenge is to support other, more mainstream uses of the system, such as calibration and imaging of data from synthesis telescopes. The purpose of this newsletter is to inform our users, both current and future, |
about AIPS++: what is there now,
what is planned, and experiences
using the system. In this edition,
we focus on the current uses of AIPS++,
and on what is to be expected in the
forthcoming third pre-release, and the
limited public release expected this summer.
|
|
Single Dish Developments Bob Garwood, NRAO-CV | |||
The single dish group in AIPS++ consists of Bob Garwood and Jeff Uphoff at NRAO-Charlottesville and Joe McMullin at NRAO-Green Bank. The two primary goals of this group are to produce a GUI-based single dish environment, currently called dish, and to support the GBT during construction, commissioning and operation. The work on dish is guided by Harvey Liszt. Harvey serves both as a designer of the interface and as a tester on the work done in implementing that design. Joe is the person with primary GBT support responsibilities.The focus of this group over the past year has been the development of dish. Initially, we are attempting to create an environment which is capable of doing traditional single-dish spectral-line analysis with an emphasis on individual spectra. In the long run, dish will be capable of handling continuum data and of making continuum and spectral-line images using a variety of techniques. Our goal has been to produce a GUI interface which is intuitive as well as obvious about what it has done to the data. Results are displayed in a plotter window as they are generated. | These results are also available at the Glish command-line for users to access outside of the GUI interface. The interface has a "trace" mode which shows the underlying Glish commands which are executed when each operation is applied through the graphical interface. The hope is that this will provide a window to the command line functionality behind the GUI so that users can construct their own operations on the data using the same tools that dish is using. Advanced users are able to add their own GUI operation to the suite of available operations in dish.As of today, dish can do some data selection followed by averaging of the selected data. Baselines (sinusoidal and polynomial) can be fit to and removed from the data. Several smoothing functions (boxcar, Hanning, Gaussian) are supported. The data can be browsed and arbitrary, user-defined functions can be applied to the data through the GUI interface. By early February 1998, dish will also contain two ways of reducing in-band frequency switched data and an initial attempt to allow users to sequence any of these operations into a string of operations which could |
then be invoked through a single press of a button.
This could also be used to apply a sequence to an
entire data set at once. Currently, all input data
must be in a simple AIPS++ table constructed
directly from a single dish FITS (SDFITS) file.
This initial phase of development (simple spectral-line
operations) will continue through the next several months.
A version will be available with the planned AIPS++
limited public release in June. This will contain,
in addition to the above operations, component fitting
and moments-plus the ability to save the state of the
dish environment from one session to the next. It
will also be possible to process data found in an
AIPS++ MeasurementSet as well as a spectral line
cube. | |
| February 1998 | AIPS++ Newsletter | Page 3 |
Software OverviewThe flow chart gives an overview of the realtime reduction system. The overall system is managed by a Glish script which presents a graphical interface to the outside world. The Glish script manages several clients which are named in boxes in the flow chart. All the clients are written in AIPS++, with the exception of Multibeam View which is written in C using the karma software system.The AIPS++ clients are responsible for reading the raw data as it is being collected, doing a robust bandpass correction for each of the 13 beams and two polarizations, and writing the data out into an AIPS++ measurement set. At a later stage the measurement sets are written to single-dish FITS files and archived on cdrom. MultibeamView is a replacement for AIPS++ AipsView and allows the observer to view the processed, and continuously updated, data cubes whose axes are frequency, time and beam number (for each of two polarizations). Other AIPS++ clients allow us to grid the data into cubes in RA-Dec-Velocity where we are able to look for our galaxies! More can be found in the Parkes multibeam documentation folders. The AIPS++-related multibeam software suite was largely written by David Barnes, Mike Kesteven, Tom Oosterloo, Richard Gooch, Lister Staveley-Smith and Taisheng Ye. It has been a pleasure to develop this software in association with the talented AIPS++ team.
|
 Figure 1: Real Time Flow
|
| February 1998 | AIPS++ Newsletter | Page 4 |
| The Role of AIPS++ in Commissioning DZB/TMS J.E.Noordam, NFRA | |||
The WSRT is the first radio synthesis instrument that really depends on AIPS++. Sometime during 1995, the decision was made to use the AIPS++ MeasurementSet (MS) as the prime receptacle for the uv-data from the new correlator (DZB). In addition, the AIPS++ Measures system was adopted for all coordinate conversions, etc., of the new Telescope Management System (TMS).Although it is true that AIPS++ has developed a little more slowly than expected (hoped) at the time, this decision was not really a blind leap of faith. The Table system on which the MS is built had been in a good state for a long time, and we had privileged access to its creator - Ger van Diepen. We had equal confidence in Wim Brouw's Measures System, which already existed as a stand-alone package and has since been put through its paces by the Parkes effort. We also took care to make sure that we had working conversion programs to existing packages like NEWSTAR, AIPS, and DIFMAP. We decided to use AIPS++ at two levels. For astronomical reduction we would rely completely on the "official" package, as generated by the AIPS++ Centre in Socorro. We would of course continue to contribute to the infrastructure (mainly related to the Table system), and make sure that the package was suitable for reducing WSRT data by giving lots of user feed-back. On the other level, we would locally generate the tools that were needed during commissioning and |
close to the
telescope: uv-data inspection,
basic calibration for setting up the
instrument before observation, and archiving.
This strategy is generally working out, give or take the usual bumps
in the road.
|
used for other instruments.
On the whole, we still feel that we made the right decision in betting on AIPS++. At the very least, we now have a powerful software environment in which to build WSRT applications, and to develop our next instrument (SKAI). Its cornerstone is probably Glish, which combines the convenience of an interpreted language with the power of Distributed Objects. (People will soon discover that this adds a completely new dimension to data reduction). If AIPS++ indeed develops into the World Reduction Package (e.g. because of its broad functionality, its gleaming user interface, or other less rational reasons), this will add considerably to the success of the WSRT upgrade.
| |
| February 1998 | AIPS++ Newsletter | Page 5 |
| AIPS++ Testing at NFRA Robert Braun, NFRA | ||
The AIPS++ testing/use activities in the Netherlands are in two main areas, and for both of them slow data access has so far limited what we can do.An attempt is being made to do some preliminary data inspection and instrumental calibration on the data stream coming out of the new correlator at the WSRT. Our new on-line system now writes only AIPS++ Measurement Sets. Inspection and calibration has proved impossible within AIPS++ during the first few months of this effort. Our normal observing mode is a 12 hour observation with 91 baselines, | 2 polarizations and 256 spectral channels using 30 or 60 second averaging. Initial experiences, were that observations longer than a few hours could not be accessed successfully at all due to excessive memory requirements. Even those observations which could be accessed required inordinately long execution times for even the simplest data inspection.Much work is now being done to speed up and limit data access. We will start again when this is sufficiently improved. In the mean time, we now proceed immediately to Classic AIPS via UVFITS in order to do anything useful. By summer 1998 the correlator capacity will be up to its full level of | 8 times the data-rate indicated above, and AIPS++ data access will need to cope well with this level of data flow.Rene Vermeulen, Huib van Langevelde and I are trying to do AIPS++ beta testing as time permits. Rene and I have worked with VLA and WSRT data, while Huib has worked with some simple JIVE VLB correlator output. We all will try to do more in the coming weeks and months, but since all of us are actually trying to commission real instruments and observing modes that are not being supported yet within AIPS++ this is very tough. [Ed. Note: Progress has been reported on the problem of data access speed for the TMS, and will be further described in the future when it has undergone a full range of testing.] |
|
AIPS++ Testing in Socorro Bob Hjellming, NRAO-AOC | |||
Organized testing of AIPS++ at the NRAO Array Operations Center evolved beyond incidental work by one or two people in December 1997. Anatharamaiah, Bagri, Hjellming, and Rupen have been using AIPS++ for a mixture of testing and work on specific projects. Regular meetings between these testers and AIPS++ staff at the AOC occur on Thursdays.Anantha's testing has been a mixture of trying out the usual sequence of operations turning visibility data into images, and exercising the features of Glish and AipsView to analyze spectral line image cubes. Bagri has been working on a specific project to try to understand the causes of the apparent time variability of polarization leakage terms for VLA | antennas, taking advantage of the fact that AIPS++ is the only software package with the separation of calibration terms (the different Jones matrices) suitable for this type of problem. Hjellming has been doing a mixture of exercising visibility and image handling in AIPS++, and importing Green Bank Interferemeter data into Tables for data manipulation and plotting. A series of recipes for doing these things in AIPS++ is under construction; these are mainly self-reminders about how to do things. Rupen has been excercising both general capabilities and those specific to visibility data.The testing at the AOC deals with beta, weekly, and daily versions of AIPS++ to provide immediate feedback while things are in early stages of development. |
| |
| February 1998 | AIPS++ Newsletter | Page 6 |
|
The Quality Assurance Group Ralph Marson, NRAO-AOC | |||
The quality assurance group is a sub-group of AIPS++ developers who, in addition to their code development duties, undertake to ensure that AIPS++ code is of a high quality. This is done in a number of ways.* Ensuring that new coding standards are developed and maintaining existing standards. Coding standards are necessary to maintain a uniform approach in the design and implementation of AIPS++ code. Otherwise there would be a vast range of different styles making code maintenance by anyone other than the original author difficult. The quality assurance group often does not develop new coding standards. That task can be done by any developer in consultation with all other interested parties. Instead it ensures that existing policies are kept up to date and decides when new ones are necessary. This task in undertaken by the "Rules Boss"; currently this is Wim Brouw. * Ensuring that new code meets the current coding standards. We have had within AIPS++ a peer review procedure for all code. This review process is mediated by a member of the quality assurance group called the "Code Cop." When a developer has finished a suitable portion of code it is submitted to the code cop who finds a suitable reviewer. Usually this is someone from a different AIPS++ site who is interested in using the newly developed code.
| The reviewer ensures that new code meets the current coding standards. The reviewer also checks whether the code performs its intended task and if the documentation is adequate.* Performing the unit testing of code. The reviewer and reviewee interact, usually via email, until the reviewer is satisfied with the code. The code cop ensures that the review process does not stall and restarts it if necessary. We have found that this process always produces higher quality code with better documentation and fewer bugs. But it takes time. One of the coding standards we use is that all pieces of code submitted for review must include a test program. This program exercises all the functions and most (more than 80%) of the lines of code. The program tests whether the results produced by all functions agree with expected results that are hard coded into the test program. The test programs are maintained with the AIPS++ source code and compiled and executed weekly. This is semi-automatic as all the test programs adhere to a defined format. Email is sent to AIPS++ developers responsible for designated areas (called "Module Bosses") with the results of the current unit tests. Last week's unit tests had 164 of 165 test programs compile and execute successfully.
| The module bosses, whom are often not the original authors, are responsible for correcting problems with the unit testing. The code cop overseas this process. * Undertaking the user-level testing of applicationsMost newer AIPS++ applications incorporate a graphical user interface. Testing this code is more difficult as it involves user interaction rather than the automated execution of test procedures. We plan to address this by having interested users try applications, often with their own data sets and data reduction goals, in the month or so preceding a release. For this to be successful we need a diverse group of people from various astronomical institutions. The "Chief Tester" coordinates this group of testers. Jan Noordam is the current chief tester. The testing described above is wider in scope than has, to my knowledge, ever been undertaken by a radio astronomical data reduction package. It is our expectation that it will result in code with far fewer bugs.
| |
| February 1998 | AIPS++ Newsletter | Page 7 |
| GB Tipper Project J.P. McMullin, NRAO-GB | |||
The 86 GHz radiometer was developed to better understand the atmospheric observing limitations in Green Bank. We are determining whether frequencies as high as 115 GHz can be realistically observed with the GBT. AIPS++ software is used for some portions of the Tipper Project.The 86 GHz radiometer was completed in early December by Mike Stennes, Bob Simmons, Lisa Wray, John Ford and Joe McMullin (with advice from a host of others). At its current location, just north of the Jansky Lab, its mirror scans a full 360 degrees every second, taking measurements on the sky and the internal calibration loads. It has a bandwidth of 3 GHz and a typical system temperature of 850 K. These measurements are integrated for one minute and then written to a FITS file. A new FITS file is generated every 2.5 days. Every hour a cron process executes a Glish script to 'fill' the FITS data into an AIPS++ table. This is then manipulated and parsed out to an ASCII file using the AIPS++ measurement and utility modules. This file is read and processed by a program which computes the opacity values and generates plots of the data with the PGPLOT libraries. It then generates and updates an HTML-style page for viewing the information. The tipper data may be accessed through the main Green Bank web page as one of the Resources. Current information provided includes the current 86 GHz measurement and quality of fit, plots of the opacity for the last day, last week, and last 2 weeks. | Plots of the data by month, with information on the percentage of time the atmospheric opacity was less than a certain value, and a link to current weather information are also available at this site. This page is updated every hour.We also include an extrapolation of the 86 GHz opacity to other frequencies of interest to the GBT, based on Erich Grossman's AT (Atmospheric Transmission) Program. These extrapolations are rough as they are based on manipulating the amount of precipitable water vapor to match the measured 86 GHz opacity. The opacity at 86 GHz can be dominated by the O2 opacity in dry air. Refinements will be added in the future. Based on the current 86 GHz, we plot the atmospheric transmission from 1 to 150 GHz as an aid to planning observations. Results thus far are very encouraging for 3 millimeter work in Green Bank.
The MS Distributed Object
The AIPS++ Distributed Object (DO) dealing with aperture synthesis data is named MS (MeasurementSet). The MS DO is accessible from Glish by default. You create an MS object from an existing MeasurementSet, from an MS stored in Tables on disk, or from a UVFITS file, using Glish commands like:
myms:=ms('X1354.ms',
|
ms2:=ms('SN1987A.ms', | |
| February 1998 | AIPS++ Newsletter | Page 8 |
|
What's new in AIPS++? Tim Cornwell, NRAO-AOC | |||
This column highlights recent developments in AIPS++ that may be of interest to users. In the future, this will include information useful to both astronomers and programmers, but for this edition, we concentrate on the former. The following features are available now in the development systems installed at the various consortium sites, and will be available in the next beta release scheduled for March 1998 (contact Tim Cornwell if you are interested in participating in our beta testing program).- The Glish scripting language has undergone a major release (2.6) that includes many new facilities compared to 2.5. The Glish User Manual has been changed to reflect these changes. Glish is available as a separate product: see the Glish Home Page. - Measures is a Glish application for handling physical quantities with units, coordinate systems, and reference frames. It is built upon the C++ Measures classes and makes most of the C++-based functionality available to Glish users. It can be used either from the Glish command line or from a Glish-based graphical user interface. It has many important capabilities but most interesting perhaps is the ability to do conversions between all major types of astronomical coordinates. It has the JPL DE200 solar system emphemeris built in so that you can, for example, find the position of Jupiter in Galactic coordinates! - Pgplotter is a Glish-based graphical user interface to the PGPLOT library that enables plotting with Glish commands using C++ code. | Our plan is that this will become the standard way of generating plots. Plot commands can be entered from the Glish command line, from the Pgplotter GUI, and from C++-based applications. A plot can be saved to either a ``plot'' file or a PostScript file Any plot file can be reopened and edited to change or add to the plot. Thus users can easily customize plots with commands similar to the well-known PGPLOT function calls, which have been slightly modified for Glish compatibility. More details can be found in the Reference Manual section on Pgplotter.- A new Tablebrowser (Newtb) has been released. This has improved performance and many new features, such as editing of entries, plotting of columns using the Pgplotter mentioned above, user control of the appearance of the table, and the use of queries to generate new tables. See the Reference Manual for more details. - Sky is a Glish-hosted imaging application that enables processing of visibilities into images, including deconvolution. It is based on the Hamaker-Bregman-Sault Measurement Equation for synthesis telescopes. It has many capabilities such as joint processing of all Stokes parameters, robust weighting, flexible windowing in deconvolution, spectral imaging, a novel sort-less gridding algorithm, specification of control parameters such as phase shifts in the measures system, very flexible multi-field processing, support for primary beam correction (including simple mosaicing of multiple pointing observations), and flexible choice of images size (powers of two are no longer required). Plotting of various products is done by the Pgplotter mentioned above. | Sky and Calibrater applications, now under development, will become the primary tools for synthesis imaging and calibration. More details on Sky are found in the Reference Manual .More information on these and other developments are summarized quarter by quarter in the AIPS++ Quarterly Reports.
| |
