 News |
FAQ
|
| Search
|
Home
|
| Getting Started | Documentation | Glish | Learn More | Programming | Contact Us |
|
| Version 1.9 Build 1367 |
|
T.J. Cornwell, NRAO
1997 March 27
AIPS++ passed an important milestone in the beta release of the system in February 1997 to a small number of users. AIPS++ has been used by an external group as the platform for reduction of data from a scientific project: a group at Parkes using a multi-beam system to survey extra-galactic HI elected to use AIPS++.
This report describes the progress in the last quarter, the results from the beta testing, the plans for the next quarter, and the revised AIPS++ development plan.
The most important event has been the beta release, made on February 26. This is covered in more detail in section 3 below.
In Single Dish support, we spent a significant fraction of our available time completing our promised support for the Parkes Multibeam 21cm observations. This included the ability to read and write SDFITS format to and from the MeasurementSet. We've also progressed on the single dish analysis program, SDCalc. A browser is now available, the limited plotter has been replaced by the binding of PGPLOT to Glish, and a demonstration script has been checked in to the system so that others may try out the existing program.
In Synthesis support, we worked extensively with scientists at the AOC to improve the documentation. We added spectral line support in the Measurement Equation, though this is still undergoing some revision. We added a variant of the Clark Clean to the existing imager, allowing cleaning of either I alone, I and V or I,Q,U, and V simultaneously. We started work on sort-less gridding for spectral line data, and disk-based FFTs, both of which benefit considerably from the advanced capabilities of the AIPS++ library classes. We completed working versions of fillers to the MeasurementSet for WSRT and BIMA formats. We also completed a writer of UVFITS from a MeasurementSet, needed for interoperability with existing analysis packages.
In Measures, we finished the development of a GUI front end to the Units and Measures classes. Testing indicates that this general tool should now be supplemented by more specific-purpose guis for particular purposes. We wrote very extensive documentation for the measures object in glish.
In Glish support, we fixed a number of bugs, added cursor support for the PGPLOT binding, added a number of additional tk widgets, investigated alternatives to the transport layer SDS, added command and output logging, and made a number of other miscellaneous changes in response to testing and use of Glish. In AIPS++ Infrastructure, we introduced a new improved Coordinates class. We added new capabilities to the Distributed Object system such as a progress bar. We have defined and described a standard framework for guis in AIPS++.
In Visualization and Image Analysis, we continued development of the visualization tool-kit. We added a number of convenience features to Aipsview, the principal one being the ability to control the line-graphics from a menu. We also gave Aipsview the ability to display 2- or 3-D sections of images with more than 3 dimensions (this is vital for spectro-polarimetric synthesis imaging where the canonical image has 4 dimensions: RA, Dec, Stokes, and Frequency). We have largely incorporated the Glish-PGPLOT binding in the system, the work outstanding being on the documentation.
In the System area, we enabled installation of AIPS++ from a binary release (needed for the beta release). We completed a port of AIPS++ to a Pentium Pro 200 MHz machine running Linux. This machine fits our definition of a canonical machine, costing less than US$5000 and having memory of 64Mbytes and disk space of several gigabytes. It is a very satisfactory AIPS++ platform running at roughly twice the speed of the Sparc 20 in Socorro, for both compiling and production.
In Documentation, we spent a considerable amount of time working to improve various introductory material. We added a capability to search the entire documentation tree. The bug tracking system has a new interface. Bug reports may be entered directly from glish.
The beta release was announced on February 26 1997. The sites involved are all consortium sites, and Caltech, University of Iowa, Kapteyn Astronomical Institute, and the National Optical Astronomy Observatory. The release notes are available at:
http://aips2.nrao.edu/aips++/docs/project/releasenotes/releasenotes.html
The release notes give detailed information on what is required for installation, how to install the system, what capabilities are available, known problems, the nature of the feedback that we would particularly like to receive, and how to submit problem reports. This is a binary release, although source code is included, with binaries for Solaris, Linux, and HP/UX.
Most feedback on the beta release is conducted either via the AIPS++ Bug Tracking system at:
http://tarzan.aoc.nrao.edu/cgi-bin/gnats_home.pl
or via an e-mail exploder that is archived at:
http://aips2.nrao.edu/aips++/mail/aips2-beta.
The responses to beta release can be summarized as follows:
http://aips2.nrao.edu/aips++/mail/aips2-beta/17
and the reply at:
http://aips2.nrao.edu/aips++/mail/aips2-beta/19
We plan to address the three major deficiencies noted above in the next few months: documentation, user interface, and speed and memory use of the synthesis code. We believe the limited public release should be made only once these three deficiencies have been satisfactorily rectified. As a result, we plan two more beta releases before the limited public release, the next targeted towards adding spectral line capabilities and improving memory use and speed, and the second targeted towards improvements in the user interface. Documentation will be addressed in both. It is possible that one more release might be needed after 0.95, and we'll keep that possibility open.
The degree and usefulness of feedback from testers varies considerably. Some sites have only sent in one or two questions or comments, while others have generated many tens of significant questions, bug reports, requests for enhancements. The most prolific and useful tester has been a non-consortium astronomer, Tim Pearson at Caltech. The absence of spectral-line capabilities is the key reason why testers have not been attracted to the system, with a number of sites promising more active testing once version 0.9 is available.
The need for extensive support of the Parkes Multibeam project has essentially terminated now that they have completed their analysis package and started observations. Over the last year, our support of the MB project has consumed between 1 and 1.5 person years of our time in providing parts of the necessary infrastructure. Since nearly all of these would have been needed anyway, the cost to us has been a change in priorities to move some things (such as SDFITS support, and AIPS++ table locking) ahead in time. We are pleased that the MB group was able to make effective use of many different components of the AIPS++ system. As well as helping provide a platform to do astronomy, this is important in that it is the first demonstration that the system is programmable by an external group of astronomers.
More details on the Parkes MB project can be found at
http://www.atnf.csiro.au/Research/multibeam/
and a live display of spectra from the receivers can be found at:
http://wwwpks.atnf.csiro.au/people/multi/public_html/live/multibeam_live.html
Further details on the use of AIPS++ can be found in the ATNF contribution below.
The long-term plans for the development of AIPS++ are spelled out in the development plan at:
http://aips2.nrao.edu/aips++/docs/notes/202/202.html
This covers major milestones and events up until late 1999, when we expect to release version 3.0.
The principal activity for the next quarter will be support of beta testing, and preparation of the next release.
In Single Dish support, we will continue the development of SDCalc, adding more functionality in addition to the selection, browsing and averaging capabilities now present.
In Synthesis support, we will continue with improving memory usage and speed for both continuum and spectral line processing. We will improve the plotting and editing capabilities. We will add a gui. We will add fitting of Component Models to represent the sky brightness.
In Measures, we will develop methods for distributing time-dependent data inside AIPS++.
In Glish support, we will release version 2.6. We will add more tk widgets as required.
In AIPS++ Infrastructure, we will deploy the current implementation of table locking now being code-reviewed. We will review the capabilities of the current Lattice/PagedArray/PagedImage classes. We will develop a tool for image-based mathematics. We will continue to augment the services (e.g. plotting, display) available to applications writers via the Distributed Object mechanism.
In Visualization and Image Analysis, we will continue development of the visualization tool-kit. We will finish the binding of PGPLOT to Glish. The major remaining development scheduled for Aipsview is the addition of labelling of curvilinear coordinates.
In the System area, we have no substantial plans beyond consolidating various ports.
In Documentation, we will improve the introductory material as suggested by beta testers. This is an area where we would particularly benefit from the close involvement of astronomers.
In Management, we will enlist the help of someone with experience in operational support of a existing large package to help define how operations of AIPS++ will occur. This will cover user and programmer support, testing policies, code acceptance procedures, etc.
The ATNF currently has 5 people working in aips++. These are Neil Killeen (25-75%, also local manager), Wim Brouw (50%), Mark Calabretta (50%), Tom Oosterloo (75%) and Mark Wieringa (30%). Additionally, there is a collaboration between the aips++ project and the ATNF headed Parkes 21cm multibeam receiver project which is contributing substantial effort to using and developing code within the aips++ environment.
Mark Calabretta will shortly be leaving the project. Mark has been with aips++ since its inception and has been responsible for the system distribution and system generation software. I would like to thank Mark for his many contributions to aips++ over the years. Mark will continue in his role as aips++ system manager at the ATNF. The loss of his 50% contribution to aips++ will be largely cancelled (although in different work areas) by an increase of Wim Brouw's contribution to a nominal 90%.
Later in the year, probably around August, Tom Oosterloo will also leave the project and the ATNF. Although the timing for this is not very good, we believe enough of the display library will be in place before he leaves. We will refill this position on a term appointment.
This problem highlights that aips++ is vulnerable to the fact that the providers of operating systems are decoupled from those that provide some of our tools.
We have installed the beta release and have a number of testers; Imke de Pater (joint IQUV deconvolution of Jupiter), John Dickey (spectral-line processing of ATCA data), Raffaella Morganti (calibration and imaging of continuum VLA data), Ray Norris (documentation perusal), Lister Staveley-Smith (images module evaluation). So far our contribution to the evaluation of the beta release has been small.
In late January, Neil Killeen visited the Kapteyn Institute as a follow up to the visit by Ron Ekers and Wim Brouw in 1996 in which they discussed the possibility of the K.I. contributing to aips++. The outcome of this most recent visit was that Hans Terlouw would install the beta release and explore it. In addition, he would investigate the idea of making an interface to Gipsy via either a derived Image class, or a Gipsy storage manager approach.
Mark Calabretta's main responsibility is to the code distribution system. His time in the last quarter has been spent on:
Wim Brouw's responsibility is mainly to designing and
implementing the Measure and related classes. His time has been spent
as follows.
)
Tom Oosterloo is working on visualization and image analysis
software. Additionally he has been seconded to the multibeam group to
help them with their aips++ software effort.
Mark Wieringa finished putting spectral line capabilities into
the MeasurementEquation and related code in late December. At the same
time he speeded up the processing, but it was still too slow for any
real spectral line work (more work will be done on this of course).
Mark did not work on aips++ again until mid February when he worked on
speeding up the gain matrix solution process for (self)calibration. A
first pass at this improved speeds by up to a factor of 10, but further
work is needed to check the accuracy of the solutions in various cases.
Neil Killeen spends his aips++ time attending to local ATNF aips++ management
issues as well as generating image applications. He has reduced his
aips++ fraction from 75% in the previous quarter to about 25% this quarter
as the former high level is to the detriment of the rest of his job
at the ATNF ! He spent his time on
We currently have 7 people (4.15 full-time equivalents) working on AIPS++: Dick Crutcher (25%), Yanti Maio (50%), John Pixton (100%), Harold Ravlin (100%), Doug Roberts (65%), Peter Teuben (25%), and Taisheng Ye (50%). Accomplishments during the past quarter are described under each person.
Dick Crutcher served as local AIPS++ project manager and as BIMA director in charge of our participation in AIPS++.
Yanti Miao worked on getting all pgplot functions to be callable from tkglish, documenting this, and translating pgplot demo programs into glish scripts to make examples of using TkPGPLOT and to test TkPGPLOT.
John Pixton worked mainly on implementation of the AIPS++ Image Display Library, including preliminary documentation. Specific areas include implementing keyboard and mouse event handling, handling graphic attributes and text, fonts, and annotations, writing some general widget utilities that will facilitate the widget layer, adjusting the PixelCanvas interface to have callback "classes" rather than functions, and adding some more test programs. He also worked on fixing gplot1d and AIPSview glish and coordinate bugs.
Harold Ravlin administers our AIPS++ workstations. He also worked on fixing AIPSview bugs which had been reported, getting AIPSview support libraries ready for the next distribution, getting AIPSview to run under linux, and adding a number of new features. These new features include:
Doug Roberts wrote a document describing table editing. He also wrote a system to allow import of MIRIAD tables written as ASCII files into AIPS++ for editing and manipulation and wrote documentation for editing external tables from MIRIAD and AIPS in AIPS++ including examples. He updated the AIPSview User's Guide to include the improvements and increased functionality added recently. Many graphics have been added to assist in the description of the documentation functionality. He wrote a preliminary User Specification for the AIPS++ profile-fitting tool, which Taisheng Ye will implement. The specification includes a review of profile fitting programs in other packages in order to identify which features should be included in AIPS++. This document also lists uses which astronomers will likely have for such a tool as well as specific features that must be in the tool. He installed the AIPS++ binary beta releases on our Sparc Ultra and on our linux machine. He also ported the AIPS++ beta release source code to the SGI using the gnu compilers and started a similar port using the SGI native compilers. (The native SGI compilers may be necessary to give optimal performance when we begin AIPS++ parallelization.)
Peter Teuben finished the bimafiller. Reading miriad data needs to be done with the existing miriad library. A small clone of miriad was created with a C++ callable interface. He added all miriad C routines to the C++ callable miriad library, hence making it complete. He tested the filler with various types of BIMA datasets. Multiple-polarization cases are not yet properly handled. More fine tuning is needed on handling multi-source, multi-array and multi-polarization, which will be done in parallel with various upcoming calibration issues for BIMA data.
Taisheng Ye started active AIPS++ work here in late February. He has been given an initial tasks of writing MathTool for manipulating images and a profile fitting tool for fitting spectral line profiles.
Local project members: Ger van Diepen (GVD), Jan Noordam (JEN, local manager), Friso Olnon (FMO), Henk Vosmeijer (HJV).
Beta testing has started at NFRA and JIVE, with STAG Chairman Robert Braun taking an active role for NFRA. It has become clear that speed, memory use and user interface are the areas in which improvement is the most urgent. It is expected that the testing intensity will increase after the release of the spectral line reduction.
After the beta release, the current status of AIPS++ and the way ahead are being discussed as planned. Jan Noordam has been working with Ger van Diepen on a demonstration of an alternative implementation of the Measurement Equation, which they will discuss in Socorro in April.
The NFRA AIPS++ group can now hire an extra AIPS++ application programmer, after an analysis of the AIPS++ functionality that has to be implemented locally to replace IWOS and NEWSTAR. The group is now actively looking, but it is difficult to find someone with the right kind of experience. In the meantime, JIVE has hired its own person with special responsibility for the link with AIPS++. This represents an increase in the local AIPS++ capability in Dwingeloo.
The data-conversion interfaces between AIPS++ and TMS and NEWSTAR are virtually working now, in accordance with the schedule. However, the arrival of the first module of the DZB correlator has been postponed for two months, until June 1st. The following uv-data conversion programs now exist:
The local uv-data visualiser/flagger has enough functionality (and user friendliness) to support the TMS/DZB introduction in Wbork until the official AIPS++ functions can take over.
Because of the beta release, a high pace of change continued in this quarter. Especially the preparation of the HPUX branch of the release and the setup of the proper environment for the testers took quite some time of both Friso Olnon and Ger van Diepen.
The routine of weekly inhales and sneezes takes less and less time. Occasionally the HPUX branch coughs a bit, but the problems are always solved relatively quickly - helped by the high responsivity of the AIPS++ people abroad, in Socorro and Sydney. Olnon spent more time in trying to get the AIPS++ documentation built locally; he has not yet succeeded in doing that, mainly due to our local LaTex setup.
Ger van Diepen has extended the Table Query Language in several ways:
The possibility of controlled concurrent access to tables has been added by means of a locking/synchronization mechanism. This allows to lock a table permanently (for most efficient access), to do it manually (for transaction-like processing) and to let the table system do it itself. When a lock is freed, modified data are flushed to disk. When a lock is acquired, the internal caches are refreshed when needed. At the moment the code is being `copped' and almost ready for check-in.
The Record classes have been extended somewhat to make it easier to handle derived Record classes in the base RecordInterface class.
The software tools Purify, PureCoverage and Quantify have been evaluated extensively, and appear to be very useful. They revealed leaks in a few programs. They also assisted in understanding a strange error which appeared in optimized HP code. It looks as if the g++ compiler deletes temporaries so early that they could not be used with the at_c functions.
Joe McMullin joined us as a scientist in Green Bank in early March from the SMTO. He'll be working as a liasion between the single dish group in Socorro and the members of the Green Bank Telescope Project. Jeff Uphoff made a lateral move to AIPS++ from the CV system administration group. Jeff brings a very strong background in Linux, and security issues in Unix. To fill a vacant position in Socorro, we are currently looking for scientist with experience in software development and interferometry. The net result of these changes will be to increase the number of scientists in the AIPS++ group by two as the expense of the number of software engineers.
In addition, as part of the on-going gradual transition from AIPS to AIPS++, Pat Murphy will be spending 30% of his time helping us with system support, and Athol Kemball will be spending some fraction of time on VLBI development in AIPS++. For both these people, the priority remains highest for AIPS support so that, for example, we expect Athol to spend whatever time is necessary supporting OVLBI in AIPS.
The core NRAO AIPS++ group is now Cornwell (100%), Garwood (90%, 10% going to Unipops support), Glendenning (100%), Marson (100%), McMullin (100%), Schiebel (100%) and Uphoff (100%). In addition, Young of the AOC Computer Group works with us, and as described above, Murphy and Kemball also contribute. We have one unfilled position, in Socorro, for a VLBI software specialist. In addition, a number of scientists participate at various levels: Rick Fisher, Bob Hjellming, Harvey Liszt, Michael Rupen.
Tim Cornwell works on documentation, the synthesis code, and general project management. Bob Garwood directs the single dish effort in Charlottesville and Green Bank. Brian Glendenning is the prime architect of many different parts of the system, acts as pinch-hitter in helping out in various areas, oversees development of the library code, and work on general project management. Ralph Marson works on the synthesis code and the Lattice/PagedArray/PagedImage classes, and is the AIPS++ Code Cop, to boot. Joe McMullin works in single dish support for the GBT. Darrell Schiebel is our Glish guru and is responsible for maintaining, adding new features to, and removing bugs from Glish. Jeff Uphoff works on the GUI side of the single dish package. Pat Murphy, Darrell Schiebel and Jeff Uphoff all work on system support. Wes Young works on documentation, gui development, and VLA support via AIPS++.
We now have a variety of platforms available to run AIPS++ on:
In addition, a SGI Power Challenge is being purchased for OVLBI support. We will have access to this machine for a limited amount of testing.
We have investigated the use of various code testing products. We've found Purify and Quantify by Pure Atria to be very useful and plan to purchase a number of licenses for these despite the unencouraging licensing provisions. These tools have enabled us to find quickly a number of otherwise difficult to diagnose problems, such as memory leaks, memory accesses out of bounds, etc..
In Single Dish support, we did continue the development of SDCalc, adding more functionality in addition to the averaging capability now present.
In Synthesis support, we did continue with alpha testing. We started development of a GUI for the imager module in accordance with the STAG's recommendations. We did not start on adding VLBI support to the synthesis system.
In Measures, we did continue and finish this stage of the development of the GUI front end to the Units and Measures classes.
In Glish support, we did not yet release version 2.6. We expect to continue bug fixing for quite some time.
In AIPS++ Infrastructure, we did make a number of changes and additions to the Image class to support the application development in Image Analysis.
In Visualization and Image Analysis, we did continue development of the visualization tool-kit, but did not yet produce a first application in this time. We did add a number of convenience features to Aipsview, the principal one being the ability to control the line-graphics from a menu. We are still progressing in incorporating the Glish-PGPLOT binding in the system.
In the System area, we did support the beta and limited public releases.
In Documentation, we did continue adding content but we did not incorporate tutorials on basic synthesis imaging.
In this section, we give the names of people in the various AIPS++ groups and the nominal fraction of time allocated to AIPS++.
The ATNF group is: Neil Killeen (25-75%, also local manager), Wim Brouw (50%), Mark Calabretta (50%), Tom Oosterloo (75%) and Mark Wieringa (30%).
The BIMA/NCSA group is: Dick Crutcher (25%), Yanti Maio (50%), John Pixton (100%), Harold Ravlin (100%), Doug Roberts (65%), Peter Teuben (25%), and Taisheng Ye (50%).
The NFRA group is: Ger van Diepen (100%), Jan Noordam (25%), Friso Olnon (50%), Henk Vosmeijer (50%).
The NRAO group is: Tim Cornwell (100%), Bob Garwood (90%), Brian Glendenning (100%), Athol Kemball (30% now, rising to 100% by October 1997), Ralph Marson (100%), Pat Murphy (30%), Darrell Schiebel (100%), Jeff Uphoff (100%) and Wes Young (90%). Athol and Pat both have responsibility for maintaining the AIPS system and will be diverted back there if necessary. We have one open position in GB for an astronomer to work with AIPS++ and the GBT, and one in Socorro. In addition, a number of scientists participate at various levels: Rick Fisher, Bob Hjellming, Harvey Liszt.
