 News |
FAQ
|
| Search
|
Home
|
| Getting Started | Documentation | Glish | Learn More | Programming | Contact Us |
|
| Version 1.9 Build 1367 |
|
T.J. Cornwell, NRAO
1999 April 12
This quarter, we have focused intensively on preparation for the first release. This includes finishing areas of functionality, integrating different applications, improving robustness and documentation. The code freeze for the release occured on April 1 (two weeks later than originally planned about 6 months ago). We expect testing and debugging to be complete by mid May, in time for a release at the Chicago meeting of the American Astronomical Society starting May 29.
In Single Dish support, much work was done on preparing the dish package for the release.
Both Bob Garwood and Joe McMullin continue to be involved in supporting the GBT. This includes regular 2 to 3 day visits by at least one of them to Green Bank for most weeks during the quarter. The AIPS++ position in Green Bank was filled during this quarter and the new person, Jim Braatz, starts in the next quarter. The primary focus during this quarter was the complete rewrite of the GBT filler and the reworking of the dish plotter.
The GBT filler rewrite was nearly complete by the end of this quarter. This was necessary in order to improve the speed of the filler in time for planned tests of the GBT holography backend in April. The rewritten filler is about 50 times faster than the old filler and holography data is supported as of the end of the quarter. Some additional fine tuning of the filler for additional backends and to be more flexible and automatic while handling on-line data remain to be done in the next quarter.
Other work in support of the GBT included further iterations with the Green Bank staff on GBT pointing issues. Two avenues for solving the pointing equations have been demonstrated and documented. Some small additions to the Green Bank tipper data reduction were also implemented. The old plotter used with the gbtlogview utility was replaced by the pgplotter tool.
In Synthesis support, the overall development priorities have remain unchanged during this quarter, with a continued focus on expanding the high-level scientific capabilities and user base of the package. We have continued the strategy of working on high-level imaging and calibration facilities to test the completeness of the design and to provide advanced capabilities, while simultaneously allocating resources for completing "thin-path" capabilities for end-to-end reduction for the simpler scientific cases.
An additional, and specific focus during this quarter has been the process of preparing for the public release. This has included the incorporation of integration developments adopted since the Applications Integration meeting held last November, and expanded documentation and testing. A high-level cookbook for synthesis reduction is under development in this regard, and will be integrated with the "Getting Started" tutorial document (Anantharamiah, NRAO) and the new GUI-based recipes contributed by R. Hjellming (NRAO). Synthesis has benefited from the re-structuring of the user interface, described elsewhere in this report, and the associated rationalization of object-oriented terminology at the highest scientific interface to the package.
The "thin-path" development efforts have included the VLA filler, for which an initial implementation has been completed (R. Marson) and is under testing, and an initiative to incorporate uv display capabilities in the Display Library (P. Barnes, in collaboration with ATNF). The filler will be in the public release. It has been designed to be easily extendible to real-time filling, and to form the basis for the VLBA filler.
The high-level imaging and synthesis capabilities developed during this release include a significant continuing effort in mosaicing (M. Holdaway and T. Cornwell), and wide-field imaging (T. Cornwell). Developments in mosaicing include enhanced primary beam correction options, an implementation of MEM, Lattice-based CLEAN and the incorporation of multi-scale CLEAN in mosaicing. Testing of these options with real data is underway. The simulator has been upgraded (T. Cornwell), and will be developed further by M. Holdaway as part of the mosaicing development plan. T. Cornwell has expanded the wide-field prototype into a functioning capability. The projection algorithm is that suggested by W. Brouw, L. Stavely-Smith and R. Sault (ATNF), and thus differs from previous implementations in SDE and AIPS. The low-level wide-field capability has been packaged as a scientific module in Glish, called dragon, and expands and extends the previous capabilities available under this name in SDE. A collaborative arrangement with the group at NRL (N. Kassim) has been established for testing and evaluation of the wide-field software. The members of this group are primary users of the VLA at 74 MHz, and have significant experience in this type of reduction. The new AIPS++ Visiting Scientist, Kumar Golap, who joined NRAO in the parallelization group in early April, will work closely with T. Cornwell and A. Kemball in extending and parallelizing the wide-field imaging capabilities.
In support of the wide-field imaging work, we have developed tools for reading information from various radio catalogs (WENSS, NVSS, FIRST) into AIPS++ component models. This, together with the positions for standard sources available from the measures system, gives a large degree of convenience in, for example, using the outlier capability of dragon to remove confusing sources. In addition star files from AIPS, and Caltech package model files can also be read.
A meeting with the commercial Pixon company (R. Puetter and A. Yahil) took place during this quarter (T. Cornwell and A. Kemball). General terms were agreed for making the pixon deconvolution method available in AIPS++ later this year. This will include parallelization development. In broad terms, AIPS++ will provide an interface to the pixon libraries in a set of imaging applications. The end-users will be responsible for obtaining these licensed libraries from the Pixon corporation separately, to be able to use these applications.
Calibration development has continued in this quarter, primarily in increasing the robustness and cross-calibration capabilities (A. Kemball). This development has followed the plan and design outlined last year, without significant deviation. Expanded testing with real and simulated data has been undertaken during this quarter. Work has started on an assisted imaging tool (analogous to DIFMAP (Caltech)), and it is planned that a preliminary version will be included in the public release.
M. Wieringa has implemented all code infrastructure to support the new MS v2.0 uv-data format, which holds significant advantages for VLBI, advanced synthesis and single-dish reduction. This format has benefited from significant scientific contributions from throughout the consortium. In order to meet pressing priorities for this release and to ensure maximum reliability, it has been decided to adopt the format after the public release. The user impact will be transparent at that time.
NFRA (J. Noordam) have continued the development of utilities required for Westerbork commissioning, as discussed elsewhere in this report.
Work has started on integrating an existing site-testing interferometer at the VLA into AIPS++, and moving the current software into Glish and C++. This work is being undertaken by K. Desai, under partial loan from the AIPS project. On a related matter, real-time imaging remains an important area for AIPS++ after the release, and preliminary discussions are underway in this area for the VLA. The use of AIPS++ in a similar context at the ATCA (M. Wieringa) is further developed at present, and is being moved to AIPS++.
The parallelization initiative with NCSA continues to have important scientific implications for synthesis development, and a full description can be found in the appropriate section of this report. This collaboration is very helpful in opening up new challenges in radio astronomy data reduction and imaging.
An important focus in synthesis planning for the period up to the release is the expansion of the active scientific user base. To date this has involved directed testing with small groups, and beta-testing program. Expanded internal testing is planned for the period immediately preceding the release of the public CD, with follow-on visits to external sites planned for the summer (T. Cornwell and A. Kemball). We regard this as the most effective way in which to expand the number of scientific users of the package, and it will be pursued actively.
In Glish, all of the problems were worked out to allow dynamic loading the Display Library (DL) along with necessary portions of AIPS++ into the glishtk widget client. This allows the glishtk client to remain uncoupled from AIPS++ and the DL, but still permits the DL to be loaded on-demand.
Also, a garbage collector was added to an experimental version of Glish in an attempt to resolve memory leaks seen in Green Bank with long-running glish processes. These leaks are the result of cyclic references in records and functions, and as a result, require higher level analysis than is possible with the simple reference counting mechanism which Glish employs. However, as a result of the extra heap overhead which the collector requires, we haven't decided how this collector should be integrated into the main glish distribution.
Other Glish changes, accomplished during the quarter were:
In Measures, no substantial changes were made.
In AIPS++ Infrastructure, we have again focussed on integration of the the user interface. Much effort has been expended on fine details of the user interface, with the work shared by Tim Cornwell, Neil Killeen and David Barnes, and feedback coming from our testers. We have added an automated scripting capability to the GUI, with the possibility of submission to a batch system where appropriate.
The table tool can now read and write ascii files. We expect this to considerably aid the use of AIPS++ for adhoc processing and plotting.
In Image Analysis, a substantial number of new capabilities have been added via the efforts of Neil Killeen (aided by Ger van Diepen). These are described below in Neil's contribution to the ATNF report. The highlights (of a huge amount of work!) are:
In Display, the main acheivement was the successful deployment of the display library inside Glish. David Barnes did much of this work:
In addition, Harold Ravlin developed a postscript driver for the Display Library, and Darrell Schiebel worked to allow dynamic loading of the DL by Glish.
In the System The system was unstable for some period. The main reason was that shared libraries has to be used by glish to be able to dynamically load the Display Library code. Towards the end of the quarter things settled down and the Display Library could be used. However, it has not been tested on all platforms yet.
Egcs 1.1.2 has been used on HPUX10. It solved the cases where egcs 1.1.1 generated internal compiler errors.
In the area of Parallelization, the parallelization group has been involved with development, profiling, and scientific processing during the first quarter of 1999. The group is processing a large four-pointing HI data set of Dave Westpfahl (NMT) using the NCSA AIPS++ system. Roberts carried out tests on imaging and deconvolving up to 10 channels at a time (6000 x 6000 pixels on a side). Execution time appeared to be anomalously high and Young and Kemball have been carrying out profile experiments to identify inefficiencies in the serial performance using the SGI profiling tools (SpeedShop, CaseVision, and perfex).
A graduate student in the UIUC Computer Science department, Enkelejda (Ledi) Imeraj, has been working with the parallel group to put in I/O instrumentation into the AIPS++ I/O routines. She is now testing I/O performance on the NCSA Origin2000 on the test M33 data set of Dave Westpfahl. The same Origin hardware/software problems have affected her analysis. The system now seems stable enough for her to continue.
In Documentation, we revised Getting Started in AIPS++, started on a cookbook, decided on a format for GUI-based recipes, and implemented a FAQ. We also deployed an ask() function inside AIPS++ for asking help from an AIPS++ expert. We also started work on issues related to the release such as a design for the CDROM cover. We issued on edition of the newsletter (Feb99).
In Management, we started gearing up our outreach program with a presentation at the January AAS meeting, presentations inside NRAO at Tucson and Charlottesville, at the AUI Board (again), and with a visit to and presentation at Arecibo.
We have been approached by Phil Diamond, the new Director of MERLIN, concerning two points. First MERLIN will adopt AIPS++ for processing of their data. Secondly there is a distinct possibility that Jodrell Bank would like to rejoin the active members of the consortium. This would be pending a funding request.
In the Quality Assurance Group, we decided to give the code-review process lower priority until the release. While we recognize the danger in following such a policy for more than a short time, it was necessary to enable the various deadlines to be met.
The key areas of concern identified in the last quarterly report, and current status are:
PostScript: yesterday wrote a small class called the GTkPSPixelCanvas, which will be used to make an agent called the pspixelcanvas. Adjusted GTkWorldCanvas so that worldcanvas agents can be constructed on top of either pixelcanvas objects or pspixelcanvas objects. Yesterday managed to get some PostScript output, and at the same time, prompted by some problems there, made the GTk Display Library much more stable to the loss of agents. Just in the last two hours added the Print button to the viewer, and I can successfully print the contents of DisplayPanels!
This is a Milestone. It's an absolute credit to the fundamental design of the PixelCanvas and WorldCanvas system of John P. [Pixton] and Tom O. [Oosterloo], and the recent implementation work of Harold [Ravlin], that in one day I've been able to add printing capabilities to the viewer. I'm standing on the shoulders of giants here guys! Ok, ok, so you can see, I'm quite pleased with this week's work in the end!
Of course, David should take most of the credit for turning the Display Library into a polished product. As David say, Harold Ravlin has done an excellent job in providing a Postscript driver for the library.
We have now embarked upon testing and completion of documentation. The steps here are:
Most code development has now ceased and AIPS++ staff are engaged upon these tasks, with some help from our testers.
Documentation is now one of the top concerns for the release. While we have extensive reference documentation, we still struggle to provide higher level documentation. The key steps here are:
We have started consideration of the impact of supporting the release. We intend to follow many of the details laid out in the AIPS++ Operations Plan. One recent technical change is that we plan an AIPS++ Update, similar to that used in e.g. Windows, whereby changes to the system can be downloaded easily. This is technically quite straightforward, and will allow us to repond to our users more expeditiously. We expect that Update will be issued about once per month, with major releases (including source code), every 6 months.
The release will be made available on CDROM. Disk space limitations mean that only the two major distributions, Solaris and Linux, will be present on the CDROM. The other distributions, HPUX, Dec Unix, and SGI Irix, will be available via anonymous ftp. The CDROM cover has been designed: it shows the AIPS++ logo, an image of the Magellanic Stream from the Parkes Survey, and the logos of the consortium partners.
Most work for the next quarter will be focused on preparing for the release, as described in the previous section.
In Single Dish, after the release, work on dish will be concentrated on rounding out the functionality to include multi-component fitting and to allow direct interaction with an AIPS++ MeasurementSet.
Work on single dish calibration and imaging was postponed in Q1 due to the demands of the GBT filler and the planned AIPS++ release. This work will need to start this quarter. Discussions also need to being on a unified approach to holography within AIPS++.
Support for the GBT will be improved due to Jim Braatz being the new AIPS++ person in Green Bank. Joe's responsibilities in Green Bank will be gradually shifted to Jim. Jim will work closely with the GBT mockup in testing AIPS++ with other components of the GBT.
The GBT filler work will be largely completed during this quarter. This will include rounding out support for all current backends as well as migrating to version 2 of the MeasurementSet. Tests will begin on providing limited near real-time display of GBT data. The purpose of these types of display is to give a quick look at the data so that data quality can be monitored.
There are two holography tests planned in or shorter after this next quarter. A hardware test is scheduled for April. While this is primarily designed to test the GBT hardware, the filler needs to be functional in order to assess the success of this test. A second test is tentatively scheduled for shortly after this quarter. This test will be as full a test as possible using the 140' telescope. Existing UniPOPS scripts and utilities will be translated into glish so that holography images can be generated from this data.
This quarter the ATNF had 3 people working in AIPS++. These are Neil Killeen (75%, also local manager), Mark Wieringa (30%) and David Barnes (75%). Wim Brouw was still on secondment to ATOMS and spent only about 10% of his time on AIPS++ this quarter.
Our systems (Solaris/Gnu and Digital Unix/egcs) were generally stable this quarter. We now have a running Solaris egcs system as well at Epping. The Linux AIPS++ system was not yet built owing to riduculous delays imposed on us by the vendor of the Linux PC in procuring some cables. That system is now ready for an AIPS++ installation.
None
David's time was spent on the DL work described above, and also on:
Wim has spent his time this quarter on:
Neil's time was splattered over a range of things this quarter, although the main focus was, as always, images. Neil's time went on:
I am yet to provide n-dimensional functional convolution (e.g. convolver ny image by this N-d Gaussian functional) as it is a hard UI problem that I did not have time to do
This quarter Mark's time has been spent on
- Finalizing of design with Athol - Implementation of all the changed MS classes. This has been completed. - Started implementing TableMeasures and Quanta for the appropriate columns. - Updating of all files dependent on the MS classes is in progress.
Harold Ravlin installed egcs on a number of machines, most importantly on the new UIUC Sparc 250 server. Roberts installed AIPS++ under the newest version compiler from egcs (1.1.2) on the new UIUC Sparc 250 server running Solaris 2.7. Currently, all the libraries and applications built properly. Glish builds, but the Tk version of Glish does not perform correctly.
Ravlin and Roberts implemented a work around not being able to mount the master directory with setuid permissions. Now UIUC is able to mount the master disk from Socorro without setuid permissions. This isolates the systems and provided a bit better security than with the setuid permissions.
Doug Roberts visited Socorro to meet with Wes Young and Athol Kemball March 15-19, 1999. Roberts spent the majority of his time working with Young to stabilize the SGI build. Hardware and software upgrades at NCSA as well as general AIPS++ code development (especially GlishTk) has created a rather unstable system on the SGI during the first quarter of 1999, which required extensive oversight by Young and Roberts. Doug Roberts consulted with Kemball in terms of the short term goals of the parallelization project.
Harold worked on a PostScript driver for the display library. The driver now supports:
Note, output is is PostScript level 2 which means really old printers, such as early Apple LaserWriters, probably won't work.
Roberts has been working extensively as a co-editor on the ADASS'98 proceedings as well as preparing a write up of his AIPS++ Parallelization talk and assisting a write up of the Demo that Young and Roberts did at the ADASS'98 conference.
A few improvements have been made:
An article discussing the tiled storage managers has been written for the newsletter.
Functions have been added to the Image classes to be able to define and use masks and regions in an image. A PagedImage can have zero or more masks and regions defined. One mask can be designated as the mask which will by default be used when the image is opened. The class MaskedImage has been removed, because ImageInterface is now directly derived from MaskedLattice. It made the inheritance tree much simpler. LatticeApply has been changed to write the resulting mask when the output lattice has a writable mask.
The Lattice Expression Language has been extended to deal with regions and masks. Note 223 has been written explaining the LEL functionality and possibilities. After a comparison between LEL and ArrayMath, LEL has been optimized to take advantage when ArrayLattice objects are used internally.
The os client has been written to handle OS-related functions. In particular it handles file-related functions.
Michael Haller and Ger van Diepen gave a Glish course for about 20 local people. Ths first session discussed the basics and the second the gui parts. In the last session we discussed some exercises and examples. It was very much appreciated. The slices will be checked into the system.
Michael and Ger spent some time discussing the TableMeasures. It appeared that some changes were needed to make them fully suitable for MS version 2. Alas Michael left NFRA before he could finish the changes. Ger hopes to finish it as soon as possible.
The core NRAO AIPS++ group is now Barnes (100%), Braatz (100%), Cornwell (100%), Garwood (100%), Golap (100%), Holdaway(100%), Kemball(100%), Marson (100%), McMullin (100%), Schiebel (100%), Weatherall (50%), and Young (100%, Golap and Young are funded by NCSA-NRAO collaboration.
We have upgraded our Solaris server (Tarzan) to an UltraSparc 10. We have replaced two old 200MHz PCs with new 450MHZ machines. We have installed an SGI Origin 200 for use in wide-field imaging research and production.
Peter worked on provision of visibility visualization capabilities using the Display Library. This work is on-going.
Tim did the following:
Bob Garwood's primary responsibility is to oversee and contribute towards the single dish work in AIPS++. This work remains focussed on the "dish" environment and the support of the GBT. His contribution in support of the GBT is primarily through the GBT fillers which convert the GBT FITS data files to an AIPS++ MeasurementSet.
Over the past 3 months (January through March) he has done the following:
Mark Holdaway finished implementing the Cornwell and Evans maximum entropy algorithm, which is available from the deconvolver application, and will soon be available from imager. Also, Mark has altered the Clark Clean algorithm to act on Lattices instead of arrays, permitting the Clark Clean to operate on arbitarilly large images. In addition, he has performed a number of incremental improvements to the deconvolver and imager applications, including adding masks to the multiscale clean and improving progress displays for multiscale and Clark Cleans.
Mark is nearly complete with the integration of the new primary beam and voltage pattern classes with a simple user interface in the style of the RegionManager. The voltage pattern manager is called upon to set the voltage pattern descriptions approriate for a measurement set whenever voltage patterns or primary beams are required. After this work is complete, mosaicing integration will begin.
Ralph worked almost exclusively on the VLA filler, which is now checked in and being tested.
Darrell performed the work on Glish described above.
In addition to these tasks, Darrell Schiebel worked on the GBT observer table parser and a script to automatically create a beta release from an AIPS++ installation.
[Preliminary]
In this section, we give the names of people in the various AIPS++ groups and the nominal fraction of time allocated to AIPS++.
The ATNF group is: Neil Killeen (75%, also local manager), Wim Brouw (20%), Mark Wieringa (30%) and David Barnes (75%).
The BIMA/NCSA group is: Harold Ravlin (50%), Doug Roberts (50%).
The NRAO group is: Tim Cornwell (100%), Jim Braatz (100%), Bob Garwood (100%), Kumar Golap(100%), Mark Holdaway (100%), Athol Kemball (100%), Ralph Marson (100%), Joe McMullin (100%), Darrell Schiebel (100%), Kate Weatherall (50%) and Wes Young (100%). Young and Golap are funded by NCSA-NRAO collaboration.
Thus, in aggregate, we have 18 people contributing about 13 FTEs to the AIPS++ Project. Of these, 9 are employed by NRAO (one funded by NCSA), and contribute 8.5 FTEs. The numbers for the other partners are: ATNF 4 and 2, BIMA/NCSA 2 and 0.55, NFRA 3 and 1.9. There are three unfilled positions, one at BIMA/NCSA (Briggs) and two at NRAO (GBT scientist, parallel applications developer: Kumar Golap due in April).
