 News |
FAQ
|
| Search
|
Home
|
| Getting Started | Documentation | Glish | Learn More | Programming | Contact Us |
|
| Version 1.9 Build 1367 |
|
T.J. Cornwell, NRAO
1999 August 11
This quarter, we have focused almost exclusively on preparing the first release. This included testing the system, fixing defects, and preparing and testing the actual CROM release.
Other than testing and fixing defects, little new work has been performed. Most of the new work has been in the application areas of single dish, synthesis and visualization.
In Single Dish support, Jim Braatz joined the single dish group in aips++ in April. He will be working out of Green Bank, replacing Joe McMullin who is now based in Charlottesville. Garwood, McMullin and Braatz continue to be involved in supporting the GBT. This includes regular visits by Garwood and McMullin to Green Bank and by Braatz to Charlottesville.
The revised GBT filler was used in a planned test of the GBT holography backend in April. The test went well exposing one bug and several shortcomings of the filler. The filler is being revised in time for another planned holography backend test in late July. This revision will include the necessary framework to handle on-line data.
Much of the additional GBT support was focussed on transferring knowledge and responsibility of the GBT commissioning support, tipper support, and other routine Green Bank support tasks from McMullin to Braatz. This has been mostly completed.
Much of the non-GBT focus during this quarter was on finding and fixing bugs in the single dish software. The dish internal rewrite was postponed in order that dish be ready and debugged in time for the upcoming release. The dish plotter continues to be improved. A command-line Gaussian component fitting operation was added to dish using available tools. A plug-in for the dish plotter was written to use the JPL line list for quick line identifications. The documentation for dish was also improved.
Work was begun this quarter on a suite of single dish imaging tools to be used in the dish environment. Three tools have been developed so far. lookmap reads a dish data set and plots the spatial locations for all of the data in the set relative to a selected scan. spectramap reads a dish data set and plots the spectra at their relative spatial locations. contourmap reads a dish data set and plots contours of integrated intensity. These tools will be improved upon and used in conjunction with a single dish on-the-fly imaging tool to be developed next quarter.
In Synthesis support, in keeping with the rest of the package, a significant focus in synthesis this quarter has been the stabilization of current capabilities in preparation for the release. This has included extensive testing to isolate defects, and their subsequent correction. This has been a necessary and beneficial process in preparation for wider deployment of the system in the scientific community.
The context of the development that has taken place, however, remains the same as in previous quarters. The strategy remains that of developing advanced synthesis capabilities in key areas to test design completeness and sufficiency, while also developing a "thin-path" reduction sequence for connected-element interferometry. As previously noted, the resources for thin-path development have primarily been assigned at the expense of some VLBI development. This is expected to be reversed after the first release, in preparation for the second release which will contain greater VLBI support.
High-level synthesis development efforts have been concentrated primarily in the area of wide-field imaging (T. Cornwell), and in mosaicing, primary beam effects and general deconvolution (M. Holdaway and T. Cornwell). Kumar Golap, who arrived as a Visiting Scientist at the NRAO during this quarter, has provided significant testing of the wide-field imaging capability on an multi-processor SGI system in Socorro, in preparation for his future planned work in parallelization of these algorithms. This has included testing against simulated wide-field data, and the development of basic automated scripts to extract and test derived image component positions. He has also written a cookbook chapter on how to use wide-field imaging capabilities in AIPS++. The wide-field development work has included an application, called dragon, to replace and extend the capabilities previously available in SDE under this name. The NRL wide-field imaging group have agreed to assist in testing the new features using 74 MHz VLA data. The mosaicing and deconvolution work has included improved primary beam weighting, the migration of multi-resolution CLEAN deconvolution to the imager tool, and the implementation of a multi-field Hogbom CLEAN for comparison against the existing Clark CLEAN implementation. Various other changes and defect corrections of benefit to mosaicing have also been implemented. Time has also been spent on preparations for the Mosaicing Techniques meeting, planned for mid-July in Socorro, which is being arranged by M. Holdaway and M. Rupen, and is a collaborative effort between AIPS++ and the MMA project.
General defect correction and testing has continued for the imager and calibrater tools (T. Cornwell and A. Kemball), including some new features designed to improve ease of use. Synthesis has also benefited overall from the improved capabilities and features in the graphical user interface. The MS v2.0 definition has been stable in this quarter, and M. Wieringa has continued the propagation of changes required for this format revision throughout the affected classes in the library. It is planned to adopt the new format, which will allow new single-dish and synthesis reduction approaches, after the first release. User impact will be minimized in this process. There has been a significant contribution to the format specification from across the project as a whole, and we have benefited greatly from the diverse instrumental experience and opinions brought to bear in the discussion of this matter.
NFRA synthesis development has been focused on utilities required to support the commissioning of TMS at WSRT (J. Noordam and G. van Diepen). Development for WSRT will expand in the future with the hiring of new scientific staff to assist work in this area.
Thin-path development has continued during this quarter. R. Marson has further refined the VLA data filler, and expanded the supported scientific capabilities. It has been tested in the supported modes and is being compared to existing data fillers for the VLA. P. Barnes has continued work on uv-data visualization in the framework of the Display Library, in collaboration with D. Barnes, N. Killeen and A. Kemball. A basic pgplotter implementation for some of these capabilities has been completed in the interim.
The parallelization effort forms an important part of synthesis development, to address the largest, most computationally demanding applications. Work on a test mosaic dataset taken on M33 at the VLA have continued, as we build parallelized capabilities in this area.
Specialized design work in this quarter has included a holography requirements document (A. Kemball), which was submitted and accepted as part of the ALMA holography design review. Work on integrating single-dish and synthesis reduction in a common design framework has continued in the framework discussed in previous quarters, but at a slower pace this quarter.
In Glish, garbage collection was added as an optional feature to Glish. This collector cleans up memory which reference counting fails to delete. Typically this memory is allocated as part of records which are self-referencing through functions and other records.
In addition to bug fixes, some clean-up work was also done with Glish shared library creation to attach version numbers to the libraries.
In Parallelization, most of the second quarter Roberts has been working with the Pablo group at UIUC CS department to identify the IO bottlenecks in large image runs. We found a few files that did not have the proper instrumentation. Also in order to trace where IO was being carried out some event tracing needed to be introduced into the imager and pimager applications. The trace calls (IO and event) are checked into the code and can be turned on by an makedefs variable (PABLO_IO) We now have a stable version of pimager which we have been using for testing. We have been carrying out tests on dedicated machines and intend to work on a single machine for a long test in late July.
In Measures, no substantial changes were made.
In AIPS++ Infrastructure, little has happened beyond bug fixes and some minor integration improvements.
In Image Analysis, we added various new convolution capabilities, improved the interfaces for various parts of the image tool, and improved the performance of the image fitter.
In Documentation, we finished the revision of Getting Started in AIPS++, started on a cookbook, and implemented a FAQ.
In Management, we made demonstrations and presentations at the Chicago AAS, and at the Tucson meeting on Sub-Millimeter Imaging. We held discussions with the JCMT/ACSIS group on their requests for assistance. This amounts to 6-7 weeks of time spread over about 9 months.
In System, we directed nearly all of our effort to production of the initial CDROM. Some planning for the post-release phase has been completed. We expect to move towards using CVS for code management and distribution.
In June, our server tarzan was the subject of a breakin. This led to the machine being out of commission for about 6 days, during which code development at the sites was hindered. Following this attack, we have made a number of changes designed to improve the security of tarzan. The principal ones are that remote NFS is no longer permitted, and the secure shell must be used to connect to tarzan. The resulting changes in working habits for remote sites are acceptably small.
In the Quality Assurance Group, we embarked on a review of the compliance with various rules, resulting in, for example, closer adherence to our guidelines for writing Graphical User Interfaces.
The first release was originally planned for the Chicago AAS meeting in early July. As a result of the testing results, we realized that this was too aggressive and decided instead to opt for a round of pre-release testing. This pre-release testing will last for 6-8 weeks and will start with the distribution of a candidate release to consortium sites and beta testers. We expect the pre-release to happen this week and the actual release to be performed by the end of September.
The major task in preparing for the release has been the extensive testing of the package. This has been performed primarily by Project Staff and accounts for most of their work in this last quarter. Bugs reports (now known as defects) are managed using the ClearDDTS system that NRAO purchased for Socorro. This enables tracking of the history of each defect report, and relays all events to the submitter, and allows the submitter to verify that the defect has indeed been fixed. Many types of statistical reports can be generated, and we use such reports to help direct work. In particular, the current scoresheet of defects arrival and repair rates is circulated at the end of evey week. We can also look at the number of assigned defects per person and even out the load by delegating some repairs to other people.
At the time of writing, the defect arrival and repair rates per week were:
| Week | Date | # New | # Resolved | Diff | # Unresolved |
| 1 | 4-25-1999 | 9 | 0 | 9 | 9 |
| 2 | 5-2-1999 | 5 | 2 | 3 | 12 |
| 3 | 5-9-1999 | 3 | 0 | 3 | 15 |
| 4 | 5-16-1999 | 15 | 22 | -7 | 8 |
| 5 | 5-23-1999 | 35 | 13 | 22 | 30 |
| 6 | 5-30-1999 | 39 | 18 | 21 | 51 |
| 7 | 6-6-1999 | 41 | 11 | 30 | 81 |
| 8 | 6-13-1999 | 18 | 7 | 11 | 92 |
| 9 | 6-20-1999 | 21 | 11 | 10 | 102 |
| 10 | 6-27-1999 | 37 | 24 | 13 | 115 |
| 11 | 7-4-1999 | 60 | 35 | 25 | 140 |
| 12 | 7-11-1999 | 45 | 44 | 1 | 141 |
| 13 | 7-18-1999 | 18 | 27 | -9 | 132 |
| 14 | 7-25-1999 | 46 | 32 | 14 | 146 |
The above list includes all defects, ranging from severity 1 (catastrophic) to severity 5 (cosmetic). If we include the defects in the old GNATS system, in the history of the project we have processed about 1500 defects, of which about 10% are still open. Most of the unresolved defects are now of severity 4 and 5, and are relatively minor. For the severity 1 and 2 errors (the most serious), the rates are:
| Week | Date | # New | # Resolved | Diff | # Unresolved |
| 1 | 4-25-1999 | 6 | 0 | 6 | 6 |
| 2 | 5-2-1999 | 4 | 0 | 4 | 10 |
| 3 | 5-9-1999 | 3 | 0 | 3 | 13 |
| 4 | 5-16-1999 | 12 | 20 | -8 | 5 |
| 5 | 5-23-1999 | 19 | 10 | 9 | 14 |
| 6 | 5-30-1999 | 22 | 14 | 8 | 22 |
| 7 | 6-6-1999 | 26 | 10 | 16 | 38 |
| 8 | 6-13-1999 | 4 | 5 | -1 | 37 |
| 9 | 6-20-1999 | 6 | 7 | -1 | 36 |
| 10 | 6-27-1999 | 19 | 19 | 0 | 36 |
| 11 | 7-4-1999 | 28 | 21 | 7 | 43 |
| 12 | 7-11-1999 | 17 | 26 | -9 | 34 |
| 13 | 7-18-1999 | 5 | 10 | -5 | 29 |
| 14 | 7-25-1999 | 3 | 8 | -5 | 24 |
The decision to hold off on the planned late June release was based in part on these statistics and on a close perusal of the nature of the most severe defects.
The key remaining area of general concern identified in the last quarterly report was the robustness of the system to user error. The robustness issues have been addressed and a number of key bugs identified and fixed. The operational stability of Glish alone is excellent, and the stability of Glish plus the AIPS++ capabilities is now acceptable, although continuing improvement is still expected.
In documentation, we have made moderate progress:
Our major new general concern is the memory usage of the system. In a typical use of the system, the memory usage is between 110 and 120 MBytes, including about the components of Glish (about 40 Mbytes), and about 8 servers. This usage is about twice what we had projected and planned for. We have done a lot of testing and development to determine the cause of this extra usage. The use of shared libraries did not save much, and now we are experimenting with a mega-server approach. We expect that with some work like this, we can save between 30 and 60 MBytes, thus bringing the number down to a reasonable number. This work can be completed before the actual release but the pre-release will be affected by this excessive memory use.
The production of the CDROM master is now well in hand. In retrospect, it is clear that we underestimated the time needed to iron out the production difficulties. Testing on a variety of platforms takes time, especially when CDROMS must be FedEx'ed around the world. However, we have now have learnt how to make a distribution for each of Linux and Solaris that can be run from the CDROM or installed to disk and run from there. This knowledge should remain current for future releases.
Finally, we have started to prototype some of the operational systems needed. We have considered how to set up helpdesk capabilities in a what that will spread the load around and will not overwhelm the Project Center with the handling of issues that are better addressed locally. Handling of questions and defects is a priority for the whole Project. In the first instance, a defect or question will be handled by on a local and regional basis (i.e. North America, Europe, and Australia). Once a defect is determined to be a true error and not e.g. a configuration or user error, it is forwarded to the Project Center for allocation to a Project Staff member for repair. Similarly, questions beyond the competence of the regional center will be referred to the Project Center. All of these steps are handled using the NRAO installation of ClearDDTS.
There is an increasing interest on the part of various groups to collaborate with AIPS++. As reported last quarter, we expect MERLIN to apply formally to join the consortium. In addition, there is a possibility that a collaboration between MERLIN and two groups at the MPIfR-Bonn will acquire about 4 programmers to work with AIPS++. Currently we are collaborating informally with USNO/NPOI, and with JCMT.
From these examples, and other expressions of interest that we have heard, it is to be expected that the number of groups actively using AIPS++ will expand over the next few years. It is therefore important that the consortium plan any expansion carefully. The following questions will have to be answered:
Most work for the next quarter will be focused on the release. After the post-release, we expect to update our plans for the next 6 month period. Here we give details of work for which plans are already clear.
In Single Dish, work for quarter three will focus on imaging and calibration. A single dish tool capable of gridding on-the-fly data will be developed in July. An initial single dish calibration tool will be developed shortly thereafter. Work within dish will consist of rounding out the functionality to include multi-component fitting and to allow direct interaction with an aips++ MeasurementSet.
Support for GBT will expand to begin to provide near real-time display of GBT data. The purpose of these types of display is to give a quick look at the data so that data quality can be monitored. A second holography backend test is scheduled for July. The format for the spectrometer backend will be finalized and the classes written to allow the filler to fill GBT spectrometer data.
In Synthesis, we will retain the overall strategy outlined above. A particular focus will be on outreach to the scientific community, through internal workshops and external visits, and in ensuring that the first release is widely used and accepted by the scientific community. Greater emphasis will be placed on VLBI applications after the first release, in completing thin-path capabilities, such as uv-data visualization, and in finalizing and expanding automated data reduction scripts and utilities.
This quarter the ATNF had 3 people working in AIPS++. These are Neil Killeen (75%, also local manager), Mark Wieringa (30%) and David Barnes (75%). Wim Brouw continued on secondment to ATOMS and spent only about 20% of his time on aips++ this quarter. Wim will return at a higher level to aips++ in the next quarter.
Our systems (Solaris/Gnu, Solaris/egcs, and Digital Unix/egcs) were generally stable this quarter. We now have a running Linux/egcs system as well at Epping. We have some remaining file locking problems that we hope will be solved by an operating system upgrade.
Peter Barnes spent a few days here whilst on leave. He and David discussed the Display Library with which Peter is working.
Frederic Badia from ATNF/Narrabri spent a week working with Neil and David on protyping some ATCA online imaging components with aips++.
Two people are doing testing for us (as well as the aips++ group itself). They are working with dish, images and quanta/measures documentation.
David's time was spent on
Wim has spent his time this quarter on:
Neil's time went on:
This quarter Mark's time has been spent on
Harold Ravlin has put in some bug fixed into the PostScript driver for the Display Library as well as a few additional features. Harold added bounding box calculations for EPS, but these do not do the work the way they should, since the one of the first things the Display Library does is to clear the entire writable area. Harold did bug fixes on plotting dashdash lines. Harold looked into a request to be able to select that paper tray, however, this is a device dependent feature and could make a file difficult to print on other printers; Ravlin will look further into this after the release. The PostScript driver no longer appends a showpage when in EPS mode; it had been doing this as a debugging aid. Harold began looking into adding support for registering colormaps.
Aipsview will now print a warning message and exit if the default visual is not PseudoColor. Numerous changes were made to Aipsview so it will look for a PseudoColor visual if the default is something else. It appears to work as long as Aipsview is running on an SGI. There may be problems with Sun's implementation of Motif. The changes will be checked in after the release. Harold added missing copyrights and RCS information to some files.
Ravlin and Roberts spent time to work on various check in and check out problems and how the remote commands rai/rao/rau worked under secure shell.
Due to the pressures of the WSRT upgrade, the NFRA contribution to the global AIPS++ package has regrettably sunk to an absolute minimum. The only substantial contribution was made by Ger van Diepen in his role as Technical Leader, and as the man responsible for a number of essential infrastructure modules (tables, lattices, ms2uvfits, etc).
This quarter, the NFRA contribution in FTE's was as follows:
| Global AIPS++ | Local AIPS++ | Other | |
| Ger van Diepen: | 50% | 30% | 20% |
| Jan Noordam: | 10% | 60% | 30% |
The core NRAO AIPS++ group is now Barnes (100%), Braatz (100%), Cornwell (100%), Garwood (100%), Golap (100%), Holdaway(100%), Kemball(100%), Marson (100%), McMullin (100%), Schiebel (100%), Weatherall (50%), and Young (100%, Golap and Young are funded by NCSA-NRAO collaboration.
Completed initial implementation of Table/MS side of Display Library inheritance tree, concluding this stage of the DL work from the last quarter. This consisted of writing 8 classes within the tree: 4 DisplayDatas classes (TablePADD, TableAsXYPlot, MSAsXYPlot, MSAsTimeSeries) and the corresponding DisplayMethods classes; and a utility class, MSPlotter. This latter was a new class, and considerable effort was made to ensure adherence to the ideals of the PlotStyles design document, despite the fact that it is only a temporary class until the MultiWorldCanvasHolder is implemented. Most of the time was spent defining the proper parameter handling for the required plot functionality. Also wrote a test program for this class.
Began duties as AOC Solaris site manager. Checking sneeze logs, mostly, with other small duties as they arise.
Made extensive use of assay and imagetest, esp during May when Anantha was away. Most of these results were reported to aips2-lib. Began testing visplot, ms, & pgplotter, generating bug reports.
Attended Imaging '99 Conference in Tucson, June 6-9.
Jim Braatz was hired on to the AIPS++ project in April 1999. His primary responsibilities are to manage the Green Bank installation of AIPS++, to support GBT operations, and to work on single-dish applications. Specifically, Jim worked on the following tasks during this quarter:
In addition to his project management duties, Tim spent a large amount of time on managing, assigning and fixing defects. He worked with Kumar Golap on the provision of wide-field imaging capabilities within AIPS++, and with Mark Holdaway on Mosaicing, and Multi-scale clean.
Over the past 3 months (April through June) he has done the following:
In April, Kumar joined NRAO as a Visiting Scientist for 2 years. He has worked with Tim Cornwell on testing of the wide-field imaging capabilities within AIPS++. He is also responsible for testing of the SGI port of AIPS++.
Mark Holdaway has been working on mosaicing integration in the imager tool. The voltage pattern editor vpmanager, which allows the user to create a wide variety of voltage pattern and primary beam models, is complete. The table of voltage patterns it generates can be read by imager, which uses the voltage patterns when mosaicing. Default voltage patterns for each telescope and observing band can be used without accessing the vpmanager. (Presently, only data with a single voltage pattern can be correctly imaged with imager due to a current limitation in the visibility iterator.)
Mosaicing has required many minor changes to the SkyEquation/SkyModel design. One of the most important has been "Sault weighting". In Sault weighting, the noise does not flare up at the edge of the mosaic sensitivity pattern. However, under Sault weighting, the flux scale is not constant over the image, and inverse-Sault weighting must be applied for some operations.
It is very imortant for mosaicing observations to accurately image extended structure. While maximum entropy is generally the algorithm of choice for imaging extended structure, it's positivity bias often results in an overestimation of the flux, and its residuals are correlated with the true emission. The multi-scale CLEAN algorithm is the answer to this problem, succesfully imaging weak extended structure with noise-like residuals. Mark has integrated the multi-scale CLEAN algorithm into imager, including masking. (This is not checked into the system due to the "code freeze" prior to the public release.)
Another aspect of mosaicing integration is modifying the SkyEquation to perform minimum sized FFT's. When minimum-sized FFT's are in place, the cpu time required for a mosaic will vary approximately linearly with the number of pointings, a great improvement over the current code. Mark has done exploratory work on how to implement minimum-sized FFT's via the MFSkyEquation (multi-field sky equation) class.
On July 14-16, AIPS++ and the ALMA project cosponsored a Mosaicing Techniques Meeting in Socorro, New Mexico. Mark was the main organizer for this meeting, bringing about 30 researchers together for discussions on new mosaicing algorithms and the problems they are having with the existing algorithms.
Athol's work is described above, principally in the section on Synthesis development.
In the last quarter Ralph has worked on:
Joseph McMullin's main responsibilities are work on single dish applications within AIPS++.
Darrell performed the work on Glish described above.
In addition to the Glish development accomplished during the quarter, Darrell got shared library builds of AIPS++ libraries working, and he worked to get things in order for producing AIPS++ CDROMs.
[Preliminary]
In this section, we give the names of people in the various AIPS++ groups and the nominal fraction of time allocated to AIPS++.
The ATNF group is: Neil Killeen (75%, also local manager), Wim Brouw (20%), Mark Wieringa (30%) and David Barnes (75%).
The BIMA/NCSA group is: Harold Ravlin (20%), Doug Roberts (50%).
The NRAO group is: Tim Cornwell (100%), Jim Braatz (100%), Bob Garwood (100%), Kumar Golap(100%), Mark Holdaway (100%), Athol Kemball (100%), Ralph Marson (100%), Joe McMullin (100%), Darrell Schiebel (100%), Kate Weatherall (50%) and Wes Young (100%). Young and Golap are funded by NCSA-NRAO collaboration.
Thus, in aggregate, we have 18 people contributing about 13 FTEs to the AIPS++ Project. Of these, 9 are employed by NRAO (one funded by NCSA), and contribute 8.5 FTEs. The numbers for the other partners are: ATNF 4 and 2, BIMA/NCSA 2 and 0.55, NFRA 3 and 1.9. There are three unfilled positions, one at BIMA/NCSA (Briggs) and two at NRAO (GBT scientist, parallel applications developer: Kumar Golap due in April).
