 News |
FAQ
|
| Search
|
Home
|
| Getting Started | Documentation | Glish | Learn More | Programming | Contact Us |
|
| Version 1.9 Build 1367 |
|
T.J. Cornwell, NRAO
1998 April 7
Progress continues towards a beta release in May 1998 and a first release in late 1998.
In Single Dish support, we added the following to the Dish program: frequency switching support, regridding, and sequencing of multiple operations. We continued the formalization of the SDFITS standard, providing a number of test examples files. At Green Bank, we continued support of the various fillers needed by the GBT development team, and we also stated development of an RFI data base inside AIPS++ using Glish tools.
In Synthesis support, we added the following to the sky program: a graphical user interface based upon the ``auto-Gui'' (see below), simple Sault-style mosaicing, multiple field processing, imaging in any coordinate frame supported by the AIPS++ Measures system (such as RA/DEC, Az/El, Galactic, SuperGalactic, planet-based, etc.). Synthesis calibration will be carried out using a sibling of sky called cal. Currently this supports self-calibration of the form that was present in the old imager program but the intention is that this will soon be augmented by more sophisticated handling of cross-calibration. The visibility visualizer (mans) developed by Jan Noordam with C++ support from Mark Wieringa, has been released into the system. The speed has been much improved by changes to the underlying ms object. Support for various urgently-needed WSRT-specific capabilities, such as determining antenna delay offsets, phase zeroes, is also provided by mans. These will be shifted to the mainstream calibration software when possible.
We have circulated a draft specification for version 2 of the AIPS++ MeasurementSet. The intention is that this should support new scientific requirements arising from VLBI and single dish processing, and also rectify a few problems found with the existing format (e.g. the inadequacy of one pointing index to specify mosaiced data). We expect that this format will be implemented following the next beta release.
In Measures, we have improved the GUI for the measures object in Glish, adding applications of interest to end-users (e.g. rise/set times), pop-up help, and numerous clarifications. A test program for the measures object has been developed. The IGRF Earth magnetic field model is now available. UVW coordinate transformations to other coordinate systems are now supported by special UVWMachine.
In Glish support, we improved memory use, added floating help, supported private colormaps in tgpgplot, added regular expression handling, and did much of the implementation of file access. We also developed Glish-based matrix algebra classes. And as usual, we fixed a number of Glish bugs. In AIPS++ Infrastructure, we developed a Lattice Expression Language, allowing Lattices to be combined using complicated mathematical operations that are evaluated at run-time. We developed a number of important tools for the graphical user interface of AIPS++:
In addition, we have added the capability to augment objects with ``plug-in''s that add extra functionality, such as standard reductions. We expect that this will become widely used as a means for adding local customizations both by observatories and by astronomers.
We have continued our integration of Measures throughout the system by allowing Measures to be stored as a native type in AIPS++ Tables.
In Image Analysis and display, we have implemented an image calculator based upon the Lattice Expression Language. Image expressions are also now available via Aipsview so one may display a combination of a number of images without any temporary image ever being written to disk. Similarly, image expressions may also be formed from image objects in Glish.
The standard plotter, pgplotter, has been incorporated in the image object. Performance of some methods of the image object has been improved by the use of the revised LatticeApply classes.
Aipsview has continued to accumulate more capabilities: vector overlays, display of image expressions. Some debugging of the Display Library has been performed. We expect significant development of DL-based applications to start once David Barnes starts work at the ATNF this April.
In the System area, we have (almost) concluded the port of AIPS++ to the SGI Origin2000 (reports of its earlier conclusion were exaggerated). A few more tests remain to be passed.
In the area of Parallelization, we have started an effort to parallelize AIPS++ synthesis code, including gridding, FFTs and clean deconvolution. First preliminary results with the parallelized deconvolution show linear speed-up for up to 16 processors. The goal over the next few months will be to translate such speed-ups into working code in the sky object.
In Documentation, we have published the first two editions of the AIPS++ Newsletter:
http://aips2.nrao.edu/aips++/docs/newsletters/frontpage.html
We have started the construction of a user cookbook. The initial draft is being written by Michael Rupen and Anantharamaiah, and will be edited and placed in the documentation system by project staff.
In Management, we held the second meeting of the AIPS++ Scientific and Technical Advisory Group in Socorro on February 9 and 10. The STAG report and the Project response are to be found at:
http://aips2.nrao.edu/aips++/docs/notes/217/217.html
and
http://aips2.nrao.edu/aips++/docs/notes/218/218.html
While we agreed with the STAG over many of their recommendations, we believe that one major recommendation, the holding back a first release until there exists a true end-to-end data reduction path for telescope data, is is not wise. Instead, we are implementing a compromise whereby a complete path is available for data that have been initially calibrated elsewhere, and we will start work on a VLA filler as suggested.
Negotiation of an agreement between ATNF and BIMA/NCSA over the split of responsibilities in image analysis and visualization has concluded. The basic agreement is that ATNF will take the lead in this area, directing NCSA personnel on the completion of the basic Display Library, and writing the first applications. We expect that applications development based on the Display Library will later spread throughout the project.
We negotiated an agreement with JIVE whereby they will take over responsibility for the development of FITS-based archiving of MeasurementSets, something that is in their critical path. JIVE has been using AIPS++ for analysis of data from their new correlator.
A couple of presentations about AIPS++ have been given by project members: Cornwell and Glendenning at the NRAO Visiting Committee, Killeen and Wieringa at the joint ATNF/AAO symposium.
Our testing situation has improved considerably with the formation of a tester's group at the AOC. Other people in the project (e.g. Neil Killeen) have also contributed significantly. Testing of the development system at the AOC has been carried out by a number of local staff members: Anantharamaiah, Durga Bagri, Bob Hjellming, Michael Rupen, and Gustaaf van Moorsel. Members of the project (in particular, Cornwell, Glendenning, Kemball and Weatherall) meet weekly with the testers. Systematic testing of features has not yet commenced. Instead, this group has concentrated upon usability issues such as the GUI and the documentation. For example, there has been much discussion of the visibility of objects in the interface: is this acceptable? Is C++ jargon intruding where it shouldn't? Is the breakdown of functionality at the right level? Currently Anantha and Michael Rupen are writing the first draft of a cookbook that will address these issues, defining terminology and concepts that will then be retrofitted to the rest of the documentation to eliminate any inconsistencies. Neil Killeen is responsible for overseeing this part of the process. Development of the user documentation is the limiting item for the next beta release.
Following the STAG meeting, we have reconsidered our strategy for releases. The STAG had recommended that we defer any release until a full end-to-end data reduction path is available for various consortium telescopes. We have considered this carefully and discussed it quite widely with a number of interested parties. Our current plan is a compromise, whereby we will aim for end-to-end reduction of data that has initial calibration applied in another package. The intention is that after entry to AIPS++, it should be possible to go all the way through to publication quality plots. This then defines the contents of the fourth beta, which after testing will form the basis of the first release. One further recommendation of the STAG is that the developer community is sufficiently small that most interested developers can obtain the development system directly and so a release targeted towards developers is not high priority. We agree with this and have deferred the developers release accordingly.
The release schedule is now:
In Single Dish support, we will start beta testing of the completed Dish program. We will develop tools for the commissioning of the GBT.
In Synthesis support, we will place in service an improved cal object for calibration of synthesis data. We will continue to improve the capabilities of the sky imaging object and also develop a much simpler, demonstration function for straightforward imaging. We will start work on wide-field imaging (as needed by the VLA) and on mosaicing. We will start work on a VLA filler.
In Glish support, we will complete addition of file access and input/output formatting functions.
In AIPS++ Infrastructure, we will continue development of graphical user interfaces for AIPS++, adding widgets for various inputs, and providing a consolidated view of objects in Glish and files on disk.
In Image Analysis and display, we will work towards completion of the Display Library, adding 1D plotting, postscript output, binding to Glish, and various widgets. Applications development will start, probably with a simple image display tool to be used as an application service.
In the System area, we will re-arrange some personnel allocations to provide more support, probably at the level of 0.5FTE. This will enable us to fix a number of long-standing but relatively minor problems, as well as to consider the future of our code maintenance system.
In the area of Parallelization, we will work towards incorporating our parallelized code in the sky object, thus providing speed-ups in working code.
In Documentation, we will continue writing of a user-level cookbook, augmented by a scheme of ``recipes''. We will shift to publishing the AIPS++ Newsletter every other month.
This quarter the ATNF had 3 people working in AIPS++. These are Neil Killeen (25-75%, also local manager), Wim Brouw (90%), and Mark Wieringa (30%). David Barnes will join us in April 1998 (slipped two months from previous date).
Our systems (Solaris/Gnu and Digital Unix/Gnu) were stable this quarter and required close to zero maintenance. The Irix native port continues to be incomplete. Wes Young has been assisting NCSA and we believe the time is near when we can try an inhale again (although we will need to upgrade our compiler yet again).
Mark Wieringa visited NFRA briefly in January to discuss Jan Noordam's visibility visualizer development requirements. Bob Sault and Lister Staveley-Smith represented the ATNF at the STAG meeting held this quarter. Tom Oosterloo also went to the STAG on our behalf to present the Display Library. He also visited NCSA to talk with John Pixton about a range of DL issues during this trip.
In March, Neil, Wim and Mark attended the annual joint ATNF/AAO symposium, this year held at Mt. Stromlo. Here we did some talking about and demonstrating of aips++ to interested people. We showed the auto-GUI, sky, images, dish, measures GUI and PGPLotter. The reaction was generally favourable. However, in my opinion, people watching a (carefully selected) demonstration get a fairly poor flavour of what it is really like to reduce data with a system. Thus, I consider doing demonstrations largely as a (useful) public relations exercise, rather than providing useful feedback for us.
Coincident with this meeting, the AAO held its board meeting. One English member of the board expressed surprise that the ATNF was still involved in aips++, since it was his opinion that the aips++ project had folded several years ago. This emphasises the absence of any aips++ presence in the U.K.
Neil Killeen has been the only tester (not actually with the beta system but with the development system) this quarter. He has spent time using
One other person has promised beta testing time but it has not been forthcoming, despite continual guilt trips.
Most of Mark's aips++ time this quarter was again spent on the MeasurementSet DO functionality required for Jan Noordam's data visualizer. Mark's time was spent on:
Wim has spent his time this quarter on:
Vector<Quantity> in
transparent way.
Neil's time was spent on:
During the past quarter we had 6 people (3.5 FTE) working on AIPS++: Dick Crutcher (20%), John Pixton (50%), Harold Ravlin (100%), Doug Roberts (75%), Dan Briggs (100%), and Peter Teuben (5%). In addition, NCSA is funding the salary of Wes Young at NRAO-Socorro and will shortly fund the salary of a second NRAO person. Accomplishments during the past quarter are described below under each person.
Dick Crutcher served as local AIPS++ project manager and as BIMA director in charge of our participation in AIPS++.
John Pixton has focused on the AIPS++ display library. During the past quarter he completed the following tasks:
Harold Ravlin administers our AIPS++ workstations, which takes about 1/3 of his time. During the quarter we acquired 4 SGI Octane workstations with R10000 processors and 1 Sun Ultra30 workstation; these replaced Sun and SGI workstations that were typically 4-5 years old. The Octanes have the same processor and development system as the NCSA Origin2000, which will facilitate our development of AIPS++ for that system. He continued to spend some of his time on AIPSview development, maintenance, and bug fixes. AIPSview work included the following:
Doug Roberts finished a major update of the manual for the AIPSview AIPS++ visualization task. The manual update coincides with the functional update that Harold Ravlin has finished which allows a flexible command file to be used to draw vector overlays on a raster image. Doug has put together a test samples suite and worked with some Astronomy department testers to make a few example files that have been put into the appendix of the manual.
Wes Young and Doug Roberts have been working to verify the AIPS++ installation on the NCSA Origin2000 using standard AIPS++ runtests. Several problems with the AIPS++ code have been found by these tests. The version now passes most of the runtests and the skytest test suite in AIPS++. They have been continuing to collaborate with the NCSA Performance Engineering Group (PEG) on issues of performance on the Origin2000. Doug has been distributing the minutes of those meetings to the aips2-parallel exploder at NRAO. Wes Young was in Urbana for an NCSA parallel workshop in February. Doug, Dan, and Wes worked together during that time. We have continued to have bi-weekly phone meetings with Wes Young and Brian Glendenning at NRAO to keep each other informed of our progress.
Doug Roberts and Dan Briggs went to Socorro for the week of March 22-29 to discuss the parallel effort so far and to formulate the high-level plan for the next 6 months. The projects for the parallel group have been slightly changed since those decided at the end of last year. They now are directed at showing improvement within the existing "sky" task, rather than creating and testing other tasks. To that end, Wes is working on a spectral line deconvolution using PVM and MPI, Dan is working on a parallel gridding algorithm optimizing for parallelism using the native SGI compiler directive, and Doug is writing interface routines to the optimized fft functions in the SGI supplied SCSL (SGI/Cray Scientific Library). We are accumulating timing results that will be presented in a poster presentation at Alliance98 conference in Urbana in April.
Dan Briggs finished the long standing regions discussion document, though it was ultimately insufficiently practical and timely for project use; it has been superceded by the Killeen/van Diepen proposal. He worked on very basic shared memory parallelization of gridding in the SDE task uvmap, which was implemented as practice for similar implementation in AIPS++. Speedup was disappointing, leading to the conclusion that it was too fine a level for parallel distribution. Better speedup is expected for AIPS++, as more work is done per point compared to purely memory based algorithms, and the distribution level will be higher. He assisted with local SGI AIPS++ infrastructure and porting and continued work on parallel infrastructure within AIPS++. He investigated problems specific to C++ programming on the Origin 2000. (Large scale dynamic reuse of allocated memory creates difficulty in a distributed memory shared address space as found on the SGI Origin. AIPS++ may be the first project to be significantly affect by this.) He worked on the robust weighting paper which will be the only refereed reference for the technique.
Peter Teuben worked on completing the BIMA data filler in time for the next AIPS++ beta release.
Local project members: Ger van Diepen (GVD), Michael Haller (MXH), Jan Noordam (JEN, local manager), Friso Olnon (FMO).
Since NFRA has made itself dependent on it for the upgraded WSRT, AIPS++ has very much become a reality for more than just the local AIPS++ team. This is stimulating, to say the least. At the same time, JIVE is taking an increasing interest, and adds to the critical mass of local AIPS++ expertise.
The TMS and JIVE teams have been closely involved in the discussions about the new MS definition.
Three members of the NFRA AIPS++ team have paid a highly productive 2-week visit to Socorro.
Friso Olnon has transferred to from NFRA to JIVE, where he will not be involved in AIPS++ work. Ger van Diepen will be responsible for the maintenance of Friso's local archiving classes MS2FITS and FITS2MS, and for the wsrtfiller that reads data in the 'old' WSRT archive format. Harro Verkouter from JIVE will now implement MS2UVFITS, which is a high priority for JIVE.
This quarter, the NFRA contribution in FTE's was as follows:
| Global AIPS++ | Local AIPS++ | Other | |
| Ger van Diepen | 70% | 20% | 10% |
| Michael Haller | 80% | 20% | - |
| Jan Noordam | 30% | 40% | 30% |
| Friso Olnon | 20% | 60% | 20% |
Commissioning of the new WSRT backend (DZB) and on-line software (TMS) continued, and the entire system is slowly coming together. AIPS++ is beginning to play a larger role in this process: the 'visualiser' now has acceptable speed thanks to Mark Wieringa's ms DO which implements most of the data-access code in C++, and buying a fast Pentium II PC under Linux.
In addition, there are Glish applications for determining antenna delay offsets and phase zeroes etc before the observation. The next step, which has to be completed in June, is to implement some larger instrumental setup programs in AIPS++: the determination of antenna pointing errors and beamshape, precise antenna positions and deviations in the antenna mirror surfaces (holgraphy). The analysis of the problems, and part of the design, have been completed. The possibility of using these programs for other telescopes than the WSRT will be kept in mind, but is not a overriding priority.
Actual data reduction still takes place in other packages (NEWSTAR, AIPS, DIFMAP). The various data conversion programs are sufficient for the moment, but would have to be upgraded for more advanced observing modes. Therefore, processing in AIPS++ itself remains a high priority.
Ger van Diepen has overhauled the uv-data conversion program ms2scn, which converts an MS to a NEWSTAR SCN-file.
Friso Olnon introduced extra tiling control in the wsrtfiller for WSRT data archived in the 'old' format. This allows the study of the effect of various tiling schemes on the efficiency of data-access.
Rather than archiving the new WSRT/DZB uv-data as MeasurementSets, whose definition is not yet stable, they will be archived as binary-table FITS files, using the program ms2fits. They can be converted back to MeasurementSets by the program fits2ms for later reprocessing in AIPS++. Obviously the conversions must conserve all information in the datasets, and do so for a variety of MeasurementSets. That is, the conversions can handle standard MeasurementSets as well as those extended with any type of optional and user- and site-specific subtables and columns in those tables. Friso Olnon has finished these two programs, and the conversions have been successfully tested on MeasurementSets built by the wsrtfiller and by the present DZB fillers.
Michael Haller has finished TableMeasures. He also made a start with the proper implementation of the existing Glish prototype `xyftool', a general-purpose interactive data visualisation and manipulation tool. The second generation will be more ergonomic, and it will make use of the new gui utilities in AIPS++.
Jan Noordam has checked in his interim WSRT uv-data visualiser. It can be used for data of other telescopes too, but only for a maximum of 16 antennas, which have to be selected by the WSRT antenna names. This will of course be made more generic before the third beta.
Jan Noordam has checked in a collection of Glish matrix operations, which include the Direct Product and various inversion methods. The latter are the heart of another set of Glish routines (lsfit) for solving sets of weighted linear equations. The latter can be fed to the solver one by one in a program loop, and intermediate solutions are possible. It is also possible to use the same solution matrix for different 'driving vectors'. The Glish-implementation is fast enough for applications that do not have to process large amounts of data, and that involve matrices smaller than about 20x20. Eventually, the routines will have to be implemented in C++.
The rest of the items in this section are by Ger van Diepen.
A parser has been written in flex/bison to parse a string containing a lattice expression. Together with the ImageExpr class it makes it possible to construct an image object (also in glish) containing an expression of images/lattices. The function 'iif' has been added to the lattice expressions to make things like clipping (and far more complex conditionals) easily possible (see example further below).
The glish script substitute.g makes it possible to substitute glish variables or expressions in a string. A variable can also be a glish object like an image or table object. It is used in TaQL and the image/lattice expressions, so one can use a glish object directly in such expressions. E.g. to set every element of an image less than 5 to 5:
im := image("name");
im1 := expr("iif($im<5, $im, 5)");
Brian succeeded in adapting aipsview to show the result of an image
expression.
The LatticeApply interface has been enhanced to make the allocation and handling of the accumulators more flexible. Neil KIlleen has used it successfully in his ImageStatistics. Preliminary results indicate a significant improvement in performance and memory utilization compared to the old implementation where LatticeApply was not used.
At the end of the quarter the Lattices/Regions proposal has been adopted and most of it has been implemented and checked into the system. Several lattice-based LCRegion classes have been implemented (box, ellipsoid, polygon, mask). Also several region collections have been implemented (union, intersection, difference, complement, extension). The base class WCRegion can be used for regions in world coordinates, but so far no concrete classes have been implemented. The abstract base class MaskedLattice and concrete classes SubLattice and SubImage have been implemented to make use of regions in lattices and images possible. Application-oriented code (like ImageStatistics, lattice expressions) has to be changed to MaskedLattice to make full use of regions possible.
Little work has been done on the Table system. A few problems have been fixed. The most notable addition is the possibility to register a callback function when scratch tables are created/deleted. It makes it possible for the Tasking environment to delete scratch tables when a task ends abnormally.
Regular maintenance of the local AIPS++ site has cost Michael Haller less than half a day per week. He now also installs AIPS++ routinely on the Linux data inspection machine in Westerbork.
The core NRAO AIPS++ group is now Cornwell (100%), Garwood (100%), Glendenning (100%), Kemball(100%), Marson (100%), McMullin (100%), Schiebel (100%), Uphoff (100%), Weatherall (50%), and Young (100%). Peter Barnes will join us in May 1998 to work in the synthesis initiative.
Tim Cornwell's primary responsibility is Project management. His coding activities focus around the synthesis package. In the last quarter, he has:
Bob Garwood's primary responsibility is to oversee the single dish work which currently is focussed on the DISH environment as well as supporting the GBT. Over the past 3 months (January through March) he has done the following work:
Brian Glendenning works on general AIPS++ development and management in a large number of areas. In the past quarter he has:
Ralph Marson works on synthesis code and as code cop. During the last quarter, he has worked almost exclusively on the ComponentModels module. In particular he has:
As Code Cop, he has:
Joseph McMullin's main responsibilities are managing the Green Bank installation of AIPS++ and supporting AIPS++-related GBT needs.
During the quarter, Darrell Schiebel continued Glish development.
Jeff Uphoff works on single dish processing and also the AIPS++ system. In the last quarter, he did:
Kate Weatherall works on documentation for AIPS++. She has:
Wes Young has worked in a number of different areas. Over the last quarter he:
In Single Dish, we did add a number of features to the dish program in preparation for testing:
In Synthesis, we did develop mosaicing within the sky application, add sky support for components. We did not complete an image geometry conversion machine that is needed for wide-field imaging. We did continue to improve the user interface, and make refinements as suggested by our alpha-testers. We have not yet produced an initial cross-calibration capability, according to the design put forward by Kemball.
We did provide support, as needed, for the effort now underway to get WSRT/TMS Setup functions implemented in AIPS++. This will be a two-pronged effort: some functions will be implemented soon using the basic tools available in Glish, and some will be later implemented using the cross-calibration framework now being worked on by Kemball. We did work successfully to improve the performance of the Glish-based visualizer developed by Noordam and Wieringa.
In Image Analysis and Display, we did develop an image calculator based on the new LatticeExpr classes. We did not substantially increase development of image display applications using the new image display library.
In Infrastructure, we did complete the auto-GUI and but not the parameter-setting shells. We did not yet finish implementation a state-saving API for Glish functions. We did add support for region specification to the Lattice and Image classes. We did not develop an image coordinate conversion machine as needed for wide-field imaging.
In Measures, we did not implement user-specified corrections to emphemerides, as needed, for example, for comet observations. We did continue the integration of Measures throughout the system, most particularly via the TableMeasures classes and in the Glish binding of Distributed Objects.
In Glish support, we made substantial progress on but did not yet complete the addition of support for string and file handling.
In the System area, we did not implement a way of tagging good builds of the development system for preservation for use by alpha testers. We did not enable checkin and checkout of Tables. We did not investigate ways of restructuring the unix directory layout of AIPS++.
In the area of Parallelization, we did start work towards parallel deconvolution and visibility gridding applications but not yet gaussian fitting.
In Documentation, we did release a first edition of the AIPS++ Newsletter. We started on but did not finish restructuring the AIPS++ Web site, with the goals of making navigation easier and improving the presentation of information thus found.
In Management, we did hold the second meeting of the Scientific and Technical Advisory Group, scheduled for February 9-10 in Socorro.
We did not release the third beta release, scheduled for March.
In this section, we give the names of people in the various AIPS++ groups and the nominal fraction of time allocated to AIPS++.
The ATNF group is: Neil Killeen (25-75%), Wim Brouw (90%), and Mark Wieringa (30%). David Barnes will join the group in mid April.
The BIMA/NCSA group is: Dick Crutcher (20%), Dan Briggs (100%), John Pixton (50%), Harold Ravlin (100%), Doug Roberts (75%), and Peter Teuben (5%).
The NFRA group is: Ger van Diepen (100%), Michael Haller (100%), Jan Noordam (100%), and Friso Olnon (100%).
The NRAO group is: Tim Cornwell (100%), Bob Garwood (90%), Brian Glendenning (100%), Athol Kemball (100%), Ralph Marson (100%), Joe McMullin (100%), Darrell Schiebel (100%), Jeff Uphoff (100%), Kate Weatherall (50%) and Wes Young (100%).
Thus, in aggregate, we have 24 people contributing 18.6 FTEs to the AIPS++ Project. Of these, 10 are employed by NRAO, and contribute 9.4 FTEs. The numbers for the other partners are: ATNF 3 and 1.7, BIMA/NCSA 6 and 3.5, NFRA 4 and 4.
