 News |
FAQ
|
| Search
|
Home
|
| Getting Started | Documentation | Glish | Learn More | Programming | Contact Us |
|
| Version 1.9 Build 1367 |
|
T.J. Cornwell, NRAO
1997 July 15
In the second quarter of 1997, we continued working in a wide variety of different areas. The main thrust was to prepare for future beta releases by, for example, fixing bugs, improving the performance of the synthesis code, and working towards a general graphical user interface. In addition, we continued development of the single dish package. We also gave a number of presentations about AIPS++ in various meetings.
In Single Dish support, we continued the development of the Single Dish analysis program. This work, directed by Harvey Liszt, is aimed at creating a graphical user interface for quick interactive processing of single dish spectra and other one-dimensional or quasi-one-dimensional data sets. The current version of the program allows selection, averaging, baseline fitting, smoothing and application of a function to the data. It is tightly integrated with a plotter based upon the PGPLOT library. Browsing of records is well-supported, via either textual or graphical display. The Measures system is utilized to allow the user to choose the units and coordinates in the displays. A demonstration Glish script for the Single Dish analysis program has been checked in (as sddemo.g).
The Parkes Multibeam project is now observing using software based in AIPS++. There has been no further substantial work from the AIPS++ side to support this work.
In Synthesis support, as planned, we have re-written the gridding and FFT code to improve the performance for both spectral line and large continuum processing. Gridding is now performed using a novel sort-less algorithm. In this approach, the gridded plane is divided into small patches, a cache of which is maintained in memory using a general-purpose LatticeCache class. As the (time-sorted) visibility data move around slowly in the Fourier plane, patches are swapped in and out as necessary. Thus, optimally, one would keep at least one patch per baseline. The FFT classes were also rewritten to perform multi-dimensional disk-based transforms. Since the AIPS++ Table system has excellent support for efficient tiled access of large, multi-dimensional datasets, this FFT code was extremely straightforward to write. Optimization of both gridding and FFT support required some tuning changes to the basic underlying libraries, such as the choice of tile shapes for Images, and modifications to the Table locking strategy.
We also changed the synthesis code by adding support for off-line Doppler tracking, by improving speed of access to the MeasurementSet by avoiding unnecessary re-formatting, and by simplifying the way that model and calibrated data are handled in the MeasurementSet.
In April, we held a mini-review of the state of the synthesis code, specifically of the capabilities of the current implementation of the Measurement Equation. An alternative implementation, the so-called ``MEGI Number Servers'', had been prototyped by Noordam and van Diepen. Although this has some attractions for particular types of processing, we decided to stay with the current implementation until a deficiency is demonstrated. In a discussion of the applications areas that the synthesis code will have to support, no deficiency was discovered. We plan to continue this review process over the next few months with the help of Bob Sault. In addition, Brian Glendenning will conduct a review of the C++ design of the classes.
In Measures, we have spent considerable effort considering how the data tables needed for the Measures system are to be distributed and updated. This initiative has been spear-headed by Wim Brouw. The implications for AIPS++ are quite large and a sufficiently-detailed proposal has taken much discussion. We expect to implement this over the next few months. AIPS++ will then contain mechanisms for delivering not only data needed for internal testing, but data needed for a multitude of purposes.
In Glish support, our main activity was to respond to bug reports. This generated a number of substantial changes. The transport layer in Glish was re-written to correct a number of deficiencies. Trapping of floating point signals was introduced, with considerable difficulty on Dec Alpha machines. A long-standing but evidently seldom-triggered event blocking bug was fixed. We also completed support for the binding of the PGPLOT widget (kindly provided by Martin Shepherd of Caltech) to Glish. In addition, a number of minor Glish language issues were addressed (e.g. conversion of strings to booleans). In AIPS++ Infrastructure, our work continues to be driven by the needs of applications development. Thus in the Table system, we added a moderate number of changes as needed by applications development. For example, as we had known would be necessary, we added Table locking to allow access to Tables by multiple processes (e.g. one writer and multiple readers). The Distributed Object system has been revised to allow support for locking. Another example of a change required for applications development is the additional of caching to the Incremental Storage Manager, thus considerably speeding access to Table columns stored using this manager.
We also continue to develop a graphical user interface for standard AIPS++ Distributed Objects, based upon the tk widgets now bound to Glish. The prototype allows interaction with the Image Distributed Object that provides access to and manipulation of AIPS++ Images from Glish. An additional component of the interface is a general purpose plotter, based upon PGPLOT, that can be used from both Glish and C++ programs. This plotter has a display list that can be edited interactively by the user, thus allowing great flexibility in the plots that can be generated. These GUIs are based upon a flexible record-based GUI factory that we expect to be useful in other contexts.
We have developed a system for invoking AIPS functionality from within AIPS++. This is based upon a quite general client-server mechanism that has a number of possible applications, but currently we can use it to start AIPS tasks. The record-based GUI described in the previous paragraph has been used to provide a GUI interface to AIPS tasks.
In Visualization and Image Analysis, we continued development of the image display library, a joint undertaking of ATNF (Tom Oosterloo) and BIMA/NCSA (John Pixton). We expect the first applications using the library to be developed in the next quarter. More information on this work can be found in the ATNF contribution below.
We have continued performing bug fixes to aipsview, which we intend to remain as the workhorse image display application for some time. We have also added a number of new capabilities, such as the ability to annotate curvi-linear coordinates. In addition, the User Manual has been revised and improved.
In the System area, we started (and are closing to finishing) a port to the IRIX native compiler, including optimizations for parallelization.
We also started an evaluation of the Kuck and Associates C++ compiler. This includes a number of optimizations not present in other C++ compilers and may improve the performance of our code. If this is successful, we plan to use this compiler for generating binaries for distribution.
In Documentation, we continued adding material to the Reference manual and conducted a review of the documentation therein.
In Management, this last quarter was an especially busy time for meetings at meetings. In early April, we held a number of very useful intra-Project discussions in Socorro. These covered a number of topics that are hard to discuss other than in person.
The following presentations to external groups were made:
As described in the BIMA/NCSA contribution, Tim Cornwell and Gareth Hunt attended a meeting of the newly-reorganized (and re-funded) NCSA (shortly to be renamed the National Computational Sciences Alliance), NRAO is a strategic partner in the Scientific Instruments Applications Team. This brings extra personnel to the AIPS++ Project (since we plan to hire a person to work on parallelization), a high-speed link to NCSA from NRAO, and access to the considerable knowledge and facilities of the NCSA.
Tim Cornwell and Gustaaf van Moorsel have made progress towards an Operations plan for AIPS++: Note 205.
Progress on the next beta release has been less than anticipated. This delay is largely due to personnel overloading. Currently only a couple of project members are sufficiently knowledgable about the synthesis code to be able to make the changes to the gridding and FFT routines that were required, and both of these people (Tim Cornwell and Mark Wieringa) had conflicting priorities. We plan to address this particular bottleneck by allocating Wim Brouw and Athol Kemball to work on synthesis development. In principle, the new hire in Socorro will also be available for synthesis development but we have, as yet, had no success in finding someone suitable for this position.
As a result, the next beta release has been delayed from May to July. The time-line is now as follows:
Comments on the release continue to come in. Some more organizations have joined as beta test sites: NRL has delivered very useful comments, and MPIfR has just started to use the system.
The principal activity for the next quarter will be support of beta testing, and preparation of the next release.
In Single Dish support, we will continue development of the Single Dish analysis program, with the goal of making this part of the third beta release.
In Synthesis support, we will continue development of the synthesis code, starting the design of a fully-fledged calibration system using the current Measurement Equation implementation. We will finish a visibility data visualizer being developed by Noordam. We will add the various capabilities needed for dealing with list of discrete components, as needed for WSRT calibration.
In Measures, we will implement the data distribution scheme now under discussion. We will incorporate JPL planetary data in the Measures system. We will add support for Measures in Tables.
In Glish support, we will continue to fix bugs as they occur in testing. We will issue release 2.6 of Glish.
In AIPS++ Infrastructure, we will continue to track the needs of applications programmers. We plan a number of revisions to the Lattice, Image, and Table classes. We will improve the table browser.
In Visualization and Image Analysis, we will continue development of the image display library. We expect to have a complete design and description, an implementation of the major components, and some example applications.
In the System area, we will complete the port to the KAI compiler. We will simplify and consolidate the disposition of Glish script files in the code management system.
In Documentation, we expect to make progress on the intermediate level documentation requested by our beta testers (cookbooks and migration guides).
In Management, we will demonstrate AIPS++ at the ADASS in Germany in September and conduct an ADASS Birds-Of-a-Feather session on AIPS and AIPS++. We will give talks on AIPS++ at Jodrell Bank and MPIfR, Bonn. We will proceed with the planned name change. We will complete the operations plan.
People
The ATNF currently has 4 people working in AIPS++. These are Neil Killeen (25-75%, also local manager), Wim Brouw (90%), Tom Oosterloo (75%) and Mark Wieringa (30%). Additionally, there is a collaboration between the AIPS++ project and the ATNF headed Parkes 21cm multibeam receiver project which is contributing substantial effort to using and developing code within the AIPS++ environment.
As noted in the previous report, Tom Oosterloo will leave the project to return to Italy. He will in fact be in Italy for all of July, and then return to the ATNF in the second week of August and remain until the end of September, before finally departing. His position has been readvertised, and we are currently in the selection process.
System
Visits
In mid April, Neil Killeen, Tom Oosterloo and Mark Wieringa visited the AOC in Socorro. In addition, Jan Noordam and Ger van Diepen (NFRA) and John Pixton (NCSA) attended this meeting. We discussed a variety of issues ranging from implementation of the Measurement Equation, the display library and image analysis implementations. We all found this meeting very valuable and I can only recommend that we try to do it more often. The ratio of time that it takes to negotiate complex issues or solve complex problems when undertaken with email compared with face to face is close to infinity. We particularly enjoyed the Tequila at Tim's party...
John Pixton has been visiting the ATNF for all of June to work on the display library design and implementation with Tom Oosterloo.
Display Library
It is probably useful to summarize the status of the display library that Tom Oosterloo and John Pixton are working on at this point since they will both depart the project in the next few months.
The design is now essentially fully fleshed out with no remaining ``perceived'' hard problems to solve. Implementation of the library from the pixel canvas level and below is essentially complete (with an X11 implementation). Work is now focussing on the higher level classes above the pixel canvas notably the world canvas, world canvas holder and display data classes (the pixel canvas is usable now if you don't care about the handling of coordinates; this is what the world canvas looks after). The world canvas should be essentially finished within the next 2-3 weeks (as of end June). Following that, as a test program, a colourmap editor using the world canvas will be made by John who will then work on coordinate labelling routines. This will take us the end of July and early August. On Tom's return from Italy in August Tom will continue working on the derived display data classes (currently raster display done, next is contours and then probably a volume renderer) plus the zoom and position handler.
The major areas that have not yet received substantial attention are the widget user interface and the integration with Glish and Glish/Tk (to enable users to call display library routines directly from Glish and to enable display library canvasses to be embedded in Glish/Tk widgets). The widget user interface is basically straightforward. The Glish/Tk integration is likely to be harder.
By the time Tom and John depart, we should have a) an essentially complete design and description of the design, b) an implementation of the major components, c) some example applications. We will need to assess how far along the widgets and Glish integration will be in mid August.
Individuals
Wim Brouw's responsibility is mainly to designing and
implementing the Measure and related classes. His time has been spent
as follows.
Tom Oosterloo is working on the display library. Tom's time
has been spent on
Mark Wieringa works with Tim Cornwell implementing the Measurement
equation. Mark's time has been spent on
Neil Killeen spends his AIPS++ time attending to local ATNF AIPS++ management
issues as well as generating image applications. He spent his time on
In May Tim Cornwell and Gareth Hunt of NRAO visited NCSA for an "all hands" meeting to plan activities for the National Computational Sciences Alliance, which the National Center for Supercomputing Applications will become in October as a result of being one of the two continuing NSF supercomputer centers. NRAO is one of the strategic partners with NCSA in the Scientific Instruments Applications Team. As a result, we expect NRAO to receive about $250,000 per year in funding from the NCSA NSF grant for cooperative work on advanced scientific computing. This will fund two people at NRAO for AIPS++ supercomputer development work and for distributed computing use of AIPS++, as well as high-speed network connection between Socorro, NM and Urbana, IL. Additionally, we expect Dan Briggs (an expert in synthesis-imaging algorithm development and a former PhD student of Tim Cornwells) to join our group in Urbana in August for AIPS++ algorithm development work.
We currently have 5 people (3.25 full-time equivalents) working on AIPS++: Dick Crutcher (25%), John Pixton (100%), Harold Ravlin (100%), Doug Roberts (75%), and Peter Teuben (25%). Yanti Miao finished her M.S. degree in computer science in May and has accepted a postion in private industry. Taisheng Ye has shifted to working on the BIMA real-time data transfer and archive system. AIPS++ accomplishments during the past quarter are described below under each person.
Dick Crutcher served as local AIPS++ project manager and as BIMA director in charge of our participation in AIPS++. In addition to usual duties, he attended a workshop in May at the Owens Valley Radio Observatory where NRAO, Caltech, and BIMA representatives discussed joint development work on the MMA, including AIPS++ software work.
Yanti Miao finished her work on getting all pgplot functions to be callable from tkglish, documenting this, and translating pgplot demo programs into glish scripts to make examples of using TkPGPLOT and to test TkPGPLOT.
John Pixton has focused primarily on the Display Library. The PixelCanvas colormap system has been designed and implemented. This provides all client applications with the ability to change the size of the colortable at runtime. An OpenGL-style caching mechanism was designed and implemented to give the client application control over what is cached while hiding details of the caching system. Event-handling system was modified to use classes to allow a client to introduce extra data or functions required to handle the passed event. The first PixelCanvas client, the WorldCanvas, has been designed and is roughly a quarter of the way completed. He also spent some time with gplot1d, aipsview, and AIPS++ management. He will complete a 5 week visit to the ATNF in early July, during which he worked with Tom Osterloo and Neil Killeen on the visualization toolkit.
Harold Ravlin administers our AIPS++ workstations, which takes about 1/3 of his time. Primarily, he has worked on Aipsview. Much work was on fixing bugs reported by beta testers. He also added curvilinear axis drawing, relative indexing for animation, printing of simple statistics and made changes so Aipsview will automatically convert from m/s to km/s in image and profile windows. In preparation for the next stand alone release, he updated the Aipsview web page and ftp site.
Doug Roberts has focused on porting the AIPS++ code to the SGI Origin2000. The first port was with the GNU compiler system which is the standard for the AIPS++ package. In order to optimize performance, he has moved to using the SGI native CC compiler, which has parallelization optimizations. He first completed the port with the AIPS++ beta release of March, and has just completed the port of the current state of AIPS++, in anticipation of the spectral-line beta release of AIPS++ at the beginning of July. He is now investigating changes that might be made for optimization and parallelization on SGI parallel-processor supercomputers. He visited Australia in May 5-19 for observing with the Australia Telescope Compact Array and meeting with AIPS++ programmers. While there, he met with Tom Osterloo and Neil Killeen discussing the status of the AIPS++ visualization toolkit and the future of AIPS++ visualization software, especially end user applications. He has also updated the users manual for Aipsview, the AIPS++ visualization software system.
Local project members: Ger van Diepen (GVD), Michael Haller (MH), Jan Noordam (JEN, local manager), Friso Olnon (FMO), Henk Vosmeijer (HJV).
Jan Noordam and Ger van Diepen have visited Socorro in April, to take part in discussions about image analysis and synthesis processing.
The NFRA AIPS++ group has hired an extra AIPS++ application programmer: Michael Haller (who is an Australian living in NL) will join the group on July 1st, for a period of 2-3 years. He is an experienced programmer, but is new to aperture synthesis.
The commissioning of the first module of the new DZB correlator will now certainly happen in July/August. Thus, the timing of the beta-2 release at the end of June is excellent. Robert Braun will again lead the NFRA beta-testing effort.
Friso Olnon continued his work on the dzbfiller, a program which creates AIPS++ MeasurementSets from "header" data gotten from the TMS (Telescope Management System) database and "actual" data from the new digital backend, DZB. The first realistic MeasurementSets have been made with dummy data (but obtained throught the proper interface layer). Although the dzbfiller is a separate program, it still shares a number of classes with the wsrtfiller (meant for handling WSRT archive data). Some of these are changed as a result of the DZB work; hence the small amount of work for the wsrtfiller. A significant fraction of the filler-development time goes into keeping the various local test and development programs working. These programs depend on two systems-in-development and are therefore even more sensitive to problems, not in the last place the awfully time consuming building process.
``Regular'' maintenance of the local AIPS++ system still costs Friso Olnon more than one day a week. In addition, Ger van Diepen and Henk Vosmeijer spend some time on it. We suspect that this load will not decrease to the targeted 1 day-a-week or less.
The inhale and sneeze operations in the weekends are unsuccessfull half the time. Although the causes of failure are usually minor glitches that are easily diagnosed and repaired, it takes about all of Monday to rebuild the system. The system obviously is very sensitive to such small problems. A nasty side effect of the failures is that the HP machine, which we share with the TMS group, is heavily loaded with a task that we promised to run outside normal working hours.
The decision to add local support for the linux version of AIPS++ (our third system branch) may be a good one in view of the load on our current AIPS++ host machines, but it also means more maintenance. Jan Noordam also uses Linux and Glish on his portable.
In the beginning of April 2 weeks have been spent in Socorro. It was a fruitful visit with lots of useful discussions about tables, images, and related topics like tiling.
The locking/synchronization mechanism in the Table system has been finalized and checked into the system. Recently it appeared that the default AutoLocking mode can generate quite some overhead. A few solutions are being considered. The limitations on changing the shapes of indirect arrays have been removed. When the data manager can handle it, one can now change the shape. Almost all storage managers can handle it.
Some work has been done on the Lattice classes. Subsectioning has been added to the TiledStepper class. The TiledStMan algorithm to calculate a suitable tile shape has been improved to try to find exactly fitting tile shapes.
A proposal has been made for support of measures in tables. It will soon be implemented by Michael Haller (a new NFRA hire).
Some work has been done on demonstrating a very basic Megi Number Server. The tests were encouraging.
The core NRAO AIPS++ group is now Cornwell (100%), Garwood (90%, 10% going to Unipops support), Glendenning (100%), Kemball(100%), Marson (100%), McMullin (100%), Schiebel (100%) and Uphoff (100%). We are very pleased that after some time working with us, Athol Kemball has joined the AIPS++ group. In addition, Young of the AOC Computer Group works with us, and as described above, Murphy of the CV computer group also contributes. We continue to have one unfilled position, in Socorro, for a VLBI software specialist. Attempting to hire into this position has been an extremely frustrating exercise. In addition to these full-time AIPS++ staff, a number of scientists participate at various levels: Rick Fisher, Bob Hjellming, Harvey Liszt, Michael Rupen. In a recent development, Michael will be devoting about 50% of his time to helping AIPS++ in the areas of testing, documentation, and advising on scientific priorities.
Tim Cornwell works on documentation, the synthesis code, and general project management. Bob Garwood directs the single dish effort in Charlottesville and Green Bank. Brian Glendenning is the prime architect of many different parts of the system, acts as pinch-hitter in helping out in various areas, oversees development of the library code, and work on general project management. Athol Kemball is in charge of the synthesis calibration development. Ralph Marson works on the synthesis code, and is the AIPS++ Code Cop. Joe McMullin provides AIPS++ single dish support for the GBT. Darrell Schiebel is our Glish guru and is responsible for maintaining, adding new features to, and removing bugs from Glish. Jeff Uphoff works on the GUI side of the single dish package. Pat Murphy, Darrell Schiebel and Jeff Uphoff all work on system support. Wes Young works on documentation, GUI development, and VLA support via AIPS++.
We have a variety of platforms available to run AIPS++ on:
Thus NRAO has examples of all architectures for which AIPS++ is currently targeted, although the HP and SGI installations have yet to be made.
Most of the NRAO staff work on general AIPS++ capabilities. The exceptions are Joe McMullin, who although he is an AIPS++ Project member, concentrates on Green Bank and GBT support concerns, and Wes Young, who is a member of the AOC Computer Division, and thus spends a substantial (and increasing) amount of his time on VLA-specific developments.
Joe joined us in March and so his first quarter has been spent learning his way around AIPS++ and GB activities. He has worked on providing documentation of AIPS++ capabilities for GB staff, and on bringing some of the Glish scripts that have been written at Green Bank into the AIPS++ system.
Wes has worked on a number of VLA specific capabilities in AIPS++, the most recent of which is a filler from VLA archive tapes into AIPS++ Tables. When completed, this will give VLA engineers the capability to examine VLA monitor data using AIPS++ in much the same way that GBT engineers can examine GBT monitor data. From there, it is a relatively short step to fill into a fully-fledged MeasurementSet, and thus give a path for visibility data directly from the Modcomps into AIPS++.
The principal activity for the this last quarter were planned to be support of beta testing, and preparation of the next release.
In Single Dish support, we did continue the development of SDCalc.
In Synthesis support, we did continue with improving memory usage and speed for both continuum and spectral line processing. We did not substantially improve the plotting and editing capabilities. We did not add a GUI (but instead choose to do GUI development work first on the Image Distributed Object). We started to add fitting of Component Models to represent the sky brightness.
In Measures, we did start to develop methods for distributing time-dependent data inside AIPS++.
In Glish support, we did not release version 2.6. We did add more tk widgets as required.
In AIPS++ Infrastructure, we did deploy table locking. We did review the capabilities of the current Lattice/PagedArray/PagedImage classes. We started but did not finish development a tool for image-based mathematics. We did continue to augment the services (e.g. plotting, display) available to applications writers via the Distributed Object mechanism.
In Visualization and Image Analysis, we did continue development of the visualization tool-kit. We did finish the binding of PGPLOT to Glish, and we did complete addition labelling of curvilinear coordinates to Aipsview.
In the System area, we had no substantial plans beyond consolidating various ports.
In Documentation, we did not yet improve the introductory material as suggested by beta testers. This is an area where we would particularly benefit from the close involvement of astronomers.
In Management, we did enlist the help of someone with experience in operational support of a existing large package to help define how operations of AIPS++ will occur: Gustaaf van Moorsel.
In this section, we give the names of people in the various AIPS++ groups and the nominal fraction of time allocated to AIPS++.
The ATNF group is: Neil Killeen (25-75%), Wim Brouw (90%), Tom Oosterloo (75%) and Mark Wieringa (30%).
The BIMA/NCSA group is: Dick Crutcher (25%), John Pixton (100%), Harold Ravlin (100%), Doug Roberts (75%), and Peter Teuben (25%).
The NFRA group is: Ger van Diepen (100%), Michael Haller (100%), Jan Noordam (25%), Friso Olnon (50%), and Henk Vosmeijer (50%).
The NRAO group is: Tim Cornwell (100%), Bob Garwood (90%), Brian Glendenning (100%), Athol Kemball (100%), Ralph Marson (100%), Joe McMullin (100%), Pat Murphy (30%), Darrell Schiebel (100%), Jeff Uphoff (100%) and Wes Young (90%). We have an open position in Socorro for a VLBI software specialist, and an open position funded by the NCSA-NRAO collaboration for someone to work on parallelization of AIPS++ code.
Thus, in aggregate, we have 25 people contributing just over 18 FTEs to the AIPS++ Project. Of these, 11 are employed by NRAO, and contribute 9.1 FTEs. The numbers for the other partners are: ATNF 4 and 2.45, BIMA/NCSA 5 and 3.25, NFRA 5 and 3.25.
