Version 1.9 Build 000

News FAQ Search Home

AIPS++ Quarterly Report: 1996 Q1

Tim Cornwell, AIPS++ Project Manager

April 15, 1996

Introduction

This report summarizes the status of the AIPS++ Project at the end of the first quarter of 1996. It describes the achievements during that quarter and gives a list of planned developments for the second quarter of 1996. It also describes long-term plans for the Project. Reports from each active AIPS++ site are included in appendix A.

Developments in 1996 Q1

In Single Dish, a development plan was formulated and fixed for the time being. This plan includes work needed for both the GBT development and the Parkes HI survey. Work has continued in a number of areas:

• Changes to the AIPS++ data store: the MeasurementSet (see below in Synthesis for more details).
• A plan for the development of the GUI for single dish processing,
• Development of specific Glish clients needed for processing,

David Barnes of the ATNF Parkes HI survey team spent 8 weeks working with the SD group in Charlottesville.

In Measures, Wim Brouw froze the design and worked on implementation. Completion of all major features except high precision VLBI support is expected in May. Parts of the code are now being reviewed.

In Synthesis processing, a demonstration is available at:

http://www.cv.nrao.edu/aips++/docs/uvdemo/101/101.html

This shows the use of the imager program to self-calibrate VLA data solving for both normal antenna-based gains, and polarization leakage effects (the "D-terms"). A substantial amount of effort was expended on optimization of this program, with results that should benefit general applications development in AIPS++. Revised specifications for various telescopes were incorporated into a development plan. The design for the MeasurementEquation was refined, splitting the ME into two parts, a SkyEquation and a VisEquation, thus easing the eventual application of this formalism to single dish processing. In coordination with the single dish group, the MeasurementSet was redesigned, moving away from one big AIPS++ table for all data to a more conventional main table / sub tables scheme. This redesign was easily implemented and is now in use in synthesis processing. Calibration information was made persistent by writing to AIPS++ tables, in which form, it can be viewed by tools in Glish/Glishtk such as the tablebrowser and the plotting utilities. All these are steps on the way to a full calibration package. This work has been performed in collaboration with Mark Wieringa, who is spending six months working at the Center with the synthesis group (this visit is funded by ATNF and NFRA).

In Visualization and Image Analysis, the NCSA group made various improvements to the aipsview visualization tool with the goal of a beta release by April 15. The major area of improvement is the inclusion of contour plotting via the PGPLOT library.

In AIPS++ Infrastructure, a large number of improvements were made in response to the requirements arising from applications development:

• The capabilities of Glishtk were expanded to include nearly all of the tk widgets,
• A improved and redesigned tablebrowser was implemented in Glishtk,
• AIPS++ table system underwent a number of improvements, the most important being a Tiled Storage Manager to allow efficient access to multidimensional data sets along various axes.
• A number of miscellaneous utilities such as least squares and interpolation classes were developed.
• The implementation of the Tasking system was completed and coding of applications based upon it is now proceeding.

In Documentation, we decided on an overall approach to programmer documentation, and began to implement it. An initial version of a revised programmers manual is now in place and being improved.

In the System area, we worked on the implementation of shared libraries under the Sun native compiler, and changed to rely upon native exception handling. In addition, we have made ports to various compilers:

• HP/UX, required for NFRA, is basically successful and is now being completed,
• After a considerable effort of trying a port to the Dec Alpha compiler, required for ATNF, we decided to abandon it since it went into a release cycle with bugs that were too serious for us to work around.
• In order to have a working port for the ATNF Dec Alpha, we persisted with a port to the GNU Project g++ compiler. This is proceeding well and we expect to have a working system soon. This will give us the ability to port to many architectures easily.
• SGI, desired by NCSA, is proceeding.

In Management, we continued the implementation of a more formal scheme for proposing and accepting changes to AIPS++.

Long-term plans

The long term plans for the AIPS++ project were first presented in the previous Quarterly report. Here we present an updated list of AIPS++ goals in the next table:

As before, the reliability gives an assessment of the probability of meeting that goal. This is based upon a number of factors: how well specified the goal is, the difficulty of the goal, and the intrinsic priority of the goal to AIPS++.

Only one major milestone has been passed since the last report: the first synthesis application (denoted by the mark).

Plans for 1996 Q2

The previous section gave long-term plans. Here we summarize the expected developments over the next quarter.

In Single Dish support, we will continue with the work required to support the friendly observer test at the 140' telescope in June 1996. This includes writing a number of Glish clients for analysis, and putting together all the key elements in the AIPS++ GUI.

In Synthesis support, we will continue the development of the polarization deconvolution and selfcalibration task, adding a simple user interface based upon the Tasking design, and a GUI based upon Glishtk. We will continue development of simulation software for testing and verification. We will write a simple mosaicing task to test the applicability of the concepts in the MeasurementEquation formalism. We will finish the WSRT filler into the MeasurementSet and perform some trial reductions of WSRT data. We will write new tools for gridding and fast Fourier transformation of visibility data. We will begin an investigation of the adaptation of the synthesis code to parallel processing.

In Measures, we will complete the implementation, save for the highest precision modes needed for VLBI and pulsar observations. We will write a modified Image Coordinate System incorporating the Measures system.

In Glish support, we will work towards improving the stability and robustness of Glish and Glishtk. We will revise the documentation of Glish to bring it up to date. We will develop a test suite for Glish capabilities. We will issue a new release of Glish.

In AIPS++ Infrastructure, we will develop a Table data manager for sparse data, allow the use of Record (the AIPS++ heterogeneous container) in Tables. We will make a number of improvements to the tablebrowser, for both functionality and efficiency.

In Visualization and Image Analysis, we will issue a beta release of aipsview complete with contouring and other improvements. We will study the possibility of incorporating into aipsview support for AIPS++ Images and full non-linear coordinates. We will issue a development plan for the ATNF/NCSA collaboration on Visualization and Image Analysis. We will continue work targeted towards a beta release of Inventor AIPSView in July 1996.

In the System area, we will investigate possible schemes for reducing the deleterious effects of the wide-spread use of templates in AIPS++.

In Documentation, we will continue the revision of the programmers manual.

Project-wide concerns

Inadequate staffing

In the last report, it was noted that the Project was inadequately staffed. In particular, the following positions were deemed to be particularly important:

• an AIPS++ scientific programmer at GB,
• a scientific programmer at NFRA to help with the AIPS++ support of the TMS commissioning,
• scientific programmers at the AOC and at ATNF to help with development of Synthesis processing.

None of these have come about. The danger to the Project is obvious: a possibility that we will miss some of the major milestones.

Transition to an operational system

According to the timetable presented above, AIPS++ will begin to be available for end-users in early 1997. It is vital that we start planning for this now. The most pressing concerns are to:

1. Ensure that interesting functionality is present for some users,
2. Improve user and programmer documentation,
3. Improve end-user installation procedures
4. Ensure adequate testing prior to any release.

Addressing some of these concerns will require additional staff. 1 is implicitly addressed in the various development plans. The functionality targeted in the beta release of AIPS++ will be specified in June 1996. The user side of 2 is to be addressed by Alan Bridle, who plans to resume work inside the Project in May. The one person currently dedicated to programmer documentation is soon to be removed from the AIPS++ group. The inadequacy of the end-user installation procedures is well demonstrated (see e.g. the ATNF and BIMA/NCSA site reports). Inside NRAO, we expect to draw on the experience of the AIPS group in end-user installations. 4 could usefully be addressed by scientists from the various consortium members. An example of this type of work is current testing of the Glish system where Harvey Liszt and Darrell Schiebel are scheduled to work together for a few months with the specific goal of finding and fixing bugs. Similar efforts should be started with other parts of the system later in the year.

Standard hardware configuration for AIPS++

We have attempted to define a standard configuration for a machine that is to run AIPS++. The basic formulation is that AIPS++ should perform well for problems of a normal size on a Unix-based workstation costing $5,000. An exemplar of such a machine is a Sun Sparcstation 4, with 64 Mbyte of main memory, and a few gigabytes of disk storage. Our main concern is that the memory will be too small for interesting problems. We will soon have to fix a policy for the use of memory inside AIPS++ programs. We plan to continue to track this issue closely over the next 6-12 months.

Scheduling

To better quantify our ability to meet targets, we have reviewed the success of the target dates scheme that we use to track progress. In the 9 months since we instituted the target dates in May 1995, we have completed 111 targets. Of these, 3 were in advance, and 48 were more than 10 (calendar) days late (where the average duration of a target is about 20 days). The 48 late targets can be categorized in the following way:

19 Underestimated difficulty of target (e.g. complex design issues)

19 Person obviously overloaded (usually due to unforeseen problems or, more rarely, other targets in conflict)

4 Obviously bad choices for target dates (e.g. inappropriate dates for some external event such as ADASS)

3 External factors (e.g. waiting for someone outside the project to do something)

3 Person culpable

The division is highly subjective, and the analysis is clearly limited. However, the basic conclusions are that we need to pay more attention to the first two types of problem. Fixing the first (underestimating the difficulty) will require more intermediate targets as checks. The second (overloading of time) is harder to deal with since it tends to be interrupt-driven, as key people are called upon to help diagnose or fix urgent problems. Perhaps the best approach here is to ensure that everyone is aware of this latter effect.

Reliance on the Internet

AIPS++ is suffering from the side-effects of the vast increase in interest in and use of the Internet. We rely upon the Internet for a number of purposes:

• General communication inside the Project is accomplished by person-to-person email and by the use of e-mail exploders.
• Code distribution and management. Revision control is accomplished via direct access to the master repository of code in Charlottesville using an NFS mount.
• Project members often need to work on machines at other sites e.g. for porting purposes.

Both the latter uses are being adversely affected by the general increased load on the Internet. One answer inside NRAO may lie in the tentative proposal for an NRAO Intranet. This will do little to address difficulties for external sites and so we anticipate having to do some work make our code management and distribution system more useable over a poorly responsive network. Note that a decision on the possible move of the code master to Socorro has been deferred until the question of an NRAO Intranet has been settled.

Scientific and Technical Advisory Group

The member of the advisory group is now mostly settled, with a few remaining possible additions. See Appendix D for the list of members. The group is expected to meet first in September or October. The purpose of the group is to advise the Project Manager on scientific and technical matters relevant to the Project.

Appendix A: Site reports

ATNF

The ATNF currently has 5 people working in AIPS++. These are Neil Killeen (local manager), Mark Calabretta, Wim Brouw, Mark Wieringa and Tom Oosterloo. Additionally, there is a collaboration between the AIPS++ project and the ATNF headed Parkes 21cm multi-beam receiver project, which includes Taisheng Ye working primarily in AIPS++.

In Epping we have acquired a new Sun Ultra 140 server specifically for AIPS++. It is configured with 192Mb of memory (essential for AIPS++ C++ compilations to run at a reasonable speed) and a fast-wide 9Gb disk.

Mark Calabretta's main responsibility is to the code distribution system. His time in the last quarter has been spent on:

• Implementing a simple system for putting template instantiations in the class library.
• Working on Glish within AIPS++.
• uprev to v06.
• Setting up new AIPS++ server for Epping.
• Answering the usual requests for bug fixes or new features and local programmer assistance.

Mark has spent approximately 50% of his time on AIPS++ related work. His nominal AIPS++ allocation is 50%. At present, this level will be sustained indefinitely.

Wim Brouw's responsibility is mainly to designing and implementing the measure and related classes. His time has been spent on:

• With the aid of the St Petersburg Astrophysics Institute, and the Mt Stromlo library, Wim finished the basic coordinate conversion routines. They (precession, nutation, annual aberration and solar position for gravitational bending) are almost finished for both J2000 and B1950.
• Speeding up the unit and quantum speeds. Large improvements were made, and checked in.
• Being involved in some of the measurement set related discussions.

However in trying to finalize the full measure class and its conversions, Wim encountered an obscure cfront compiler error. With help from NRAO, this was eventually bypassed. This plus some other pressing work has delayed delivery for the full system (due 22/mar/96).

Wim notes that his productivity was badly damaged by spending about 40% of his AIPS++ time on system/compiler related issues. He has spent 22% of his time on AIPS++ related issues in this quarter.

Mark Wieringa is a part of the team working on the synthesis related classes. He is currently in Socorro on his second 3-month stay there working with Tim Cornwell. He has spent his AIPS++ time on

• Reworking the MeasurementSet. It is now a table with a collection of sub-tables for antenna, feed, frequency etc information.
• Updating the single source fits filler, the visibility iterator and the VisSet to use the new MeasurementSet.
• Working on simulation software so that the MEGI formulation can be well explored.

Mark has spent 75% of his time on AIPS++ in this quarter. His nominal AIPS++ allocation is 50%.

Mark's involvement in AIPS++ is negotiated yearly with Narrabri since he has been seconded to the project from the Narrabri computer group. Currently the agreement is until mid-1996.

Tom Oosterloo is working on visualization and image analysis software. This work involves collaboration with the NCSA AIPS++ group. His time in the last quarter has been spent on

• Reviewing Aipsview v1.0
• Brief review of Aipsview v1.1
• Outlining the case for Aipsview to be restructured as a widget set (image display library)
• Writing the user specifications for volume rendering
• Planning the general layout of the 21cm multi-beam display software
• Working on the user specifications for the multi-beam visualisation software multi-beam
• Reading (object oriented methods (book rumbaugh), AIPS++ docs, c++, etc)

Tom's nominal AIPS++ allocation is 75%. He has spent about this amount on AIPS++ in the last quarter.

Neil Killeen spends most of his AIPS++ time attending to local ATNF AIPS++ management issues but is also beginning to plan some simple image analysis software. Neil writes that he has spent his AIPS++ time on

• Procurement of the Ultra 140 server
• Liaison with multi-beam group
• Discussing with NCSA and Tim Cornwell, the future of the visualization and image analysis collaboration.
• Working on a development plan for image analysis and visualization.
• Annual ATNF Steering Committee report on AIPS++
• Annual report to Australian Government Department of Industry, Science and Technology for obtaining AIPS++ travel related funding.
• Discussion with Tom Oosterloo about the visualization software
• Measurement set discussions.
• Began to learn the Glish language with which I will write some simple image analysis applications. I also started to work out what infrastructure is available to me from Glish. In this regard, I found the documentation of AIPS++ rather poor. Glish has an excellent manual, but in order to answer very basic questions I was forced to ask people, mainly Brian Glendenning. I also find in general that the documentation pages are poor. It is easy to find out what is something that is a future target, but it is hard to find out what is done and how is it accessed. My questions were:
  • How do I read a FITS image into AIPS++ ?
  • How do I access an AIPS++ table from Glish ?
  • How do I access the 1-D plotting tool plot1d from Glish ?
  • How do I display an image with Aipsview via Glish ?

I have no nominal AIPS++ time allocation, and the load varies. I would guess that averaged over the last quarter, I have spent some 30% of my time on AIPS++.

The multi-beam received project is using AIPS++ as its software base. Their main experience comes from David Barnes who spent a few months in Charlottesville starting in early 1996. They report:

• AIPS++ is very hard to install. This is because of the complexity of the procedures, as well as the fragility of the tools (C++ compilers).
• We are very happy with support received at Charlottesville -this aspect of the collaboration worked well. It also enabled the previewing of unreleased software such as Glishtk and we hope that this can continue.
• David Barnes was able to put together a skeleton reduction system relatively quickly (filler-calibrator-browser). Taisheng Ye in Epping has already modified the filler to read multi-beam data and to write out a SD MS.
• We are not happy with delays in definitions with the SD MS and the SD FITS formats. This may cause us to diverge from the AIPS++ definitions if this is not resolved soon. From the astronomers point of view, these should be the first things to be defined, not the last. Our writing of code is being delayed by this uncertainty.

BIMA/NCSA

The BIMA/NCSA effort the past quarter continued to concentrate on visualization development. New features and bug fixes for aipsview, continued development of Inventor aipsview, AIPS++ graphics, and installation of a full AIPS++ system at NCSA were the major efforts. Individual efforts are listed below:

Dick Crutcher (25% time): Time was spent on AIPS++ management, testing aipsview, and writing aipsview user documentation.

Polly Baker and George Baxter (50% time each): In the area of 3D visualization, they continued work toward a July beta release of the Inventor aipsview program. They did considerable cleanup to the code supporting Inventor aipsview, started a review of the in-code documentation, built a tool for reading the VRML files produced by Inventor aipsview into NCSA's virtual environments: the CAVE and ImmersaDesk. In part, this served as a warm-up exercise to building a tool that can support Inventor aipsview's data visualization functions in these environments.

Harold Ravlin (80% time): He is the main person responsible for aipsview maintenance and improvements. The major effort has been to add the pgplot vector graphics package to aipsview. Major new features which have been added to aipsview this quarter are the ability to plot contours, to show a box around an image display area with axis ticks and coordinate annotation, to show a colormap wedge when an image is shown as a raster and the axes are displayed, and to produce a postscript file (for printing) of raster images and/or contour representations of images, with or without axis notations. Contour values may be set as the user chooses, contours may be displayed alone, on top of the associated raster image, or on top of another raster image. Contours update just like rasters when animations are run. Axes annotations are controlled by defaults. Various minor features and bug fixes were also added to aipsview.

John Pixton (100% time): He has worked mainly on producing a plot1d widget using pgplot that will replace the commercial widget now used in AIPS++ for general vector graphics work. This is expected to be finished by the middle of April. In addition, he worked on a design for an interface between aipsview and external programs, that might be used to interface existing fortran or C analysis codes with aipsview. He also worked to fix some aipsview bugs, and assisted Randy Sharpe in the installation of a full AIPS++ Solaris system at NCSA.

Randy Sharpe (20% time): He installed a full AIPS++ installation at NCSA on a Solaris system and began work on a port of AIPS++ to our SGI Power Challenge Array. This work will be taken over by Doug Roberts, who will become a 50% AIPS++ worker during the second quarter, with a focus on parallel processor applications.

Peter Teuben (10% time): He worked on the specification of BIMA requirements for calibration of BIMA data. This should be finished by the end of the quarter.

NFRA

Local project members: Ger van Diepen (GVD), Jan Noordam (JEN, local manager), Friso Olnon (FMO), Jayaram Chengalur (JNC, local Project Scientist)

General

It has not been possible to hire an AIPS++ application programmer at NFRA in time to play a significant role in the commissioning of the first production version of the new WSRT Telescope Management System (TMS) in august 1996. In view of the financial situation of NFRA, it is also increasingly doubtful that such a person can be hired in 1997, despite general agreement that it is highly desirable.

Jan Noordam has checked in version 2.0 of AIPS++ Note 185.

AIPS++ applications

Friso Olnon has delivered a working prototype of the WSRT data filler. The work has been delayed somewhat by the revision of the MeasurementSet.

AIPS++ Site in Dwingeloo

End of February the regular maintenance of NFRA's AIPS++ site was transfered from the local AIPS++ project group to the general R&D maintenance group (FAM), consisting of Henk Vosmeijer and Friso Olnon. As part of this transfer activity, Olnon wrote the "NFRA/AIPS++ Configuration Manual" conform the NFRA standard, which is essentially an introduction to the AIPS++ System Manual by Mark Calabretta.

At the same time the NFRA installation was extended to include an HP Unix host, which is required for the new WSRT on-line system TMS. The environment for AIPS++ work now exists, but not all standard AIPS++ code compiles and links on the HP. As long as the Center does not have an HP itself, the NFRA group will continue to assist the Center in supporting the HP platform in the future.

The AIPS++ installation on the Sun/Solaris host keeps working without much trouble. That is the platform on which all current local AIPS++ development is done. At the time of writing (March 5) we have problems building Glish clients, but the FAM group is working on that with the help of Darrell Schiebel.

Ger van Diepen has conducted two C++ workshops at NFRA. This included advanced subjects like "counted reference", "copy-on-write" and "expression tree".

Infrastructure software.

Ger van Diepen has put considerable work into documenting the new Table software. He also implemented some small changes in the Table system at the request of some users.

He has developed a framework for data format conversion (e.g. VAX<->IEEE). This will be used in the MeasurementSet filler, and perhaps by TMS. The student Anco Boersma has developed an IO framework, which will allow transparent IO in which the conversion classes can be easily integrated.

The development of the new Record classes is almost finished. With these classes, arbitrary recursive structures can be specified, instantiated and made persistent. The Table system supports these structures.

Miscellaneous

NFRA shares with ATNF the cost of sending Mark Wieringa to the Centre.

NRAO

Personnel

Wes Young has been reassigned back to the AOC Computer Division, effective May 1, probably for a period of about two months, but only as a backup for system administration.

Appendix B: Summary of AIPS++ Personnel

In this section, I give the names of people in the various AIPS++ groups and the nominal fraction of time allocated to AIPS++.

The ATNF group is: Mark Calabretta (50%), Wim Brouw (50%), Mark Wieringa (50%) and Tom Oosterloo (75%), with Neil Killeen as the local Manager.

The BIMA/NCSA group currently has 4 active workers, as well as Dick Crutcher, the local manager. These are Polly Baker (50%), John Pixton (100%), Harold Ravlin (80%), and Doug Roberts (50%); In addition, Jim Morgan and Peter Teuben at Maryland follow AIPS++ developments and expect to become directly involved in the future.

The NFRA group is: Ger van Diepen (100%), Jan Noordam (25%), Friso Olnon (50%), and Jayaram Chengalur (25%)

The NRAO group is: Tim Cornwell (100%), Bob Garwood (90%), Brian Glendenning (100%), Ralph Marson (100%), Tim Roberts (100%), Darrell Schiebel (100%), Paul Shannon (100%), Shelby Yang (100%), and Wes Young (90%). In addition, a number of scientists participate at various levels: Alan Bridle, Rick Fisher, Bob Hjellming, Harvey Liszt.

Appendix C: Review of developments promised for 1995 Q4

This section has been written to mirror the section in the last quarterly report on plans for 1995 Q4.

In Single Dish support, we did implement an improved tablebrowser, based upon the Glishtk GUI machinery. We did not yet write a first GUI with simple astronomical task, but we did complete approximately 50% of the Glish C++ clients needed for the observations. David Barnes did visit Charlottesville to work with the group for 8 weeks on support of the Parkes Multi-beam observations.

In Synthesis support, we did finish a task to do polarization leakage self-calibration and imaging, we did improve gridding and the use of FFTs, we did ensure persistence for calibration objects, we did develop capabilities for simulation of synthesis observations, but we did not write a first GUI-based imaging task. In preparation for the commissioning of the WSRT TMS, we did write a prototype filler to load WSRT data into a Synthesis MeasurementSet. We did not yet start a collaboration between the Synthesis group and NCSA to investigate and possibly implement selected synthesis algorithms on parallel processing machines.

In Measures, we did complete a re-design of the classes. A completed Measures implementation will not be available by the end of the quarter but we expect it to be finished mid-April.

In AIPS++ Infrastructure, we did continue work on the Table system, finishing the Tiled storage manager. We deferred improving the existing Run-Length-Encoded storage manager (called the "Miriad" storage manager). We did complete a re-write of the File and I/O classes. We did implement a system allowing time-changing data such as IERS data to be available inside AIPS++ via the Tables system. The VLA calibrator list is one of the first sources of data included. The implementation of the tasking system (http://www.cv.nrao.edu/aips++/docs/notes/185.ps) did continue, first with the control hub and C++ run-time system, and then with a sample application to the Table system.

In Visualization and Image Analysis, we have make progress but did not yet finish re-implementing the gplot1d Glish client on top of the PGPLOT graphics widget. We did investigate and proceed with putting PGPLOT graphics capabilities directly into Aipsview, and we did add hardcopy output to Aipsview.

In the System area, we did partially implement shared libraries. We did not add automatic registration for the AIPS++ e-mail exploders (a "majordomo"), nor transfer the AIPS++ master code repository to Socorro. We did attempt ports to the g++ compiler, and to Dec Alphas, SGI and HP/UX configurations. We did implement the now-standard compiler exception mechanism.

In Documentation, we did revise the programmer's section of the on-line documentation, identify and remove out-of-date documents. We started on but did not complete a revision of the format of source code documentation. We did put programming entries in the AIPS++ Glossary. We did fully implement and start to use the gnats bug reporting system.

In Management, we did form and schedule a meeting of a Scientific Advisory Group for AIPS++. Tim Cornwell did not visit Pune for discussions with GMRT staff.

In addition, Allen Farris of STScI did re-design and re-implement the FITS classes used by AIPS++. We will benefit directly from this work.

Appendix D: Membership of AIPS++ Scientific and Technical Advisory Group

The following have agreed to serve:

Robert Braun NFRA

Jay Chengalur NCRA/NFRA

Roger Foster NRL

Walter Jaffe Univ. Leiden

Harvey Liszt NRAO

Lee Mundy U. Maryland

Bob Sault ATNF

Lister Stavely-Smith ATNF

Dave Shone NRAL

Huib van Langevelde JIVE

Tony Willis DRAO