Getting Started Documentation Glish Learn More Programming Contact Us
Version 1.9 Build 1556
News FAQ
Search Home

NOTE 226 - AIPS++ RELEASE PLAN V1.3

Tim Cornwell
National Radio Astronomy Observatory
P.O. Box O, Socorro, NM, 87801, USA

October 1 1999

A postscript version of this note is available (126kB).

Contents

Introduction

The purpose of this document is to outline the deliverables in the forthcoming release of version 1.3. This updates the previous development plan..

The immediate priorities for the AIPS++ project are:

Release 1.2 is now going out the door. It has a stable Glish, a reasonable GUI, lots of synthesis imaging, a thin-path for calibration, a version of the single dish analysis program dish, extensive image analysis capability, two versatile display programs, and lots of miscellaneous functionality. The documentation framework is now all in place but the documentation itself must be filled out.

We intend to settle down on a 6 month plan-develop-test-release (PDTR) cycle. The time scale is as follows, counting from the date of the previous major release.

+0 weeks
Revise planning documents
+2 weeks
Finalize planning, assign targets
+2 weeks
Start development, testing
+2 weeks
Initiate completion of testing of existing capabilities
+4 months
Freeze new checkins, start dedicated testing
+5 months
Hard freeze, uprev to next versions of release and develop masters.
+5.5 months
Cut final CDs
+6 months
Ship CDs

The times required for each step may vary but we intend to stick to this regular major release schedule. Note that patches to the previous release will be issued as needed, probably on a 4-6 week interval.

In the medium term (end of 2000), we expect that the third release (probably 1.4) will have most of the synthesis calibration and imaging tools present for all consortium telescopes. Following that we will next target a developer's release.

We must act with discipline in setting and acheiving targets. In particular, at this point in the cycle, we must not schedule too much to be completed before the freeze. Accordingly, I have changed some of the submitted plans to defer some work until the next release.

Applications integration (Tim Cornwell)

A major push to integrate applications was started in November 1998. For the most part this initiative was successful and led to considerable improvements in the stability and useability of the system.

(a) the current state of the library

The basic Tasking capabilities exist can provide a number of services for C++ programmers. These include getting and returning parameters, showing progress meters, asking the user questions, and displaying plots. Passing arbitrary types through the Parameter system is possible but clumsy. The Tasking code lacks some test programs and has not yet been reviewed.

The Glish side of the user interfaces is in reasonable shape but there is still a lack of uniformity and a dearth of good high-level widgets,

(b) the current state of applications

Not applicable.

(c) scientific priorities

Some types of interaction with applications are still lacking.

The main GUI interface is basically satisfactory. The scheme of an autogui controlled by meta-information seems to work for many different contexts. The Copy-Paste idiom for passing information around seems to be viable for simple entities. One problem is that the GUI widgets are incomplete. We need a wide range of more useful widgets:

Passing information around the user interface is still quite clumsy. For example, to get a region from an image on disk requires far too many operations. Part of this is due to the confusion between e.g. disk image file or Glish image tool. One obvious approach is that since we know the type of a table (e.g. Image, MeasurementSet, Calibration), we should automatically open the corresponding tool when an image is needed; Thus the catalog should be extended to show the connection between a Table and the corresponding tool. In general, converting from a data file to a tool should be a one-click operation both in the catalog and in the spanner.

We are still missing a high-level concept to bind the interface together. We've called this a workspace in previous discussions.

(d) deliverables by first release

We delivered the following:

High
Investigate use of TiX widgets. It seems that binding the widgets will consume a considerable amount of Darrell's time.
High
Decouple Auto-GUI from objectcatalog. This and a general restructuring was performed.
High
Parameter saving, restoring, transferring. A simple scheme is now in place.
High
Define format for Auto-GUI widgets, implement a couple. This occurred.
High
Improve help presentation in AutoGUI. We've moved to displaying most of the help via browsers.
High
Standardized parsing. The entryparser has improved matters somewhat.
High
Improved stability and robustness of the user interface. The GUI and Glish can now run for many hours between failures. The remaining problems are quite obscure and are hard to repeat.
High
Memory use: the memory use is still too high. The megaserver brought some improvements but is not a long-term solution.

(d) deliverables by second release

All of these should be completed early in the development cycle to allow propagation to the rest of the package.

High
Protocols for ``connecting elements of the interfaces''. This is a key issue for easing the use of the interface. It should become less difficult to do common operations. This includes but is not limited to shortcuts in the toolmanager. (Tim Cornwell, 3w)
High
Widgets for Measures, Quanta, Selections. The first two are probably best accomodated by special widgets that call the measures and quanta tools directly, rather than by using the measures and quanta guis. The Selection widget should compose a TaQL query for a set of possible columns specific to a given type of table. (Tim Cornwell, 3w) This requires the next:
High
Name registry for types of Table (Tim Cornwell, 1w)
High
Propagate history information from Glish to ApplicationInfo (Tim Cornwell, 1w)
Medium
New C++ services: display of an image, interrupt a running process, get new parameters. (Tim Cornwell, 1w)
Medium
Prototype of workspace (Tim Cornwell, 3w)
Low
Propagate unset values to C++. This is needed to avoid the horror of special values being used for logically unset values. (Tim Cornwell, 2 w)

Deferred but important

Change servers.g to use subsequences
This is vital to resolve a complete mismatch between the models used. Glish is event-driven, servers is functional, and the GUIs are event-driven. servers should be changed to be event driven as well. I think this is a lot of work to propagate everywhere.

Synthesis (Athol Kemball)

Purpose

The purpose of this section is to update the existing synthesis development plans (Cornwell 1997, Kemball 1998) as part of the current planning cycle for the second release. This section summarizes the current state of the synthesis library and applications, defines the scientific priorities, and identifies deliverables for the second release. Infrastructure required from other parts of the project is explicitly identified. Items which were listed in earlier development plans but which are not yet complete are in the most part carried over to this development plan, unless they have been replaced by higher priority items.

The identification of synthesis goals and priorities is a continuous process, influenced by outside advice and consultation with users and external committees alike. This section identifies the priorities as currently assessed for this time scale. It is not a list of all possible synthesis development priorities for the project.

Current state of the synthesis library and applications

Data formats
The existing MS v1.0 data format was revised to v2.0 to accommodate unified synthesis and single dish processing, and to allow for additional synthesis capabilities. The new format has been adopted, and all changes to infrastructure classes have been made, with the exception of changes to the fillers themselves, although these changes will only be checked in to the system after the first release.

The capability to read and write data in UVFITS format has been expanded in the last development cycle to support a broader set of data fields or sub-tables. The full MS archive utility using FITS binary tables, already existing in the package, has been reviewed and kept current in the last development cycle.

Data fillers
Data fillers exist for WSRT, ATCA, BIMA and the VLA. The VLA filler was added in the last development cycle; the existing fillers have been updated in the same period. The library infrastructure developed for the VLA filler was designed to allow re-use for future fillers, including the one required for the VLBA.

Uv-data visualization and editing
A development effort with ATNF in the area of uv-data visualization has led to a requirements document for these capabilities, a design within the current Display Library (DL), and some preliminary implementation code. This uncompleted item is a major priority in this plan.

A preliminary data lister has been developed in C++, but requires integration and expansion. Simple editing capabilities have been added to visplot and a new editing tool flagger.

Calibration
Basic cross-calibration suitable for connected-element arrays has been added to the system. This has included the use of an expanded calibration table format covering both image-plane and uv-plane calibration components, as well as other capabilities. Basic flux bootstrapping utilities have been added to the calibration system, for cases where that is appropriate. Calibration data visualization and editing is rudimentary, and awaits progress in the general area of uv-visualization described above.

Image-plane calibration has been significantly expanded with the addition of more versatile primary beam models and corrections as part of the mosaicing development, which can be specified using the voltage pattern manager.

Imaging
The mosaicing capabilities in the package has been expanded to include enhanced primary beam corrections and weighting, as well as new or improved deconvolution methods. A Lattice-based Clark CLEAN is now available in imager, as well as multi-field Hogbom deconvolution and multi-resolution CLEAN algorithms. Discussions with Pixon LLC regarding use of their code in aips++ have continued. An image-plane deconvolution tool, deconvolver, has been added to the system, and includes MEM deconvolution. In addition, general infrastructure has been improved or expanded in support of mosaicing development.

Wide-field imaging has been added using the Sault-Brouw common projection algorithm, and has been extensively debugged with simulated and real data. It is an integral part of the general imaging code. Preliminary on-the-fly (OTF) code has been developed in prototype form.

An automated wide-field imaging script dragon has been developed and added to the system. The imaging wizard simpleimage has been upgraded to use more recent infrastructure in the package.

Documentation
A user-level cookbook has been written in a first draft, along which will be combined with several contributed recipes.

Simulation
The structure and infrastructure used by the existing simulator has been upgraded, and it is being revised to accommodate more physically meaningful calibration errors, particularly those required by ALMA simulations.

General
Synthesis applications are fully integrated into the new user interface provided by the toolmanager development.

Development priorities

The preceding plan identified broader scientific use, the finalizing of interfaces and development for time-critical local needs as overall development priorities. It is appropriate to revise these in the current plan, although we do not shift focus from the key identified objectives.

At the current stage of development of the project, and for the planning interval covered by this document, the following key priorities are identified:

Deliverables for the second release

The deliverables for the second release are listed below:

Single Dish (Bob Garwood)

(a) Current state of the library

The current state is generally good. However, all of the deficiencies in the library outlined in the previous version of the development plan remain.

FITS
This continues to drag out. The random access abilities apparently available in the most recent version we received from Ferris (which has yet to be incorporated into the aips++ system due to lack of documentation and a few concerns about the code) would be very useful to use in the GBT filler.

Fitting
We continue to have a need for a general purpose multi-component fitting class.

Calibration
No work has been done so far on single dish calibration.

MS columns
Parkes multi-beam data archived in the SDFITS convention preserves information about the first version of the MeasurementSet. As we migrate to the second version of the MeasurementSet, sdfits2ms needs to be able to do the right thing as it converts this original SDFITS data into the future versions of the MeasurementSet.

(b) The current state of applications

General single dish applications consist of dish and its plotter, dishplot and the data iteration client which dish uses, sditerator. This client, sditerator, expects the data to be in a flat Table made directly from a FITS binary table following the SDFITS convention. It can not interact with a MeasurementSet. The plotter is built using the pgplotter widget with some additional features appropriate for single dish data. Dish has the following operations:

GBT specific tools consist of two data fillers, a tool for viewing an interacting with the engineering log data, and a set of utilities for use during commissioning of the GBT. One filler puts engineering log data into a flat Table and the other fills astronomical data into a MeasurementSet. The commissioning tools remain largely untested. The astronomical filler can currently only fill DCR and holography backend data.

(c) Scientific Priorities

(d) Deliverables by first release - 1 October 1999

(e) deliverables by second release - 1 April 2000

Image Analysis (Neil Killeen, Ger van Diepen)

Current Status

People

The resources dedicated to image analysis are modest. Currently only about 0.75 FTE (N Killeen) is provided directly on the high level applications. A further 0.3 FTE (G van Diepen) has been available in the last year for low level infrastructure work (lattices, regions, LEL). Finally, there is 0.75 FTE (D Barnes) working on the Display Library and applications, some of which are related to image analysis or used by image analysis applications.

Library

AIPS++ provides access to images via a range of C+ + classes. These fall into the following basic categories

Lattices
These define and provides access to regular n-dimensional lattices (disk or memory based)

Regions
These support image regions-of-interest and image masks.

Lattice Expression Language
These classes support calculation of mathematical expressions of lattices.

Coordinates
These support non-linear coordinate systems. The sky coordinate conversions are handled with the WCS system (Calabretta and Greisen).

Images
These combine lattices and coordinates to define and access images.

FITS
classes are provided for converting bewteen AIPS++ images and FITS images.

In addition, there are what we might call analysis classes. The functionality revolves around

Applications

The majority of the library capability is available to the user via three Glish tools; Image, Regionmanager and Imagefitter. Their actual functionality is listed below under deliverables for the first release.

In addition, there is some small amount of non-library functionality in the Image tool that should be moved to the library:

Scientific Priorities

Development has been heavily focussed on providing the infrastructure to make application development straightforward. We are now in the position to develop more applications, although some substantial infrastructure work is still required. Of course, development of an application usually means doing the real work in a C+ + class with a thin binding to the Image tool, so that application work sometimes appears as infrastructure work.

Image analysis is driven by scientific needs extending over a very broad front; this makes it quite hard to prioritize some of the competing required functionality. However, functionality that will have a large user base is generally developed first.

Deliverables by First Release 1999/10/01

The image analysis functionality is provided through three tools; Image, Regionmanager and Imagefitter. The first provides basic access and services for images. The Regionmanager manages the creation of image regions-of-interest, and the Imagefitter is a specialized application fitting source models to the sky.

Image tool

The Image tool provides the Glish user access to all of the functionality in the Library.

The functionality of the Image tool is as follows.

Conversion
Interconvert between AIPS++ images and FITS files Also, more basic structures such as shapes and arrays can be converted to images.

Inquiry
Inquire basic information about the image such as its name, shape, type, header information.

Pixel access
Access (get and put) directly the pixel and pixel mask values.

Image calculator
Manipulate mathematical expressions involving images.

Coordinates
Basic access to the coordinate system and inter conversion between pixel and world coordinates.

Convolution
Separable convolution by parameterized functions and convolution by a user given kernel.

Analysis
Computation of statistics and histograms, moment analysis and fitting of models to the sky.

Display
Display the image (raster or contour) with the Viewer. The user can also use AipsView.

Reshaping
Images can be reshaped in a variety of ways (subimage, padding, concatenating).

Masks
Multiple image pixel masks are supported. They can be set, copied and deleted. Some basic capability is available to change mask values.

Utility
A range of utility services are available; open, close, rename, and destroy this tool. View the history, and locking.

Custom GUIs
There are some custom GUI interfaces developed for specific functions. These are for handling masks, separable convolution, and moment analysis.

Cookbook
chapter on image analysis
Extensive documentation
for images module, image, regionmanager and imagefitter tools.
test script
for image tool
test/demo
programs for the majority of the lattice and image C++ classes in trial. Some classes ready for review.

Regionmanager

The Regionmanager tool manages image regions of interest. It has a custom GUI interface.

The functionality of the Regionmanager tool is as follows.

Imagefitter

This tool, written purely in Glish, but drawing on many other AIPS++ components (Image, ComponentList, Viewer tools) offers interactive fitting of models to sources in an image.

The functionality of the Imagefitter tool is as follows.

Display
The image is displayed with the Viewer providing full control over the display.

Regions
Interactive cursor-driven specification of regions-of-interest to be fitted. Regions can be recovered from a Table and automatically refit.

Fits
Accepted model fits are stored and returned in a Componentlist tool. Fit parameters are listed and residuals are displayed via an image, histogram and statistics. Model parameters can be held fixed in the fitting process and the fitted model can be subtracted from the image

Deliverables by Second Release 2000/04/01

The next development cycle will see continued in both infrastructure and applications.

There is some dependency on other people that must be agreed upon. In particular ComponentList [rm] and image overlap a fair amount, and some services I wish to provide in the image tool probably should be supported by ComponentList (the model stuff). Also the ImageRegrid class has been developed by [mh], and partly further developed by him, it may be more effective for him to finish this work than me [nebk] attempt to take it over. Some of [gvd] needed for region and LEL work. Some of [drs] so that pgplotter objects can be passed through to C++ pgplotter.

Visualization (David Barnes)

Current status

Library

Presently, the AIPS++ Display Library (DL) provides an object-oriented approach to data display. The library defines a PixelCanvas interface, of which we have implementations for two devices: X Windows and PostScript. The DL defines and implements a WorldCanvas on top of the PixelCanvas interface. DisplayData objects, of which there are two existing implementations (LatticeAsRaster and LatticeAsContour), can be registered for display on WorldCanvas objects via a mechanism implemented in the WorldCanvasHolder class. A sophisticated colormap management system is available, as well as event handlers for the X Windows PixelCanvas, the WorldCanvas, and the DisplayDatas.

While visualization development in the last 10 months has been mostly directed at applications work, in that time the following improvements have been made to the library itself:

Applications

Most visualization development in the last 10 months has been directed towards the delivery of the AIPS++ `Viewer,' a new image viewer for AIPS++. The Viewer is implemented as a dynamically loaded, Glish/Tk agent-based interface to the DL, and the application (``tool'') itself is put together with Glish code. Consequently, the Viewer is available for use as a (quasi) stand-alone tool, or as a module for inclusion in other AIPS++ tools, such as the Imagefitter and Simpleimage. This is what distinguishes the Viewer from previously available visualization programs such as AipsView and MultibeamView.

Scientific priorities

There are two scientific driving forces behind the DL:

1.
The provision of AIPS++-native display applications that can be tightly coupled to all stages of data processing: editing, imaging and analysis. This includes display ``components'' that can be ``plugged-in'' to objects needing display services, but without the ``added baggage'' of a complete AIPS++ application.
2.
The provision of programmability of the DL at the Glish command-line level. The drive here is to enable the typical Glish programmer--or keen astronomer with a new idea and minimal Glish knowledge--to put together innovative visualisation applications that extend or complement the functionality of the applications provided by AIPS++ in 1. above.

First release: 1999/10/01: Deliverables

For the first release, the objective was to deliver a replacement for AipsView. This objective has mostly been met by the provision of the Viewer tool, which has the following display capabilities:

The Viewer falls short of full AipsView replacement level in only a few areas, such as animation.

Second release: 2000/04/01: Deliverables

With the AipsView replacement well in hand, the highest priority for the second release must be to demonstrate that new and innovative capabilities are possible with the existing DL infrastructure. The main goal, therefore, will be the provision of new and interesting DisplayDatas:

1.
Design oversight review of UV-related DisplayDatas: A major new thrust for the DL is the development of non-image-based DisplayDatas, as are needed for example in UV plane visualization. David Barnes will be the key design reviewer. High, ongoing review role, David Barnes.
2.
TrueColor et al. support in X11PixelCanvas. This is essential for acceptance of the viewer as the prime visualization tool. High, 4w Harold Ravlin?
3.
The addition of HSV and RGB multi-channel overlay facilities for the LatticeAsRaster class is highly desirable. This would enable quite sophisticated representations of data, especially for complex or multi-waveband images. Medium, 4w David Barnes, 4w Harold Ravlin?
4.
A rudimentary DisplayData for plotting catalog information already exists. Provided the required library enhancements are made, this too could be made available in the second release. Medium, 4w, David Barnes, 2w Harold Ravlin?
5.
Design and implementation of improved animation capabilities Medium, 2w David Barnes
6.
Support for masks in viewer. Medium, 2w David Barnes, Harold Ravlin

Deferred

1.
The development of an interactive overlay DisplayData would also be very nice. The user would be able to make annotations on the display, and have them drawn correctly as they zoom in or out, and save the annotations to a Table. This DisplayData could be used by various tools to show the location of ImageRegions, or fit parameters, for example.

Dependencies

Glish (Darrell Schiebel)

The only changes foreseen here for the second release are:

Control of memory use
Both glish and glishtk use large amounts of memory. High, DRS, 2w
Destructors
A mechanism for destruction of glish entities such as closures is needed.High, DRS, 2w
Implement binding to TiX widgets
. The TiX widgets provide a mechanism for improving the quality of our graphical user interfaces.Medium, DRS, 4w
Local/global eval
Low, DRS, 2w

Measures (Wim Brouw)

(a) the current state of the library

1.
Time, 'infinite', constant directions, 'stable' Earth platform, magnetic field available
2.
Moving directions: support for major solar system objects

(b) the current state of applications

Applications are supportive only (conversion guis; specialised conversion machinery). There are conversion guis available for time, frequency, velocity, positions on Earth and in the sky, and also available where requested (e.g. frequency; uvw).

(c) scientific priorities

(d) deliverables by first release

See (a).

(e) deliverables by second release

(f) assumptions about progress in other areas

Infrastructure (Ger van Diepen)

(e) deliverables by second release (2000/04/01)

The Fitting classes must be finished. These have many different applications.

A number of changes are needed in the Table system. All of this work will be done by Ger van Diepen unless noted otherwise.

The following are deferred until the third release (version 1.4)

Documentation (Kate Weatherall, Wes Young)

Current state

The overall documentation framework is now in place: we have getting started documents, a cookbook, a Glish manual, a user reference manual, a FAQ, background documentation, documentation search, defect reporting and tracking, and a glossary. Virtually of this has been developed by project members, the most notable exception being the ``Getting Started in AIPS++'' contributed by Anantha.

Scientific priorities

Deliverables by first release

Deliverables by second release

(f) assumptions about progress in other areas

Documentation (cookbook chapters and recipes) must come from our users.

Quality Assurance Group (Ralph Marson)

Given the limited resources available to individual AIPS++ team members, the QAG has to concentrate its efforts in these areas that will aid in producing a stable system. The major effort will therefore be 'testing'. We view testing mainly as 'pro-active' bug-fixing, rather than the, maybe more visible, 'reactive' bug fixing. We believe that it will have a higher efficiency, and that it has a better chance of keeping homogeneous, reliable code.

This type of testing will include some centralised attending to conformance of rules. The production of test programs can, at a later time, be used during code reviews.

Centralised testing and the writing of test scripts will take a significant fraction of the time of many of the AIPS++ programmers, and in particular the QAG members.

The QAG will not neglect its duties in other areas, namely code reviews and maintenance of coding "rules". But, unlike testing, we do not envisage major new initiatives in these areas over the next six months.

(a) Current state of the library

Too small a fraction of the current library has been reviewed. Most of the classes (504 to 415) and lines of code (296K to 236K) still reside in trial whereas the majority of the test programs (166 vs 121) are in aips.

(b) Current state of the applications

Not relevant to QAG.

(c) Scientific priorities

Improve the reliability of AIPS++, through more rigorous unit testing of glish and C++ code.

(d) Deliverables by the first release

1.
100% successful completion of all test programs in the aips package for both linux and solaris architectures.
2.
Successful completion of the assay for both linux and solaris architectures.
3.
Well established code review system
4.
Extensive set of templates to guide programmers in what we consider good coding style.

In the above we define linux and solaris to mean RedHat-5.1 and Solaris-2.6. In absence of test machines that are readily accessable to QAG members we cannot guarentee anything about other processors or different versions of the operating systems.

All of the above targets have currently been met with the exception of the second one. There is a continuing problem which we hope will be resolved by the first release.

(e) Deliverables by the second release

Parallelization (Athol Kemball, Doug Roberts)

Purpose

The purpose of this document is to update the existing development plan for the AIPS++ parallelization initiative (Kemball 1998) to cover the next six months, which specifically includes the second release planned for 15 March 2000. The current status of the parallelization effort libraries and applications are described, the scientific priorities are listed, and deliverables for the second release are specified. Infrastructure required from the rest of the project is listed separately.

Current status of the parallelization effort

At present the parallelization initiative has achieved the following goals:

1.
Porting of AIPS++ to IRIX and the SGI native C++ compiler. This includes verifying a stable SGI build at the AOC, and ensuring a repeatable stable build at NCSA. Significant progress has been made in the last development cycle to achieve this goal. An Origin200 has been obtained at the AOC, which is used to generate the primary SGI build; parity in IRIX and compiler revisions is maintained between NRAO and NCSA.

2.
Development of a applicator/algorithm class structure, which encapsulates the parallel transport layer, for use in the parallelization of existing AIPS++ algorithms. Demonstration of this capability, using MPI transport, for the embarrassingly parallel problem of multi-channel CLEAN deconvolution on an NCSA Origin2000 system.

3.
Implementation of fine-scale parallelization of FFT transforms using the SGI parallel library (SCSL), and their evaluation.

4.
Implementation of a batch utility and scripter to allow use of the large NCSA SGI Origin2000 systems.

5.
Collaboration with the parallel I/O ET team in the NCSA Alliance to successfully instrument AIPS++ I/O using the Pablo libraries. Compilation of I/O instrumentation data for a selection of large-scale runs.

6.
Initial imaging of a large multi-configuration dataset taken on M33 by the VLA, to be used as a test case of AIPS++ parallelization capabilities.

7.
Initiation of an effort to port AIPS++ to Windows NT, for use of the NCSA NT supercluster. A significant fraction of the AIPS++ standard computational library has already been ported to Visual C++ in this effort.

8.
Establishment of a collaboration with the Albuquerque High Performance Computing Center (AHPCC) to allow AIPS++ to be installed on their Linux supercluster in a research project to investigate AIPS++ parallelization performance in cluster environments. This work, along withe the NT port and supercluster work, is part of a CS MS project.

9.
Investigation of porting requirements to move to the 64-bit SGI compiler ABI.

10.
Profiling of the serial performance of key algorithms targeted for parallelization, including mosaicing and wide-field imaging.

11.
Debugging and testing of the wide-field imaging implementation in AIPS++, in preparation for its parallelization.

Development priorities

The AIPS++ parallelization effort forms part of the Radio Astronomy Imaging Group, which in turn is one part of the Scientific Instrumentation Application Technology (AT) team in the NCSA Alliance. Members of the AT team include NCSA, BIMA and NRAO. The overall goal of the Radio Astronomy team is the development of data reduction capabilities for the demanding imaging applications of radio astronomy. The elements of this goal are:

1.
Development of a complete and sophisticated data processing system (AIPS++) for full, final data processing, visualization and analysis.

2.
Enhancement of critical tasks in the AIPS++ package to take advantage of high-performance parallel computing environments, such as those available at NCSA.

3.
Development of a system for near real-time transfer of data from a remotely located radio synthesis array telescope.

4.
Automatic production of first-order images from the observed Fourier visibilities.

5.
Archiving of raw data and first-order images.

6.
Development of a digital library in which final images are available to the community.

The parallelization initiative has the following overriding goals:

Deliverables for the second release

The deliverables for the second release are listed below:

Development plan: prerequisites

The parallelization development effort may require assistance with the transition to 64-bit compliance throughout AIPS++, depending on the scope of the problem as finally determined.

References

Cornwell, T., 1997, AIPS++ Note 201, http://www.aips2.nrao.edu. Kemball, 1998, previous parallelization development plan, Dec. 1998. Kemball, 1998, Synthesis plan at integration meeting, Nov. 1998.

System (Ger van Diepen)

Current system

AIPS++ can be executed on any UNIX system with X-windows. It has proven to run under SUN Solaris, HP-UX, Linux, SGI/Irix, and Digital Unix on a variety of machines. It requires little extra software. The public domain package Tcl/Tk is needed for the graphical interface provided by glish. Netscape is needed for browsing the documentation.

Developing in AIPS++ requires much more software, all of it available in the public domain area. The following software is needed:

The offical project C++ compiler is egcs 1.1.2, which works fine for all supported UNIX platforms. GNU and egcs have merged again, so the newest g++ compiler is 2.95, which will be adopted in the future by the AIPS++ consortium. New compilers are often based on the EDG front-end. Ports to the KAI and SGI compilers, based on EDG, are mostly completed.
The C++ code is quite heavily templated. Templates are explicitly instantiated, because automatic instantiation led to excessive link times.
AIPS++ has some Fortran77 code, which can be compiled with g77 or a commercial Fortran compiler.

In October 1999 the first release of AIPS++ was distributed on CDs, which contain executables, libraries, documentation, and source code. It is possible to get updates via ftp.
Sites actively developing AIPS++ can obtain and update a development version via ftp.

Future developments

C++ standard and Compilers

C++ has now an official ANSI/ISO standard. Several new possibilities in the C++ standard are already used in AIPS++.

Some other options are of interest and might be used in the future.

The AIPS++ consortium sites use the products purify, purecoverage, and quantify to check the code thoroughly. Unfortunately Rational only supports egcs 1.0.3 for these products. Therefore only those C++ constructs can be used that are supported by egcs 1.0.3 or can easily be emulated.

Porting to Windows NT

A port to Windows NT is underway as part of the parallellization efforts at NCSA to use AIPS++ on an NT cluster. Apart from that it is foreseen that NT will be an important platform. It raises the problems how to map the UNIX and X11 functionality to Windows NT.
Windows NT has the POSIX standard implemented, but when using it the Win32 API cannot be used anymore. This limitation is not acceptable.

AIPS++ uses several UNIX system and library calls. The most important are file handling, pipes, sockets, time, environment variables. X11 is used heavily in aipsview. It is also used in the Display Library, but the class structure is such that it should be easy to use classes dedicated to NT.

There are a few public domain and commercially available packages being able to aid in a port. Amongst them are:

When doing the port it has to be decided what is the best way to go. There are a few scenario's possible.

A possible porting strategy could be:
1.
Handle parts of the basic library not dependent on UNIX or X11.
2.
Handle UNIX-dependent library code.
3.
Handle glish. Tcl/Tk has already been ported to Windows NT, so the glish graphics should not be a problem.
4.
The final step is porting X11-dependent code.
It has to be decided if and which porting tools will be used. Using Win32 calls directly results in best performance, but tools make the porting easier.

As soon as code development will be done on Windows NT, a code management system (integrated with the UNIX side) and a build system should also work on it.

Code Management

The current configuration, code management, and build system has a few problems: It has to be investigated whether current subsystems have to be replaced by other packages (public domain or commercial).

Another problem is the size of the libaips library. It is already quite large and will get twice as large once the trial code is moved to aips. It requires that the aips package is split into a few subpackages. A requirement is that they are layered and that a lower level layer is not dependent on a higher level layer. This may be hard for DO implementations which tend to use a lot of classes.
Splitting the library can be done in a simple way by having a few packages at the aips level. A more fundamental approach is to have a subpackage layer under aips. That makes it possible that other packages (nrao, dish) are split in subpackages.

An extra directory level could also be useful in the apps tree. One can think of an application group (with maybe some common source files) and in subdirectories the individual applications.

Shared libraries and dynamically loadable code are used to make dynamically configurable code possible. The glishtk part of glish is relying on shared libraries.
Some experiments have been done with using shared libraries to reduce the size of the system. So far they have not been successful, because too many seldomly used code got linked into the shared libraries. Also the use of static variables gave problems. After splitting the libraries another attempt can be done.

Documentation

The current user documentation system, based on latex and latex2html, is fragile and hard to configure. It needs to be stabilized. At the moment Kate Weatherall and Wes Young are investigating other methods like SGML-based systems..

Care should be taken that no latex constructs are used which are not (yet) supported by latex2html. Having to do very frequent latex2html updates can be a pain for some users.

It should regularly be checked if the search facility for the AIPS++ html pages is selective enough.

The code documentation system is based on the Perl script cxx2html which extracts declarations and comments (with special tags) from the header files. It is doing a very good job, but needs some changes to make nicer html output.

Release support

On a regular basis AIPS++ releases are done on CD. In October 1999 the first release was done. Users can get updates via ftp. In order to reduce ftp traffic a mechanism is needed to make the updates selective, i.e. that only the parts needed are ftp-ed. It would also be nice if the dependencies are such that linking a program is based on changes in the source code and not on changes in a library. In that way a program is only linked when really needed, thus also ftp-ed when really needed.

Support of industry standards

There are several industry standards which are of use to AIPS++. It has to be decided if and how they are supported in AIPS++.

Priorities for system

After the release in October 1999 two formal releases are foreseen until the end of 2000. The last release is meant to be used by any programmer who likes to develop code using AIPS++ classes.

Before the next release (1.3) the following needs to be done:

1.
Makedefs reconcilation High GVD 1w
2.
Implement data proposal of Wim Brouw High WB 2w
3.
Automatic tools for patch generation. e.g. Have a better dependency analysis to link based on source code changes instead of library changes High GVD 2w
4.
Investigate ways and advantages of partitioning libraries. Medium GVD 2d
5.
Improve using native exceptions. To prevent memory leaks, all variables on the heap should be put into an AutoPtr object until their pointer is taken over. Low GVD 4w

Before the following release (1.4):

1.
Use threads in DO's to make them non-blocking. It requires that static data is thread-safe.
2.
Port to Windows NT.
3.
See if automatic template instantiation can be used. This is needed when heavily templated classes in Blitz++ or STL are going to be used.
4.
Replace some of our classes by classes from the C++ Standard Library. Candidates are String, Complex, Map, Block. Other classes like valarray should also be looked at.
5.
Use Blitz++ or techniques used in it to speed up the Array arithmetic.
6.
Investigate how CORBA, DCOM, OLE, XML, ODMG, and Java can be used.

Other work (Tim Cornwell)

Support of HIA ACSIS project

This section outlines the deliverables to be expected from AIPS++ in support of the HIA ACSIS project. This follows a dicussion in Socorro on July 13 between Tim Cornwell, Athol Kemball, Darrell Schiebel (AIPS++), Peter Dewdney, Gary Hovey, and Tony Willis (HIA/ACSIS).

Document XSysEventSource (DRS: 1d, due 1 Sept 1999)
Current documented example evidently does not compile

Debug Glish Shared clients (DRS: 3w, due 1 Nov 1999)

1.
The link operator does not work for shared client when 'from' client is attached to in a script other than the original launching script.

2.
A shared client is launched in script a and then attached to by script b. Script b interacts with the client and then exits. If script a is now exited the client continues to run, unattached to any script. The client should die when all scripts to which it has been attached have finished.

3.
general improvements to manual in areas relating to distributed clients.

4.
Debug shared clients on original or copy of working system

5.
class library needed to set up handles for events from multiple scripts for a shared client - i.e. translate most of 'multiplexer.cc' into a class library that can be used with GlishSysEvent.

Allow Glish out-of-band signals (DRS: 3w, due 1 April 2000)
How to get an abort, or stop, to the head of the message queue?

User control of Glish polling (DRS: 3w, due 1 April 2000)
Allow user to set how often glish polls other systems to see if they are still running - ie. override 5 sec default, which is currently fixed in the code. The default seems to generate too many spurious warning messages when clients are running on multiple machines.

Catch of "connection terminated" conditions (DRS: 2d, 1 April 2000)
In glish polling of remote machine, how to catch a polling event and tell tasks that a remote machine has vaporized? How to test 'shared client' or 'handle' case that a remote machine has gone away?

Improved Glish documentation of communications and clients (DRS: 1w, 1 April 2000)
A good description of the communication protocol(s) and routines underpinning GLISH and Glish Clients is required.

Appendix 1 - Detailed Development Areas for Image Analysis

This is a fairly detailed description of the areas of development that are expected for the next year or so in Image Analysis. A subset of these will be targeted for the second release (see above). Some of this will not be of general reader interest. The time estimates are generally bare minimums to get something rudimentary going.


next up previous
Please send questions or comments about AIPS++ to aips2-request@nrao.edu.
Copyright © 1995-2000 Associated Universities Inc., Washington, D.C.

Return to AIPS++ Home Page
2006-10-15