News
| Getting Started | Documentation | Glish | Learn More | Programming | Contact Us |
|
| Version 1.9 Build 000 |
News
|
AIPS++ Quarterly Report: 1996 Q2
Tim Cornwell, AIPS++ Project Manager
July 17, 1996
In the second quarter of 1996, we proceeded with
work aimed towards a beta release of AIPS++ targeted for early
1997. We also worked towards supporting the use of AIPS++ for
a number of applications: an HI survey using a multibeam system
on the Parkes telescope, as the prime data reduction system for
the GBT, and used as the prime on-line data inspection system
for the new WSRT on-line system, TMS. We have continued development
of the infrastructure needed to support these activities, in the
Table system, in the User interface, in the Measures system, and
in documentation. Work in all of these areas has proceeded reasonably
well, although continuing problems with compilers have generally
slowed the Project. The most significant obstacle has been continuing
difficulties in some areas of design, most particularly the design
of the single dish analysis package, this resulting in a delay
of a few months in the first tests of a single dish package for
the GBT. A cross comparison of synthesis processing by Newstar
and AIPS++ has been performed and confirms that the AIPS++ synthesis
package can reproduce the results from Newstar even at very high
levels of precision. First tests of programming AIPS++ by non-expert
programmers have also been encouraging, showing that the promises
of object-oriented programming are to some extent being realized.
Planning for the beta and full releases is continuing, and a detailed
list of the targeted contents of the first beta release has been
drawn up.
This report summarizes the status of the AIPS++ Project at the end of the second quarter of 1996. It describes the achievements during that quarter and gives a list of planned developments for the third quarter of 1996. It also describes long-term plans for the Project. Reports from each active AIPS++ site are included in appendix A.
In Single Dish, after some unsuccessful attempts,
we produced, with the help of Harvey Liszt, a design for SDCalc,
an application for essentially interactive processing of one-dimensional
data sets. SDCalc is principally intended for single dish spectral
analysis but we anticipate that it will also be used for many
different applications that require interactive perusal and processing
of data from a telescope.
In Measures, we completed the implementation
of the Measures system, omitting for the moment the highest precision
support needed for VLBI and pulsar observations. This latter will
be added once the specifications are more fully known, probably
later this year. We have developed a system for automatically
acquiring and loading IERS data.
In Synthesis , we converted the synthesis
code to use the AIPS++ Distributed Object system whereby objects
in the system are accessible from Glish. This has three immediate
advantages, first a simple and flexible user interface is available
using Glish and a Distributed Object GUI (the DOG) written in
Glish, second, each of the methods of the DO are available from
Glish thus allowing great flexibility, and third, the methods
of the various objects can be invoked from Glish functions or
scripts thus allowing programming of significant applications
solely from the Glish level.
To test and demonstrate the synthesis code, we processed
a 21cm WSRT observation of 3C147, correcting for a number of effects:
antenna-based gains, polarization leakage, and correlator-based
gains. This was performed using a script written in Glish that
constructed the necessary distributed objects and then invoked
the various methods as required. This processing was originally
performed at the Center, but is now possible at Dwingeloo on the
HP/UX systems using the Gnu compiler. In addition, we processed
data from a variety of different VLA observations to test the
capability to solve for polarization leakage. We also processed
some 20cm observations of Jupiter (kindly provided by Imke de
Pater) to test the efficacy of the joint deconvolution of Stokes
I,Q,U, and V. Satisfyingly but perhaps not too surprisingly it
appears that the results from joint deconvolution are superior
to those obtained from separate deconvolution.
We developed a design for cross-calibration of synthesis
data using the synthesis code. This draws upon the various strengths
of the AIPS++ Table system to provide simple data selection capabilities.
This design is now being implemented.
In a test of the capability of non-experienced programmers
to add significant new features to the synthesis code, Jan Noordam
wrote a "MeasurementComponent" to include ionosphere
corrections into AIPS++, as will be needed for the new UHF receivers
in Westerbork.
In Visualization and Image Analysis, the aipsview
program has proceeded to beta release after the incorporation
of contour plotting capabilities, postscript generation, and control
via Glish. The Glish plotting client, gplot1d, has been rewritten
to be based upon the well-known PGPLOT library developed by Tim
Pearson at Caltech, using a Motif widget written by Martin Shepherd.
The development of the advanced visualization program
OpenInventor aipsview has been discontinued due to Polly Baker
leaving the NCSA group. In this context, it is now clear that
we need a C++-based tool-kit for image visualization. We investigated
the possibility of adapting the aipsview library to this purpose
and concluded that it is not feasible. Hence our final major changes
to aipsview will be the addition of AIPS++ Image and Coordinate
class support to allow reading of AIPS++ native images, and fully
accurate coordinate display.
Image analysis development has proceeded at a low
level, mainly via the work of Neil Killeen, as detailed in the
ATNF site report given below. One interesting aspect of that work
has been Neil's assessment of the difficulty of programming in
C++ using the AIPS++ library of classes:
"I have found that writing top-level C++
application quite straightforward (once you know your way about
AIPS++). Since at the top level one must write something procedural,
this does not require any great depth of knowledge of C++ or OO
design."
In AIPS++ Infrastructure, the Glish
program has been through a process of improving robustness and
stability that seems to have significantly improved it. The documentation
has been revised to incorporate a number of changes made recently,
the most notable being the incorporation of the Tk-widget set,
thus allowing GUIs to be constructed from inside Glish. We expect
that a release of Glish 2.6 will be made in the next quarter.
The Table system has seen a number of substantial
improvements and additions. It is now possible to retrieve an
entire table row at once (both from C++ and from Glish). Tables
now have type information easily accessible. A number of implementation
changes have been made in response to timing and performance information
from the various uses of the Table system that are now occurring.
We finished the design and implementation of the
PagedImage class. We developed a class for Linear fitting. We
completed version 1 of the Distributed Object system and implemented
a GUI component (the DOG) written in Glish to allow interactive
selection of DO methods and associated parameters. A filechooser
and datachooser have also been written and incorporated in the
DOG.
In Documentation, we developed programmer
documentation, checking in a Programmer's Reference Manual. We
started implementation of a powerful system, designed by Paul
Shannon, for providing various levels of user help from the Glish
prompt.
In the System area, we continued development
of the support needed for the Gnu C++ compiler. Since the Gnu
compiler does not perform automatic instantiation of the templates
required for a program, this burden falls upon the programmers.
We wrote tools to aid this work. Although the extra work involved
in thus instantiating templates is not negligible, the general
reaction to the Gnu compiler has been positive since the operation
of the compiler is very straightforward. From a project perspective,
one key attraction of the Gnu compiler is that it does allow straightforward
porting of AIPS++ to new platforms. All consortium sites except
NCSA are now running Gnu systems. We plan to make it the default
compiler so that all AIPS++ code must run under Gnu.
We moved the master repository of AIPS++ code to
Socorro. This move was quite painless and has had few unforeseen
consequences.
We implemented some tools for determining the user
of templates in executables, and then evaluated the impact of
the widespread use of templates on executable size. We have developed
strategies for reducing executable size but these have had to
be deferred temporarily for the lack of a person to implement
them.
In addition, we made many other improvements to the
code management system. For example, the e-mail archives are now
search-able by content.
In Management, we made presentations to the NRAO Visiting Committee and the Users Committee and to the BIMA board. The first two were accompanied by demonstrations by Rick Fisher and by Tim Cornwell respectively.
Here we present an updated list of AIPS++ goals in
the next table:
| January 96 | First synthesis application polarization self-calibration and imaging | High |
| May 96 | Graphics system independent of Xrt/Graph | Medium |
| April 96 | Simulation package for Measurement Equation | High |
| April 96 | First version of tasking system and user environment | High |
| May 96 | Measures system largely complete | High |
| August 96 | Initial cross-calibration package | Medium |
| Mid 96 | Table system largely complete | High |
| Mid 96 | Framework for user and programmer documentation complete | High |
| Q3 96 | Support "friendly" observer at 140' | Medium |
| Q3 96 | First image analysis application | Medium |
| September 96 | Support use of TMS at WSRT simple calibration and editing of synthesis data present | High |
| October 96 | Support multi-beam observing at Parkes | High |
| October 96 | First meeting of AIPS++ Scientific and Technical Advisory Group | High |
| December 96 | Measures support for VLBI | Medium |
| October 96 | Wide-field imaging for VLA data | Medium |
| December 96 | Mosaicing applications | Medium |
| Q1 97 | First beta release of AIPS++ | Medium |
| Q3 97 | First full release of AIPS++ | Medium |
| Mid 97 | Support GBT commissioning fully capable Single Dish package present | High |
| Early 98 | Complete calibration and imaging for VLBI | Medium |
| Late 99 | Complete migration of AIPS, Miriad, NewStar capabilities to AIPS++ | Medium |
As before, the reliability gives an assessment of the probability
of meeting that goal. This is based upon a number of factors:
how well specified the goal is, the difficulty of the goal, and
the intrinsic priority of the goal to
AIPS++.
A number of milestones have been passed since the last report (denoted by the mark). Some have been shifted in time.
The previous section gave long-term plans. Here we
summarize the expected developments over the next quarter.
In Single Dish support, we will produce a
prototype of the SDCalc single dish analysis application. We will
support the use of AIPS++ for the analysis needs of the Parkes
multi-beam survey. We will finish a reader and a writer for single
dish fits (SDFITS), conduct interoperability tests with other
observatories, and write a paper describing the format.
In Synthesis support, we will implement the
cross-calibration design, add a GUI and write documentation. We
will add fitting of components to both images and visibility data.
We will support investigations of time-variability of VLA D-terms
using the code. AIPS++ synthesis code will be used as for first
pass analysis of data from WSRT as needed for the commissioning
of TMS. Sanjay Bhatnager of GMRT will come to Socorro for 3-4
months to work with the Synthesis group on polarization calibration
and imaging. Cornwell and Wieringa will present a paper on ADASS
on the design and implementation of the synthesis code.
In Measures, we will add support for use of
IERS bulletins. We will complete the review of the submitted code.
In Glish support, we will develop a test suite
and make release 2.6. We will start work on object orientation
inside Glish.
In AIPS++ Infrastructure, a Table storage
manager, the IncrementalStMan, for handling columns in which the
information changes infrequently will be completed. We will allow
Records (containers of heterogeneous information) to be stored
in Tables. We will write and document version 2 of the Distributed
Object system. We will develop a class for non-linear fitting,
and add constraints to the existing linear fitting classes. We
will implement a modified strategy for coding exceptions in C++
code. We will design and implement new Gridding and FFT classes
to replace the ones currently used since they have a number of
serious deficiencies.
In Visualization and Image Analysis, we will
finish incorporation of AIPS++ Image and Coordinates classes into
aipsview, and make a beta release of the new aipsview. We will
start work on a C++-based tool-kit for visualization. We will
finish an image statistics program.
In the System area, we will improve end-user installation procedures, particularly for installation from binaries alone.
In Documentation, we will finish the user
help system and start to write end-user documentation using it.
We will finish the initial version of the Programmer's Reference
Manual.
In Management, we will organize the first
meeting of the Scientific and Technical Advisory Group.
We have to ensure that the beta release will be attractive
for both astronomers, primarily, and programmers, secondarily.
Given the desire to release in about 6 months and to test adequately,
we have to ensure that the targeted software is either present
now in some form or is feasible to develop in a few months. Bearing
this in mind, the following areas make sense as explicit targets:
Notes: we intend to advertise AIPS++ as containing powerful general purpose tools for the manipulation and plotting of tabular data. We will document and demonstrate this capability explicitly rather than expect people to see it buried in the other parts of the system.
Note: The key elements will be the rewrite of gridding/fft (needed for speed) and the expansion of MeasurementEquation to allow spectral line and Stokes I.
Notes: These are the least developed and will require the most work to get done in time.
In addition, we need to perform adequate testing
prior to any beta release. This should be on a canonical machine:
US$5000 machine with 64Mbytes memory and few GBytes disk space.
The intention is that the beta release will not occur until all these items (except those marked "if possible") are complete. For beta sites, we foresee a number of categories:
We intend to limit the list of beta sites to 10-15
at most.
The platforms supported will be:
The ATNF currently has 5 people working in AIPS++. These are Neil Killeen (local manager), Mark Calabretta, Wim Brouw, Mark Wieringa and Tom Oosterloo. Additionally, there is a collaboration between the AIPS++ project and the ATNF headed Parkes 21cm multibeam receiver project (see report below for details of people) which is contributing substantial effort to
using and developing code within AIPS++.
We continue to struggle with the weekly inhale which is built
on our Ultra server 140. We build the native, gnu and cfront systems.
There are problems with the Ultra that regularly cause the rebuild
to fail. SUN are investigating with reasonably high priority at
this point. It is still often Tuesday or Wednesday before we have
the system operational. We will implement a dual, leap-frogging
system in the near future so that we will always have one functioning
system.
Regardless, it continues to emphasise that AIPS++ extends the
host computing environment significantly, even for the most uptodate
computers, and this is of concern not only for the developers,
but also for the non-consortium customers, who will seize upon
such issues as reasons not to bother with AIPS++. We have learnt
an important lesson at the ATNF through other software: it is
hard to recover users once they have an initial, bad experience,
even if you address all of their problems. First reactions linger.
The port to the DEC Alpha (Unix) using the Gnu compiler is very close to succeeding. This is essential for the multibeam project which will use AIPS++ on an Alpha at Parkes for its data pipeline. The remaining problems revolve about idiosyncrasies of the Alpha that have to be worked around. At present, AIPS++ is compiled but not installed.
Mark Calabretta spent 6 weeks working at the consortium center
in Socorro. Mark continued his work in the system area of AIPS++
during this time but worked more closely with the NRAO team.
Brian Glendenning visited the ATNF for 4 weeks to work with the multibeam group, the image analysis group and also to talk with the ATNF ATOMS (redesigning online control software for ATNF telescopes) group. He also gave a well received talk on the AIPS++ experience. Brian's visit was very valuable to us all and we enjoyed his stay.
Most of the AIPS++ group attended a C++ course help in mid June. The general feeling following the course was that it was valuable to most of us. Most important is the timing. You must do a course in which you learn skills which you will then continue to use.
Mark Calabretta's main responsibility is to the code distribution system. His time in the last quarter has been spent on
Mark has spent approximately 75% of his time on AIPS++ related
work. His nominal AIPS++ allocation is 50%. At present, this level
will be sustained indefinitely.
Wim Brouw's responsibility is mainly to designing and implementing the Measure and related classes. His time has been spent as follows.
Wim has spent 37% of his time on AIPS++ related issues in this
quarter. His nominal allocation is approximately 50%. This level
will be sustained indefinitely.
Mark Wieringa is a part of the team working on the synthesis related
classes. He has just returned from his second three-month stay
in Socorro (February April) where he worked with Tim Cornwell.
He has spent his AIPS++ time on
Mark was doing AIPS++ work for April only of this quarter during
which period 100% of his time was spent on AIPS++.
Mark's involvement in AIPS++ is negotiated yearly with AT/Narrabri
since he has been seconded to the project from the Narrabri computer
group. For the future following Mark's return to Australia, we
have agreed that he will continue to be an important player in
AIPS++ , but his nominal contribution will be of the order of
30% from now on.
Tom Oosterloo is working on visualization and image analysis software. Additionally he has been seconded to the multibeam group to help them with their AIPS++ software effort. His time in the last quarter has been spent on
Tom's nominal AIPS++ allocation is 75%. He has spent about this
amount on AIPS++ in the last quarter.
Neil Killeen spends his AIPS++ time attending to local ATNF AIPS++ management issues but is also now coding simple image analysis applications. He has spent his AIPS++ time on
Neil has no nominal AIPS++ time allocation, and the load varies.
Averaged over the last quarter, he has spent some 40% of his time
on AIPS++.
The multibeam received project is using AIPS++ as its software base. A team consisting of Lister-Staveley Smith (ATNF), Tai-Sheng Ye (ATNF), David Barnes (University of Melbourne) and Tom Oosterloo (ATNF) is working on a data pipeline within AIPS++ taking data from the telescope format to SD FITS in real time. They report:
Polly Baker has left the BIMA/NCSA AIPS++ group in order to pursue other visualization activities at NCSA. Graduate student George Baxter was unwilling to switch to AIPS++ visualization work from the Inventor Aipsview project he and Baker were working on. The loss of these personnel made it necessary to terminate the Inventor Aipsview software project. Active AIPS++ workers are now Dick Crutcher (the local manager), John Pixton (100%), Harold Ravlin (80%), and Doug Roberts (50%). In addition, Peter Teuben at Maryland follows AIPS++ developments and will start work on a BIMA AIPS++ filler later this summer. Yanti Maio will join the NCSA AIPS++ visualization team at the 50% level in July while she pursues an advanced degree in CS, after finishing a PhD in radio astronomy. Two additional full-time positions in the NCSA AIPS++ team are available and will be filled as soon as possible.
A beta 1.1 version of Aipsview was released. This version fixed a number of bugs and added a number of new features: glish control, postscript printing capability, and pgplot graphics capability for contouring and line profile plotting and coordinate axis annotation. In addition, the pgplot-based gplot1d widget was finished and checked in. Ravlin and Pixton visited Socorro for a week to work with the AIPS++ center; they are now working on a significant revision of Aipsview which will tie it more fully into AIPS++ and allow direct use of AIPS++ images and coordinates.
NRAO and NCSA signed a MOU for closer cooperation in radio synthesis imaging. NRAO will be included in the NCSA NSF proposal for its 10-year renewal, with funding requested via an NCSA subcontract to NRAO for two AIPS++ personnel at Socorro and a dedicated high-speed Socorro-Urbana data link.
A 21cm WSRT observation of 3c147 has been successfully reduced
and displayed by AIPS++, both in Socorro an Dwingeloo. This required
flagging of bad data, and solutions for Image Sky Model, gain
(GJones), leakage (DJones) and multiplicative interferometer errors
(MJones). The end result has the same quality as the NEWSTAR result
with the same dataset, i.e. very close to the theoretical noise.
It reproduces the very low-level extended emission in Stokes Q.
The exercise constituted a useful check of AIPS++ genericity for
different telescopes, and for the various sign conventions between
WSRT archive format and AIPS++. It gives confidence that the `alpha'
release of AIPS++ will have sufficient functionality to support
the first version of the new WSRT on-line system (TMS) in august.
The main worry in this respect remains the C++ compiler situation.
Jayaram Chengalur will move to the GMRT in early July. Although
other work has prevented him from getting deeply involved, he
will continue taking an interest in AIPS++ in Pune. He will maintain
close links with the AIPS++ group in Dwingeloo.
There has been no further progress on acquiring an extra AIPS++ application programmer in Dwingeloo.
Friso Olnon has corrected and adapted the prototype WSRT filler
in response to the needs of the above-mentioned 3c147 reduction.
The filler will be made more general, and upgraded with new record
and IO-classes, in time for TMS commissioning.
Jan Noordam has written an FJones solver, by adapting just one of the methods of the GJones solver. This demonstrates the reuse capability of Object-Oriented programming. It is part of his efforts to include ionosphere corrections into AIPS++ in time for the new UHF receivers in Westerbork by the end of 1996.
The CenterLine C++ compiler used in Dwingeloo will soon disappear.
In anticipation of the elevation of the GNU compiler as `default'
compiler for AIPS++, Friso Olnon and Ger van Diepen are adapting
the local AIPS++ installation (and the AIPS++ code) for its standard
use, also on the many HP hosts used at NFRA. In the meantime,
we have to keep alive three parallel AIPS++ systems, so that development
work by Ger van Diepen and Jan Noordam can continue. We only just
succeed in doing that, more or less. Especially the HP/GNU branch
is still quite rudimentary, and takes (too) much time to get working.
At the same time more and more AIPS++ applications (Glish clients mostly) become available, with which we can inspect and reduce our data ourselves, and even compose our own procedures. These applications also require extra attention in terms of supporting programs and configuration.
Ger van Diepen has improved the table system in several ways:
In April Ger has spent 3 fruitful weeks in Socorro and Charlottesville, which gave a good idea of the progress that had been made. Face-to-face discussions are so much more efficient than email
discussions.
Tim Roberts left NRAO in late June. His position will be moved
to Green Bank to allow hiring of an astronomer for AIPS++ development
and support for the GBT. Alan Bridle rejoined the AIPS++ group
to work 2 days per week on user documentation.
The NRAO group is split into a number of areas of responsibility. Cornwell works on management and synthesis code development, Garwood on single dish code, Glendenning on management and infrastructure, as well as pinch-hitting in most areas, Marson works on synthesis code and also acts as code-cop, Schiebel is responsible for Glish development, Shannon works on single dish code and general GUI issues, Yang works on primarily mathematical classes for the library, and Young works on documentation and user interface issues. NRAO is planning to hire an astronomer to act as AIPS++ contact for the GBT.
In this section, we give the names of people in the
various AIPS++ groups and the nominal fraction of time allocated
to AIPS++.
The ATNF group is: Mark Calabretta (50%),
Wim Brouw (50%), Mark Wieringa (30%) and Tom Oosterloo (75%),
with Neil Killeen as the local Manager.
The BIMA/NCSA group currently has 4 active
workers, as well as Dick Crutcher, the local manager. These are
John Pixton (100%), Harold Ravlin (80%), and Doug Roberts (50%).
The NFRA group is: Ger van Diepen (100%),
Jan Noordam (25%), Friso Olnon (50%), and Jayaram Chengalur (25%)
The NRAO group is: Tim Cornwell (100%), Bob Garwood (90%), Brian Glendenning (100%), Ralph Marson (100%), Darrell Schiebel (100%), Paul Shannon (100%), Shelby Yang (100%), and Wes Young (90%). We have one open position in GB for an astronomer to work with AIPS++ and the GBT. In addition, a number of scientists participate at various levels: Tony Beasley, Alan Bridle, Rick Fisher, Bob Hjellming, Athol Kemball, and Harvey Liszt.
This section has been written to mirror the section
in the last quarterly report on plans for 1996 Q2.
In Single Dish support, we did continue with
the work required to support the friendly observer test at the
140' telescope in June 1996. However, problems with the design
of the principal application, SDCalc, has led to a postponement
of this test for a few months..
In Synthesis support, we did continue the
development of the polarization deconvolution and selfcalibration
task, adding a simple user interface based upon the Tasking design,
and a GUI based upon Glishtk. We did continue development of simulation
software for testing and verification, but not at the level anticipated.
We did finish the WSRT filler into the MeasurementSet and perform
some trial reductions of WSRT data. Due to the ongoing work with
the Measures system, we did not write new tools for gridding and
fast Fourier transformation of visibility data. We did not begin
an investigation of the adaptation of the synthesis code to parallel
processing.
In Measures, we did complete the implementation,
save for the highest precision modes needed for VLBI and pulsar
observations. We did write a modified Image Coordinate System,
but it does not yet incorporate the Measures system.
In Glish support, we did work towards improving
the stability and robustness of Glish and Glishtk. We did revise
the documentation of Glish to bring it up to date. Both the test
suite and the new release have been deferred until this next quarter.
In AIPS++ Infrastructure, we started but did
not finish development of a Table data manager for sparse data.
We did not allow the use of Record (the AIPS++ heterogeneous container)
in Tables. We deferred improvements to the tablebrowser.
In Visualization and Image Analysis, we did
issue a beta release of aipsview complete with contouring and
other improvements. We studied the possibility of incorporating
into aipsview support for AIPS++ Images and full non-linear coordinates.
We did not issue a development plan for the ATNF/NCSA collaboration
on Visualization and Image Analysis. We discontinued work targeted
towards a beta release of Inventor AIPSView in July 1996.
In the System area, we did investigate possible schemes for reducing the deleterious effects of the wide-spread use of templates in AIPS++, but work to implement the changes has been deferred.
In Documentation, we did continue the revision
of the programmers manual.
The following have agreed to serve:
Robert Braun NFRA
Jay Chengalur NCRA/NFRA
Roger Foster NRL
Dennis Gannon UIUC
Walter Jaffe Univ. Leiden
Lee Mundy U. Maryland
Bob Sault ATNF
Lister Stavely-Smith ATNF
Dave Shone NRAL
Doug Tody NOAO
Huib van Langevelde JIVE
Tony Willis DRAO