 News |
FAQ
|
| Search
|
Home
|
| Getting Started | Documentation | Glish | Learn More | Programming | Contact Us |
|
| Version 1.9 Build 1367 |
|
It is clear that making early releases of a new software system can have both positive and negative results. If the release is scoped properly and made available to ``friendly users'' (i.e., people willing to work closely with the software development group and provide positive feedback) it gives the development team needed inputs for subsequent improvements and extensions to the system. However, if the contents of the release are oversold, or expectations exceed what can be delivered, an early release can do more damage than good. Moreover, the damage can have a long-lasting effect on user perceptions of the system. A poorly prepared early release will alienate much of the potential user community.
Of course, delaying software releases too long also undermines the credibility of the software development project. However, it is our opinion that the potential harm of a premature release is the greater risk. AIPS++ should not promise a release to users outside of the immediate software development community (i.e., the Consortium sites) until it has a robust set of utilities and a (limited) number of demonstration applications. These applications need not be complete, but must show the potential of the new system and provide some incentive for users to make the transition from the old to the new technology.
The relatively early availability of the BIMA visualization software meant that there was a clear opportunity to demonstrate some of the emerging AIPS++ data visualization functionality to the wider community, even if it does not meet the longer-term requirement for AIPS++ compliance. Development could then have been guided by feedback from real user requirements and experience, rather than more abstract and idealised considerations. This particular strategic opportunity was not exploited, if even it was perceived by the Consortium, because of the technical rather than user focus that dominates much of the project's thinking. The response of the review (observer) audience to the demonstration of the BIMA software is evidence enough to support this particular supposition.
Astronomical research is rarely confined to a single instrument or wavelength domain, and astronomers have to use data analysis tools from a variety of systems and environments in order to develop a comprehensive understanding of the physics of astronomical objects. AIPS++ strives to be THE reduction and analysis tool for radio astronomy (for both filled and synthetic apertures), yet it must work in conjunction with other astronomical data analysis packages and public domain or commercial software. The Project does not seem to have given much attention to this situation. That is, how might astronomers make use of AIPS++ facilities without having to import, install, and learn the entire AIPS++ environment? And conversely, how might an AIPS++ user access tools from outside the environment, and easily transfer data from one system to another? We believe that the AIPS++ design is flexible enough to facilitate open exchange of software and data, but simply want to emphasize the long-term importance of providing an open system. It would be a mistake to implement AIPS++ under the assumption that the system provides all capabilities, or isolates the user from either the host operating system or other applications packages.
The Panel is concerned with the implications of the frequently stated ``Polish graduate student'' model (whereby all software required for the AIPS++ Project, other than compilers, must be non-commercial), which appears to one of the very few identifiable strategic drivers for the AIPS++ project. The ``Polish Graduate Student'' is a powerful, effective metaphor that may have been too effective in guiding this project into unfortunate, wasteful redevelopment of difficult software. This ``parsimony-driven'' view of the development of an important (software) ``scientific instrument'' seems highly anomalous, given the cost in human and hardware terms of conducting most radio astronomy research. This tacit strategy has been costly to the project in that it has forced the developers to eschew, as a matter of principle, highly appropriate commercially available software, which has had to be rebuilt from scratch or acquired from other research groups (with very considerable long-term support and evolution implications).
The technical development has proceeded in the absence of a clear understanding of the ordering or priority of applications to be supported by AIPS++. Technical staff have thus applied their best effort to anticipate what infrastructure might be needed. At least five different criteria have been used in discussions for determining or suggesting the proper ordering and priority of AIPS++ application development.
1. AIPS++ must provide at least one end-to-end analysis path. In favor of this criterion, people argue that in order to earn credibility with the client community AIPS++ must show that it really does something. Furthermore, AIPS++ must show that it does that something correctly and with reasonable performance. The use of the DDT test set was suggested as a means to cross-validate AIPS++ against Classic AIPS. While a comparison of results on the DDT test set or some other test suite might be useful for validating that AIPS++ tools do valid computations, the distributed architecture of AIPS++ may make it very difficult to obtain meaningful performance benchmarks. Discussions with project management suggest that while DDT might be a reasonable partial strategy for accomplishing this objective, DDT does not trace the whole end-to-end data analysis path of interest. This criterion emphasizes the importance of early prototype tools, even ones that do not use the full capabilities of the envisioned computing infrastructure. In addition, the observation that astronomers tend not to comment on design documents but they evidently respond viscerally to effective demos argues that a rapid prototyping strategy would be particularly effective in this community.
2. AIPS++ must provide the core functionality of AIPS. Besides the ambiguity of what the ``core'' functionality of AIPS might be, there is doubt as to whether this criterion yields the most effective use of project effort. On the one hand, duplicating AIPS functionality might not be an effective goal since AIPS exists and does support much work. In addition, it is clear that the AIPS core is itself inadequate for the analysis of interferometric data from a number of telescopes. On the other hand, duplicating AIPS functionality might provide an entry path to AIPS++ for the largest number of users.
The next three criteria would lead the AIPS++ project team to produce capabilities that would be unique to AIPS++ in some way. The arguments for these and against the above criteria are that astronomers have software that supports many of their needs now, so AIPS++ would have a greater impact by providing something not available in other software packages, whether this be unique preprogrammed functionality, unique flexibility, or unique integration of available but currently separated tools. In favor of the above criteria are the arguments that the above criteria would engage one larger user community and the experience with existing tools would provide a better cross- validation of the AIPS++ infrastructure than would completely new development.
3. AIPS++ must provide a way for the community to understand and access AIPS++ infrastructure capabilities, thereby allowing the community to develop their own data analysis methods. The review panel was told several times that user programmability is the key potential advantage of AIPS++. This capability would be a major attractor of a user community to AIPS++ and would set up AIPS++ to attract graduate students and build a long-term future in the community. Users interested in such interactions are interested in direct access to their data and the ability to develop new reduction and analysis algorithms, and not necessarily in the niceties of GUIs nor in advanced computing technology in and of itself. If this criterion is adopted, it will be crucial to devise an effective and efficient strategy for communicating to users a vision or model and a direction of development of the infrastructure.
We note that if direct manipulation of data is adopted widely in the astronomy community it may dramatically change the way astronomy is performed since it will be necessary to describe and justify algorithms mathematically rather than by reference to a particular standard analysis program.
4. AIPS++ must provide something in a form that is better in some way than what other packages provide. The idea is that to attract a user community whose needs are already at least partially satisfied by existing software packages, AIPS++ must provide versions of the tools that are more accurate, more robust, better integrated, easier to use or customize, or faster than the existing tools.
5. AIPS++ must provide some things that are best-in-the-world implementations. This objective would be most readily realized by associating AIPS++ with an emerging project for which AIPS++ would provide primary support. Such an arrangement would likely associate AIPS++ with a small but loyal user community. Having any community willing to proclaim the merits of the AIPS++ project and show real results obtained from it would be an advance over the current situation.
The review panel was told that the view of project management is to push single-dish applications now (especially NRAO instruments) and in 1995 focus on core interferometry applications. The direction of the infrastructure design has occurred primarily by consensus and persuasion rather than, for example, by explicit evaluation of options by a task force, critical evaluation of the decisions, and final decision by the technical leader.
Recommendations:
The most important recommendation is that the AIPS++ Project must have a clearly defined strategic plan -- to date, there appears to be none. We suggest that the AIPS++ Steering Committee and Project Management define specific goals for each phase of AIPS++ development. These goals should be based on functionality and content, not the calendar. The goals should be chosen for their strategic interest to the AIPS++ project, i.e., to demonstrate both the practicality of the object-oriented software development methodology and the potential for flexibility and new functionality as a result of the object-oriented paradigm. The Project has already identified several possible applications that could form the basis for the initial releases, such as single dish (on-the-fly) mapping, image deconvolution (various CLEAN algorithms), and general purpose image manipulation via the glish CLI. We suggest that one or two facilities of this sort be defined as the goal of each phase, and that the Project enlist the help of astronomers at the Consortium institutions and within their user communities to provide feedback.
On the more general issue of which of the general criteria discussed above should form the primary basis for defining a strategic plan, the Panel takes no position. No doubt more than one can be adopted. We observe, however, that the community appears for the most part to be willing to be patient with the project and accept the Project's priority and ordering decisions when such decisions are indeed made and communicated to them so they can know what to expect from AIPS++. Regardless of the criteria selected, the AIPS++ project management must identify the infrastructure elements that are really needed by the initial applications and get those developed rapidly.