News
| Getting Started | Documentation | Glish | Learn More | Programming | Contact Us |
|
| Version 1.9 Build 000 |
News
|
This is a working proposal for the AIPS++ information system. It is a live document that is still being refined by the Documentation Group as the design and priorities evolve. Comments from the AIPS++ community are therefore especially welcome.
This section lists what is needed in the Information System.
The AIPS++ mission statement states that AIPS++ must be programmable to a high degree. It must be as easy as possible for as many people as possible to realize (new) ideas with AIPS++.
This implies that the system should not be designed for too narrow a purpose, that anybody is a prospective user of any part of AIPS++, and that this philosophy must be supported by the documentation.
But there will be a wide variety of levels at which users will seek programmability in AIPS++.
Initially, most astronomers will want simply to know what tools and techniques are already present in AIPS++. They will not care what language AIPS++ is coded in, or that it is object-oriented. They will first ask what can I do now in AIPS++, and how do I do it?
For many users, programmability in AIPS++ means flexibility in analyzing images once they are constructed satisfactorily by standard means. These users will not wish to delve into the AIPS++ classes. They want tools that will allow them to do image calculus flexibly without detailed knowledge of the object-orientation within AIPS++.
Only a few users (initially) will want to change how data are calibrated and/or turned into images, to experiment with new algorithms for calibration and image-making as well as in image analysis. This small but ultimately crucial group will need documentation of the AIPS++ classes and libraries along the same lines as those needed by the full-time AIPS++ programmers.
Whether AIPS++ succeeds in improving programmability will depend on:
AIPS++ will have all previously known categories of users of astronomy software: novices, experts, application programmers, system programmers, and system managers. But our documentation must no longer target them separately. Instead, we must cross the traditional boundaries of user categories and in doing so help individual users to do the same.
Astronomers need:
The natural directory tree structure for such documents will be based on application arenas that are meaningful to astronomers.
Programmers need:
The natural directory structure for these documents will be based on the AIPS++ class library and the organization of the application code by place of origin within the consortium. Note that this not necessarily, indeed will often not be, the same tree as the application arena tree. Cross-linking the documentation in the same way as the final applications cross-link the AIPS++ classes will be an important step in building a viable information system.
System managers need:
The system-management have historically been quite separate from those dealing with astronomical applications that are run once the code has been installed. This may continue to be true for AIPS++.
All three categories of user need on-line descriptions of the problems (bugs) currently known to afflict the software.
The traditional documentation model has been off-line printed manuals and on-line plain text. Often, one set of manuals was written by system builders, another by expert users. The paper documents were hard to create and to keep both up-to-date and compatible with the on-line text. Their bulk inhibited people from reading them, especially if they were obsolescent or the information in different manuals were incompatible! Because the information was on several media and in several formats, it was hard to link it all together. Discrepancies tended to increase in time.
From these experiences and the mission statement, we set the following goals for the AIPS++ documentation system:
A key to success in both of these areas will be how well we learn to use hypertext.
A hypertext file structure combined with a versatile browser is a powerful way to realize an information network with the desired properties. Hypertext technology was used successfully at NFRA to convert the old-fashioned NEWSTAR documentation quickly into a good-looking and heavily used information network.
Hypertext solves many of our problems because it recognizes that connections between pieces of information are at least as important as the information itself. It makes it easier to incorporate new documents and to update old ones. It allows new documents to link directly to existing ones rather than duplicating them or, still worse, contradicting them. It also enables one to reorganize an existing body of information by rearranging its links, or by opening new paths through it.
Coupling hypertext tutorials to form-based software interfaces also opens an arena of interactive documentation. Short demonstrations of the software using sample databases can be built into documents where appropriate. Future machine-readable versions of lectures in data-processing workshops could contain interactive "examples".
Finally, the World Wide Web lets us expand the outer boundary of the network via links into other, related documentation systems.
Many different users with different goals may use the same hypertext information base, each in his or her most appropriate style. Multiple links can help users to navigate documentation in ways that match their own interests or level of understanding. In principle, we can do this now by hopping between pages and chapters in traditional (linear) documents, if these documents are well indexed.
A good hypertext system informs and accelerates this process by making it more interactive. Users can plunge into such a system from a variety of starting-points and still get relevant information quickly.
A bad hypertext system can lose users in a maze of linkages, however! Users may get "lost in hyperspace" once freed from the linear structure of a printed manual. When a hypertext system contains a large body of information, it is essential to provide navigational aids such as maps, indexing tools and traditional containers such as tables of contents.
Hypertext "trees" can be climbed and descended easily if they are not too tall. Hypertext "webs" full of powerful but ad hoc leaps may be exciting to create, but can ensnare users and make sure they go nowhere! Questions like "Where am I?", "Is this something for me?" and "Where do I go next?" must be answered readily by the system. We believe that trees and webs will both have their place in the AIPS++ Information System. Identifying where to use each to best effect will be an important part of the final design.
The AIPS++ Home Page is intended to address two questions:
The second group now includes direct links to
The links from the Home Page to the rest of the AIPS++ documentation are currently ad hoc. Much of the current documentation was in place before any plan to organize it as an on-line system. AIPS++ workers have become used to finding it directly from the Home Page. This practise should end. The On-Line Documentation Menu should be the prime organizer for detailed documentation. The Home Page should be simplified to address its two main functions more clearly and with fewer sub-menus.
These are currently TeXinfo documents about basic aspects of the AIPS++ system. They are all available as PostScript. Some are also in (experimental) hypertext form served from Texinfo sources:
The first two documents are kept up-to-date, but the others are outdated and should be replaced as soon as possible. All should be in native hypertext (HTML) form.
These are an inhomogeneous collection of documents about the design of AIPS++. They are simple-text, TeXinfo and LaTeX documents available either as simple text or as PostScript files:
The distinction between Notes and Memos is unclear: some of the most useful documents are actually Notes. Anyway, most of these documents are mainly of historical interest.
Programmers document their code according to certain rules, and the documentation extractor generates information for a programmer's reference manual directly from that code. This information is currently stored as HTML files under the /aips++/docs/aips/implement directory, and is referenced directly by the top levels of the WWW-visible documentation system.
The AIPS++ Programming Manual lists standards and guidelines for coding and documentation. The rules are still subject to change and the relevant sections of the manual will shortly be updated. Right now, any new code is inspected by a code-cop to see whether the code and the internal documentation conforms to current standards.
Many developments in AIPS++ start with E-mail discussions, and quite often remain hidden in that corner for a long time -- sometimes for ever. Ideally all relevant information should find its way into one of the more permanent information repositories. Unfortunately, this does not always happen, and it may sometimes be worthwhile to browse through the E-mail archives.
AIPS++ initially adopted TeXinfo as a documentation format, but HTML combined with WWW-compatible viewers is better. Even the TeXinfo documents that were automatically converted into HTML documents are much more readable than the originals.
HTML is an SGML-compliant language that leaves as much control as possible over the details of document formatting in the hands of the browser. This is not everyone's favorite philosophy but it does impose some uniformity on documentation from a wide range of different sources, as we may encounter in AIPS++. The main technical deficiencies of HTML are fluidity in, and gaps in, the current (Level 2.0) language standard. Some essential tools for technical documentation, such as mathematical formalism, captioned images and resizable tables, are missing. The HTML Level 3.0 standard promises much of what we need, however. (Alan Bridle's memo "HTML for AIPS++ User Documentation" has details).
CERN's Arena browser displays HTML Level 3.0 (mathematical formulae, resizable tables, etc.) now, but is a test-bed beta release that lacks many features beloved by users (index searching, hot-lists). Its default colors and fonts are also barely legible. It does not support forms. It is useful for testing the proposed math and table capabilities of HTML and possibly for validating HTML files (it has a built-in error reporter and editor) but is not yet ready for workhorse duty.
Netscape Navigator is the favored browser du jour of the World Wide Web. It is full-featured, has attractive displays and a user-friendly caching system. It also responds to some legal HTML Level 2.0 tags that are not yet realized predictably in NCSA Mosaic ("mailto:") and produces better Postscript output from its file-printer. Its popularity should not seduce us into using its non-standard HTML extensions in AIPS++ documentation, however. The initial version (1.1N) is free to individuals and educational establishments and for trial by other users.
NCSA Mosaic is licensed to Spyglass for commercial development. Free versions are still available from NCSA but will lag the Spyglass product. Mosaic is buggier than Netscape Navigator as a level 2.0 browser, and some of its bugs are noticeable in AIPS++ use. (You cannot, for example, read a local file from a directory tree containing "aips++", or read the aips++.html home page in some versions.) Its Postscript printer has problems aligning italic, bold and normal text that are solved by Netscape Navigator and Jan Karrman's HTML-to-PostScript filter html2ps. It does not yet support the "mailto:" URL predictably.
AIPS++ should plan to support any widely-available browser that fully realizes the HTML Level 3.0 standard when this is stabilized, so we can display algorithmic aspects of the code clearly to astronomers. Right now, Arena falls short by lacking important features. We should therefore support the area of overlap between Netscape Navigator, NCSA Mosaic and the HTML 3.0 standard and hope that this area soon expands to meet all of our needs. We should also test examples of our HTML products for portability to all three browsers mentioned above.
We will need several tools to help users find their way through a big hypertext system.
One tool is our own HTML style. At high levels in the documentation system, we should resist the temptation to create hypertext "webs" in favor of clearly-structured "trees". We should ensure that low-level documents, which may benefit from a smaller-grained web structure, contain clear links to "container" or "contents" documents higher in the system.
It may also help to have a graphical mapper to show the main links within our hypertext system. This would show where the current document is situated in the system on a linked-box map. It would also let a user jump across the system by clicking on this display. This could let a user who has become "lost" reach a "contents" document nearer their topic of interest, whether or not the currently open document contains a suitable link. Darrell Schiebel's summer student, Ed Gray, has made progress on such a tool as a graphical class browser. This should be evaluated in real-world use and perhaps be generalised to become a hypertext link geographer.
We also need an indexing tool to help users find information from a WAIS-like search of the contents of the HTML files. Ideally this would respond to a keyword search by returning an HTML file that lists all possibly relevant links (up to a maximum number set by the user).
Hypertext information systems do not eliminate the need for printed documentation. People want the option to read documentation when away from the computer, and on-screen presentations of information also have well-known limitations of legibility, field-of-view and eyestrain relative to printed manuals. We need a good tool for printing multiple hypertext files as collated, indexed, paginated, single paper documents. There are surprisingly few such tools now available that can read HTML.
There will be two main sources of information in the AIPS++ documentation system.
Darrell Schiebel's home-brewed documentation extractor now provides this part of the information system. It is an integrated part of the system generation procedure.
Much of the astronomical tutorial information, in the arena of synthesis imaging, may be harvested from the TeX input files for the NRAO Summer Schools, which are available to the AIPS++ documentation effort. Integrating these with our documentation system may indeed be an immediately attractive feature to users. There is no comparable motherlode of tutorial information for the single-dish arena, however.
The documentation system must be integrated well with the AIPS++ user interfaces. Note that these interfaces are plural, as plainly demanded by the mission statement.
An HTML-based documentation system will couple particularly closely to a user interface based on HTML forms, as currently planned for some applications. A form-based interface is really an active part of the documentation that calls a script to pass parameters to the code as if these parameters were entered on the command-line.
The HTML forms in this interface will link down to details about input parameters in the reference information and up to discussions of astronomical concepts and strategies in the tutorial information. Context sensitivity is built-in, as it is provided by the user simply by reading down a particular form. This interface will be the easiest one to connect to the rest of the documentation system. It will also be the first to be outgrown by novice users on the way to becoming experts!
We can expect the AIPS++ wizards, or at least those who think they are wizards, to prefer the command line interface. We need to connect them to an appropriate part of the documentation system as soon as their wizardry encounters one of its limits.
At its simplest, we could trigger the document browser to display documentation on appname in response to appname ? or some such from the command line.
Possibly demi-wizards would like an option to keep a browser window open so that it can bring up documentation for appname whenever they type a valid appname at the command line?
It will be important to ensure that documentation for all interfaces, however it is presented to the user, derives from the same source information as documentation for all the others, to ensure consistency. The multiple interfaces required by the AIPS++ specifications must not become a new source of user confusion, like the multiple media that once housed documentation!
An important part of the documentation is what we do when things go wrong, either because a user specified inappropriate input parameters or because they triggered a fatal bug in the code. Wherever possible, we should avoid presenting the user only with an obscure message from an operating system utility that they may never have seen before. One goal should be to recover from most errors by popping open a browser window that contains a document with advice on how to avoid the error in the user's next attempt.
When all else fails, we need a way for users to report difficulties or bugs to the AIPS++ team. The report should be acknowledged and recorded automatically, then (ideally) answered in a reasonable time. We will need a mechanism to classify the bugs, assign them for response, and make sure the response occurs in a time commensurate with the priority and degree of difficulty of the fix. It is critical to recognize that documenting the current bug collection is part of documenting the project. It helps if users can consult a database to see what bugs have been reported for a particular application, and what the status of fixing them is. We should plan to build such a database from bug reports. Both NFRA's NEWSTAR and classic AIPS have experience in this area, and we should draw from that.
Some bugs are user errors. At a certain level, any recurrent user error points to a failure of the documentation. The part of the bug collection that is labeled "user errors" is therefore also worth keeping. It should be shown to people who write the code comments and the tutorial information in the associated application arena!
Hypertext lends itself well to supporting dictionary/glossary functions. A glossary is needed to provide a set of definitions that standardizes usage throughout a documentation system and, ideally, the code. Alan Bridle is compiling a glossary of astronomical and computing terms used by AIPS++ that should be linked to all tutorial levels of the documentation. It will be important to ensure that these terms are used consistently throughout the AIPS++ documentation, and tools for providing links to it from automatically-generated HTML would be useful.
A good hypertext documentation system for code requires that the documentation corresponds as closely as possible to the code it informs about. Only then is it possible to maintain code and documentation simultaneously and so ensure that the two are always in sync. Therefore, the system will have the following components:
In the current AIPS++ system the close correspondence between code and documentation is implemented by two parallel directory trees with essentially the same structure.
...aips++/code/ ...aips++/docs/
package*.h package*.html
package*/ package*/
implement/ implement/
module-group*.html
module*.h module*.html
module*/ module*/
file-group*.html
file*.h file*.html
file*.cc
Module documentation is handled in almost the same way as the class-level documentation. For each module a header file is written which is then run through the extractor to produce the HTML file. The extractor automatically adds a "table of contents" which consists of hypertext links to the lower-level HTML files.
Package documentation has not yet been written. I guess that it will initially be created by hand (as a tutorial-type HTML file) but in the end we should write package header files and extend the extractor so that it can automatically append index information, analogous to the module files.
Both the reference- and tutorial-type HTML files end up in the same directory tree. This reflects the basic idea that together they form a single information system. The only clear difference is the way in which they are generated: Reference files are extracted from the corresponding files in the "code" tree, whereas tutorial files are checked in as HTML files in the source tree "code/doc" and then moved to the corresponding position in the "docs" tree.
Otherwise, reference and tutorial info are not clearly different, and the boundary may shift (as for the package files). Some reference information is very tutorial-like (such as the synopsis sections) and may even evolve into real tutorials as the code evolves. On the other hand, some reference-type information might very well be supplied in the form of tutorials, or tutorials may be formatted as in-code comments to make their extraction and maintenance automatic.
The top level seen by the astronomers will contain broad-based tutorial documents that describe large application arenas. The main application arenas unique to synthesis imaging might be:
Single dish application arenas might be:
These arenas might be in common:
There would be one "tutorial" per application arena. These arena tutorials would be generic documents, equally applicable to Classic AIPS, AIPS++, or MIRIAD. Much material for such tutorials can be found in the NRAO Summer Schools and similar workshops.
Below the tutorials we need a documentation layer that interfaces broad principles to specific applications. In Classic AIPS these were EXPLAIN files, one per application. explain.html in AIPS++ documentation should cover how the fundamentals described in the arena tutorial are realized in a specific AIPS++ application. It will focus more on the specifics of the application than on general astronomical background, because it can use hypertext links up into tutorial.html.
A crucial function of explain.html is to be an astronomically- oriented pointer to the list of input parameters that the user should provide when running the application. explain.html point to all possible user inputs, but should distinguish those that are mandatory from those that are optional, and suggest defaults, short-cuts and astronomically-oriented strategies for assigning inputs.
We will also need one file per input parameter (or per small group of connected input parameters) to tell the user clearly what each parameter describes, in what units, and with what format and range restrictions (if any). These inputs*.html files will be very numerous. To be useful, they must reflect the currently compiled code seen by the user. Thus they are another type of reference information that must be generated by an automatic extractor from comments in the code and updated simultaneously with relevant changes to the code. The pointers to the inputs*.html files from an explain.html should be more stable in name and number than the content of the inputs*.html, and should be maintainable "by hand" for the major application arenas. We may however want to consider maintaining them automatically in some cases.
Here is a possible file structure for documentation on the deconvolution arena, containing application tools for CLEAN and for the Maximum Entropy Method (MEM):
...aips++/docs/
deconv/ for all deconvolution docs
tutorial.html intro to deconvolution in general
clean/
explain.html intro to AIPS++ CLEAN
form.html form-based interface to CLEAN
show.html interactive demo of CLEAN?
inputs*.html auto-generated
mem/
explain.html intro to AIPS++ MEM
form.html form-based interface to MEM
show.html interactive demo of MEM?
inputs*.html auto-generated
...
This outline also recognizes two other types of file that are suited to this part of the AIPS++ directory structure.
form.html carries the form-based user interface for the application. Its design requires close cooperation between an astronomer-user and the programmer. It will be tightly linked both to the inputs and the explain and tutorial files. It will likely be used only by novice users, but they will need access to guidance of all kinds from the documentation!
To clarify what sorts of information would sit where in this structure, let's look at what might be in the description of a simple CLEAN tool.
The user would probably start at the form-based interface to the CLEAN tool. This would contain HTML links to the /clean/inputs*.html files wherever input parameters are requested. Everywhere the form asks for an input parameter we must a link to a description of what that parameter specifies within the code, in what units, with what format, and with what range restrictions. These descriptions must come from the programmers, because they may change any time the code is changed. We must get them from comments in the code and update them automatically. The input form itself will be hand-generated but it must be able to specify fixed links to an inputs.html layer that is regenerated whenever code that specifies the inputs is changed.
The form-based interface would also contain links into the /clean/explain.html file. For example, suppose we allow the user to specify a optional CLEAN "box", which can be defaulted to some fraction of her image. When should she be careful about specifying the shape and size of the box? That's astronomical documentation that must be provided by explain.html. Suppose the CLEAN box is initially a square. inputs.html then tells her that the input parameters are xmin, ymin, size in pixels and they must be integers. Later we allow the box to be rectangular with four inputs, specifying xmin, ymin, xmax and ymax. Its astronomical purpose doesn't change, so explain.html stays the same. But inputs.html must change as soon as the rectangular method appears in the compiled code at her site.
explain.html gives an astronomical context for all the input parameters that she could specify in order to do a CLEAN deconvolution in AIPS++. It can be thought of as an introduction or preface to form.html, though they should probably be separate files. It will also review the approach taken by the program to finding CLEAN components, introducing algorithm-specific concepts like the loop gain, CLEAN box, etc. But it does not dwell on the general principles and limitations of deconvolution. For these it links to tutorial.html above it. This level may be where the user is reminded that if her target source is really big and smooth she might want to use MEM not CLEAN anyway!
show.html is included in the scheme to recognize that we may also use the AIPS++ forms interface to give novice users interactive demonstrations of data processing. These would use standard data sets with inputs selected by the users or by the tutorial or explain files. (These may not be part of an initial AIPS++ documentation system but may be worth exploring once the more basic structure is fully in place.)
To make the above scheme work we need a subdirectory per application task. It is in this subdirectory that the information system must make an interface between tutorial information such as explain.html that need revision on a moderate cycle and reference information like inputs*.html that must be revised whenever code that accepts user inputs is changed.
There will eventually be a tutorial.html file for every code grouping of some size and for every astronomical application arena. Most of these will be written by expert users.
Most scientists who will contribute such information to AIPS++ soon are more likely to know TeX or LaTeX than to know HTML. Until the HTML Level 3.0 standard emerges fully, TeX and LaTeX will also be the most effective tools for generating highly mathematical (symbol-rich) text. Even when HTML 3.0 browsers are common, we can expect to get much more input from astronomers as LaTeX than as HTML. simply because many use LaTeX to submit scientific papers to journals electronically or as paper "manuscripts". We cannot expect "volunteers" to learn a new markup language just to provide AIPS++ with documentation. Instead, we must expect either to translate from TeX/LaTeX to HTML 3.0 or to incorporate their documents as PostScript files.
Converting TeX-based input to HTML 3.0 will eventually be preferable. Links between the tutorials and the rest of the system can then become two-way and so be more effective. Internal linkages within the incoming document can also make it easier to browse on-line.
Converting whole TeX and LaTeX documents to PostScript files lets them be browsed by a helper application such as Ghostview and be referred to by HTML documents. But the PostScript documents cannot directly refer back (re-link) to the HTML-based system. This route is suitable for "one-way" access to entire symbol-rich background documents, but will not let the tutorials play the role envisaged for them above. This situation may change if Adobe PDF (an enhanced PostScript that will support hypertext links) becomes common and particularly if it becomes well-integrated into tools used by the World Wide Web.
The bottom line is that we do not have the technology to start on the tutorial.html files satisfactorily with either option at the moment. We should not do more than prototype them in a few demonstration arenas (or arenas that do not need symbol-rich documentation) until we either have HTML 3.0 or we know the response of WWW browser-writers to Adobe's PDF tools.
explain.html and form.html files are also written by hand but contain pointers for detailed contact with the code via the inputs*.html files. As applications are developed, there should be close contact between an astronomer-user and the programmer about the structure and content of these files so that they make sense to both. We should concentrate on forming such astronomer-programmer teams for prototype applications and generating explain and form samples.
Most AIPS++ users will be astronomers who will not initially need to know, nor will they care, about the class structure or even the object-orientation in the code. They will only ask about the precise meanings of, formats of and restrictions on, inputs that they have to provide to run AIPS++ applications. Later, if they want to change or expand tools for themselves, they will need class-based documentation.
The only way to merge both sorts of documentation into one system is to have them "meet" at the application level. In the scheme presented here, this meeting occurs where the explain files provide pointers into the inputs* files. We should prototype this linkage between the reference information and the tutorial information for a few applications as soon as possible. The success of the documentation system will depend significantly on how well we automate it.
We will need a tool that pipes information from the class-based reference information into the inputs*.html files for every application. We will have to establish and maintain a web of connections between the inputs*.html files and the present class-oriented documentation. This web must be able to link any and all changes to code that collects user inputs promptly to all relevant inputs*.html files. The web must be maintained as automatically and rigorously for the documentation system as it is in the code compiler. Much of the success or failure of AIPS++ will depend on the still-to-be-written tool that we will use for this. We should give high priority to designing this tool and using it for the prototype linkages referred to above.
The new tools that are needed by the AIPS++ documentation system are listed below according to our view of their priority:
Wherever possible, we should use existing tools rather than invent our own, of course. If anyone knows of tools that can fill these needs, please tell us!
There is also a is large body of "old" documentation, namely the stuff presently in /aips++/docs (and associated) directories. What should be done with it?
Further developments depend on the detailed reactions of users and project workers to this outline and to the definition and scheduling of action items by project management.
The first design for the documentation system was suggested by Friso Olnon (NFRA) in September 1994. He discussed the priorities mainly from a programmer's perspective and made the case for a hypertext system (in a hypertext memo now superseded by this document). In March 1995, Alan Bridle (NRAO) became an editor for astronomical information in AIPS++; he produced a similar plan for the scientific documentation. Olnon and Bridle merged their plans into the first draft of this document.
Since early 1995, Darrell Schiebel (NRAO) has worked on tools for automatically extracting and navigating HTML documentation directly from the C++ code.
In August 1995 Wes Young (NRAO) took responsibility for further design of the programmer documentation. Alan Bridle (abridle@nrao.edu), Darrell Schiebel (drs@cv3.cv.nrao.edu) and Wes Young (wyoung@nrao.edu) were designated as the AIPS++ Documentation Group. This document became the repository for their conclusions as they relate to the overall design of an AIPS++ information system.
You could now go back to the:
Copyright © 1995,1999,2000 Associated Universities Inc., Washington, D.C.
abridle@nrao.edu