How to Make an AIPS++ Release

B.E. Glendenning

 

Introduction

Making AIPS++ Releases is conceptually fairly simple:

    1. When a release is imminent, tell programmers to be cautious about new checkins.
    2. When the system appears to be sufficiently stable, do an uprev. This changes the major version number of the system, and copies off the base-release from the development system so programmers can get back to normal development.
    3. Make the *.tar.gz files (source and binary) for the base release. Do a test install.
    4. Test the base release (use the test installation), and acquire source-code changes for it. In practice it is unlikely that you will not find problems with the base release, so you should anticipate releasing the system at patch level 1.
    5. Create the patch, rebuild the .tar.gz files, and announce your system to the world.
    6. As new patches come in, repeat steps 4 and 5 as necessary.

Of course the devil is in the details. Much of the following could and should be automated, although it's really not all that bad.

The Uprev

Choose the installations from which you are going to generate the initial binaries. For example, in Socorro we used daily/sun4sol_gnu and daily/linux_gnu for AIPS++ 1.1 (beta) release. Make sure that the sources, generated documents, and binaries will be copied during the uprev for these installations by checking the .aipsrc variables inhale.base.code.preserve, sneeze.base.*.preserve, and sneeze.base.docs.preserve are not false (this is described under the inhale and sneeze sections of the System manual).

I recommend that you do a normal cumulative inhale before the uprev as a last chance to find problems - it is easier to fix problems with the normal ai/ao mechanism than by applying patches after the uprev. To speed this cumulative inhale along you may want to consider only building the optimized libraries during this cumulative inhale (by entering "defeat" in the debug column for all libraries in the site makedefs). Remember to set them back after the uprev has happened! To clean out any cruft, you might want to think about deleting libexec/* and all the binaries from architecture/bin/* before this cumulative as well. If you delete the binaries, make sure you do a "gmake allsys" in code/install/codemgt for all architectures to make sure that inhale, sneeze, and friends are available for the inhale! After the inhale completes, exercise it fairly thoroughly. It should at least be equivalent to a "stable" release before proceeding with the uprev.

To perform the actual uprev, as aips2adm on the AIPS++ master machine (currently tarzan), first manually edit the file ~ftp/pub/master/VERSION and increment the major version by one and change the minor version to a dash ("-"). For example, when doing an uprev from version 10 to 11, the VERSION file contained the line:

11.-.00 Fri 1998/10/02 00:30:21 GMT

Then either wait for exhale to run, or start one manually if you are in a rush. Make sure there is lots of free space available on the aips2adm filesystem. 500M is not enough, 1G is. If you need to free up more space, consider cleaning out the aips2-inhale messages from mail, and moving ~aips2adm/archive to another filesystem are good possibilities. (Actually, the size of the partition should be increased).

After the inhale, check that the base-release appears to be complete. In the above example, the base release would be a directory named base-11 under the top-level directory you have inhaled (that is, it is at the same level as code). For some reason the generated docs directory was not copied into the base release at the last uprev. If this happens again you should copy them (with tar | tar) into the base-release directory.

At this point, the base release is ready to use. You can have people start testing it while you prepare the binary release. I suggest you just make a symlink to the base release, perhaps something like /aips++/beta, and tell people to start using it (i.e. just source /aips++/beta/aipsinit.{sh,csh}) and start running it, while you prepare the binary release. Announce to programmers what policy they should adopt for patches, typically to send them to aips2-cop in a shar file rooted at code with a description of what bug or new feature the changed files implement.

Preparing the Initial Binary Release

Even if you intend to make a public release at patch level 1, it is useful to create an initial binary release to, if nothing else, test the release procedures. You should do the following tasks as aips2mgr.

I suggest that you first preserve the base release to enable you to recover from errors. You can do this as follows:

cd /aips++/daily/
tar czf /somewhere/base-11.original.tar.gz base-11

(This assumes that you are using GNU tar that can compress on-the-fly with the "z" option). This compressed tar file will require ~200M if the base release contains two architectures.

Next strip all the binaries for the base release. You must strip the binaries on the architecture that built them (i.e., you cannot strip Linux binaries on a Solaris system).

cd base-11/sun4sol_gnu/bin
strip * # Errors can be ignored

Next we need to make the .tar.gz files for the generated documentation (docs, as opposed to doc!) and the .tar.gz files for the binary architectures you are going to distribute. The docs are easy:

cd /aips++/daily/base-11/code
tar cf ~/ttmp/docs-11.000.tar ../docs
gzip -best ~/ttmp/docs-11.000.tar

Note that these .tar.gz files are made relative to the code directory, and also that it is best to use the –best option with gzip for things that users will ftp to minimize their download time (even though it takes considerably longer than compressing at the default level). The docs .tar.gz file will require about 30M.

Preparing the .tar.gz files for the binaries is almost the same:

cd /aips++/daily/base-11/code
tar cf ~/ttmp/bin-sun4sol_gnu-11.000.tar \
--exclude-from ~/ttmp/excludesun4sol \
../sun4sol_gnu/{aipsrc,bin,doc,lib,libexec,makedefs,VERSION}
gzip -best ~/ttmp/bin-sun4sol_gnu-11.000.tar

Here the file excludesun4sol contains a list of files that we do not wish to include in the tar file. The exclude-from switch is a GNU tar extension. The contents of this file for the 1.1 beta release are given in the appendix.

Repeat the above recipe for all binary architectures you wish to support. The exclude file, if any, will have to be different for each architecture since the prefix (e.g., ../sun4sol_gnu) differs for each architecture.

Next we need to create the ftp directories to hold the distribution files, and copy the binary and source files into it.

cd ~aips2adm/pub/RELEASED
mkdir aips++1.1
cd aips++1.1
cp ~aips2adm/pub/code/*11* . # This copies in all the sources
cp ~bglenden/ttmp/{bin,docs}*.tar.gz . # Copy binary/docs
cp /aips++/daily/base-11/code/VERSION .
cp /aips++/daily/base-11/code/install/configure .
cp /aips++/daily/docs/project/releasenotes11.ps . # daily/docs!
gzip –best releasenotes11.ps

Edit a file called README to just tell the readers to download the release notes and follow directions. Later you can edit the README to contain late-breaking information.

Have some people (preferably naïve!) make a test installation to test both the instructions and the system. A common problem is that additional shared libraries might have to be distributed with some architectures. In this case, just put the extra .so files in the appropriate architecture’s lib directory, and remake the binary .tar.gz file for that architecture.

Creating a Patch

Note that AIPS++ patches are cumulative, so if this is not the first patch you should add the new files to the existing patch directory. It makes life much easier for you if you insist that patches be submitted as shar files created with the –B option rooted at the code directory. Despite your insistence, some people will not follow directions.

OK, suppose that you now have a set of "patch" files under a directory hierarchy rooted at code. As a sanity check it is useful to check that all the files have corresponding files in the standard code directories. This can happen if the shar files you are sent start a level too high or too low. You can do this as follows:

cd …/patches/code
find . –type f –print > /tmp/files
for i in `cat /tmp/files`
do
if [ ! -f /aips++/daily/code/$i ] ; then echo "$i" ; fi
done

This will list all the files in the patch that do not have a corresponding file in the master. Of course these could be legitimately new files, but this is a good check to make. Make sure there is a VERSION file in the patch code directory (i.e. parallel to trial, aips, etc.), and increment its patch number by one and set the time.

cd patches/code/
tar cf ~/ttmp/11.000.01.tar .
gzip -best ~/ttmp/11.000.01.tar

Now we have to apply the patch to the base-release and rebuild the binaries and documents. If the rebuild is successful then we will "publish" the patch. There can be a problem with tar refusing to write a new source file over an older one if the old file is read-only, as the AIPS++ sources generally will be. A trivial script applypatch will overcome this problem. It can be found in ~aips2adm/pub/patches.

cd /aips++/daily/base-11/
. aipsinit.sh
applypatch …/ttmp/11.000.01.tar.gz

OK, now that we have applied the patch, all we should have to do is remake everything.

cd code
gmake allsys > ../make.log 2>&1 &

If the patch was very large you might want to force a complete rebuild (removing all the libraries is an easy way to do this) before issuing the gmake. (Of course if you have put any extra required shared libraries in lib you should not remove them!). You need to rebuild all architectures of course.

Check the log carefully. If the build was not clean, you need to correct the patch until it does build cleanly. After a clean build, you need to get the patched base-release should be well tested internally. After the patch passes, you should copy the .tar.gz file to ~aips2adm/pub/patches. This file will be used by any sites that want to patch their base-release directly from the patch. This will not be a common occurrence.

Updating the Binary Release

We need to update the .tar.gz in pub/RELEASED. First copy the existing files to a safe place. I suggest making a directory called .old (so that it will be hidden) and make a new subdirectory every time there is an update. For example, before we update for patch #1:

mkdir .old .old/11.00
mv * .old/11.00

Next regenerate the .tar.gz files for all the sources (i.e. with the patch applied).

cd /aips++/daily/base-11/code
cp VERSION ~aips2adm/pub/RELEASED/aips++1.1
for i in admin aips atnf bima contrib dish doc drao install \
nfra nral nrao synthesis tifr trial vlbi
do
echo $i # Follow progress
tar cf ~aips2adm/pub/RELEASED/aips++1.1/$i-11.000.tar $I
gzip –best ~aips2adm/pub/RELEASED/aips++1.1/$i-11.000.tar
done

Regenerate the docs and binary .tar.gz files the same way you did before. As a sanity check, check the sizes of all the .tar.gz files with their previous size (i.e. in the .old directory). Investigate the cause of any major change, especially if the newer file is smaller than the previous one!

Further Releases

To create releases beyond patch level #1, you merely have to repeat the procedures outlined in "Creating a Patch" and "Updating the Binary Release" as described above.

Appendix: Excluded Binaries

For the 1.1 beta release of AIPS++, the following binaries were excluded from the binary .tar.gz file for both (sun4sol and linux) architectures are as follows (except for linux linux_gnu replaces sun4sol_gnu).

../sun4sol_gnu/bin/otfgridder
../sun4sol_gnu/bin/holog
../sun4sol_gnu/bin/sdimaging
../sun4sol_gnu/bin/mdl2csm
../sun4sol_gnu/bin/gbtmsfiller
../sun4sol_gnu/bin/ms2scn
../sun4sol_gnu/bin/j2convert
../sun4sol_gnu/bin/ms2fits
../sun4sol_gnu/bin/wsrtfiller
../sun4sol_gnu/bin/fits2ms
../sun4sol_gnu/bin/gbtfiller
../sun4sol_gnu/bin/fillpointings
../sun4sol_gnu/bin/gbtlogfiller
../sun4sol_gnu/bin/sddfiller
../sun4sol_gnu/bin/vlafiller
../sun4sol_gnu/bin/atabd
../sun4sol_gnu/bin/vlacalflux
../sun4sol_gnu/bin/vlacalcat
../sun4sol_gnu/bin/gbtspec2table
../sun4sol_gnu/bin/duplicates
../sun4sol_gnu/bin/unused
../sun4sol_gnu/bin/used
../sun4sol_gnu/bin/reident