Version 1.9 Build 1556

News	FAQ
Search	Home

Next: Environment Up: Data Repository Previous: Contributing to the Repository

Subsections

Repository Conventions

There are certain conventions which must be followed when contributing to the repository. In general, the makefiles which are used to build the data repository are much simpler, by design, then those used to build the AIPS++ system. This is possible because the data repository makefiles only build the repository on one system and the work they must do is much more limited than that of the AIPS++ makefiles.

Organization

All of the scripts, makefile includes, etc. which are used to build the data are kept beneath config directories. The directories which may be found beneath config are bin for executable scripts, libexec for scripts which are included rather than executed directly, and makedefs for the small bundles of functionality which are included by other makefiles to allow them to do their work. A config tree can occur anywhere in the data repository source tree. The principal is that the config files should be placed as far down on the source tree as possible. For example, a script would only be placed in the top level data/config/libexec directory if it is included by scripts in more than one of the broad top level categories, e.g. measures_util.g is used in both ephemerides and geodetic so it must be placed in this top level config directory. If it were used only in geodetic, it would be moved down to the data/geodetic/config/libexec directory. The reason for this rule is so that wherever possible checking out the lowest level config directory along with the top config directory and the source for a particular data component of the repository will be sufficient to build that particular component.

Using Scripts

In many cases, an AIPS++ table is simply copied from the data source tree to the repository. There are, however, instances where an AIPS++ table is created (and updated) from files retrieved from remote sites. During the build, all of the tools available in an AIPS++ installation will be available for the makefiles and scripts to use. However, the plan is that no compilation, e.g. of C++ code, should be necessary to build the data repository. It should all be done with scripts. Currently, it is done with Glish scripts. Should there be cases where this is impossible this rule can be revisited.

When using scripts which build a piece of data, the exit status of the script indicates the result. In particular, an exit status of 0 indicates that the script completed successfully and that the data item was updated, an exit status of 1 indicates that the script completed successfully but the data item was not updated, and finally, an exit status of -1 indicates that an error occurred during the execution of the script. The exit status of the scripts is used to generate logs containing only a list of the updates or errors which can then be used to keep tabs on the system without getting daily email messages.

Makefiles

In general, there is a one-to-one mapping between directories in the data source tree and directories in the data repository tree. The one slight deviation from this general rule is with tables. For tables, there is a replication of the table directory. So for example if we have a table called demo/3C273XC1 in the repository tree, this table would actually correspond to demo/3C273XC1/3C273XC1 in the source tree. The makefile, scripts (which need not be beneath a config/libexec directory), text files, etc. which install and perhaps create this table are found in the first 3C273XC1 directory while the 3C273XC1 subdirectory only contains the table which is installed in the repository.

When writing makefiles, there are some basic tools which should be used. $(RUN) must be used to execute the scripts used to generate a data component from scratch. This collects the exit status of the script and then uses the status to produce the update and error logs. Another tool which is often useful is $(MKHIER). When a fully qualified path is passed in, this will create the entire path if it doesn't exist, e.g. a makefile rule like:

   doit:
           $(MKHIER) /tmp/just/trying/it/out

would cause all of the directories in /tmp/just/trying/it/out to be created.

In addition to these tools, there are some variables which are useful:

$(DATA_ROOT) fully qualified path to the root of the data tree, set in each makefile.
$(DATA_THISDIR) fully qualified path to the current directory, set in each makefile.
$(DATA_INSTALLTHISDIR) the install point in the data repository which corresponds to the current directory.
$(AIPSROOT) fully qualified path to the root of the AIPS++ installation available for use in building the data repository.
$(AIPSARCH) fully qualified path to the architecture directory of the AIPS++ installation.

When writing makefiles in the data system, there are two targets which are recursively built these are build and install. build is the target which builds data elements from scratch. install is the target which moves the data element from the source tree to the mirror point for the repository. Obviously all makefiles will specify an install target, but not all makefiles will specify an build target.

Much of the standard work done by the makefiles is found in makedef components which are included in the makefiles for individual data elements. All makefiles include a standard preample which does the minimum setup required to find the makedef include files:

   DATA_THISDIR  := $(shell pwd | sed -e 's=^/tmp_mnt/=/=')
   DATA_ROOT := $(shell echo $(DATA_THISDIR)/ | sed -e '{s@/data/.*$$@/data@;}')
   include $(DATA_ROOT)/config/makedefs/basic

All makefiles should simply include this at the beginning. The basic makedef sets up all of the standard variables and rules for recursively building the build and install targets. Including the install.table makedef defines the rule which will copy a table from the current directory to the repository. It is assumed that the table is beneath the current directory and has the same name as the current directory. Including the install.fits makedef defines the rule which will copy all FITS files in the current directory to the corresponding directory in the repository. By convention, all FITS files have the .fits suffix.

So for example, the makefile to just copy a table which is not updated as part of the build process to the repository would look like:

   DATA_THISDIR  := $(shell pwd | sed -e 's=^/tmp_mnt/=/=')
   DATA_ROOT := $(shell echo $(DATA_THISDIR)/ | sed -e '{s@/data/.*$$@/data@;}')
   include $(DATA_ROOT)/config/makedefs/basic
   include $(DATA_ROOT)/config/makedefs/install.table

The makefile to install FITS files would look like:

   DATA_THISDIR  := $(shell pwd | sed -e 's=^/tmp_mnt/=/=')
   DATA_ROOT := $(shell echo $(DATA_THISDIR)/ | sed -e '{s@/data/.*$$@/data@;}')
   include $(DATA_ROOT)/config/makedefs/basic
   include $(DATA_ROOT)/config/makedefs/install.fits

and the makefile to build and install a table might look like:

   DATA_THISDIR  := $(shell pwd | sed -e 's=^/tmp_mnt/=/=')
   DATA_ROOT := $(shell echo $(DATA_THISDIR)/ | sed -e '{s@/data/.*$$@/data@;}')
   include $(DATA_ROOT)/config/makedefs/basic

   build:
           @$(RUN) glish IERSeop97.g

   include $(DATA_ROOT)/config/makedefs/install.table

A good example of how to create and update tables can be found in the scripts and makefiles beneath data/geodetic/IERSeop97. In this case, Glish is used to create and maintain the IERSeop97 table.

Next: Environment Up: Data Repository Previous: Contributing to the Repository Contents Index