Time and time again, we have seen a resistance to the use of Autoconf because of its (seemingly) arcane language constructs. While macro languages are difficult to use and sh(1) syntax is not immediately intuitive, neither tool can be considered arcane. Indeed, sh(1) is a much more powerful language than most other shell scripting languages, but because of its unique syntax, people tend to shy away from its use.

Further complicating the issue is the complexity of Autoconf in general. It contains many, many macros, and it defines rules for the order of executing those macros. In order to use Autoconf effectively, build system developers must read a fair amount of documentation (all of which is readily available). In our experience, however, many people feel that development build system software should be immediately obvious^[1] and should not require much effort to learn.

As a result of these issues, we have attempted to make Doozer++ easier to use than “raw” Autoconf-based <>configure scripts. Doozer++ macros hide some common details that often trip up configure script authors. Many utility macros are provided to reduce code duplication between configure.in files. For example, code for verifying that an existing installation of a software package meets a version requirement is provided. The code for performing the check is modularized to separate the common tasks involved with this. Namely, Doozer++ provides macros comparing two version numbers, acting on the results of the comparison, and caching the results to speed up future tasks. Each macro is used as the basis for the next so that users have the freedom to choose how they want their code to behave. We feel that this offers a level of flexibility not available with Autoconf 2.13.

Because VR Juggler has always been a cross-platform tool, we have had the need for a cross-platform way to build it. The VR Juggler 1.0 build system achieved this to a limited degree, and Doozer++ goes a step further toward true platform independence. At this time, Doozer++ makes use of software utilities found on all modern operating systems. The list of utilities includes m4(1), sh(1), Perl, Autoconf, and GNU make. We have avoided the use of platform-specific tools because it is all too easy for the platform-specific parts of a build system to get out of sync with each other. Furthermore, we have tried to avoid the use of platform-specific code whenever possible. This is a more difficult goal to achieve, but later sections will address the extent to which platform-specific code can be reduced.

As a cross-platform C++ library, VR Juggler must be compatible with the prevailing C++ compiler on a given platform. In the case of IRIX, that is SGI's MIPSpro Compilers. Similarly, the use of Microsoft Visual C++ is crucial on Win32 platforms. Open source operating systems such as FreeBSD and Linux use GCC, so we have not ignored that compiler whatsoever. Due to our limited needs, however, we have focused on compilers for the C, C++, Java, and IDL languages. Nothing prevents Doozer++ from being extended to allow compiling of code written in FORTRAN, Ada, Pascal, etc.

In Doozer++, an m4 macro sets up the basic platform-specific pieces. User-level code (in an Autoconf configure.in file) then uses that information to execute other macros that pick the appropriate compiler. This builds upon the foundation provided by Autoconf for detecting installed programs, but it goes further by allowing users to associate one or more compilers with a given platform. The end result is a “fallback” system wherein users specify the preferred compiler and zero or more alternatives if the preferred compiler is not available. To achieve this flexibility, the Doozer++ m4 macros must have no compiler-specific bits.

Language freedom is slightly more complicated than compiler freedom. As will be discussed more fully in the following section, Doozer++ separates its work into two pieces: configuration and compilation. During configuration, the tools for compilation are chosen; during compilation, the chosen tools are put to use.

The primary project building tool is GNU make. We chose GNU make initially because of its portability and because it offered useful features over most basic make(1) implementations. Other implementations, such as BSD make, offer even more features, but they are not as portable as GNU make.

Because Doozer++ uses GNU make, users must write makefiles. Doozer++ provides a number of modular makefile “stubs” that collect common functionality. In particular, these stubs provide support for compiling code written in Java, C, C++, and IDL. Adding support for other languages can be done within Doozer++ or within user-level makefiles.

The design of Doozer++ involves a distinct separation between the tools used and the tasks performed. As a result of this separation, there exists the possibility to extend Doozer++ to use tools other than Autoconf and GNU make. Moreover, different combinations are allowed. For example, we have found that make(1) does not deal well with compiling Java code in general. A more appropriate tool for this task has come out of the Apache Jakarta Project called Ant. Within Doozer++, we could make use of Ant instead of or in addition to GNU make to compile Java software. Nothing restricts users to Autoconf and/or GNU make.

The basic idea behind the separation is the following: detect and configure the static information needed to execute tools that compile a (potentially complex) software system. This leads to a two-step process: configuration and compilation. These steps are described in the following subsections.

#if defined(HAVE_HASH_MAP)
#include <hash_map>
#elif defined(HAVE_EXT_HASH_MAP)
#include <ext/hash_map>
#elif defined(HAVE_HASH_MAP_H)
#include <hash_map.h>
#else
#error "std::hash_map is not available with this compiler"
#endif

#if defined(__GNUC__)
#  if __GNUC__ < 3 && __GNUC__ >= 2 && __GNUC_MINOR__ >= 95
#     include <hash_map>
#  elif __GNUC__ >= 3
#     include <ext/hash_map>
#  else
#     include <hash_map.h>
#  endif
#elif defined(__MSVC_VER__)
#  if __MSVC_VER__ >= 7
#     include <hash_map>
#  else
#     error "std::hash_map is not available with this compiler"
#  endif
#elif defined(__sgi__)
#  include <hash_map>
#else
#  error "std::hash_map is not available with this compiler"
#endif

There are many long-term goals for Doozer++ that are beyond the scope of this document. As should be evidenced from the previous sections, however, some near-term goals include the use of project builder tools besides make(1).

Due to the size of the VR Juggler 2.0 source base (nearly 500,000 lines of C++ and Java by one estimate), it is important that the complexity of compiling be centralized rather than spread out over the entire source tree. Centralizing the complexity allows most of the work to be done once, the results of which can then be reused by the rest of the build code. This is the key tenet for the other goals.

Centralization of complexity allows the VR Juggler build to follow the goal of Doozer++ wherein tools and tasks are separated based on what they do. The result is that each of the module's has an Autoconf- and Doozer++-based configure script that bears the brunt of the work needed to hide complexity. Then, each module has one or more “glue” makefiles that pull everything together. These glue makefiles typically direct a given module's build process so that steps occur in the correct order. Because they oversee the whole compilation process, they are often large files with many targets, each of which is responsible for a specific task. With a configure script and a glue makefile, all that remains is a listing of source files that must be compiled. The makefiles that list source files are often very, very short and easy for anyone to extend.

With this foundation, the complexity is separated thusly:

Other steps have been taken to offload complexity or repeated tasks into centralized components. For example, the directory juggler/macros contains Autoconf/m4 macros that can be used in configure scripts. Most of the macros are used to detect and provide information about the various modules that comprise VR Juggler. There are a few helper macros that build on top of Doozer++ and common idioms we have used to simplify configure scripts further.

As has been discussed at great length previously, it is important that users of the build system be freed from remembering many command-line arguments or special steps that must be taken to build a given piece of software. With several modules to configure and build, there is a lot to know about compiling VR Juggler. Most modules have external dependencies, the paths to which must be specified using command-line arguments. Furthermore, some modules may be compiled differently based on configuration-time settings. The so-called “global build” manages all of this, and so it has the responsibility of reducing the amount of information users must remember.

To that end, the global build offers some features for remembering command-line arguments so that users do not have to. Many of the configure scripts for the individual modules are written such that the default values for arguments are reasonable enough that users rarely have to pass them. The combination of these two features usually allows the typical developer to run the global configure script with no arguments. While this is not always the case (for example, on Win32, the path to NSPR must be specified unless the user has done something special in his or her environment), it happens often enough to keep people reasonably happy.

VR Juggler as a whole is a cross-platform software system. Porting to new operating systems is a non-trivial task, and spending time porting the build system to new platforms is time that could be spent more effectively. Speaking less abstractly, before the source code can be ported to a new platform, the build system has to be capable of compiling the source code on the new platform. For that reason, it is important that the entire build system can be put to use right away so that the attention can be devoted to the more difficult task of porting C++ code.

The VR Juggler build system is limited in its portability by the portability of Doozer++. To its credit, Doozer++ is more portable than VR Juggler at this time, so the VR Juggler team still has some room to move. Even so, the VR Juggler build has its own quirks, and thus, people writing the code to build VR Juggler must always have portability in mind. For example, “BASH-isms” must never appear in VR Juggler configure scripts or makefiles. Most Linux distributions may use BASH for /bin/sh, but that certainly does not meant hat all operating systems vendors follow that unfortunate trend.

In keeping with the Doozer++ goal of separating tools and tasks, the VR Juggler build system offers good portability by putting all of the platform- and software-specific pieces in Autoconf configure scripts. In so doing, the makefiles rarely, if ever, have to be modified when a new platform is added to the list. Furthermore, makefiles for GNU make lack sufficient programmatic constructs to provide developers with the ability to write tests that provide more than limited portability. While the Doozer software is relatively portable through its use of GNU make and platform-specific makefiles, it is missing the expressiveness of something based on Autoconf (or a similar tool). As discussed in the previous chapter, the use of make(1) alone for portability requires much effort to make hard-coded, platform- and site-specific makefiles.

In order for the global build to work, the modules it wraps must follow certain rules. If a module does not comply with all the rules, there is no guarantee that it will be able to compile under all circumstances. In other words, “rogue” modules that only implement a few pieces of the puzzle quickly become the weak link in the chain, and module build system authors who want to live outside the rule set complicate matters for everyone else.

Because the VR Juggler build system is a complicated dance, there are many rules that must be followed. For example, certain targets must be defined so that recursive make(1) calls can proceed through the entire source tree. These targets include 'release', 'install-debug', 'links', and 'buildworld'. Other rules include restrictions on the names of files or directories, use of platform-specific conventions, the presence of a working -config script (see the section called “The *-config Scripts and the *.m4 Macro Files”), and provisions for the detection of a usable installation. The full list of rules are provided in the following subsections.

#include <defines.h>
#include <vpr/Sync/Mutex.h>
#include <vrj/Kernel/Kernel.h>

The script configure.pl is a Perl script written to act as part of a build wrapper around an arbitrary collection of software modules. The modules are linked through some sort of (conceptual) dependency graph, in this case specified by a simple configuration file (see the section called “Build Configuration File”). configure.pl reads the configuration file and proceeds to configure the individual modules in an order that satisfies the dependencies. Along the way, environment variables are set or extended so that each subsequent module is configured with the correct settings. The processes of managing the dependencies and performing the configuration are the topics of this section.

Build Configuration File

Before explaining how configure.pl manages its dependencies, it will be helpful to understand the role played by the configuration file from which the dependencies are read. In the VR Juggler build system, the file is called juggler.cfg, and it is located in the top-level source directory. At a very high level, the configuration file defines one or more modules that must be configured and compiled. The modules may be independent of each other, or they may form a dependency graph. In the latter case, one module states that it depends on one or more other modules. The following code block shows an example of this:

module VPR                                                 
{
   external;                                               
   modules/vapor;                                          
}

module Tweek                                               
{
   depend VPR;                                             
   modules/tweek;                                          
}

	These lines declare two modules named “VPR” and “Tweek” respectively.
	These lines list directories upon which the containing module depends. Each of these directories must contain a script called configure that can be executed. This method of listing dependencies requires explicit paths, and it helps greatly if those dependencies exist within the local source tree.
	This line indicates that the Tweek module depends on the VPR module. Here, note that depend is a keyword with special significance. In effect, the VPR module is included inside the Tweek module definition so that it picks up all of VPR's dependencies.

Going deeper into the module definition, we find that environment variables can be set with each directory listing using a comma-separated list. These variables provide extra information about the configuration environment after the configure script has completed successfully. To illustrate this, we extend the above example as follows:

module VPR
{
   external;
   modules/vapor: VPR_CONFIG=vpr-config, VPR_BASE_DIR=instlinks;          
}

module Tweek
{
   depend VPR;
   modules/tweek: TWEEK_CONFIG=tweek-config, TWEEK_BASE_DIR=instlinks;    
}

As before, these lines list directories upon which the containing module depends. This time, we have added environment variable settings for the variables $VPR_CONFIG, $VPR_BASE_DIR, $TWEEK_CONFIG, and $TWEEK_BASE_DIR.

While any environment variable can be set in the configuration file, those shown above have special significance^[3]. Those variables ending in _CONFIG set the corresponding environment variable to include the full path to the named file (despite the fact that the full path is not given in the assignment). The path is constructed using the associated directory dependency. This directory is also added to the script's execution path. This extra little bit is needed so that a given -config script can be executed by another -config script if necessary. (The details about why all of this is necessary are discussed in the section called “The *-config Scripts and the *.m4 Macro Files”.)

Those variables ending in _BASE_DIR define the installation directory for the given module. In the VR Juggler build system, this is necessary for dependent modules to find the headers and libraries they need to compile. (Again, more information about this is given in the section called “The *-config Scripts and the *.m4 Macro Files”.) Once again, the value being assigned has special significance. If the value is the token instlinks, it is taken to mean that the full path to the installed module is in a directory relative to the current directory called instlinks. Any other value is used verbatim as the value of the environment variable.

The various *-config scripts (vpr-config, tweek-config, vrjuggler-config, etc.) play a vital role in the design of the global build. Unfortunately, this is also where the global build gets complex. Here, code that is intended for use by users of VR Juggler and associated modules is put to use by the code that compiles everything. It makes use of a strict set of behaviors wherein various environment settings and command-line options form a hierarchy of preferences and fallbacks. If anything goes wrong with a user's configuration process, it is almost always related to a .m4 file or a -config script giving unexpected results because of a misused command-line option or a “dirty” environment.

At various times in the history of the VR Juggler project, the developer “pseudo-installation” has been a topic of great controversy. Questions (or arguments, depending on your perspective) regarding its usefulness and its differences from a release installation come up repeatedly. The reason for its existence is quite simple: to simplify the lives of developers. The differences between the developer installation are also fairly simple. In a nutshell, a developer installation uses debugging libraries by default and links applications statically. A release installation, on the other hand, uses optimized libraries by default and links applications dynamically. The reasoning behind this is a little more difficult to nail down, and for that reason, we will say that it is beyond the scope of the document.

To satisfy the only goal of the developer installation (simplification of developers' lives), the developer installation must act exactly like a release installation, but it must be inside the build tree. The developer installation is created automatically as part of the build process, and ideally, its construction is faster than that of a full release installation. In any case, if all goes well, a developer can treat this pseudo-installation as if it were a real installation for the purposes of running tests.

Prior to early August 2002, the developer installation was created separately from a release installation. This was done through the use of symlinks on UNIX-based platforms and file copies on Win32. Since August 2002, the developer installation still uses symlinks or file copies in the same manner, but there is no longer a separation between creation of the developer installation and the release installation. In other words, the release installation is used to make the developer installation, but it is directed to install into the build tree.

The decision to use symlinks or file copies is based on the host platform and on the use of the 'links' target that every module must define. Using a custom Perl script, bsd-install.pl, written to be fully compatible with BSD install(1), symlinks may be created instead of using file copies. (The bsd-install.pl script comes with Doozer++ 1.5 and beyond.) Within the script, a test is performed to determine if the host platform is a Win32 system. If so, copies are always used because there are no symlinks on Win32 file systems. All of these decisions had been made in each module's build system, but they have since been offloaded into bsd-install.pl. This is in keeping with the Doozer++ goal of centralizing complexity.

At one time, there was a long-term goal for this global build script, or “project builder”. In conjunction with cvs-gather.pl, a semi-arbitrary collection of software packages would be downloaded and compiled. The commonality between them would be the use of configure scripts that would be invoked by the project builder. The dependencies would be specified through some configuration file, possibly written in XML, that would be constructed on the fly based on the cvs-gather.pl dependency file.

These lofty plans have not materialized, and it is unclear whether the need for such a tool still exists. Nonetheless, the idea of a highly generalized project builder played a key role in the way that configure.pl was written. In particular, its highly generic nature was motivated by the potential for downloading arbitrary source code and running a configure script. The makefile generated by configure.pl initially followed this goal, too, but it has since had its scope narrowed to deal with the specific conventions of the VR Juggler project.

Any module with DocBook-based documentation can be added to the documentation build. As with the source code build, there are rules that must be followed. With this smaller, less complicated build system, there are fewer things to deal with.

At this time, most of the documentation in the Juggler Project is written using DocBook. Because of that, the settings for building documents from DocBook files are centralized in the file juggler/doc/docbook.mk. This file is parameterized to the extreme so that including makefiles can override its default settings easily. Usually, makefiles that include docbook.mk direct the build to use OpenJade, Saxon, or Xalan to process the DocBook input and PassiveTeX, FOP, or XEP to create PDF files. Further configuration can be done that chooses different versions or installations of the DocBook style sheets, Saxon, Xalan, FOP, etc. The following is the makefile used to generate HTML and PDF versions of this document:

default: html

docs: html chunk-html pdf                                            
install-docs: install-html install-chunk-html install-pdf            

NAME=		build.system                                                  

XML_FILES=	$(NAME).xml                                               
HTML_FILES=	$(NAME).html                                             
PDF_FILES=	$(NAME).pdf                                               

XSLT_TOOL=	Saxon                                                     

# Fill these in!!  Together, they make up the installation prefix.
webroot=	$(HOME)/public_html/jugglerweb                              
instdir=	docs/juggler.build.system                                   

prefix=		$(webroot)/$(instdir)                                       
INSTALL_FILES=	$(webroot)/base_style.css                             

NEED_DB_IMAGES=	1                                                    

$(NAME).html: $(NAME).xml                                            
$(NAME).pdf: $(NAME).xml $(NAME).fo                                  

include ../docbook.mk                                                

	These are the required targets that all makefiles in the documentation build must have. Refer to the section called “Required Makefile Targets” for more information on these targets.
	This variable lists the base name of the source document. Since the generated documents differ only in final extension, this variable is used internally as the basis for the name of the source file and the generated documents. If there are multiple source documents, multiple variables (such as $(NAME1), $(NAME2), etc.) would be used.
	This variable lists all the DocBook XML source files that will be used as input to the XSLT processor. Each source file should have corresponding output files, also listed in this makefile.
	The source document can be rendered as HTML, and so this variable is used to list the output file. All HTML output files must be listed in this variable. The names must reflect the non-chunked output file names. Chunked HTML output is handled separately since the file names cannot be listed easily in this context.
	Similar to $(HTML_FILES), this lists the PDF file(s) that can be rendered from the XML input. This has the same semantics as $(HTML_FILES).
	The variable $(XSLT_TOOL) is used to tell docbook.mk which XSLT processor should be used. The default tool is Xalan, but Saxon can be used instead, as shown above.
	These variables deal with the installation process. The critical variables are $(webroot) and $(prefix). The variable $(webroot) will be overridden by the calling makefile (the top-level documentation makefile) to provide an alternate base directory as necessary. Setting it here provides a good default value to use when writing documentation and testing rendering. The $(prefix) variable is what tells docbook.mk where the document(s) will be installed. It must be set in order for an installation to succeed. Finally, $(INSTALL_FILES) provides any extra files that must be installed. In this case, we want to copy over the common stylesheet used by the rest of the VR Juggler website.
	Defining the variable $(NEED_DB_IMAGES) informs docbook.mk that the DocBook images are needed by rendered versions of the source document(s). Symlinks will be created as part of the rendering process, and the images directory will be copied over during installation.
	These lines simply list dependencies as a way to help make(1) in doing its job effectively. Here, the $(NAME) variable again plays a useful role.
	Finally, the last line (and it should always be the last line) includes the docbook.mk makefile stub. All of the settings above have an effect on the way that the targets defined in docbook.mk behave.