platypope.org

[This is an only slightly edited repost from an earlier incarnation of my blog. The archives from that blog didn't make the transition, but I'm planning to continue the series, so here's this post again.]

How does one build a piece of software from source? The superficial answer is run the tools1 on the source and intermediate files in the proper order. I'm sure we've all slapped something in a single C source file and just run the compiler on it from the shell. The next logical step beyond this would be a batch mechanism like a shell script running all the necessary commands. Perhaps worthy of a post to worsethanfailure.com if I still had access to the original, I've seen production binaries built with a file like:

#! /bin/sh

cc -c source1.c
cc -c source2.c
# ...
cc -o product source1.o source2.o # ...

In practice most of us use higher-level tools (thank god). The make tool represents the obvious next step "Unix philosophy"-wise. It assembles a directed acyclic graph of files in the build and performs the steps necessary to build the parts of the graph not yet present, leveraging the POSIX shell to represent and execute each build step. Some projects — it seems to be especially popular for kernels — use what I'll call Plain Old Make (POM), applying make "directly."

I put "directly" in quotation marks because most large POM projects put a fair amount of engineering effort into constructing higher-level abstractions applied consistently and automatically throughout the project. In standard Unix fashion make provides "mechanism not policy." In this case make implements the "mechanism" of bringing all the DAG nodes up to date, but requires make users to specify the "policy" of how to create each of those nodes from their dependencies2.

This leads to two interesting observations:

• Most projects — especially large ones — want factored build policies (node patterns) they can apply consistently throughout the project.
• Most projects share an awful lot of policy in common.

The GNU version of make includes extension to help with the first3, but for the second we move on to yet higher-level tools.

Most open source projects these days describe their builds using the GNU autotools: autoconf, automake, and libtool. The autotools provide abstract build policies which factor reusable build logic. They provide these policies in two ways. First, the autotools "know about" certain kinds of common build targets — they include the logic necessary to build a "program" or a "shared library" on supported platforms, requiring the developer only to specify the kind of target desired, not all the steps necessary to achieve it. Second, the autotools provide a policy for producing certain kinds of variant builds — they probe for features of the target platform and implement a policy for reacting to those features4.

Unfortunately, the autotools limit their abstract policy capabilities in an important way: the inability to implement new abstract policies. The autotools make it possible to descend to the mechanism level of make, m4, and the POSIX shell, but only in the same concrete way those tools do on their own. The autotools provide a collection of abstract policies, but no real policy abstraction.

Build tools which implement real build policy abstractions move us out of the realm of current popular use and on into part 2 of this series...

1 e.g., compiler, linker, Whiz-bang Source Frobnicator Enterprise Edition

2 The pedantic will note make support for "pattern rules," but I consider the fact that POSIX make pattern rules don't account for e.g. the different ways an object file needs to be generated for inclusion in an executable vs. a shared library enough for me to ignore them in this discussion.

3 $(include) and friends, although they're pretty clunky in practice.

4 This would be your config.h and its HAVE_FOO_H etc. macros.