Tclmake

The agmake tool

General description

This is the tool that should replicate the whole functionality of the standard POSIX make tool, with even most possibly similar syntax. Of course, this tool defines only the method of declaring the build rules - it does not provide various text processing functions that the make tool provides (such as $(wildcard ... ) for example). In this case, the user is encouraged to use standard Tcl functions (or even other Tcl libraries), or maybe also some convenience functions provided by make.tcl

The general idea of this tool is that the whole contents of the "Makefile" is a normal Tcl script with no unusual processing. The only difference is that some additional commands are available. The form of Makefile (here named Makefile.tcl) is also in order to make things most possibly similar to the POSIX make tool. This script could be, of course, a self-running script, which just imports the "make tool" extensions by a usual Tcl declaration using source, or even better package require command - although the method of having a Makefile.tcl file was chosen just to remind the POSIX make tool as much as possible.

No matter that, you are still free to use not only any available Tcl command, but you can also add extra Tcl packages, if you find it useful.

Basic additional commands

There are two most important commands for Makefiles:

• rule: defines a rule with file-based target, dependencies, and the command to produce the target
• phony: either define a purely phony target with only dependencies (integrator) or make an existing file-based target a phony target

Imagine now a typical set of statements in a POSIX Makefile:

CC:=gcc
program: program.c util.h
        $(CC) -o $@ $<

The same thing written in Makefile.tcl file would be:

set CC gcc
rule program program.c util.h {
        $CC -o $@ $<
}

Important things to pay attention here are:

• variables are just normal Tcl variables available in the same scope as where the rule is defined
• the $@ and $< (and $? not present here) have the same meaning as in Makefile. The Tcl interpreter just ignores them (they are not treated as anything special), it's the make.tcl processing facilities to expand them into special items
• the command to build should be at best in { ... }; as you know, this is passed in Tcl language as a single argument with nothing inside expanded (although here the make.tcl's facilities do variable expansion on their own)
• the rule command treats the last argument as the command to run
• in contrast to Makefile, there are no whitespace characters of special meaning (in particular, the "Tab" character required before the command lines), except the end-of-line character

There's also additional command, which simulates the "generic rule" feature of Makefile, however it doesn't work the same as in make, although it can be found more useful: the rules command.

This command allows to define multiple targets with their own dependency list, with one command common for all:

rules {
      {program.o program.cc util.h}
      {util.o util.cc util.h}
} {
      $CC -c -o $@ $<
}

The rules command has also an alternative syntax, which is useful for standard dependency generators in compilers, which generate the rules specifically for Makefile:

rules {
     program.o: program.cc util.h
     util.o: util.cc util.h
} {
      $CC -c -o $@ $<
}

Additionally, you can define a generic rule:

rule *.o *.cc util.h {
     $CC -c -o $@ $<
}

This differs in syntax a little bit to Makefile, where you'd have %.o or something like that. However the generic rules are just rules that can be defined and applied everywhere, unless there's an explicit rule already defined.

There are also additional things you can do in the command argument. Normally this encloses commands to be executed by a surrounding shell. There are several special symbols that make the line be treated differently:

• ! - special command, which's name follows the symbol directly:
  • !link <target> - requests that for that target it should use exactly the same command as in given target. If there are special variables, they obviously get expanded within the rules of the current, nor referred target
  • !tcl <command...> - executes a Tcl command.
• % - same as !tcl
• = - same as !link

The ability to run Tcl command are important for the very special autoclean command. This is typically predicted to be used this way:

rule clean {
      !tcl autoclean program
}

You don't have to think, which possibly files are intermediate files to be deleted. This rule simply walks thru the target definition and finds all files, for which the build rules are defined. All such files are being deleted. Please note that it may not always do exactly what you want. If you are unsure, there's an additional command autoclean-test, which does the same search, but only informs which files would be deleted.

The autoclean command's first argument is the target that should be used to start looking for dependencies. Additional optional arguments are flags, possibly prefixed by - character - it makes it "negative" flag, otherwise it's a "positive" flag. They can be used to qualify additionally the files defined as targets to be "cleanable" or not:

• Targets that contain the flag passed here as negative flag are excluded from deletion candidates
• Targets that contain the flag passed here as positive flag are deletion candidates, even if they are found as not generated (they are found in dependencies, but not in rules)

You can try out any internal command from make.tcl command line using the -x option.

Command-line options

The make.tcl command line supports the following options:

• -C <dir>: cd to <dir> before looking for makefile
• -f <makefile>: use <makefile> as build configuration file (NOTE: if used with -C, relative to that directory)
• -k : even if any target failed, continue building of targets that can be built
• -j <jobs>: allow to run maximum <jobs> in parallel (use 'j' as <jobs> as the number of cores of the current machine)
• -x <command>: execute a Tcl command available in the context of Makefile.tcl (NOTE: pass it in quotes, this must be one single argument)
• -v: display verbose messages
• -d: display debug messages

Migration from Makefile

There are several things you need to know how to construct Makefiles for make.tcl.

Generally Tclmake uses exactly the same philosophy that stands behind the standard POSIX make tool - that is, it supports primarily defining rules connected to files. So,the following statement defines the rule:

rule a.out file1.cc file2.cc {
     g++ -o $@ $^
}

As you can see, it also honors the standard make's special variables, $@, $<, $^ etc. If you don't want to make the target of a rule be connected to file, add the following declaration:

phony <target>

If you want to declare a purely phony target, you can provide only the phony declaration, optionally followed by dependent targets.

The most important things you need to know is how to manage the "scriptable" features of make and how to translate them to Tcl.

1. Tcl does not support lazy variables (those that are assigned by = in Makefile). The variables are simply assigned and they hold the same value when assigned unless reassigned. If you want to have some symbolic name that will expand to something meaningful only in the last possible time, you have to declare a procedure without arguments, just like normal procedures defined by proc command. Then you can use your SYMBOL procedure not as in Makefile by $(SYMBOL) or ${SYMBOL} - but as [SYMBOL]. There are small usable shortcuts to that, like pdef that allows you to follow the name by any number of words that will be resolved at definition time and returned in this form when called. For really lazy evaluation you can use pdefx, this time with the contents in { ... }, which should be a Tcl expression.
2. The assignment expression, be it with = or := can use multiple arguments in the same line. This doesn't work this way with the standard variable assignment command in Tcl, set. So, you cannot do set CXX g++ -std=c++11 - you have to do set CXX "g++ -std=c++11". There are some convenience commands provided for this occasion: pset accepts multiple arguments following the variable name and it glues them together, so pset CXX g++ -std=c++11 works correctly.
3. Comments are not honored in every possible place - in most of the places, especially within {...} the # symbol is a character like any other. You have to know how particular expression in braces will be interpreted and whether it will do appropriate #... lines removal. This is done, for example, in the "body" part (last argument) of the rule command. For custom data - such as, for example, if you want to set a list of files to a variable and want to make it in a column, with possibly comments, then the best way is to use the plist command. Note that you should set the result of this command to a variable.
4. The functionality of += operator is best replicated by lappend command. Note however that this command treats the current value of given variable as a list, so it must be a form that will be correctly parsed as a Tcl list. If it's not, this command will result in error. In practice it means that the value of the variable should not contain any unbalanced braces or double-quote characters with glued-in "trailing characters". In order to have the same convenience as with pset, but for appending, you can use pset+.
5. Tabulators don't matter. You can use them just on your own for clarity.
6. Prerequisites are using the same syntax as in make - with pipe ( | )
7. Nonexistent variables don't expand to an empty text - unlike in Makefile, in Tcl they cause exception! Every expression like $VAR will result in a valid - including empty - value only when it's set. In practice, if you want to use optional variables, in Tcl you'd normally have to check if [info exists VAR] and only after this is confirmed can you try to evaluate $VAR. For convenience you have two additional commands: pget returns the value of the variable, or empty string if it doesn't exist, and phas, which is very good for testing for variables expected to be set to 1 or not existent.

Cached dependencies

The standard make tool allows you to use dependencies stored in files.

This feature is quite weird and requires a little bit side-tool support and a little bit hacking. A simple method to use a rule stored in a depfile is:

include(options.d)
<tab>$(CC) -c options.c

This actually will work only if you have manually maintained dependency files (*.d files are manually created and kept in the repository), but it won't work, if you use generated dependency files. In this case, the dependency file doesn't exist at the first time, so this include instruction will fail.

If you want to use generated dependency files, you should do it this way:

%.o: %.c %.d
include($(wildcard *.d))
<tab>$(CC) -c $<

In this case, make will try to include all dependency files found by $(wildcard *.d), which may also be an empty list. If a rule for a particular source file was found, it will undergo the dependency rules as defined in a file. If not, then it will undergo the generic rule (first line), while requesting to build a dependency file first. This ensures that unless you mess up with files manually, there's never a situation that you have the *.o file, but not *.d - so the generic rule is used only if the *.o file does not yet exist, so it will be compiled anyway.

You could theoretically do the same also in Makefile.tcl, for example

set options_d {options.o: options.c}  ;# fallback, if depfile not found
catch {set options_d [exec cat options.d]}
rule $options_d {
    $(CC) -c $<
}

But Tclmake has a little bit better and more direct support for dependency files. Of course, there is one important difference to make's depfiles: Depfiles for Tclmake do not contain the target specification, only the source files.

The general syntax, stating that you have a depfile - here named options.dep - for building options.o from options.cc, you can specify the rule this way:

rule options.o <options.dep {
       command
}

Which replaces any long dependency specification:

rule options.o options.cc ...lots of header files... {
     command
}

Stating that options.o should be built from options.cc, which includes defines.h file, your *.dep file should contain:

options.cc defines.h

You can have this dependency files stored permanently, just like source files, or you can have a rule for them, just like any other rule. Some compilers, notably gcc with -MM option, provide ability to generate the dependency specification - however it's suited for Makefile, that is, it contains TARGET: as the first word, so it will generate for our file:

options.o: options.cpp file.h

Tclmake requires a file with dependency specification without target. Additionally, to keep the dependency list visually nice, the backslash-line-extensions are generated. These two problems have to be taken care of, and there's a builtin command to do it: gendep.

So, after passing it throug gendep, your dependency file should contain in this case:

options.cpp defines.h

And the gendep command should get the name of the dependency file to write and the command to generate dependencies.

Normally you don't have to use the gendep command directly because there's a shortcut for that: the dep-rule command. Its syntax is much like rule command, but it gets exactly 3 arguments: the dependency file, the source file and the command to generate dependencies. For example:

dep-rule options.dep options.cpp {
       g++ -MM options.cpp
}

The action in this case is exceptionally just to generate the "standard" dependency file - this will add the filter that will postprocess it and write into the dependency file. Once you have that, you can now use your dependency file as a source of dependency information:

rule options.o <options.dep {
      g++ -c options.cc
}

Of course, you need additionally the dep-rule command. Actually it doesn't do anything magic - the above dep-rule is a shortcut to:

rule options.dep options.cc {
      !tcl gendep options.dep g++ -MM options.cpp
}

The motivation for having a special support for dependency files was a better support for Silvercat, which has an option to use dependencies cached in files. This way it can support dependency files defined per single file, as well as the *.o file defined separately to the dependency specification (it's important because autogenerated specifications for *.o files in Silvercat use *.ag.o form).

This is more-less what would be generated from Silvercat when you use default -depspec value, auto. For cached it's even simpler: the dependency file is generated in one step when compiling the *.cc file. The gcc compiler has -MMD -MF <depfile> option syntax to specify the explicit dependency filename. When done so, the compilation rule usually looks this way:

rule options.ag.o options.cc <options.ag.dep {
       g++ -c -MMD -MF options.ag.dep options.cc -o options.ag.o
}

Note though that not every compiler supports this feature.