Académique Documents
Professionnel Documents
Culture Documents
4)
The Waf Book (v1.5.4)
Thomas Nagy
C opy righ t © 2008-2009 Thomas Nagy
C opies of this book may be redistributed, v erbatim, and for non-c ommercial purposes. The
lic ense for this book is by -nc-nd lic ense
Table of C ontents
Prefac e
1. Gettin g Started
Obtaining W af
Inst alling
R unning W af
8. C and C ++ p rojec ts
freehackers.org/~tnagy/…/single.html 1/43
10/10/2010 The Waf Book (v1.5.4)
11. Using the d ev elopment v ersion
List of T ables
Preface
Table of C ontents
Software is now used to ex press the process of building software, it can take the form of
simple sc ripts (shell scripts, makefiles), of co mpilers (C Make, Qmake), or c omp lete
applications (SC ons, W af). The term " build sy stem" is used to design th e tools used to build
applications.
There are so many different languages that c reatin g a build sy st em to support all the
possible kind of projec ts out-of-the-box is likely to be difficult. It looks much more
interesting to c reate one whic h c an be ex tended easily .
The W af approac h is to prov ide a minimalistic and v ery flex ible framework, whic h is p ac kaged
it with a signific ant amount of default build rules to be usable for most p rojec ts, and many
recipes for the most co mplic ated sc enarios.
The first c hapters cont ain a d etailed descrip tion of the basic use of W af. The c hapters on
C onfiguration, Tasks, and Task generators desc ribe the internals. These c hapt ers should be
read c arefully before try ing to ex tend W af. The nex t c hapters c ontain ex amples for the
languages such as C ++, and for special sc enarios whic h may occ ur in the most c omplic ated
builds. The last c hapters are prov ided as a referenc e for t hose who may need to mod ify W af
itself.
Obtainin g W af
In stalling
Running W af
Obtaining Waf
The W af projec t is located on Google C ode The c urrent W af v ersion req uires P y thon ≥ 2.3
c ompiled with support for the bzip2 c ompression algorith m. Installing W af is op tional, if y ou
want to install W af on the sy stem, ju mp to the nex t sect ion.
Installing
W af inst allation is optional an d there are many adv antages of no t installing W af, amo ng
them:
If, despite the adv antages enumerated prev iously , y ou really want to in stall th e W af
application on the sy stem, y ou may use the following c ommand on Unix -like sy stems such as
Linux :
Root priv ileges may be required dep ending on the final loc ation.
W af c an be uninstalled using:
Running Waf
The W af sc ript can be used direc tly :
By default, the W af sc ript unc ompresses its library in the c urren t direc tory , in a hidden
fo lder. This means the W af sc ript must resid e in a writable folder. The naming is allow
different W af v ersions to run from the same folder:
Sy stem files: files and folders whic h do not belong to the project
Sou rce directory : direc tory con taining the sourc e files for an applic ation, it is meant to
be ultimately pac kaged, and redistributed to other dev elopers or to end u sers.
freehackers.org/~tnagy/…/single.html 3/43
10/10/2010 The Waf Book (v1.5.4)
Build direc tory : all files produc ed by W af will be outp ut in that fold er
The W af philosophy is to av oid the pollution of the sourc e direct ory by letting all files into
the build direc tory . Th e build directo ry c an be loc ated on the sy stem, out of the sourc e
directory (like in /tmp for ex ample).
W hen W af is lau nched, it looks for the user-defined W af sc ripts whic h are files written in the
Py thon languag e. The most important one is the top-lev el W af sc ript file in wh ich sev eral
fu nction s and attributes must be prov ided for defining a v alid W af project:
A simple empty W af project c an be declared using a wsc ript file c ontaining the following
Py thon c ode:
C onfiguring the project: searc hing for sy stem parameters and c ompilers
Building th e project: building the software
W e will now pro v ide a more c omplicated wsc ript file, ex ec ute the ty pical c ommands, and
look at the outp ut:
freehackers.org/~tnagy/…/single.html 4/43
10/10/2010 The Waf Book (v1.5.4)
3 4
A build d irec tory was c reated with the name giv en in the wsc ript file. It c ontain s a
c onfiguration log, and v arious c ache files. The last command produc ed a tarball of th e
projec t (an arc hiv e) c ontaining the sourc e files. For now it only c ontains the wscript file.
freehackers.org/~tnagy/…/single.html 5/43
10/10/2010 The Waf Book (v1.5.4)
D ec laring the location of sc ripts that p rov ide other build fun ctions
D ec laring the ac tual targ ets to build
Subdirectory recursio n
The func tion build may reference build func tions in wsc ript present in sub fold ers, for
ex ample:
The top-lev el wsc ript may ind icate th ere is a build func tion to ex ec ute in src /wsc ript using:
The wsc ript files that are not on the top-lev el often c ontain a large build funct ion, with no
additional c onfiguration or c ommand-line op tions. In this c ase th e file only c ontain on e
method and the indentation is an annoy anc e. For this reason, the wscript_build files
freehackers.org/~tnagy/…/single.html 6/43
10/10/2010 The Waf Book (v1.5.4)
c ontain the body of a build func tion defined in a wsc ript file. A folder may only contain either
a wsc ript or a wsc ript_build but not both.
Intermediate resu lts (objec t files) may be produc ed to speed up th e build (no need to
rebuild ev ery thing when small c hanges occ ur)
Sou rce files may be proc essed in parallel to sp eed up the build
Sou rce files may be tran sformed into o ther so urce files (seq uenc es of transformations)
In a more general sc ope, the input n ec essary for a build may co me from arbitrary data
instead o f files (database, c ommands to ex ec ute). The co ncepts of input, output and
transformatio n and the transformation co nstraints (in parallel, in sequenc e) remain
identical.
In W af, the operation of tran sforming data is performed by tasks , whic h are instances of
the c lass TaskBase (in the W af library ). The tasks in the following ex ample illustrate the
declaration of Tasks from the base c lass. The task instanc es must be loc ated in a build
c ontex t (the fu nction build) to be taken int o ac count in the build phase, and t o obtain the
referenc e on the sourc e and build folders:
3 4
3 4
This ex ample prov ides a mod ified co nfigure func tion, whic h is the topic of the nex t sec tion.
The task generator sy stem an d the default generators will be rev iewed in detail in C hapter 6,
Task enc apsulation using task generators .
Configuring a project
The func tion co nfigure is used for four main purposes:
Indic ating new sc ripts that prov ide c onfiguration fun ctions to c all
Pro v iding c onfiguration c ontex t s to ho ld persistent d ata
Pro v iding c onfiguration helpers
freehackers.org/~tnagy/…/single.html 7/43
10/10/2010 The Waf Book (v1.5.4)
Loading v arious W af tools
The top-lev el wsc ript may ind icate th ere is a c onfigure func tion to ex ec ute in src/wsc ript
using:
As a mat ter of fac t, any wscript file may ind icate a sub-c onfiguration fu nction to ex ec ute.
3 4
This c onfiguration modifications (on c onf.env ) are persistent, they are reused during the
build (using bld.env ). Let us look at the files produc ed:
freehackers.org/~tnagy/…/single.html 8/43
10/10/2010 The Waf Book (v1.5.4)
3 4
By default, tools are loaded from the W af library , but it is possible to load tools from the
sourc e d irec tory using the fo llowing:
3 4
freehackers.org/~tnagy/…/single.html 9/43
10/10/2010 The Waf Book (v1.5.4)
The func tion set_options may referenc e bu ild func tions in wsc ript present in sub folders, for
ex ample:
The top-lev el wsc ript may ind icate th ere is a build func tion to ex ec ute in src /wsc ript using:
Installing targets
Targets are installed when W af is run with t he following c ommand-line waf install. If the
targets are not built, they will be installed as soon as they are built. W e will now desc ribe the
three main way s of installing the targ ets.
The v ariables in curly brac kets are substitut ed by v alues found in bld.env , for ex ample
${PREFIX } is substituted by the v alue of bld.env['PREFIX '] . An env ironmen t parameter
may be prov ided if nec essary :
Though the substitutio n is on ly performed on the first parameter, it is possible to re-use the
method bld.get_install_path or the func tion Utils.subst_vars
The v ariables in curly brac kets are substitut ed as desc ribed in the prev ious paragraph, using
main.env instead of bld .env . The chmod attribute is optional, and is used to c hange the
permissions of t he inst alled file (by default it is set to 064 4).
To prev ent the installation of a target (by d efault, programs and shared libraries are
freehackers.org/~tnagy/…/single.html 10/43
10/10/2010 The Waf Book (v1.5.4)
installed), set install_path to None
Custom tasks
Task installation is disabled by default for most task ty pes. If a method named install is
prov ided, it will be ex ecuted immediately after the task is suc c essfully ex ec uted. Unlike the
c ompilation, the file installation is not run in parallel (there is little gain in c opy ing files in
parallel).
Sev eral c onfigu ration files are produc ed and the c onfiguration parameters are stored in the
fo lder c4che loc ated under the build direc tory
The file default.cache.py is called a c onfiguration env iron ment, and it may be edited by
hand. W e will giv e the details on the following sect ion. The other files are not meant t o be
modified by hand, ev en if they are editable.
freehackers.org/~tnagy/…/single.html 11/43
10/10/2010 The Waf Book (v1.5.4)
By default, the c onf object c omes with a default built-in c onfiguration env iron ment:
To replac e the default c onfigu ration env ironment, a new one mu st be c reated and assigned:
In the build sec tion, the v alues are retriev ed using bld.env_of_name , for ex ample
sev eral different c onfig uration helpers may be c alled: the first one for finding t he c ompiler
and the others for finding c ompiler flags. In some c ases, it is interesting to disable some tests
(unnecessary , time c onsuming, etc ), to insert new tests, or to modify the ex isting tests. The
check_tool met hod has a parameter for passing the list o f c hec ks to perform:
3 4
If no chec k is p erformed, the c onfiguration helpers are still attac hed to the co nf object, and
may be used later:
freehackers.org/~tnagy/…/single.html 12/43
10/10/2010 The Waf Book (v1.5.4)
To ease the c reation o f projects split into modules, conf.c hec k_tool will not load the tools
twice for the same env ironment and the same parameters.
An error handler attac hed to the con f objec t is used for c atc hing the C onfiguration
ex ceptions and proc essing the errors. Here is how to replac e the default configuration error
handler by a c ustom method which may mo dify the list of tests, stop th e ev aluation, or re-
raise the ex ception:
3 4
The following diagram illustrates the test ex ecution loop performed from conf.c hec k_tool
Then the new c onfiguration t est may be used easily in wsc ript:
freehackers.org/~tnagy/…/single.html 13/43
10/10/2010 The Waf Book (v1.5.4)
By default, the archiv e name uses the v ersion and the projec t n ame found in the top-lev el
wsc ript file
The default c ompression format is bzip2. It may be changed to g zip by u sing the following
c ode:
Custom archives
C ustom archiv es c an b e c reated by prov iding a func tion named " dist" in the t op-lev el sc ript
file. The following ex ample prints a c hec ksum immediately after the arc hiv e is c reated:
A func tion may be defined for ex clud ing or adding more files to the arc hiv e. The hook is
ex ec uted from the temporary direct ory after the dist func tion h as finish ed to run:
W hen c opy ing the files, the d ist func tion automatic ally ex c ludes temporary files. The
fu nction dont_d ist in t he mod ule Sc ripting does the filtering, the c ode is reproduc ed below:
3 4
To ex c lude new files, it is possible to modify the ex tensions, or t o prov ide a new method
do nt_dist , for ex ample, to disable most of the filtering:
3 4
freehackers.org/~tnagy/…/single.html 14/43
10/10/2010 The Waf Book (v1.5.4)
Cleaning the project tree
By default, all the files produc ed by W af are produc ed into the b uild direc tory . An additional
file named .lock-wscript is c reated to store information on the build direc tory :
The c ommand waf distclean remov es these two elements from the filesy stem. Afterwards, it
is necessary to rec ompile the projec t again.
To remov e the build files without remov ing the con figuration, use the c omman d waf clean
instead. The files c reat ed in the build direc tory may be remov ed manually , but the
modific ations will not be detec ted by W af.
C reating all tasks by hand is a tedious proc ess that the task gen erators (C hapt er 6, Task
encapsulation using task generators ) may automate. Before starting the build, W af asks eac h
task generator to prod uce the c orrespondin g tasks. If W af is launc hed from a sub folder
inside the sourc e direc tory , it will try to av oid the c reatio n of the tasks that are not relev ant
fo r that particu lar sub folder (optimization).
Once the tasks are c reated, W af will rev iew eac h o f them one by one to dec ide whether to
ex ec ute them o r not. A summary c an be found on the following diagram:
Task execution
Ex ecutin g a task c onsists in c alling the method run on that task, and setting the task
ex ec ution state. The fo llowing diagram is a summary of the proc ess:
freehackers.org/~tnagy/…/single.html 15/43
10/10/2010 The Waf Book (v1.5.4)
The met hod post_run can be used to c hec k if the files hav e been produc ed, it must throw
an OSError if th e task has no t c omp leted properly .
By default, W af does n ot allow consumer threads t o ac cess the tasks direc tly :
There is little need for parallelizing the c omput ation of the nex t task to ex ecute,
c hoosing t he nex t task is fast enough
The thread issues are limited to a v ery small sec tion of the c ode
The produc er-con sumer sc heme prev ents busy waiting for the nex t task
A simple global error han dler c an be used for proc essing the errors and to dec ide to
stop the build
The following illustrates the relationship pro ducer-c onsumer performed for the builds:
freehackers.org/~tnagy/…/single.html 16/43
10/10/2010 The Waf Book (v1.5.4)
Task execution order
Running tasks in parallel is a simple problem, but in prac tic e it is more c omplic ated:
D ependenc ies c an be disc ov ered durin g the build (dy namic task c reation)
New ordering con straints c an be disc ov ered after files are c ompiled
The amount of tasks and ordering con straints (graph size) c an be huge and
performanc e may be a problem
To make the problem more simple, it is div id ed by the different c onc ern s, and the ord ering
c onstraints c an be giv en on three different lev els:
1. gro ups of tasks may run only after ano ther group of tasks has finished to run, t his
represents a stric t sequential order bet ween g roups of tasks, for ex ample a c ompiler is
pro duced and used to c ompile the tasks in the nex t group
2. task ty pes to indic ate the instan ce will run aft er other task ty pe instanc es, for ex ample
linking object files may o nly oc c ur after compiling the sourc e files
3. specific c onstraint s for task instanc es t hat c an only run after a few other task
instances
Task groups
The following is used to dec lare grou ps of tasks to run one after the other:
The effec t of task groups when runn ing tasks in parallel is illustrated by the following
diagram. Three groups of tasks hav e been added, and th e ex ec ution of the nex t group only
starts when the ex ec ution of the tasks in the prev ious group is c omplet e.
Precedence constraints
The attributes before and after are used t o dec lare ordering co nstraints between tasks:
Another way to dec lare prec edenc e constraints is to dec lare a file ex ten sion produc tion, for
ex ample:
The ex tensions hav e to matc h to add a v alid prec edence c onstraint, bu t they are only
annotations, th ey do not mean the tasks ac tually hav e to produc e files of that ty pe.
unlike the prev ious con straints, it is used on the instanc es of c lass Task which is a subc lass
of c lass T askBase
freehackers.org/~tnagy/…/single.html 17/43
10/10/2010 The Waf Book (v1.5.4)
The following diagram illustrates the task proc essing inc luding the signature, it is only v alid
fo r Task instanc e (not TaskBase instanc es):
1. ex plic it dep endenc ies: input files and dependenc ies set ex plicitly using task.deps_man
or bld.depends_on
2. implic it dependenc ies: dependenc ies searc hed by the task it self (like sourc e files
inc luded from other sourc e files).
3. parameters: c ompilation flags and c ommand-line parameters.
3 4
Although task in stances may be created direc tly , t hey ac t like global v ariables in the sc ripts,
and they may lead to the c reation o f spaghetti cod e. The c ontainers named task generators
are used for c reating t he tasks and keeping trac k o f them.
freehackers.org/~tnagy/…/single.html 18/43
10/10/2010 The Waf Book (v1.5.4)
Explicit rules for simple transformations (Make-like)
For simple transformations, a make-like sc heme en ables the creation of tasks by means of
task generators. The task is c reated lazily , for ex ample only if waf is c alled from a
subdirec tory of the projec t c ontainin g the declaration.
Simple rules
The following ex amples illustrate how to perform simple rule-based transformations, t he
attribute rule is mandatory , but the input or output files are optional:
A basic example with an input and an output. T he task is run whenever the
input or the rule change
T he task will run whenever the input or the rule change, but there is no
output
Creates a task with no input, the output is regenerated whenever the PREFIX
variable changes
Creates a task without inputs our outputs, it is run whenever the rule text
changes
Advanced features
The nex t ex amples illustrate v arious attribu tes whic h may be used to alter task ex ec ution.
3 4
freehackers.org/~tnagy/…/single.html 19/43
10/10/2010 The Waf Book (v1.5.4)
3 4
By binding the rule to the act ual dec laration, this system limits code reuse . For this
reason, it is rec ommended fo r solv ing v ery spec ific problems in whic h a library would not
make sense. The c reat ion of lots of small rules may also impact performanc e negativ ely .
In prac tic e, the c haining dec laration is located in a separate py thon file whic h may be re-
used for other projec ts.
The met hod to proc ess .c files (main.c ) is c alled automatic ally by apply_core , and is
declared in the followin g man ner:
1. If a method is mapped ( @extension ) to a name list ed in sourc es, it is c alled with that
name as a parameter
2. The names that d o not hav e a mapping are su pposed to be files and are transformed
into Node instanc es
3. For eac h node (temporary v ariable self.allno des ), a method c orresponding to the file
ex tension is applied
In this sy stem, the list of sou rces for an applic ation does not hav e to ex actly represent files.
Also, methods may re-injec t new nodes to proc ess by ex tension (in the temporary v ariable
self.allnodes ), without keeping an ex act referenc e on the method that will be c alled.
freehackers.org/~tnagy/…/single.html 20/43
10/10/2010 The Waf Book (v1.5.4)
Custom processing by task generator methods
In some cases the info rmation to pro cess is v ery spec ific and does not really fit in the list of
sourc es. The data may be att ac hed to v arious attribute of the task generator, and a distinct
task generator method may p rocess the attributes dy namic ally .
In the fo llowing ex ample, a new attribute 'tst' is added to task generato rs instanc es, and is
used to call a method to print the c ontents of that attribute at the start of the build:
To attac h a new metho d to the task_gen c lass (the ty pe of foo ), the dec orator @taskgen
is used in the following manner:
At this point, th e funct ion print_tst is attac hed to the c lass as a new method, but to proc ess
the attribute foo .tst , the method has to be c alled ex plicitly :
To sc hed ule that method for automatic ex ec ution, it is nec essary to bind it somehow. The
aliases are giv en in the task generato r attribute features :
For map ping th e method to all featu res, the wildc ard * may be used, a c omplete ex ample is
reproduc ed below:
3 4
In general, the methods c ann ot be c alled sev eral times or in random order. For ex ample, in
c ++ programming c ontex ts, the c ompilatio n tasks must be c reated before link tasks bec ause
link task will referenc e the c ompilation tasks. This leads to a part icular workflow, reproduc ed
in the following illustration:
The list o f methods to ex ecut e is kep t in the attrib ute meths of the task generator instanc e.
The list o f methods to ex ecut e may be giv en direc tly .
It is howev er more c onv enient to use the attribute feature to add the method s
automatic ally . For ex ample, for c reating a c application, the met hods to ex ec ute depend on
the operating sy stem, so the feature sy stem ac tually simplifies the c ode reuse.
The order c onstraints on the methods (after/before), are used to sort the list of meth ods in
the attribute meths . Th e sorting is p erformed onc e, and t he list is c onsu med as meth ods are
ex ec uted. Thou gh no new feature may be added onc e the first method is ex ec uted, new
methods may be added dy namic ally in self.meths. Here is how to make an infin ite loop:
To make this possible, the task ex ec ution is organized in the follo wing manner:
freehackers.org/~tnagy/…/single.html 22/43
10/10/2010 The Waf Book (v1.5.4)
Job control
Job c ontrol is related to the parallelization algorithms used for launc hing the tasks. W hile the
aim of parallelization is to max imize the amo unt of tasks in ex ec ution, d ifferent algorithms
may be used in practic e
In the NORMAL orderin g, task groups are c reated, and a topologic al sort is performed in
adv anc e. The ov erall performanc e penalty for c omplete b uilds is usually small, like a few
sec onds on builds during min utes.
In the JOBC ONTROL ordering, groups are c reated in adv anc e, and a flag indic ates the
max imum amount of jobs to be used when the c onsumer threads ex ec ute the tasks. This
prev ents parallelization of tasks whic h use a lot of resourc es. For ex ample, linking c ++ objec t
files uses a lot of R AM.
In the MAXPARALLEL ordering, Eac h task ho lds a list of tasks it must run after (there is only
one list o f tasks waiting to be ex ec uted). Though this v ersion parallelizes tasks v ery well, it
c onsumes more memory and proc essing. In prac tic e, W af may last 20% more on builds
when all tasks are up-to-date.
Warning
Because most task classes use ordering constraints, the maximum
parallelization can only be achieved if the constraints between
task classes are relaxed, and if all task instances know their
predecessors. In the example graph, this was achieved by
removing the ordering constraints between the compilation tasks
classes and the link tasks classes
freehackers.org/~tnagy/…/single.html 23/43
10/10/2010 The Waf Book (v1.5.4)
An assumption is made that all tasks hav e input and output nodes, and that ord ering
c onstraints can be deduc ed from them
D educ ing t he c onstraints from the input and output nodes ex hibits a n^2 behav iour
Note
In practice, the NO RMA L algorithm should be used whenever
possible, and the MA XPA RA LLEL should be used if substantial
gains are expected and if the ordering is specified between all
tasks. T he JO BCONT ROL system may be useful for tasks that
consume a vast amount of resources.
The following pic tures illustrate the differenc e in sc heduling a build with different independent
tasks, in whic h a slow task is clearly identified, and launch ed first:
W af prov ides a func tion for reordering the tasks b efore t hey are launc hed in the module
Runner, the default reordering may be c han ged by dy namic method replac ement in P y thon:
If the reordering is not to be performed eac h time a task is retriev ed, th e list of task may be
reordered when the nex t gro up is retriev ed:
It is possible to measure the task ex ec ution times by intercepting the func tion c alls. The task
ex ec ution times may be re-used for optimizing the schedule for subsequent bu ilds:
freehackers.org/~tnagy/…/single.html 24/43
10/10/2010 The Waf Book (v1.5.4)
The c onstructio n of c/c++ applic ations c an be quite c omplicated, and sev eral measures must
be taken to ensure coh erent interac tion with new c ompilation rules. The c anon ical co de for a
task generator building a c /c++ app lication is the following:
freehackers.org/~tnagy/…/single.html 25/43
10/10/2010 The Waf Book (v1.5.4)
automatic ally ex ported, and the dependent binary is rec ompiled when the library c hanges:
A static library
Include paths to export for use with uselib_local (include paths are not
added automatically). T hese folders are taken relatively to the current target.
A program using the static library declared previously
A list of references to existing libraries declared in the project (either a
python list or a string containing the names space-separated)
To link an applic ation against v arious system libraries , sev eral c ompilation flags and link
flags must be g iv en at once. To reduc e the maintenance, a sy stem called uselib can be used
to giv e all the flags at the same time:
Declare a few variables during the configuration, the variables follow the
convention VA R_NA ME
A dd all the VA R_NA ME corresponding to the uselib NA ME, which is 'T EST ' in
this example
The v ariables used for c /c ++ are the following: STATIC LIB, LIB, LIBPATH, LINKFLAGS,
RPATH, C XXFLAGS, C C FLAGS, C PPPATH, C PPFLAGS, C XXD EFINES, FRAMEW OR K,
FRAMEW ORKPATH, C XXDEPS . The u selib is similar to the c asc ading sty le sheet (C SS)
princ iple.
Files will be com piled in c mode, but no program or library will be produced
Different compilation flags m ay be used
T he objects will be added automatically in the link stage
W riting c onfigu ration headers can be performed using the following methods:
The c ode snipped will produc e the following co nfig.h in th e build direct ory :
Local folders are giv en relativ e to the folder of the c urren t script . The equiv alent build
directory folder is then added automatically for headers or sourc e files produc ed in the build
directory .
W af uses a prep rocessor writ ten in Py thon for adding the dependenc ies on th e headers. A
simple parser looking at #include statements would miss c onstruc ts suc h as:
Using th e c ompiler for finding the dependenc ies would not work for app lications requiring file
preprocessing suc h as Qt. Fo r Qt, special inc lude files hav ing the .mo c ex tension must be
detec ted by the build sy stem and produc ed ahead of time. The c compiler c ould not parse
suc h files.
Portability recommandations
By default, the preproc essing does not climb to sy stem headers. This may lead to missed
depdend enc ies in the c ases similar to the following:
freehackers.org/~tnagy/…/single.html 27/43
10/10/2010 The Waf Book (v1.5.4)
To write portable c ode and to ease debugging, it is strongly reco mmended to put all the
c onditions used in the projec t into a co nfig.h file.
Debugging dependencies
The W af prepro cessor contains a spec ific debugging zone:
The dependenc y c omp utation is performed only when the files are not up-to-d ate, so these
c ommands will display someth ing on ly when there is a file to c ompile.
Configuration helpers
The met hods check_cc and check_cxx are used to detec t parameters using a small build
projec t. The main parameters are the following
3 4
Here is an ex ample of a config.h pro duced with th e prev ious test c ode:
3 4
Pkg-config
In stead of duplic ating the con figuration detection in all dependent projec ts, co nfiguration
files may be written when libraries are installed. To ease the interac tion with build sy stems
based on Make (c anno t query datab ases or apis), small applic atio ns hav e been c reated for
reading the cac he files and to interpret the parameters (with names traditionally ending in -
config ): pkg-co nfig, wx -config, sdl-c onfig, etc .
W af prov ides the method check_cfg for qu ery ing c onfig parameters:
3 4
D ue to the amo unt of flags, the lac k of standards between c onfig applic ations, and to the
output c hangin g for different operating sy stems (-I for gc c , /I for msv c ), the output of pkg-
c onfig is parsed, and the v ariables for the c orresponding uselib are set in a go . The func tion
parse_flags(line, uselib, env) in the W af module c onfig_c .py performs the output
analy sis.
freehackers.org/~tnagy/…/single.html 29/43
10/10/2010 The Waf Book (v1.5.4)
Table of C ontents
This c hapter prov ides ex amples for adv anc ed scenarios to demonstrate the prac tic al use of
the W af library .
File substitutions
The following ex ample illustrates how to produc e a pkg-c onfig file from a temp late:
The v ariables must be declared in th e template file by enc losing the names bet ween @
characters (m4 sy ntax ).
The default substitutio n func tion may be replaced by changing the attribute func of the task
generator.
Compiler output
The command-o utput sy stem enab les ex ec ution o f sy stem commands without requiring a
shell. Th e following ex ample illustrates how to ex ec ute a sc ript p resent in the sourc e
directory each time it is ex ec uted.
freehackers.org/~tnagy/…/single.html 30/43
10/10/2010 The Waf Book (v1.5.4)
Variants and environments
The following co de demonstrates ho w to create a c onfigu ration for a new v ariant named
'debug'
After running waf configure , the bu ild folders c orresponding to the default and to the
debug v ariants are produc ed:
To use t he debug v ariant, th e env ironment must be giv en direc tly , by default, a cop y of
the env ironmen t named default is set:
Cloning targets
Task generators may be c opied for different v arian ts easily using the clo ne method. The
paramet er is the name of the env ironment to use (not the v ariant). The named
env ironments are c reated during the configuration, as shown in the prev ious subsec tion.
It is also possible to c lone all task generators from a projec t using a c ode similar to the
fo llowing :
This sc enario demonstrates the bld.add_gro up to segment the build in to stric tly seq uential
parts:
freehackers.org/~tnagy/…/single.html 31/43
10/10/2010 The Waf Book (v1.5.4)
3 4
freehackers.org/~tnagy/…/single.html 32/43
10/10/2010 The Waf Book (v1.5.4)
3 4
Task c reation is performed through a new task generator method (proc ess_sv n) whic h is
ex ec uted only when the task generator c reates all its tasks. This organization makes it
possible to c ompile only the c ode nec essary for a spec ific target (tasks are c reated lazily ,
and are grouped by task gen erators). To ad dress the scenario desc ribed, a new task ty pe is
c reated, and methods are replac ed dy namic ally : runnable_status (ex ec ute eac h time), and
post_run (c hange the signature of the outputs acc ording to the contents of the file
c reated).
A c ompiler produces source files (.c files) for whic h tasks must be c reated
The sourc e file names are no t kno wn in advance
The task must be run o nly if necessary
Other tasks may depend on the tasks proc essing the sou rce produc ed (c omp ile and
link the .c files)
The main difficu lty in this scenario is to store the information on the sourc e file produ ced
and to c reate the c orrespond ing tasks each time.
freehackers.org/~tnagy/…/single.html 33/43
10/10/2010 The Waf Book (v1.5.4)
freehackers.org/~tnagy/…/single.html 34/43
10/10/2010 The Waf Book (v1.5.4)
3 4
The following ex ample demon strates how to c reate a task ty pe fulfilling the requirements,
and how to use it.
freehackers.org/~tnagy/…/single.html 35/43
10/10/2010 The Waf Book (v1.5.4)
3 4
ex ec uting the sc ript with "foo" as argument will produce the following output
The parameter ctx in the func tion foo abov e is prov ided to enable rec ursion and sharing
data with the sc ripts. In the following ex ample, the script subdir/wscript_foo will be looked
up, and ex ec uted if found. Else, the sc ript subdir/wscript will be looked up and loaded as a
module. If suc c essfully loaded, the func tion foo will be ex ecuted, else an ex c eption is raised.
For bac kward c ompatibility reasons, the c ommand s init, shutdown, dist and distcheck do
not ac c ept a co ntex t as parameter y et.
freehackers.org/~tnagy/…/single.html 36/43
10/10/2010 The Waf Book (v1.5.4)
Chaining commands
Sinc e W af 1.5.4, it is possible to c hain sev eral c ommands, for ex ample:
The ex ec ution will stop as soo n as an error is enc ountered, for ex ample if distc lean fails, the
c onfiguration and the build will not b e ex ec uted, so the c omman d is equiv alent to:
A frequent usec ase is to c hain the ex ec ution of bu ild and c leaning steps:
As a special ex c eption, the func tion init is alway s prepended to the commands to ex ecute,
and the functio n shutdown is alway s appen ded.
C ustom c ontex ts may be prov ided for the function s co nfigure and build .
In jec ting new c ommands is u seful for writin g testc ases. By ex ec uting waf test , the following
sc ript will c onfigure a projec t, create source files in the sourc e d irec tory , build a program,
modify the sourc es, and rebu ild the program. In this c ase, the program must be rebuilt
because a header (implic it dependenc y ) has c hanged.
freehackers.org/~tnagy/…/single.html 37/43
10/10/2010 The Waf Book (v1.5.4)
3 4
To display all the trac es (useful for bug reports), use the following flag:
The W af module Logs replac es the P y thon module logging. In the sourc e c ode, traces are
prov ided by using the debug func tion, they must obey t he format "zon e: message" like in
the following:
3 4
Zone Meaning
command-lines executed (enabled when -v is provided without
runner
debugging zones)
task_gen task creation (from task generators)
action functions to execute for building the targets
env environment contents
envhash hashes of the environm ent objects - helps seeing what changes
build build context operations such as filesystem access
deps implicit dependencies found (task scanners)
Benchmarking
The scrip t utils/genbench.py generates a simple benc hmark for W af. The habitual use is the
fo llowing :
The projec t c reated c ontains 50 libraries with 100 c lasses for eac h, eac h sourc e file hav ing
15 includ e head ers pointing t o the same library and 5 headers pointing to the headers of
other lib raries in the projec t.
The time taken to c reate the tasks and to resolv e the dep endenc ies can be ob tained by
freehackers.org/~tnagy/…/single.html 38/43
10/10/2010 The Waf Book (v1.5.4)
injec ting c ode to disable the ac tual c ompilation, fo r ex ample:
Profiling
Profiling information is obtain ed by c alling the module cProfile an d by injecting spec ific c ode.
The two most in teresting met hods to profile are flush and compile . The most important
number from the profiling information is the amount of func tion calls, and reduc ing it results
in noticeable speedups. Here is an ex ample on the metho d c omp ile:
3 4
Here the output obtained on a benc hmark build created as ex plained in the prev ious sec tion:
3 4
From the profile information, it appears that the ho t spots are, in order:
The persist enc e implemented by the cP ickle module (the c ache file to serialize takes
about 3Mb)
Ac c essing c onfiguration data fro m the Env iron ment instanc es
C omputing implicit dependencies (the test projec t co ntains lots of interleav ed
dependenc ies)
In prac tic e, the time taken by these methods is not signific ant enough to justify code
c hanges. Also, profiling must be carried out on builds which last at least sev eral sec onds.
After ex ecuting a full build, a trac e of the ex ecutio n is ou tput in /tmp/test.dat ; it may be
proc essed by o ther applic atio ns suc h as Gnuplot:
freehackers.org/~tnagy/…/single.html 39/43
10/10/2010 The Waf Book (v1.5.4)
To av oid regenerating W af eac h time, the env ironment v ariable W AFD IR c an be used to
point to the direc tory containing wafadmin:
Although the W af binary depends on Py thon 2.3, the W af sourc e code depen ds on P y thon
2.4. W hen generating W af, a parser modifies the sourc e c ode and perfo rms the following
operatio ns:
Programming constraints
Though W af is written in Py th on, additional restric tions apply to the sourc e c ode:
Identation is tab-o nly , and the max imum line length should be about 200 charac ters
The dev elo pment code is kept c ompatible with Py tho n 2.3, to the ex ception of
dec orators in the Tools d irec tory . In partic ular, the W af binary c an be generated using
Py t hon 2.3
The wafadmin modules must be insulated from the Tools modules to keep the W af c ore
small and language independent
Api c ompatibility is maint ained in the c y c le of a mino r v ersio n (from 1.5.0 to 1.5.9)
More c ode alway s means more bugs. W hen ev er possible, unnecessary c ode must be
remov ed, and the ex isting c ode base should be simplified .
Module Role
Defines the build context class, which holds the data for one build
Build
(paths, configuration data)
Contains the configuration context class, which is used for
Configure
launching configuration tests, and the extension system
Constants Provides the constants used in the project
Contains a dictionary class which supports a lightweight copy
Environment
scheme and provides persistence services
Logs Provide a logging system
Node Contains the file system representation class
Provides a custom command-line option processing system based
O ptions
on optparse
Runner Contains the task execution system (threaded producer-consumer)
Constitutes the entry point of the Waf application, use the
Scripting
command-line for launching the configuration, the build, etc
Provide the task generator system, and its extension system based
T askGen
on method addition
T ask Contains the task classes, and the task containers.
freehackers.org/~tnagy/…/single.html 40/43
10/10/2010 The Waf Book (v1.5.4)
Contains the support functions and classes re-used in other Waf
Utils
modules
The essential c lasses and methods from the core library are represented on the following
diagram:
A ttribute Information
root Node representing the root of the file system
srcnode Node representing the source directory
bldnode Node representing the build directory
node_sigs File hashes (dict mapping Node ids to hashes)
node_deps Implicit dependencies (dict mapping Node ids)
Implicit file dependencies which could not be resolved (dict mapping
raw_deps
Node ids to lists of strings)
Signature of the tasks previously run (dict mapping a T ask id to a
task_sigs
hash)
Sequence for generating unique node instance ids (id of the last Node
id_nodes
created)
freehackers.org/~tnagy/…/single.html 41/43
10/10/2010 The Waf Book (v1.5.4)
Build context access
In prev ious W af releases, the build c ontex t was su pposed to be a uniqu e object (one build
activ e at a time). To enable the use of W af as a lib rary , the dependency on the singleton
Build.bld was remov ed. This implies that eac h objec t should be able to obtain its build
c ontex t from its attributes. Here are a few ex amples:
Parallelization co ncerns
Build con tex ts p erform an os.chdir c all before starting to ex ec ute the tasks. W hen running
build con tex ts within build c ontex ts (tasks), the c urrent working directo ry may c ause
v arious problems. To work around them, it may be nec essary to c hang e the c ompilat ion
rules (co mpile from the file sy stem root) an d to inject c ode (replac e bld.c ompile).
D irec t Node instances are not used any where in the W af c ode. Instead, eac h b uild c ontex t
c reates a new Node subc lass (bld.node_class), on whic h t he build c ontex t instance is
attac hed as a c lass att ribute.
Threading concerns
Nearly all the c ode is ex ec uted in the main thread. The ot her threads are merely waiting for
new tasks, and ex ecut ing the methods run from the task instanc es. Suc h meth ods should
c ontain as little code as possible, and ac cess the BuildC ontex t in a read-only manner.
If the ru n meth ods hav e to modify the build c ontex t, it is rec ommended to ov erload the
method get_out of the sc heduler and to ex ec ute methods in an ev ent-like manner (data is
attac hed to the task, and the metho d get_out ex ecutes the cod e). Adding more tasks from
a runnin g task is demonstrated the section c alled “A c omp iler produc ing sourc e files with
names unknown in adv ance”.
Task classes
The whole proc ess of generat ing tasks thro ugh W af is performed by methods added on the
c lass task_gen by c ode inject ion. This often puzzles the programmers u sed to static
languages where new func tions or c lasses c annot b e defin ed at runtime.
The task generators automat ically inherit th e build c ontex t attribute bld when c reated from
bld.new_task_gen. Likewise, tasks created from a task generator (create_task) automatic ally
inherit th eir generator, and th eir build c ontex t. D irect instantiation may result in problems
when running W af as a library .
The tasks c reat ed by task generator methods are automatic ally stored on the build c ontex t
task man ager, whic h stores the task into a task group. The task groups are later used by th e
sc heduler to obtain the task which may run (state machin e). Target (un)installation is
performed right after a task has run, using the method install.
freehackers.org/~tnagy/…/single.html 42/43
10/10/2010 The Waf Book (v1.5.4)
For huge projec ts, c alling make rec ursiv ely is nec essary for flex ibility , b ut it hurts
performance (launch many proc esses), and C PU utilizatio n (runn ing tasks in parallel). Make-
based bu ild sy stems suc h as C Make or Autotools in herit the limitations of Make.
Though W af uses a similar design as SC ons, W af is about 15 times faster for similar features
and without sac rificing build ac curac y . The main reasons for this are the following:
The W af data struc tures (file sy stem representation, tasks) hav e been c arefully c hosen
to minimize memo ry usage and data duplic atio n
For a projec t of the same size, SC ons requires at least 10 times as many func tion c alls
The W af hashing sc heme uses the md5 algorithm p rov ided by the Py thon distribution . It is
fast enough for up to about 100Mb of data and ab out 10 000 files and v ery safe (v irt ually no
risk of c ollision).
If more than 10 0Mb of data is present in th e project, it may be necessary to use a faster
hashing algorithm. An implementation of the fnv algorithm is present in the W af distribution,
and c an replac e md5 without really d egrading acc urac y .
If more than 10 000 files are present, it may be nec essary to replace the hashing sy stem by
a file name+size+timestamp hash scheme . An ex ample is pro v ided in the c omment
sec tion o f the module Utils.py . That scheme is more effic ient b ut less ac c urat e: the W af
c ac he should not be used with this sc heme.
Link Description
http://code.google.com/p/waf The Waf project page
The Waf wiki, including the frequently
http://code.google.com/p/waf/w/list
asked questions (FA Q )
http://groups.google.com/group/waf-
The Waf mailing-list
users
Glossary
W af essential terms
A task gen erator is an ob jec t in stance of the c lass Task.task_gen. The task generators
enc apsulate the c reation of v arious task instanc es at a time, and simplify the creation
of ordering constraints between them (for ex ample, c ompilation tasks are ex ecu ted
before link tasks).
Task
Tool
A W af tool is a py thon module c ontain ing W af-spec ific ex tensions. The W af tools are
loc ated in the fold er wafadmin/Tools/ and usually c ontain a global v ariable detect
whic h may referen ce fun ctions to ex ec ute in the c onfiguration.
freehackers.org/~tnagy/…/single.html 43/43