Darwinbuild
Updated 2017-12-19

1. Overview
  1.1 Availability, Bug Reports, Contributions, and Discussion
2. Installation
  2.1 Creating the Build Directory
3. Building Darwin Projects (darwinbuild)
  3.1 Install Headers
  3.2 Chroot Builds
4. Build Environment Variables
  4.1 Build Aliases and Alternate Targets
5. Tips and Techniques
  5.1 Private Headers
  5.2 Build Tools
  5.3 Static Libraries
  5.4 Alternate BuildRoot Storage
Appendix
A. darwinxref

===========
1. Overview
===========

Darwinbuild is a collection of tools that assist compilation of
the many projects contained in Darwin[1], the open source base of Apple's
macOS operating system[2].  Apple publishes the sources of these projects
in an archive format (.tar.gz).  An archive is published for each project
version on Apple's site[3].

These tools will provide the proper build environment as well as help to
resolve any necessary dependencies prior to building.

[1] <http://developer.apple.com/darwin/>
[2] <https://www.apple.com/macos/high-sierra/>
[3] <http://www.opensource.apple.com/>


1.1 Availability, Bug Reports, Contributions, and Discussion

Due to the constant development of darwinbuild, the only version you should
be working with is the master branch of the Git repository[1]. This should
always be stable enough for building projects.

[1] <https://github.com/PureDarwin/darwinbuild>

===============
2. Installation
===============

You can install by checking out the source and building darwinbuild yourself.
You would want to run the following two commands:

  % cd <path to top of source>
  % sudo xcodebuild install DSTROOT=/ -scheme World

This will install darwinbuild and its supporting tools into /usr/local/bin.

If you want to manage darwinbuild with Homebrew, you can try this technique.
First, open Xcode and perform an archive. There is no good way of doing this
from the command line, unfortunately. Then export the generated archive bundle
(it should appear with the name "world", after the scheme that created it) to a
location such as the Desktop. (Really, anywhere would do.) Then run the following
commands:

  % mkdir -p /usr/local/Cellar/darwinbuild/HEAD
  % cp -R <path to archive folder>/Products/usr/local/{bin,share} /usr/local/Cellar/darwinbuild/HEAD
  % brew link darwinbuild

You don't need to copy the include directory, and the rest of the exported archive
can be deleted afterwards.

2.1 Creating the Build Directory

After installation, you must initialize the build directory using darwinbuild.
Assuming you wanted to build projects from the SUFuji16E195 plist[1] (which is
what most current development is targeting):

  $ mkdir ~/SUFuji16E195
  $ cd ~/SUFuji16E195
  $ darwinbuild -init https://github.com/csekel/PureDarwin-System-Plist/raw/master/SUFuji16E195.plist
  Creating build root disk image ...
  Attempting to download https://github.com/csekel/PureDarwin-System-Plist/raw/master/SUFuji16E195.plist ...
  Download complete
  Initialization Complete
  $ ls
  .build  Headers Logs    Roots   Sources Symbols

[1] <https://github.com/csekel/PureDarwin-System-Plist/blob/master/SUFuji16E195.plist>

After initialization, the build directory will contain the following:
.build     contains private data for the darwinbuild system
Headers    contains the resulting header files from previous builds
Logs       contains logs of previous build attempts
Roots      contains the finished products of previous successful builds
Sources    contains sources downloaded from the web
Symbols    contains debug symbol versions of previous build products

When using the darwinbuild command, it is necessary that the current
working directory be this build directory, or alternatively,
that the DARWIN_BUILDROOT environment variable be set to the absolute
path of the destination directory.  The example usages in this document
assume the current working directory is this build directory.


=========================================
3. Building Darwin Projects (darwinbuild)
=========================================

To build a Darwin project, for example xnu, the darwinbuild script can be
used in the following manner:
  $ darwinbuild xnu

The darwinxref tool is consulted to find the version that corresponds to the
build specified when the build directory was initialized.  It is necessary
to run the darwinbuild tool as root so that projects can set the proper
ownership and permissions on the build results.

darwinbuild first looks in the Sources directory for a directory containing
the sources to be built (Sources/xnu-3789.51.2), or a .tar.gz archive
containing the sources (Sources/xnu-3789.51.2.tar.gz).  If neither is found,
darwinbuild will attempt to download the sources from the web (including
from GitHub Releases).

If it does not already exist, a BuildRoot directory will be created.  This
is where the build will actually take place.  By default, darwinbuild does
not change the root directory to BuildRoot any more (see the chroot(8) man
page for details).  This is because of incompatibilities between a chroot
environment and the current version of the Xcode build tools.  Darwinbuild
can modify the tools to work correctly in a chroot, but *this is unsupported*
and may break at any time.

The build output will be written to the console, and additionally logged
into a file in the Logs directory.  The above example produces the
following file:
  Logs/xnu/xnu-3789.51.2.log~1

The ~1 indicates that this log file corresponds to the first attempt to
build xnu version 517.11.1.  Each subsequent attempt will add one to this
build number.

If the build succeeds, the finished product will be copied out of the
BuildRoot directory and into the Roots directory:
  Roots/xnu/xnu-3789.51.2.root~1
After the copy, darwinbuild traverses the directory and records all files
found in the darwinxref database.  This makes it possible to query which
project a file is produced by.  When a Mach-O executable, library, or bundle
is found during the traversal, the dynamic library load commands are recorded
in the darwinxref database.  This makes it possible to query which additional
projects are required to run an executable produced by the project.

Additionally, any products containing debug symbols will be placed into
the Symbols directory.


3.1 Install Headers

Passing the -headers flag to darwinbuild will start an alternative build
style where only the project's headers are produced.  This is useful for
working around various circular dependencies while compiling low level
projects like xnu, cctools, Libc, etc.

The results of a -headers build are placed in the Headers directory, in
the same style as the Roots directory for a regular build.  For example:
  % sudo darwinbuild -headers IOKitUser
  % ls Headers/IOKitUser/IOKitUser-184.hdrs~1


3.3 Logging Dependencies

Passing the -logdeps flag to darwinbuild will build the specified project
while recording the paths of all files openened and all executables invoked.
These paths are written into the Logs directory, and if the build is
successful, will be imported into the darwinxref database.


==============================
4. Build Environment Variables
==============================

At the minimum, the build environment consists of the creation of SRCROOT,
OBJROOT, SYMROOT, and DSTROOT environment variables.  These variables contain
an absolute path to a directory which must exist prior to invoking the Makefile.
User configurable environment variables, such as the target architecture,
can be set in the build plist file that is loaded via loadIndex.  The
darwinbuild script creates the aforementioned directories, sets the environment
variables, and issues the appropriate make command.

Within the plist file, the RC_ARCHS variable indicates which target
architectures should be included in the build.  Currently only ppc and i386
are available.  For each architecture in RC_ARCHS, and equivalent RC_{arch}
should be set to YES (i.e. RC_ARCHS="ppc" RC_ppc=YES).

The RC_NONARCH_CFLAGS specify additional command line flags to be passed to the C
compiler during the build.  On Darwin, the -no-cpp-precomp flag should be
passed since cpp-precomp is not available.  During the build, the RC_ARCHS and
RC_NONARCH_CFLAGS variables will be combined to create the RC_ARCHS environment
variable; a -arch flag will be appended for each architecture listed in RC_ARCHS.

The MACOSX_DEPLOYMENT_TARGET variable should match the major macOS release
that is being targeted, such as 10.2 or 10.3.


4.1 Build Aliases and Alternate Targets

Some projects may produce different results based on the contents of
the RC_ProjectName variable.  When the same source archive is used
to create more than one component of Darwin, it's referred to as a build
alias.  Because of this, it is important to always provide accurrate
project name and version information in the environment.  By default,
darwinbuild sets these variables appropriately.

By default, "install" (for non-headers builds) is the first argument passed
to the make tool (gnumake or xcodebuild).  However, some projects produce
different results based on alternate targets.  If the project in the property
list contains a "target" attribute, that string will be passed instead.

=============
A. darwinxref
=============

The darwinxref tool allows you to query which source version of a Darwin
project is present in a particular macOS build.  It also stores information
about what dependencies a particular project has, and what files the project
produces.  As each macOS release is made available, Apple publishes a
property list file containing the project names and versions in that release.
These property lists are read by the darwinxref tool to seed its internal
database.  Property lists are available via Subversion[1].

The darwinxref tool uses SQLite[2] to maintain its database of projects,
versions, files and dependencies.  This distribution includes a pre-built
version of the sqlite library in the file libsqlite3.a, and its associated
header file, sqlite3.h.

[1] <http://svn.macosforge.org/repository/darwinbuild/trunk/plists>
[2] <http://www.sqlite.org/>

An example of downloading, installing, and querying a property list:
  $ curl -L https://github.com/csekel/PureDarwin-System-Plist/raw/master/SUFuji16E195.plist > \
      plists/SUFuji16E195.plist
  $ darwinxref loadIndex plists/SUFuji16E195.plist
  268 of 268 projects loaded.
  $ darwinxref -b SUFuji16E195 version xnu
  xnu-3789.51.2

To list all projects in a build, use the special project name '*':
  $ darwinxref -b SUFuji16E195 version '*'

To register the results of a previous build with the database use the
register command.  Note this is done automatically by darwinbuild:
  $ darwinxref register adv_cmds 63 Roots/adv_cmds/adv_cmds-63.root~1
  /bin
  /bin/ps
  /bin/stty
  ...

To find which project produces the 'whois' command by searching the
list of previously registered files:
  $ bin/darwinxref findFile whois
  adv_cmds-63:
          /usr/bin/whois