Skip to content

Commit

Permalink
docs: advanced build options (#624)
Browse files Browse the repository at this point in the history
  • Loading branch information
@wolfv
wolfv authored Feb 12, 2024
1 parent fbaa36e commit 8532797
Show file tree
Hide file tree
Showing 3 changed files with 220 additions and 120 deletions.
202 changes: 202 additions & 0 deletions docs/build_options.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
# Advanced Build Options

There are some specialized build options to control various features:

- prefix replacement
- variant configuration
- encoded file type

These are all found under the `build` key in the `recipe.yaml`.

## Always include and always copy files

There are some options that control the inclusion of files in the final package.

The `always_include_files` option can be used to include files even if they are
already in the environment as part of some other host dependency. This is normally
"clobbering" and should be used with caution (since packages should not have any overlapping files).

The `always_copy_files` option can be used to copy files instead of linking them.
This is useful for files that might be modified inside the environment (for example configuration files).
Normally, files are linked from a central cache into the environment to save space – that means
that files modified in one environment will be modified in all environments. This is not always
desirable, and in that case you can use the `always_copy_files` option.

??? note "How `always_copy_files` works"
The `always_copy_files` option works by setting the `no_link` option in the
`info/paths.json` to `true` for the files in question. This means that the
files are copied instead of linked when the package is installed.


```yaml title="recipe.yaml"
build:
# include files even if they are already in the environment
# as part of some other host dependency
always_include_files: [glob]

# do not soft- or hard-link these files, but always copy them was `no_link`
always_copy_files: [glob]
```
!!! note "Glob patterns"
Glob patterns are used througout the build options to specify files. The
patterns are matched against the relative path of the file in the build
directory.
Patterns can contain `*` to match any number of characters, `?` to match a
single character, and `**` to match any number of directories.

For example:

- `*.txt` matches all files ending in `.txt`
- `**/*.txt` matches all files ending in `.txt` in any directory
- `**/test_*.txt` matches all files starting with `test_` and ending in `.txt` in any directory

## Merge build and host environments

In very rare cases you might want to merge the build and host environments to
obtain the "legacy" behavior of conda-build.

```yaml title="recipe.yaml"
build:
# merge the build and host environments (used in many R packages on Windows)
merge_build_and_host_envs: bool (defaults to false)
```

## Prefix detection / replacement options

During installation time the "install"-prefix is injected into text and binary
files. Sometimes this is not desired, and sometimes the user might want closer
control over the automatic text/binary detection.

The main difference between prefix replacement for text and binary files is that
for binary files, the prefix string is padded with null bytes to match the
length of the original prefix. The original prefix is the very long placeholder
string that you might have seen in the build process.

On Windows, binary prefix replacement is never performed.

```yaml title="recipe.yaml"
package:
name: mypackage
version: 1.0
build:
# settings concerning the prefix detection in files
prefix_detection:
# force the file type of the given files to be TEXT or BINARY
# for prefix replacement
force_file_type:
# force TEXT file type (list of globs)
text: [globs]
# force binary file type (list of globs)
binary: [globs]
# ignore all or specific files for prefix replacement`
ignore: bool | [path] (defaults to false)

# wether to detect binary files with prefix or not
# defaults to true on Unix and (always) false on Windows
ignore_binary_files: bool
```
## Variant configuration
To control the variant precisely you can use the "variant configuration"
options.
A variant package has the same version number, but different "hash" and
potentially different dependencies or build options. Variant keys are extracted
from the `variant_config.yaml` file and usually any used Jinja variables or
dependencies without version specifier are used as variant keys.

Variant keys can also be forcibly set or ignored with the `use_keys` and
`ignore_keys` options.

In order to decide which of the variant packages to prefer and install by
default, the `down_prioritize_variant` option can be used. The higher the value,
the less preferred the variant is.

More about variants can be found in the [variant documentation](variants.md).

The following options are available in the `build` section to control the
variant configuration:

```yaml title="recipe.yaml"
build:
# settings for the variant
variant:
# Keys to forcibly use for the variant computation
# even if they are not in the dependencies
use_keys: [string]
# Keys to forcibly ignore for the variant computation
# even if they are in the dependencies
ignore_keys: [string]
# used to prefer this variant less
down_prioritize_variant: integer (defaults to 0, higher is less preferred)
```

## Dynamic linking configuration

After the package is build, rattler-build performs some "post-processing" on the
binaries and libraries.

This entails making the shared libraries relocatable and checking that all
linked libraries are present in the run requirements. The following settings
control this behavior.

With the `rpath` option you can forcibly set the `rpath` of the shared
libraries. The path is relative to the install prefix. Any rpath setting is
ignored on Windows.

The `rpath_allowlist` option can be used to allow the `rpath` to point to
locations outside of the environment. This is useful if you want to link against
libraries that are not part of the conda environment (e.g. proprietary
software).

If you want to stop `rattler-build` from relocating the binaries, you can set
`binary_relocation` to `false`. If you want to only relocate some binaries, you
can select the relevant ones with a glob pattern.

To read more about rpaths and how rattler-build creates relocatable binary packages,
see the [internals](internals.md) docs.

If you link against some libraries (possibly even outside of the prefix, in a
system location), then you can use the `missing_dso_allowlist` to allow linking
against these and suppress any warnings. This list is pre-populated with a list
of known system libraries on the different operating systems.

As part of the post-processing, `rattler-build` checks for overlinking and
overdepending. Overlinking is when a binary links against a library that is not
specified in the run requirements. This is usually a mistake because the library
would not be present in the environment when the package is installed.

Conversely, overdepending is when a library is part of the run requirements, but
is not actually used by any of the binaries/libraries in the package.

```yaml title="recipe.yaml"
build:
# settings for shared libraries and executables
dynamic_linking:
# linux only, list of rpaths relative to the installation prefix
rpaths: [path] (defaults to ['lib/'])
# Allow runpath / rpath to point to these locations
# outside of the environment
rpath_allowlist: [glob]
# wether to relocate binaries or not. If this is a list of paths, then
# only the listed paths are relocated
binary_relocation: bool (defaults to true) | [glob]
# Allow linking against libraries that are not in the run requirements
missing_dso_allowlist: [glob]
# what to do when detecting overdepending
overdepending_behavior: "ignore" or "error" # (defaults to "error")
# what to do when detecting overlinking
overlinking_behavior: "ignore" or "error" # (defaults to "error")
```
137 changes: 17 additions & 120 deletions docs/recipe_file.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -336,26 +336,6 @@ build:
string: abc
```

A hash will appear when the package is affected by one or more variables from
the conda_build_config.yaml file. The hash is made up from the "used" variables
- if anything is used, you have a hash. If you don't use these variables then
you won't have a hash. There are a few special cases that do not affect the
hash, such as Python and R or anything that already had a place in the build
string.

The build hash will be added to the build string if these are true for any
dependency:

* package is an explicit dependency in build, host, or run deps
* package has a matching entry in conda_build_config.yaml which is a pin to a
specific version, not a lower bound
* that package is not ignored by ignore_version

OR

* package uses `{{ compiler() }}` jinja2 function


#### Dynamic linking

This section contains settings for the shared libraries and executables.
Expand All @@ -379,7 +359,6 @@ build:
- bspatch4 = bsdiff4.cli:main_bspatch4
```


### Script

By default, rattler-build uses a `build.sh` file on Unix (macOS and Linux) and a
Expand All @@ -390,8 +369,16 @@ scripts for different platforms.

```yaml
build:
# A very simple build script
script: pip install .
# The build script can also be a list
script:
python setup.py install --single-version-externally-managed --record=record.txt
- pip install .
- echo "hello world"
- if: unix
then:
- echo "unix"
```

### Skipping builds
Expand Down Expand Up @@ -436,17 +423,17 @@ build:
evaluate to `true` in the platform it is built on, which probably will result
in incorrect/incomplete installation in other platforms.

<!--
### Include build recipe

The full boa recipe and rendered `recipe.yaml` file is included in
the package\_metadata by default. You can disable this with:
The recipe and rendered `recipe.yaml` file are included in
the package\_metadata by default. You can disable this by passing
`--no-include-recipe` on the command line.

!!! note
There are many more options in the build section. These additional options control
how variants are computed, prefix replacement and more.
See the [full build options](./build_options.md) for more information.

```yaml
build:
include_recipe: false
```
-->

Requirements section
--------------------
Expand Down Expand Up @@ -1173,93 +1160,3 @@ Experimental features

- [`load_from_file`](./experimental_features.md#load-from-files)
- [`git.*` functions](./experimental_features.md#git-functions)

<!--

Experimental features
---------------------

### Build time features

```{warning}
This is an experimental feature of boa and may change or go away completely
```

With boa, you can add "build-time" features. That makes building packages from
source much more flexible and powerful and is a first step to enable a true "source"-distribution on top of conda packages.

```yaml
name: libarchive
...
features:
- name: zlib
default: true
requirements:
host:
- zlib
run:
- zlib
- name: bzip2
default: true
requirements:
host:
- bzip2
run:
- bzip2
```

This adds two "features" to the boa recipe. These features can be enabled / disabled when invoking boa:

`boa build . --features [zlib, ~bzip2]`

This would compile libarchive with the zlib compression mechanism enabled, and bzip2 disabled. If a feature is not specified, the default value is used.
A feature can add additional requirements to the build/host/run section, and adds some environment variables to the build script invocation. In our example, the FEATURE_ZLIB environment variable will be set to `1`. This information can be used in the build script to enable or disable configuration and compilation flags.

For this libarchive recipe, the `./configure` call might look like this:

```
${SRC_DIR}/configure --prefix=${PREFIX} \
$(feature $FEATURE_ZLIB --with-zlib --without-zlib) \
$(feature $FEATURE_BZIP2 --with-bz2lib --without-bz2lib) ...
```

In this case we're using a special shell function `feature` to select between the enabled and disabled flag (similar to a ternary operator). The `feature` shell function is automatically added by boa into the shell environment.

One could similarly use bash `if / else` to set flags based on the `$FEATURE_...` variable.

If you want to depend on packages and require a specific set of features, you can use the following syntax:

```yaml
requirements:
host:
- libarchive [static, zlib, bzip2]
```

Sometimes you might want to depend on the `static` build _only_ if the package we are compiling is also compiled as a static package. In that case, you can use the `&` modifier:

```yaml
package:
name:
- libarchive
requirements:
host:
- bzip2 [&static]
# - bzip2 1.2.3 [&static] is also valid
features:
- name: static
default: false
```

Now, by default building `libarchive` will use the dynamic build of `bzip2`. When building `libarchive --features="[static]"` boa will instead use the `bzip2-static` package as requirement.

### How we think features will work

Features work by requiring a special build string. When compiling with features, the build string will be composed of all features and the build hash. For example, the following active features [zlib, bzip2] will be first alphabetically sorted and then concatenated to a build string of the form `+bzip2+zlib_h123123_0`. A deactivated feature will be prefixed with a `-`. To match packages against required build time features boa will compose a regex-based match string. E.g. when asking for at least `[bzip2]`, boa will use a build string of the form `+bzip2*`.

-->
1 change: 1 addition & 0 deletions mkdocs.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ nav:
- Home: index.md
- Highlevel overview: highlevel.md
- Recipe file: recipe_file.md
- Advanced options: build_options.md
- Jinja functions: available_jinja.md
- Experimental features: experimental_features.md

Expand Down

0 comments on commit 8532797

Please sign in to comment.